Position: Sr. Databricks Architect (Data Governance & AI)
Location: Houston, TX (4 days hybrid onsite)
Job Summary:
We are looking for a Senior Databricks Architect with strong expertise in Data Governance and exposure to AI/ML tools. The ideal candidate will design and implement scalable data platforms, ensure data quality and compliance, and support advanced analytics and AI initiatives.
Key Responsibilities:
- Design and architect scalable data solutions using Databricks (Lakehouse architecture)
- Implement and manage data governance frameworks (data quality, lineage, catalog, security)
- Define and enforce data standards, policies, and best practices
- Work with stakeholders to ensure compliance with data privacy and regulatory requirements
- Build and optimize ETL/ELT pipelines using Spark and Databricks
- Integrate AI/ML tools and frameworks for advanced analytics and automation
- Collaborate with data engineers, analysts, and data scientists
- Monitor performance, cost optimization, and reliability of data platforms
Required Skills:
- Strong experience with Databricks, Apache Spark, and Delta Lake
- Hands-on experience in Data Governance tools (Unity Catalog, Collibra, Alation, etc.)
- Proficiency in Python, SQL, and data engineering concepts
- Experience with cloud platforms (Azure, AWS, or Google Cloud Platform)
- Knowledge of data security, data privacy, and compliance frameworks
- Exposure to AI/ML tools (MLflow, Azure ML, or similar)
Preferred Qualifications:
- Experience with Generative AI or advanced analytics use cases
- Strong understanding of data modeling and architecture patterns
- Certifications in Databricks or cloud platforms
Soft Skills:
- Strong communication and stakeholder management skills
- Problem-solving and analytical thinking
- Ability to lead and mentor teams
Nice to Have:
- Experience with real-time data processing (Kafka, streaming)
- Familiarity with DevOps and CI/CD for data pipelines