Azure Databricks Engineer Jobs in Texas (NOW HIRING)

POSITION OVERVIEW

The Senior BI Lead & Databricks Engineer is responsible for serving as the technical authority and team leader for Business Intelligence and data platform initiatives. This is a dual-function role combining expert-level Databricks platform engineering with senior Power BI development and offshore BI team leadership. The role owns the architecture, governance, and delivery roadmap for both the Databricks Lakehouse environment and the Power BI reporting ecosystem, while guiding an offshore team of BI developers to ensure consistent quality and on-time delivery.

PRINCIPAL RESPONSIBILITIES AND DUTIES

Databricks Platform Ownership

• Serve as the primary subject matter expert (SME) and internal champion for Databricks across the organization.
• Architect, build, and maintain scalable data pipelines using PySpark, Delta Lake, and Databricks Workflows.
• Implement and enforce the Medallion architecture (Bronze / Silver / Gold) for structured data layering.
• Manage Databricks Unity Catalog for data governance, access controls, and data lineage tracking.
• Optimize cluster configurations, job scheduling, and compute costs to ensure platform efficiency.
• Establish CI/CD best practices for Databricks notebooks and jobs using Azure DevOps or equivalent tooling.
• Partner with data engineers, architects, and business stakeholders to define data contracts and SLAs.
• Monitor pipeline health, data quality, and platform performance; resolve issues proactively.

Power BI Leadership & Development

• Lead the design and delivery of enterprise Power BI reports, dashboards, and semantic data models.
• Define and enforce Power BI development standards including naming conventions, DAX best practices, and deployment pipelines.
• Architect and manage the Power BI semantic layer (datasets and dataflows) connected to Databricks as the source of record.
• Review and approve all BI deliverables produced by the offshore development team prior to release.
• Provide hands-on development for complex or high-priority reports and datasets as needed.
• Manage the Power BI Premium or Fabric workspace environment, capacity planning, and licensing.
• Collaborate with business stakeholders to gather requirements, prioritize the BI backlog, and deliver actionable insights.

Offshore Team Leadership

• Provide day-to-day technical direction, task assignment, and code review for offshore BI developers.
• Establish and communicate sprint-level priorities, delivery timelines, and quality standards.
• Conduct regular team check-ins to remove blockers, ensure alignment, and support developer growth.
• Create and maintain BI development documentation, standards templates, and reusable component libraries.
• Onboard new team members and establish repeatable, documented development processes.
• Escalate capacity, quality, or scope issues to leadership proactively and with recommended solutions.

Data Strategy & Stakeholder Engagement

• Act as a strategic advisor to business leaders on data visualization, self-service analytics, and data-driven decision making.
• Participate in roadmap planning for the overall data platform and BI tool ecosystem.
• Translate complex technical concepts into clear, business-facing communication for non-technical audiences.
• Identify and evaluate opportunities to leverage AI/ML capabilities within Databricks and Microsoft Fabric.
• Perform other related duties or special projects as assigned or directed.

EXPERIENCE AND OTHER REQUIREMENTS

• Minimum 5 years of hands-on data engineering experience, with 3 or more years working in Databricks at production scale.
• Expert proficiency in PySpark, Spark SQL, and Delta Lake including schema evolution, file optimization, and VACUUM operations.
• Deep working knowledge of Medallion / Lakehouse architecture patterns and best practices.
• Minimum 5 years of Power BI development experience including complex DAX, data modeling, and enterprise report design.
• Experience with Power BI Premium, Deployment Pipelines, and workspace administration.
• Proven ability to architect and manage enterprise-grade semantic models and datasets connected to cloud data platforms.
• Hands-on experience with Azure Data Lake Storage, Azure Data Factory, or equivalent cloud data services.
• Strong SQL skills including complex joins, CTEs, window functions, and query performance tuning.
• Experience with CI/CD pipelines and version control (Azure DevOps, Git) for data and BI workflows.
• 2 or more years of experience leading, mentoring, or managing BI developers; offshore team management experience strongly preferred.
• Demonstrated ability to manage workload, quality standards, and delivery timelines across distributed teams.
• Excellent written and verbal communication skills with the ability to present technical findings to non-technical stakeholders.
• Ability to manage competing priorities and deliver consistent results in a fast-paced, remote environment.

EDUCATION / CERTIFICATION

• Bachelor's degree in Computer Science, Information Systems, Data Engineering, or a related field preferred; equivalent professional experience considered.
• Databricks Certified Associate or Professional certification (Data Engineer or Data Analyst track) preferred; willingness to obtain certification if not already held.
• Microsoft Certified: Power BI Data Analyst Associate (PL-300) preferred; willingness to obtain certification if not already held.