Director Databricks Data Engineer Jobs in Missouri

The Senior Data Engineer is responsible for leading the development of scalable, data-driven solutions that enable trusted business intelligence, analytics, and enterprise data products across the Oil, Gas and Chemical (OGC) Global Practice. This role partners with engineering, construction, business, data engineering, and technology stakeholders to establish reliable data foundations that support strategic decision-making and operational excellence.

This position serves as a technical leader in advancing the organization's Databricks-based data ecosystem through modern data engineering practices, analytics, governance, and data product development. The role is responsible for designing and optimizing data pipelines, implementing data quality controls, managing enterprise data transformations, and enabling scalable solutions that support reporting, business intelligence, and analytical initiatives.

The Senior Data Engineer establishes and promotes best practices for data management, governance, metadata management, lifecycle controls, and analytical solution development while supporting the organization's long-term data strategy. Machine learning, predictive analytics, and advanced statistical techniques may be applied where appropriate to deliver measurable business value and improve decision-making capabilities.

Key Responsibilities:

Data Platform and Data Engineering

• Lead the design, development, implementation, and support of scalable data pipelines, orchestration frameworks, and data transformation processes utilizing the Databricks platform.
• Design, develop, and optimize Databricks notebooks, workflows, Delta Lake architectures, and enterprise data products to support business and project delivery needs.
• Oversee the collection, ingestion, cleansing, transformation, validation, and quality control of structured and unstructured data assets.
• Implement database-layer transformations, query optimization strategies, and automated quality controls to support a high-performance shared data environment.
• Partner with architects, developers, and data engineers to align solutions with enterprise architecture, governance standards, and platform best practices.

Data Governance and Data Products

• Establish and promote data governance standards, including data ownership, lifecycle management, metadata management, and data quality controls.
• Lead implementation of enterprise data quality frameworks and validation processes to ensure trusted and consistent data assets.
• Develop and maintain enterprise data flow documentation that visualizes data origins, transformations, dependencies, and downstream consumption across systems.
• Lead the development and lifecycle management of enterprise data products supporting analytics, reporting, operational processes, and business decision-making.
• Support Code of Account (COA) mapping initiatives by identifying, modeling, validating, and governing quantity and cost-related data relationships across estimating, engineering, procurement, and construction systems.

Analytics and Business Intelligence

• Analyze data to discover business value, trends, relationships, and opportunities that support strategic business initiatives and operational improvements.
• Develop dashboards, reports, and visualizations utilizing Power BI and related technologies to communicate technical results and business insights.
• Perform root cause analysis, trend analysis, and analytical investigations to support business performance and process improvement.
• Apply machine learning, predictive analytics, and statistical techniques where appropriate to solve business problems and improve decision-making outcomes.

Leadership and Collaboration

• Partner with engineering, construction, project delivery, and business stakeholders to understand data requirements and improve enterprise data maturity.
• Evaluate emerging platform technologies, analytical capabilities, industry practices, and data solutions for applicability to business challenges and opportunities.
• Train and mentor less experienced data professionals and business data users. Provide performance feedback to managers.
• Participate in recruitment efforts and technical candidate evaluations.
• Other duties, as assigned.

• Bachelor's degree in Analytics, Computer Science, Information Systems, Statistics, Mathematics, Engineering, or a related field; and a minimum of 8 years of related experience.
• Experience designing, developing, deploying, and supporting production solutions within Databricks environments strongly preferred.
• Strong knowledge of Databricks Lakehouse architecture, Delta Lake, Unity Catalog, Databricks Workflows, and Databricks notebooks strongly preferred.
• Experience implementing governance, security, metadata management, and data lifecycle processes within enterprise data platforms strongly preferred.
• Experience with Databricks, Python, SQL, and modern data engineering practices.
• Experience developing, supporting, and optimizing scalable data pipelines, orchestration frameworks, and cloud-based data solutions.
• Knowledge of data governance frameworks, metadata management, data quality methodologies, and lifecycle management practices.
• Experience leading data mining, advanced analytics, machine learning, predictive analytics, and statistical modeling initiatives.
• Experience developing enterprise data products and analytical solutions that support business intelligence and operational processes.
• Experience with Power BI or similar data visualization and reporting technologies.
• Experience with GCP BigQuery or similar cloud data warehouse technologies is preferred.
• Familiarity with engineering, construction, project delivery, asset, or related technical data environments is preferred.
• Ability to collaborate effectively with business stakeholders, architects, developers, data engineers, and application teams.
• Ability to translate complex data ecosystems into practical, scalable, and sustainable enterprise solutions.
• Strong problem-solving and analytical skills.
• Strong attention to detail and commitment to data quality.
• Excellent verbal and written communication skills with the ability to present technical concepts and findings to business audiences.