ZipRecruiter

Log In

1

Overnight Databricks Data Engineer Jobs (NOW HIRING)

Berkley

Databricks Data Engineer

Manassas, VA · On-site

$114.50K - $137.50K/yr

The Databricks Data Engineer will help design, build, deploy, and maintain scalable and production grade data pipelines in modern cloud environments, enabling analytics, AI, ML, and decision ...

Berkley

Databricks Data Engineer

Manassas, VA

$114.50K - $137.50K/yr

The Databricks Data Engineer will help design, build, deploy, and maintain scalable and production grade data pipelines in modern cloud environments, enabling analytics, AI, ML, and decision ...

Berkley

Databricks Data Engineer

Manassas, VA

$114.50K - $137.50K/yr

The Databricks Data Engineer will help design, build, deploy, and maintain scalable and production grade data pipelines in modern cloud environments, enabling analytics, AI, ML, and decision ...

Prophecy Technologies

Databricks Data Engineer

Marlborough, NH · On-site

$113.20K - $136K/yr

This role is for a Databricks Data Engineer with experience in designing, developing, and deploying robust, scalable batch and streaming data pipelines using PySpark, Spark SQL, and Delta Live Tables.

Horizon Industries

Databricks Data Engineer

Vienna, VA

$114.90K - $138K/yr

Who we are looking for We are seeking a Databricks Data Engineer for a full-time position supporting our various clients We are committed to your growth and success in the IT software development ...

Prophecy Technologies

Databricks Data Engineer

Charlotte, NC · On-site

$111.80K - $134.30K/yr

We are seeking a hands-on Databricks Data Engineer to design, develop, and optimize data pipelines and analytics solutions. Key Responsibilities: * Design, develop, and deploy scalable ETL/ELT data ...

Deloitte

Sr Databricks Data Engineer

Arlington, VA · On-site

$120.20K - $165.10K/yr

Share this job: Share: Share Sr Databricks Data Engineer with Facebook Share Sr Databricks Data Engineer with LinkedIn Share Sr Databricks Data Engineer with Twitter Caution against fraudulent job ...

Purple Drive Technologies

Databricks Data Engineer

Seattle, WA · On-site

$130.30K - $156.50K/yr

POSITION SUMMARY We are seeking an experienced Data Engineer specializing in Databricks platform for a 6-month contract position based in Seattle, WA. This role focuses on designing, building, and ...

Deloitte

Sr Databricks Data Engineer

Atlanta, GA · On-site

$110.10K - $132.20K/yr

As a Sr Consultant - Databricks Data Engineer, you will design, build, and optimize scalable data engineering solutions using Databricks across various cloud environments. Responsibilities : • Lead ...

Deloitte

Sr Databricks Data Engineer

San Francisco, CA · On-site

$134.90K - $162K/yr

As a Sr Consultant - Databricks Data Engineer, you will design, build, and optimize scalable data engineering solutions using Databricks across various cloud environments. Responsibilities : • Lead ...

Datavail

Databricks Data Engineer

Boulder, CO · On-site

$112.20K - $134.80K/yr

* As a Databricks Data Engineer, you will work directly with clients across multiple industries to design, implement, and optimize Databricks-based data solutions * You will be a key member of our ...

Deloitte

Sr Databricks Data Engineer

Kansas City, MO · On-site

$111.60K - $134K/yr

As a Sr Consultant - Databricks Data Engineer, you will design, build, and optimize scalable data engineering solutions using Databricks across various cloud environments. Responsibilities : • Lead ...

Deloitte

Sr Databricks Data Engineer

Boston, MA · On-site

$124.40K - $149.40K/yr

As a Sr Consultant - Databricks Data Engineer, you will design, build, and optimize scalable data engineering solutions using Databricks across various cloud environments. Responsibilities : • Lead ...

Deloitte

Sr Databricks Data Engineer

Denver, CO · On-site

$117.80K - $141.50K/yr

As a Sr Consultant - Databricks Data Engineer, you will design, build, and optimize scalable data engineering solutions using Databricks across various cloud environments. Responsibilities : • Lead ...

Deloitte

Sr Databricks Data Engineer

Manhattan, NY · On-site

$126.10K - $151.40K/yr

As a Sr Consultant - Databricks Data Engineer, you will design, build, and optimize scalable data engineering solutions using Databricks across various cloud environments. Responsibilities : • Lead ...

Deloitte

Sr Databricks Data Engineer

San Jose, CA · On-site

$134.20K - $161.10K/yr

As a Sr Consultant - Databricks Data Engineer, you will design, build, and optimize scalable data engineering solutions using Databricks across various cloud environments. Responsibilities : • Lead ...

Deloitte

Sr Databricks Data Engineer

Stamford, CT · On-site

$122.10K - $146.60K/yr

As a Sr Consultant - Databricks Data Engineer, you will design, build, and optimize scalable data engineering solutions using Databricks across various cloud environments. Responsibilities : • Lead ...

Datavail

Databricks Data Engineer

Boulder, CO

$112.20K - $134.80K/yr

* As a Databricks Data Engineer, you will work directly with clients across multiple industries to design, implement, and optimize Databricks-based data solutions * You will be a key member of our ...

next page

Showing results 1-20

People also search for

All JobsDatabricks Data Engineer JobsOvernight Databricks Data Engineer Jobs

Overnight Databricks Data Engineer information

What are the key skills and qualifications needed to thrive as an Overnight Databricks Data Engineer, and why are they important?

To thrive as an Overnight Databricks Data Engineer, you need strong proficiency in data engineering, Python or Scala programming, and experience with big data technologies, typically supported by a relevant degree in computer science or a related field. Familiarity with Databricks, Apache Spark, cloud platforms (such as AWS or Azure), and certifications like Databricks Certified Data Engineer are highly valued. Attention to detail, problem-solving, and effective communication are essential soft skills, especially for troubleshooting and collaborating across shifts. These competencies ensure reliable data pipeline management and efficient resolution of issues during off-hours, maintaining seamless business operations.

What are some unique challenges faced by Overnight Databricks Data Engineers, and how can they be addressed?

Overnight Databricks Data Engineers often work with limited real-time support, which can present challenges when troubleshooting urgent data pipeline issues or system outages. To address this, it’s essential to develop strong problem-solving skills, document processes thoroughly, and leverage automated monitoring and alerting tools. Additionally, close collaboration with daytime teams during handoff periods ensures continuity and minimizes disruptions. Building a habit of proactive communication and maintaining detailed logs helps the entire team resolve issues efficiently and maintain data quality.

What is an Overnight Databricks Data Engineer?

An Overnight Databricks Data Engineer is a professional who works primarily during night shifts to manage, design, and maintain big data pipelines and workflows using Databricks, a cloud-based data analytics platform. Their responsibilities often include developing and optimizing data processing jobs, ensuring data quality, and troubleshooting issues that arise during overnight data operations. This role is critical for organizations that require 24/7 data processing, continuous ETL jobs, or real-time analytics. Working overnight may also involve monitoring automated systems, performing scheduled data loads, and collaborating with global teams to ensure data availability and reliability.

What is the difference between Overnight Databricks Data Engineer vs Data Engineer?

AspectOvernight Databricks Data Engineer
Work EnvironmentPrimarily remote or on-site, working overnight shifts to support global data operations
CertificationsDatabricks certifications, cloud platform credentials (AWS, Azure), data engineering certifications
Tools & TechnologiesDatabricks platform, Spark, cloud services, SQL, Python, ETL tools
Industry UsageTech, finance, healthcare, retail with 24/7 data needs

While both roles focus on data engineering, the Overnight Databricks Data Engineer specializes in managing data pipelines on the Databricks platform during overnight shifts, often supporting global operations. A Data Engineer may work across various platforms and shifts, with broader responsibilities in data architecture and pipeline development. The overnight role emphasizes specific platform expertise and shift timing, catering to organizations with continuous data processing needs.

What cities are hiring for Overnight Databricks Data Engineer jobs? Cities with the most Overnight Databricks Data Engineer job openings:
What are the most commonly searched types of Databricks Data Engineer jobs? The most popular types of Databricks Data Engineer jobs are:
What states have the most Overnight Databricks Data Engineer jobs? States with the most job openings for Overnight Databricks Data Engineer jobs include:
Overnight Databricks Data Engineer jobs near you
Databricks Data Engineer

Databricks Data Engineer

Berkley

Manassas, VA • On-site

$114.50K - $137.50K/yr

Other

Medical, Dental, Vision, Life, Retirement, PTO

Posted 5 days ago

Job description

Company Details
We started in early 2019 as a small group of technologists with a passion for making insurance better. Today we are working with a team of industry experts who run five different insurance brands and collectively control $1 billion in annual premiums.
We believe in an idea and execution meritocracy. In other words, a place where the best ideas win and the people who deliver the most value get the most opportunities.
As we grow our team, we are looking for inquisitive, entrepreneurial people who are excited to reimagine the insurance industry.
Insurance is too complex. Help us make it better.
Responsibilities
This position requires on-site work Monday-Thursday at either our Manassas, VA or Chesterfield, MO location.
The Databricks Data Engineer will help design, build, deploy, and maintain scalable and production grade data pipelines in modern cloud environments, enabling analytics, AI, ML, and decision advantage at scale. This role will work with cutting-edge tools like Databricks, Delta Lake, PySpark, and AI/BI genie to transform raw data into actionable insights. As a hands-on Databricks Data Engineer with deep expertise in Azure Databricks and MLOps, this role will have the opportunity to migrate and translate legacy SSIS ETL logic into scalable, cloud-native data pipelines in Databricks. This role will partner with data engineers, data scientists, and product manager to design features, train/evaluate models, and deploy them to production using MLflow, Databricks and Workflows-with rigorous observability, governance (Unity Catalog), and CI/CD automation.
Data Pipeline Engineering
  • Design, build, and maintain high-performance, scalable ETL/ELT pipelines using Azure Databricks, Delta Lake, and PySpark.
  • Convert and modernize existing SSIS package logic into cloud-native Databricks pipelines using PySpark notebooks, Delta Live Tables (DLT), and Databricks Workflows.
  • Implement reliable batch and streaming pipelines with robust data quality and validation frameworks.
  • Optimize pipeline performance using Photon, efficient file formats, partitioning, Z-ordering, and caching strategies.

Lakehouse Platform Development
  • Develop and manage datasets within Delta Lake, ensuring ACID reliability, schema evolution, versioning, and time travel.
  • Architect feature-rich data layers including:
    • Bronze (raw ingestion)
    • Silver (validated, conformed)
    • Gold (analytics-ready and ML-ready)
  • Implement data governance using Unity Catalog for fine-grained access control, lineage, auditability, and metadata management.

MLOps & ML-Enabled Data Pipelines
  • Partner with data scientists and data engineers to create feature pipelines, model training pipelines, and production scoring pipelines.
  • Deploy and operationalize models using MLflow, Databricks Model Registry, and Databricks Workflows.
  • Use Databricks built-in AI SQL functions such as ai_query, ai_forecast, ai_analyze_sentiment to generate actionable insight from large amount of unstructured or structured raw data
  • Implement monitoring for:
    • Pipeline failures
    • Data/feature drift
    • Model performance degradation
    • Operational SLAs/SLIs/SLOs
  • Build automated CI/CD workflows using GitHub Actions or Azure DevOps for notebook deployment, pipeline testing, and environment promotion.

Data Platform, Data Security & Data Governance
  • Collaborate with data engineers to design reliable data products on Delta Lake; leverage Delta Live Tables (DLT) for declarative pipelines when applicable.
  • Enforce Unity Catalog for lineage, permissions, and audit; manage secrets, tokens, and keys securely (e.g., Databricks secrets, Key Vault/Secrets Manager).

Collaboration & Leadership
  • Work closely with cross-functional teams: data engineering, data scientist, product manager, and business stakeholders.
  • Serve as a Databricks SME-championing best practices, code standards, governance, and reusable frameworks.
  • Document architecture, workflows, data models, runbooks, and operational procedures.

Qualifications
  • Minimum of 3 years of experience in Databricks, PySpark notebooks, Python, DevOps, software development, and data engineering.
  • Certified Databricks Data Engineer Associate or Professional is a plus.

Skills & Competencies
  • Proficient in designing, building, deploying, and maintaining high-performance, scalable ETL/ELT pipelines using Azure Databricks, Delta Lake, and PySpark Notebook.
  • Proficient in building, deploying, and operating production ML models such as supervised, unsupervised, and anomaly detection, including techniques for imbalanced datasets
  • Proficient with ML engineering and MLOps, including model versioning, CI/CD for ML, monitoring, drift detection, and automated retraining
  • Proficiency in Python including Pandas and PySpark Dataframes
  • Expert level of SQL skills including Stored Procedure, experience with SSIS, SSRS, Power BI is a plus.
  • Proficient with cloud data engineering platforms, such as Azure, Databricks, Spark, or SQL, and batch and streaming pipelines
  • Familiar with Databricks AI Built-In Functions such as AI_Query, AI_Gen, AI_Classify, AI_Forecast, AI_Analyze_Sentiment, able to use them to extract actionable insights from large amount of unstructured or structured raw data
  • Experience with Python and ML frameworks, such as PyTorch or TensorFlow
  • Experience improving data quality, lineage, and observability in enterprise data environments and operationalizing rules and model-driven scoring for prioritization, routing, or case selection
  • Experience with predictive analytics, machine learning and artificial intelligence desired.

Education
  • A Bachelor's degree in Computer Science, Management Information Systems, Engineering, Math, Physics, or a related quantitative field is required (4-year degree). Master's degree preferred
  • Experience in the commercial insurance industry is a plus.

Additional Company Details
The Company is an equal employment opportunity employer.
We do not accept any unsolicited resumes from external recruiting firms.
The company offers a competitive compensation plan and robust benefits package for full time regular employees.
Base salary & Benefits include Health, dental, vision, life, disability, wellness, paid time off, 401(k) and profit-sharing plans.
The actual salary for this position will be determined by a number of factors, including the scope, complexity and location of the role; the skills, education, training, credentials and experience of the candidate; and other conditions of employment.
Additional Requirements
• Ability to travel locally and nationally up to 5% of the time
Sponsorship Details
Sponsorship not Offered for this Role

Apply