ZipRecruiter

Log In

1

Overnight Databricks Data Engineer Jobs in Edison, NJ

StratG Inc

Data Engineer

Holmdel, NJ · On-site

$127K - $153K/yr

Data Engineer Location : New Jersey who is a self-starter, fueled by collaboration, able to ... Expert in AWS Databricks, Python, Spark * Strong SQL, and UNIX Shell scripting experience

CYNET SYSTEMS

Data Engineer

Jersey City, NJ · On-site

$55 - $60/hr

... an ETL/ELT developer/data engineer. * 3+ years of Databricks data engineering experience ... Expert in AWS Databricks, Python, Spark. * Strong SQL and UNIX Shell scripting experience.

Group Nine

Databricks Data Architect

Raritan, NJ · On-site

$65.25 - $84/hr

Data Engineering Execution * Build and maintain ingestion frameworks (ADF / Databricks / Spark) * Implement Bronze -> Silver transformations aligned to architecture * Architect data quality checks ...

New

SmartIT Frame

Azure Databricks Architect

Morristown, NJ · On-site

$65.50 - $85.25/hr

Key Skills: Azure Databricks & Data Engineering Apache Spark, PySpark/Scala, SQL Delta Lake & Lakehouse Architecture Azure Data Factory, ADLS, Synapse ETL/ELT Pipeline Development & Performance ...

GEI Consultants

Data Engineer

Manhattan, NY · On-site

$90K - $150K/yr

Azure or Databricks certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect Expert, Databricks Data Engineer Professional) are a plus. We are GEI. Some of the world's most ...

GEI Consultants

Data Engineer

New York, NY · On-site

$125K - $150K/yr

Azure or Databricks certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect Expert, Databricks Data Engineer Professional) are a plus. We are GEI. Some of the world's most ...

Fusemachines

Data Engineer

New York, NY · On-site

$125K - $150K/yr

Databricks Certified Data Engineer Associate * Microsoft Certified: Azure Fundamentals * Microsoft Certified: Azure Data Engineer Associate * Microsoft Exam: Designing and Implementing Microsoft ...

Fusemachines

Data Engineer

New York, NY · On-site

$125K - $150K/yr

Databricks Certified Data Engineer Associate * Microsoft Certified: Azure Fundamentals * Microsoft Certified: Azure Data Engineer Associate * Microsoft Exam: Designing and Implementing Microsoft ...

Tror AI for everyone

Azure Databricks Engineer

Iselin, NJ · On-site

$61 - $79.25/hr

Data Pipeline Development: * Build and maintain scalable ETL/ELT pipelines using Databricks ... Azure Data Engineer Associate or Databricks certified Data Engineer Associate certification ...

next page

Showing results 1-20

All JobsDatabricks Data Engineer JobsOvernight Databricks Data Engineer JobsOvernight Databricks Data Engineer Jobs in Edison, NJ

Overnight Databricks Data Engineer information

What is the difference between Overnight Databricks Data Engineer vs Data Engineer?

AspectOvernight Databricks Data Engineer
Work EnvironmentPrimarily remote or on-site, working overnight shifts to support global data operations
CertificationsDatabricks certifications, cloud platform credentials (AWS, Azure), data engineering certifications
Tools & TechnologiesDatabricks platform, Spark, cloud services, SQL, Python, ETL tools
Industry UsageTech, finance, healthcare, retail with 24/7 data needs

While both roles focus on data engineering, the Overnight Databricks Data Engineer specializes in managing data pipelines on the Databricks platform during overnight shifts, often supporting global operations. A Data Engineer may work across various platforms and shifts, with broader responsibilities in data architecture and pipeline development. The overnight role emphasizes specific platform expertise and shift timing, catering to organizations with continuous data processing needs.

What are some unique challenges faced by Overnight Databricks Data Engineers, and how can they be addressed?

Overnight Databricks Data Engineers often work with limited real-time support, which can present challenges when troubleshooting urgent data pipeline issues or system outages. To address this, it’s essential to develop strong problem-solving skills, document processes thoroughly, and leverage automated monitoring and alerting tools. Additionally, close collaboration with daytime teams during handoff periods ensures continuity and minimizes disruptions. Building a habit of proactive communication and maintaining detailed logs helps the entire team resolve issues efficiently and maintain data quality.

What are the key skills and qualifications needed to thrive as an Overnight Databricks Data Engineer, and why are they important?

To thrive as an Overnight Databricks Data Engineer, you need strong proficiency in data engineering, Python or Scala programming, and experience with big data technologies, typically supported by a relevant degree in computer science or a related field. Familiarity with Databricks, Apache Spark, cloud platforms (such as AWS or Azure), and certifications like Databricks Certified Data Engineer are highly valued. Attention to detail, problem-solving, and effective communication are essential soft skills, especially for troubleshooting and collaborating across shifts. These competencies ensure reliable data pipeline management and efficient resolution of issues during off-hours, maintaining seamless business operations.

What is an Overnight Databricks Data Engineer?

An Overnight Databricks Data Engineer is a professional who works primarily during night shifts to manage, design, and maintain big data pipelines and workflows using Databricks, a cloud-based data analytics platform. Their responsibilities often include developing and optimizing data processing jobs, ensuring data quality, and troubleshooting issues that arise during overnight data operations. This role is critical for organizations that require 24/7 data processing, continuous ETL jobs, or real-time analytics. Working overnight may also involve monitoring automated systems, performing scheduled data loads, and collaborating with global teams to ensure data availability and reliability.
What are the most commonly searched types of Databricks Data Engineer jobs in Edison, NJ? The most popular types of Databricks Data Engineer jobs in Edison, NJ are:
What cities near Edison, NJ are hiring for Overnight Databricks Data Engineer jobs? Cities near Edison, NJ with the most Overnight Databricks Data Engineer job openings:
Overnight Databricks Data Engineer jobs near you
Databricks Data Engineer

Databricks Data Engineer

Recruit Professional Services

Bedminster, NJ • On-site

$116K - $140K/yr

Other

Re-posted 17 days ago

Job description

Databricks Data Engineer
Responsibilities:
  • Assist with leading the team's transition to the Databricks platform and utilize the newer features of Delta Live Tables, Workflows etc.
  • Design and develop data pipelines that extract data from Oracle, load it into the data lake, transform it into the desired format, and load it into Databricks data lakehouse.
  • Optimize data pipelines and data processing workflows for performance, scalability, and efficiency.
  • Implement data quality checks and validations within data pipelines to ensure the accuracy, consistency, and completeness of data.
  • Help create and maintain documentation for data mappings, data definitions, architecture and data flow diagrams.
  • Build proof-of-concepts to determine viability of possible new processes and technologies.
  • Deploy and manage code in non-prod and prod environments.
  • Investigate and troubleshoot data related issues and fix or provide solutions to fix defects.
  • Identify and resolve performance bottlenecks, which could include suggesting ways to optimize and performance tune databases and queries to enhance query performance.

Requirements:
  • Bachelor's Degree in Computer Science, Data Science, Software Engineering, Information Systems, or related quantitative field.
  • 4+ years of experience working as a Data Engineer, ETL Engineer, Data/ETL Architect or similar roles.
  • Current/active Databricks Data Engineer/Analyst certification a BIG plus.
  • 3+ years working with Databricks with knowledge and expertise of data structures, data storage and change data capture gained from prior production implementations of data pipelines, optimizations, and best practices.
  • Solid continuous experience in Python.
  • 3+ years of experience in Kimball dimensional modeling (star-schema comprising of facts, type1 and type2 dimensions, aggregates, etc.) with solid understanding of ELT/ETL.
  • 3+ years of solid experience writing SQL and PL/SQL code.
  • 2+ years of experience with Airflow.
  • 3+ years of experience working with relational databases (Oracle preferred).
  • 2+ years of experience working with NoSQL databases: MongoDB, Cosmos DB, DocumentDB or similar.
  • 2+ years of cloud experience (Azure preferred).
  • Experience with CI/CD utilizing git/Azure DevOps.
  • Experience with storage formats including Parquet/Arrow/Avro.

#RecruitPS

Apply