Databricks Data Engineer Jobs in Aurora, IL (NOW HIRING)

The Senior Data Engineer - Quality is a hands-on technical role responsible for designing and building robust, scalable, end-to-end testing frameworks for modern data pipelines. This role focuses on ensuring data quality across ingestion (APIs, SQL Server, flat files) and transformation using Medallion Architecture (Bronze, Silver, Gold).

The ideal candidate has strong experience in data quality engineering, Python and SQL, and building automated validation frameworks using tools such as Great Expectations. Experience leveraging AI-assisted development tools (e.g., Claude Code) and prompt engineering to accelerate test generation and standardization is highly valued.
This role is critical to ensuring reliable, high-quality data products that power downstream analytics in tools such as Power BI and Tableau.

• Design and implement scalable end-to-end testing frameworks for data pipelines

• Validate ingestion from APIs, SQL Server, and flat files

• Ensure data quality across Medallion architecture layers (Bronze, Silver, Gold)

• Build automated checks for schema validation, data integrity, and transformations

• Develop reusable validation patterns using Great Expectations or similar frameworks

• Leverage Claude Code and prompt engineering to accelerate test generation and standardization

• Create reusable AI-driven testing assets (.md skills, templates) and workflows for AI-assisted coding and testing

• Integrate testing into CI/CD pipelines (Azure DevOps, GitHub Actions)

• Collaborate with data engineers, Product, and DevOps to define data quality standards and acceptance criteria

• Monitor and improve data reliability, observability, and test coverage

• Investigate data quality issues and drive root-cause resolution

Education and/or Experience:
Required:

• Bachelor's degree in Computer Science, Information Technology, or related field.

• A minimum of 5 years of experience in data quality engineering, data testing, or data platform engineering

• Strong experience building automated testing frameworks for data pipelines

• Hands-on experience with data validation tools (Great Expectations, DQX - Databricks Data Quality Framework, dbt tests)

• Proficiency in Python and strong SQL skills

• Experience with AI-assisted development tools (e.g., Claude Code, Copilot)

• Strong understanding of prompt engineering and reusable AI workflows

• Experience validating data ingestion from APIs, relational databases, and flat files

• Deep understanding of data transformation validation and Medallion Architecture

• Experience integrating testing into CI/CD pipelines (Azure DevOps, GitHub Actions, etc.)

• Familiarity with data observability and monitoring practices

Preferred:

• Experience with Azure Databricks and Apache Spark (PySpark)

• Familiarity with Delta Lake and Unity Catalog

• Experience building reusable AI prompt libraries or skills (.md or similar formats)

• Hands-on experience using Claude Code for automation or framework generation

• Knowledge of data contracts and schema evolution strategies

• Experience with test data management and synthetic data generation

• Familiarity with infrastructure as code (Terraform, Bicep)

• Experience with streaming data pipelines (Kafka, Event Hubs)

Other Skills and Abilities:
The following will also be required of the successful candidate:

• Strong organizational skills

• Strong attention to detail

• Good judgment

• Strong interpersonal communication skills

• Strong analytical and problem-solving skills

• Able to work harmoniously and effectively with others

• Able to preserve confidentiality and exercise discretion

• Able to work under pressure

• Able to manage multiple projects with competing deadlines and priorities

#LI-Hybrid
#LI-OE1

The target salary range for this role is:

Salaries vary by location and are based on numerous factors, including, but not limited to, the relevant market, skills, experience, and education of the selected candidate. Our compensation package also includes bonus eligibility and a comprehensive benefits program. Benefits information can be found at Sidley.com/Benefits.

To perform this job successfully, an individual must be able to perform the Duties and Responsibilities above satisfactorily and meet the requirements. The requirements listed above are representative of the minimum knowledge, skill, and/or ability required. Reasonable accommodations will be made to enable individuals with disabilities to perform the essential functions of the job. If you need such an accommodation, please email sidleytalentacquisition@sidley.com (current employees should contact Human Resources).

Sidley Austin LLP is an Equal Opportunity Employer.