Associate Azure Databricks Jobs in Ashburn, VA (NOW HIRING)

About Agile5: Agile5 Technologies, Inc., is a Woman-Owned Small Business (WOSB) and Information Technology (IT) services firm that specializes in the design, development, testing, integration, and maintenance of enterprise software systems. We believe our employees are the company’s most valuable asset. We are invested in seeing our employees grow in their careers, while maintaining a work/life balance. We have an immediate, full-time need for a skilled, energetic, and driven Senior Data Engineer.  

Description: The Senior Data Engineer serves as the technical lead for complex data migration and pipeline engineering initiatives within enterprise data lakehouse modernization efforts. This role leads the conversion of legacy ETL artifacts into scalable Python/PySpark code, migrates database systems to cloud-native Delta Lake architectures, and implements robust data governance and automated reconciliation frameworks. Operating in a secure environment, the ideal candidate will manage Databricks infrastructure, design CI/CD pipelines, and mentor engineering personnel while ensuring complete data accuracy.

Senior Data Engineer Job Duties:

• Serve as the primary technical lead executing daily migration operations, including data profiling, schema analysis, pipeline conversion, automated reconciliation, and production deployment validation.

• Migrate legacy Hive tables to Delta Lake format on AWS S3 storage using Databricks ingestion tools.

• Convert High and Medium complexity Informatica mappings to Python/PySpark code while preserving all business logic, data quality checks, and data cleansing routines.

• Design and implement Change Data Capture (CDC) pipelines to maintain data synchronization between legacy and target environments during migration.

• Configure and maintain Databricks workspaces, clusters, notebooks, and jobs within AWS GovCloud environments.

• Implement Unity Catalog data governance, including RBAC, column-level encryption, audit trails, and data lineage tracking.

• Build and execute automated data reconciliation scripts validating 100% migration accuracy.

• Develop and maintain CI/CD pipelines for Python code deployment using Azure DevOps, checking in all converted code with comprehensive documentation.

• Integrate the Databricks lakehouse with downstream applications (such as Power BI and ESRI) and configure SSO integrations.

• Access legacy enclave environments for data profiling and side-by-side validation while collaborating daily with database managers and analysts.

• Mentor mid-level engineering staff on platform operations and migration methodologies, participate in Agile ceremonies, and support system acceptance demonstrations.

• Performs other duties as assigned.

Security Clearance Requirements:

• Public Trust / Tier 4 Eligible: No clearance required to apply; must be a U.S. citizen willing to undergo a background check to obtain a Public Trust / Tier 4 clearance.

Experience Requirements:

• Minimum experience required varies by degree level: PhD with 4 years; Master's degree with 8 years; Bachelor's degree with 10 years; or High School Diploma with 14 years of relevant experience.

• Hands-on experience with Databricks (workspace administration, notebook development, job scheduling) and proficiency in Python, PySpark, and SQL for large-scale pipeline development.

• Experience migrating data from legacy platforms (Hive, Hadoop, Oracle) to cloud-native platforms, utilizing Delta Lake or Apache Iceberg table formats.

• Practical experience with AWS services (S3, IAM, RDS, GovCloud), CI/CD pipeline tools (Azure DevOps, Jenkins, GitHub Actions), and enterprise ETL/ELT frameworks.

Education Requirements: Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related technical field (or equivalent combination of education and experience).

Desired Skills / Qualifications:

• Databricks Certified Data Engineer (Associate or Professional).

• Direct experience converting Informatica PowerCenter mappings to Python/PySpark code.

• Experience with Unity Catalog data governance, Cloudera Hive/HiveQL, Power BI/ESRI integrations, and CDC methodologies.

• Familiarity with FISMA High or FedRAMP High security requirements and SAML-based SSO/Okta integrations.

Location: Washington, DC

Status: Full time

Schedule: Day shift, Monday–Friday

Physical Requirements: Must be able to remain in a stationary position for long durations of time. Also, must be able to continuously operate a computer and other office productivity machinery.  

Travel Required: No  

This job description is subject to change at any time.

We are an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability status, protected veteran status, or any other characteristic protected by law.