Freelance Databricks Data Engineer Jobs in Virginia

We are seeking to hire a Data Engineer to our team!

Client:
CTEC develops and delivers innovative customer-centric technologies and solutions that support the Office of Personnel Management's (OPM) Health and Insurance business unit and Office of the Chief Information Officer (OCIO).

Duties and Responsibilities:

• Data Pipeline & ETL Development: Design, develop, and maintain scalable ETL pipelines and data workflows to ingest, transform, and integrate data from legacy systems and external sources into modern cloud-based data platforms.
• Cloud Data Platform & Databricks Implementation: Build, optimize, and maintain data processing solutions using Azure Databricks and lakehouse architectures to support analytical, operational, and reporting use cases.
• Data Migration Support: Support phased data migration from legacy databases and ETL tools to Azure Databricks environments, including transformation documentation and data mapping.
• Lakehouse & Data Layer Design: Implement layered lakehouse data architectures (e.g., bronze, silver, gold layers) in Databricks to support data quality, performance, and downstream reporting needs.
• Python & PySpark Development: Develop data processing notebooks, workflows, and distributed data transformations using Python and PySpark within Databricks environments.
• Data Quality & Validation: Develop data validation, reconciliation, and testing processes to ensure data accuracy, completeness, and consistency across data domains.
• Data Integration & Interoperability: Integrate Databricks data platforms with analytics and reporting tools to enable business intelligence and operational dashboards.
• Data Governance & Security Implementation: Support data governance initiatives including metadata management, data catalog integration, encryption, access controls, and compliance with federal data protection requirements.
• DevOps & CI/CD Support: Maintain source control and CI/CD pipelines for Databricks and data engineering workflows, supporting automated promotion across environments.
• Technical Collaboration: Work closely with data architects, solution architects, business analysts, and reporting teams to implement approved data solutions.
• Operational Support & Troubleshooting: Provide ongoing support for Databricks workflows, resolve pipeline failures, and troubleshoot complex data processing issues.
• Mentorship & Knowledge Sharing: Provide guidance to junior data engineers and contribute to documentation and team enablement.
• Works under minimal supervision with minor guidance from senior personnel.

Skills & Work Experience:

• Professional Experience: 7+ years of experience in data engineering, ETL development, or large-scale data integration environments.
• Strong experience designing and developing ETL pipelines and data transformations in Azure Databricks environments.
• Strong proficiency in SQL and Python, with hands-on experience using PySpark for distributed data processing.
• Experience working with cloud-based data platforms, data lakes, and lakehouse environments, preferably on Microsoft Azure.
• Experience implementing layered lakehouse data architectures (bronze, silver, gold) for enterprise analytics.
• Familiarity with Spark-based big data processing frameworks.
• Experience supporting data migration from legacy databases and ETL tools to Databricks-based platforms.
• Experience integrating Databricks platforms with business intelligence and reporting tools such as Power BI.
• Familiarity with data governance, metadata management, and data security best practices.
• Experience with source control and CI/CD pipelines for data engineering and Databricks workflows.
• Working knowledge of SDLC and Agile delivery methodologies.
• Excellent organizational, communication, and collaboration skills.

Preferred:

• Hands-on experience with Azure Databricks workflows, Delta tables, and Databricks job orchestration.
• Experience with Azure Data Lake, Azure Data Factory, or similar Azure data services.
• Experience supporting federal IT modernization or large-scale enterprise data transformation initiatives.
• Familiarity with healthcare, insurance, or benefits administration data environments.
• Experience implementing data governance or data catalog platforms.

Education:

Bachelor's degree in Computer Science, Engineering, Information Systems, or a related technical discipline. Equivalent education or professional experience will be considered in lieu of a degree.

Clearance:
Must be a U.S. citizen and be able to obtain a Public Trust clearance.