Remote Azure Databricks Jobs in Washington, DC (NOW HIRING)

The Senior Data Engineer will own the end-to-end data pipelines that ingest historical and transactional records from rigid legacy Systems of Record (SOR) and structure them into a high-performance environment on Databricks. The ideal candidate will be responsible for resolving legacy synchronization gaps, eradicating administrative tracking burdens, and establishing an auditable, compliant, and performant data foundation to feed downstream Power Platform applications and Azure AI Search-powered Retrieval-Augmented Generation (RAG) models.

Core Technical Responsibilities

• Databricks Spark Pipelines & ETL Engineering: Build and scale distributed PySpark and Spark SQL pipelines on Azure Databricks to ingest raw scheduled feeds (nightly/weekly flat files or database links) from source systems. Ensure high-performance data cleaning and schema enforcement.
• Medallion Lakehouse Architecture Implementation: Develop and maintain Delta Lake schemas across the Medallion Architecture. Maintain exact, append-only copies in the Bronze layer; build automated multi-source joins and reconciliation mapping in the Silver layer and publish optimized multi-dimensional tables in the Gold layer representing unified, historical actuals.
• Continuous Reconciliation Engine & SQL Triggers: Design and deploy Databricks automated jobs that scan the Silver layer to proactively identify and flag unmatched, misaligned, or trailing ledger transactions prior to monthly/quarterly closeouts. Write scheduled SQL trigger procedures to manage and clear 'Pending Internal Holds' inside the Azure SQL serving layer as soon as transaction matches appear in core warehouse feeds.
• Database Serving Layer Optimization & Schema Resilience: Manage the Azure SQL Serving Layer to host application metadata, scenario models, routing matrices, and transaction logs. Formulate and enforce multi-tenant partitioning strategies using mandatory Fiscal_Year and Record_Version schema columns. Drive the data ingestion granularity strategy, keeping detailed transactions in Delta Lake while rolling up aggregated monthly balances in Azure SQL to ensure sub-second query times inside Dataverse Virtual Tables and Power Apps.
• Downstream Integration & Virtual Table Provisioning: Configure and optimize SQL views exposed as Dataverse Virtual Tables, enabling the Power Platform front-end (Model-Driven power user dashboards and embedded Canvas apps) to read real-time blended actuals without data replication or excessive storage fees. Standardize environment variables inside the deployment profile framework to prevent unmanaged, hardcoded environmental settings.
• AI RAG Pipeline Data Ingestion & Security: Collaborate with the AI Engineer to feed unstructured programmatic files into Azure Blob Storage and structure them for Azure AI Search vectorization. Ensure all data structures adhere to strict Federal compliance, maintaining data boundaries within the secure FAA Azure GovCloud (GCC) tenant and conforming to FedRAMP, FISMA, and Section 508 accessibility standards.

Required Experience & Technical Skill Set

• Databricks Mastery: Minimum of 4-6 years of direct experience with Azure Databricks, Delta Lake, and Unity Catalog within a production cloud ecosystem.
• Apache Spark & Coding: Advanced proficiency in PySpark and Spark SQL for developing distributed, high-throughput ETL/ELT pipelines and handling high-concurrency relational data.
• Relational Database Expertise: Strong knowledge of Azure SQL Database / Microsoft SQL Server, including schema partitioning, performance tuning, indexed views, and complex triggers/stored procedures.
• Integration & Pipeline Patterns: Experience with automated ingestion pipelines utilizing flat files, API connectors, serverless Azure Functions, and Power Automate flow integrations.
• Power Platform Alignment: Understanding of Microsoft Dataverse Virtual Tables and Environment Variables to manage connection references and abstract code across Dev, Test, UAT, and Prod environments.
• Federal Compliance & Security: Direct experience operating within Azure GovCloud (GCC), ensuring strict compliance with FedRAMP (Moderate/High controls), FISMA ATO standards, and NIST SP 800-218 supply chain requirements.
• Federal Financial Systems (Preferred): Highly preferred experience working with federal financial systems of record such as Oracle Delphi (SGL accounting strings), PRISM, and/or federal payroll structures (REGIS/FPPS).

About YCG

YCG is a fast-growing, remote-first technology company focused on delivering innovative solutions across the Microsoft ecosystem. We specialize in Power Platform, Dynamics 365, and Azure, helping organizations modernize operations, automate workflows, and unlock data-driven insights.

Our expertise includes Power Apps, Power BI, Power Pages, and Dataverse, combined with advanced capabilities in Azure AI/ML, cloud compute, analytics, Purview, eDiscovery, and integration services. We build scalable, intelligent solutions that drive efficiency and support smarter decision-making.

We bring experience working in government contracting environments, including familiarity with agencies such as the FAA, as well as delivering solutions across finance, contracts, and program management domains. We understand the importance of security, compliance, and performance in highly regulated industries.

As a growing team, we offer a collaborative culture, flexible remote work, and opportunities to work on cutting-edge cloud and AI solutions. If you're looking to make an impact and grow with a company on the rise, we’d love to connect.