We are seeking an experienced Senior Data Engineer with strong expertise in
Azure Databricks, Spark, and Azure data services to design and deliver scalable, cloud-based data solutions for
healthcare analytics.
The ideal candidate will have hands-on experience in building modern data platforms, implementing CI/CD pipelines, and enabling advanced analytics and machine learning initiatives while ensuring data security and regulatory compliance.
Responsibilities
- Design, develop, and maintain scalable, secure, and high-performance ETL/ELT pipelines using Azure Databricks, PySpark, Synapse Analytics,
- Work with structured and unstructured data from various healthcare sources (EHR, claims data, etc.).
- Implement and optimize data storage solutions using Azure technologies such as Databricks, Azure Data Lake, Azure SQL, Synapse Analytics.
- Develop and deploy machine learning models and predictive analytics solutions to support healthcare analytics.
- Ensure data quality, integrity, and security in compliance with HIPAA and other healthcare regulations.
- Collaborate with data scientists, analysts, and software engineers to deliver high-performance data solutions.
- Automate data workflows using cicd, and other cloud tools.
- Monitor and troubleshoot data pipelines to ensure optimal performance and reliability.
- Participate in technical design reviews and contribute to data platform architecture decisions.
- Promote data engineering best practices across the organization.
Required Skills & Qualifications:
- 8+ years of experience as a Data Engineer with a focus on Azure cloud services.
- Strong understanding of Delta Lake, Lakehouse Architecture, and data warehousing concepts.
- Strong expertise in SQL, Python, Spark, and Databricks.
- Azure/Databricks certifications such as Microsoft Certified: Azure Data Engineer Associate.
- Hands-on experience with Azure Data Factory, Azure Synapse, Azure Data Lake, and Power BI.
- Knowledge of machine learning models and data science techniques in a healthcare setting.
- Experience working with FHIR, HL7, and other healthcare data standards.
- Familiarity with DevOps and CI/CD pipelines for data workflows.
- Strong problem-solving skills and ability to work in an agile environment.