Job Description:Seeking an experienced Senior Databricks Developer to lead the design, development, optimization, and management of enterprise-scale cloud data engineering solutions using Azure Databricks and Azure data services. The ideal candidate should possess strong expertise in Azure Databricks, Azure Data Lake, ADLS Gen 2, ETL/ELT development, cloud-native data pipelines, and enterprise data architecture. This role requires leading end-to-end data engineering initiatives, building scalable pipelines for structured, semi-structured, API, and unstructured datasets, supporting analytics and AI workloads, and collaborating with business and technical stakeholders to deliver reliable, scalable, and cost-efficient data solutions. The candidate will also drive data governance, quality standards, performance tuning, metadata management, and continuous improvement across cloud data environments.
Key Responsibilities:
- Lead the design, development, and implementation of end-to-end data pipelines on Azure using Azure Databricks, Azure Data Lake, and ADLS Gen 2
- Develop scalable ETL/ELT pipelines supporting structured, semi-structured, API-based, and unstructured data sources
- Collaborate with Data Scientists, Business Analysts, architects, and stakeholders to translate business requirements into scalable technical data solutions
- Optimize data workflows for performance, scalability, reliability, and cost-efficiency within cloud environments
- Establish, maintain, and enforce enterprise data quality, governance, validation, and compliance standards
- Provide technical guidance for data modeling, schema design, metadata management, and analytics-ready data structures
- Support enterprise analytics, reporting, AI, and machine learning workloads through optimized data engineering solutions
- Monitor, troubleshoot, and resolve issues related to production data pipelines and cloud-based integrations
- Recommend performance improvements and optimization strategies for enterprise data workflows
- Evaluate emerging Azure and Databricks technologies and recommend enhancements to enterprise data architecture
Additional Responsibilities:
- Support enterprise cloud modernization and analytics transformation initiatives
- Participate in architectural discussions, technical reviews, and solution planning sessions
- Collaborate with engineering and analytics teams to improve scalability and operational performance
- Maintain technical documentation, standards, and best practices for enterprise data engineering initiatives
- Support continuous improvement and innovation within Azure cloud and Databricks environments
Required Skills:
- Strong expertise in Azure Databricks and enterprise data engineering
- Experience designing and developing ETL/ELT pipelines for enterprise environments
- Strong experience with Azure Data Lake and ADLS Gen 2
- Experience working with structured, semi-structured, API-based, and unstructured datasets
- Strong knowledge of data modeling, schema design, and metadata management
- Experience with cloud performance optimization, monitoring, and troubleshooting
- Strong understanding of data governance, validation, and data quality standards
- Experience supporting analytics, AI, and machine learning workloads using cloud data platforms
- Strong collaboration, analytical, and problem-solving skills
Qualifications:
- Senior-level experience in cloud-based data engineering and Azure Databricks environments preferred
- Experience collaborating with business, analytics, and technical teams required
- Experience supporting enterprise analytics and AI-driven workloads preferred
- Strong cloud optimization and enterprise-scale data architecture experience preferred
Certifications:
- Microsoft Azure certification preferred
- Databricks certification preferred