We are seeking a skilled Data Engineer with 5 7 years of experience to design, build, and maintain scalable data pipelines and cloud-based data solutions. The ideal candidate will have strong expertise in SQL, Azure Data Services, and Databricks, with experience supporting analytics, reporting, and data-driven business initiatives. This role will collaborate closely with data analysts, software engineers, architects, and business stakeholders to deliver reliable, high-quality data solutions.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Azure and Databricks.
- Build and optimize data ingestion processes from multiple structured and unstructured data sources.
- Develop and maintain data models, data warehouses, and data marts to support business intelligence and analytics needs.
- Write complex SQL queries, stored procedures, and performance-tuned data transformations.
- Implement data quality, validation, and monitoring processes to ensure data accuracy and integrity.
- Develop and maintain Databricks notebooks, workflows, and Spark-based data processing solutions.
- Collaborate with business stakeholders, analysts, and engineering teams to gather requirements and deliver data solutions.
- Optimize data pipelines for performance, scalability, and cost efficiency.
- Support production deployments, troubleshoot data issues, and perform root cause analysis.
- Create and maintain technical documentation, data lineage, and operational procedures.
Required Qualifications
- 5 7 years of experience in Data Engineering, Data Warehousing, or related roles.
- Strong SQL expertise including query optimization, performance tuning, indexing, and complex data transformations.
- Hands-on experience with Azure Data Services including Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure SQL Database, and Azure Synapse Analytics.
- Strong experience with Databricks, Apache Spark, PySpark, and notebook development.
- Experience building and maintaining ETL/ELT pipelines in cloud environments.
- Proficiency in Python for data engineering and automation tasks.
- Experience with dimensional modeling, star/snowflake schemas, and data warehousing concepts.
- Knowledge of data governance, data quality, and metadata management practices.
- Familiarity with Git-based source control and Agile development methodologies.
- Strong analytical, troubleshooting, and problem-solving skills.
Preferred Qualifications
- Experience with Power BI or other reporting and visualization platforms.
- Knowledge of Delta Lake, Lakehouse architecture, and real-time data processing.
- Exposure to Azure DevOps, CI/CD pipelines, and Infrastructure-as-Code practices.
- Experience working with large-scale enterprise datasets and distributed data processing.
- Familiarity with data security, compliance, and access control best practices.
- Azure Data Engineer Associate (DP-203) or related cloud certifications preferred.