Overview:Job Title: Airflow Engineer - Data Pipeline Automation
Location: Phoenix, AZ (Onsite | Contract)
Employment Type: Contract
Job SummaryWe are seeking a highly skilled
Airflow Engineer with expertise in designing, optimizing, and maintaining large-scale data workflows. The ideal candidate will have deep knowledge of
Airflow architecture, DAG development, CI/CD pipeline management, and cloud integrations (Azure/AWS). This is an
onsite contract role in Phoenix, AZ.Key Responsibilities - Airflow Architecture: Design, configure, and optimize Airflow environments, including schedulers, executors (Celery, Kubernetes), and plugins.
- DAG Development: Build complex, modular, and reusable DAGs to automate data pipelines and workflows.
- Performance Optimization: Identify bottlenecks and implement orchestration and scheduling best practices.
- Cloud Integration: Connect Airflow with Azure Data Factory, Databricks, Azure Storage, AWS, and other cloud-native services.
- CI/CD Management: Develop and maintain CI/CD pipelines for DAG deployment, testing, and version control using Azure DevOps or similar tools.
- Monitoring & Alerting: Implement logging, monitoring, and alerting frameworks for Airflow jobs to ensure reliability.
- Documentation & Mentoring: Maintain technical documentation and conduct knowledge-sharing/mentorship sessions with teams.
- Troubleshooting: Provide expert-level support for Airflow-related incidents and contribute to root cause analysis.
Required Skills - Strong expertise in Apache Airflow (architecture, schedulers, executors, plugins).
- Proficiency in Python (must) and Java (nice to have) for DAG and pipeline development.
- Hands-on experience with cloud platforms (Azure, AWS) - Data Factory, Databricks, Synapse, Functions, Storage.
- Familiarity with CI/CD tools (Azure DevOps, Jenkins, GitLab CI/CD).
- Experience with Docker and Kubernetes for containerized Airflow deployments.
- Strong troubleshooting, debugging, and performance optimization skills.
- Excellent communication and collaboration skills.