Overview
CTG is seeking to fill a Lead Data Platform Engineer – Astronomer / Airflow position for our client.
Location: Atlanta, GA
Duration: 12 months
Duties:
- Lead L2/L3 support for enterprise Astro (Astronomer-managed Apache Airflow) environments.
- Own Incident, Problem, and Change Management processes for the Data Orchestration Platform.
- Perform advanced root cause analysis (RCA) for pipeline failures, scheduler issues, worker instability, and infrastructure bottlenecks.
- Improve DAG reliability, SLA adherence, and reduce MTTR across critical data workflows.
- Establish and maintain operational standards, including support playbooks, runbooks, and escalation procedures.
- Design and maintain dynamic DAG frameworks (Factory pattern) to enable scalable pipeline onboarding.
- Enable automated onboarding of new data domains, data sources, and transformation workloads.
- Support and optimize complex ETL/ELT workflows across Data Lakes (S3-based architectures) and Data Warehouses (e.g., Snowflake, Amazon Redshift).
- Collaborate with data engineering, platform, and DevOps teams to ensure stable, scalable orchestration services.
- Provide technical guidance and support on open-source platform implementation, customization, and integration.
Skills:
- Deep expertise in Astronomer-managed Apache Airflow (Astro) environments.
- Strong Python development skills for DAGs, operators, plugins, and automation frameworks.
- Experience with ETL/ELT pipeline design and distributed data processing patterns.
- Proficiency with AWS cloud services including EKS, S3, RDS, IAM, CloudWatch, and VPC.
- Kubernetes operational experience (Airflow on EKS or equivalent).
- Infrastructure-as-Code experience (Terraform or CloudFormation preferred).
- Strong knowledge of CI/CD pipelines, Git-based workflows, and deployment automation.
- Familiarity with data platforms such as Snowflake and Amazon Redshift.
- Strong analytical and troubleshooting skills for distributed systems.
Experience:
- Proven experience supporting and operating enterprise-scale Airflow/Astro environments.
- Experience leading production incident response, RCA, and operational reliability improvements.
- Hands-on experience building scalable DAG frameworks and reusable pipeline templates.
- Background in cloud-native data engineering and platform operations.
- Experience optimizing performance, scaling workloads, and improving orchestration efficiency.
- Prior experience in enterprise data platform support or SRE/DataOps roles strongly preferred.
Education:
- Bachelor’s degree in Computer Science, Information Systems, Engineering, or related field, or equivalent experience.
Excellent verbal and written English communication skills and the ability to interact professionally with a diverse group are required.
CTG does not accept unsolicited resumes from headhunters, recruitment agencies, or fee based recruitment services for this role.
To Apply:
To be considered, please apply directly to this requisition using the link provided. Kindly forward this to any other interested parties. Thank you!