Position: Data Support Engineer (Databricks experience)
Location: Chicago, IL 6060
Duration: 6 months
Job Description:
A Data Application Support role specializing in Databricks focuses onย maintaining, troubleshooting, and optimizing Spark-based data pipelines and Lakehouse architectures. Key responsibilities include resolving L2/L3 production incidents, performance tuning SQL/Python (PySpark) jobs, managing Delta Lake assets, and collaborating with data engineers to ensure data reliability.
Key Responsibilities
- Incident & Problem Management:ย Provide L2/L3 support for data applications, resolving production issues and troubleshooting Databricks jobs, notebooks, and workflows.
- Performance Tuning:ย Optimize Spark applications, SQL queries, and Delta Lake tables to improve efficiency and reduce costs.
- Pipeline Maintenance:ย Monitor and troubleshoot ETL/ELT pipelines in Databricks (including Data Factory/Delta Live Tables), ensuring data quality and lineage (Unity Catalog).
- Collaboration:ย Act as a liaison between users, data engineering teams, and platform engineering, providing technical expertise and contributing to documentation.
- Automation:ย Create tools to automate routine support tasks and enhance support team productivity.ย Databricksย +5
ย
Required Qualifications & Skills
- Technical Expertise:ย Strong hands-on experience with Apache Spark, Python/PySpark, and SQL.
- Databricks Ecosystem:ย Proficiency with Databricks Unified Data Analytics Platform, Delta Lake, and ideally Azure Databricks.
- Cloud Data Storage:ย Experience with Azure Data Lake Storage (ADLS Gen2)ย or similar data lake technologies.
- Version Control & CI/CD:ย Experience with Git/Azure DevOpsย for code management.
- Problem-Solving:ย Strong analytical skills, with the ability to diagnose complex data processing bottlenecks.ย
ย
"Nice-to-Have" Skillsย
- Data orchestration toolsย (e.g., Apache Airflow, Azure Data Factory).
- Data governance toolsย (e.g., Unity Catalog, Collibra).
- Streaming data knowledgeย (e.g., Spark Structured Streaming).ย