Job Summary:
We are seeking a Databricks Engineer with strong expertise in building scalable data pipelines, data transformation workflows, and analytics solutions using the Databricks platform. The ideal candidate will have experience with big data technologies, cloud platforms, and modern data engineering practices.
Key Responsibilities:
- Design, develop, and maintain data pipelines using Databricks, PySpark, and Spark SQL.
- Build and optimize ETL/ELT processes for large-scale data processing.
- Develop data ingestion frameworks from various sources including databases, APIs, and cloud storage.
- Implement Delta Lake solutions to ensure data quality, reliability, and performance.
- Collaborate with data analysts, data scientists, and business stakeholders to deliver data solutions.
- Optimize Spark jobs and Databricks workloads for performance and cost efficiency.
- Integrate Databricks with cloud services such as AWS, Azure, or GCP.
- Monitor, troubleshoot, and resolve data pipeline and platform issues.
- Participate in code reviews, testing, deployment, and documentation activities.
Required Skills:
- Strong experience with Databricks, Apache Spark, PySpark, and Spark SQL.
- Proficiency in Python and SQL.
- Experience with Delta Lake, Data Lake, and Lakehouse architectures.
- Knowledge of data modeling, ETL/ELT processes, and data warehousing concepts.
- Experience with cloud platforms (Azure, AWS, or GCP).
- Familiarity with CI/CD pipelines, Git, and DevOps practices.
- Strong analytical and problem-solving skills.