Azure Data Engineer W/ Python/C#
Location: 100% Remote
Duration: long term
Interview Mode: Video Need 12-13+ years candidate PySpark or Scala Spark SQL Delta lakes * Python/C#
Primary Duties And Accountabilities
- Create and maintain optimal data pipeline architecture
- Assemble large, complex data sets that meet functional / non-functional business requirements.
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and Big Data technologies.
- Deliver automation & lean processes to ensure high quality throughput & performance of the entire data & analytics platform.
- Work with data and analytics experts to strive for greater functionality in our analytics platforms.
Position Specifications
Minimum:
Preferred:
Experience in building/operating/maintaining fault tolerant and scalable data processing integrations using Azure
Strong problem-solving skills with emphasis on optimization data pipelines
Excellent written and verbal communication skills for coordinating across teams
A drive to learn and master new technologies and techniques
Experienced in DevOps and Agile
Experience using Docker or Kubernetes is a plus
Demonstrated capabilities with cloud infrastructures and multi-cloud environments such as Azure, AWS, IBM cloud environments and using CI/CD pipelines.
Experience with Delta Lakes
Experienced using Databricks & Apache Spark
Experienced using Azure Data Factory or Synapse Analytics