Python DeveloperIntroduction
As a Python Developer on our team, you will be responsible for designing, building, and maintaining high-performance ETL/ELT data pipelines using Python and PySpark. You will play a key role in developing and managing data processing jobs across large-scale datasets, ensuring data quality and contributing to platform modernization efforts.
Responsibilities
- Design, build, and maintain high-performance ETL/ELT data pipelines using Python and PySpark.
- Apply expert-level coding skills to develop and manage data processing jobs leveraging PySpark for distributed computing across large-scale datasets.
- Take full ownership of the data workflow, including getting data from multiple sources, scrubbing, and validating data to ensure the highest quality.
- Write and optimize complex, performant SQL queries for data extraction, integrity checks, and performance tuning.
- Contribute to platform modernization by exploring and increasing the adoption of AI/ML, including using tools like Copilot and Claude for acceleration, and building models to fill data gaps or improve systems.
- Collaborate with data architects by proposing ideas and great questions, taking ownership as the expert on data, pipelines, and systems.
- Implement DevOps practices for the automated deployment and orchestration of Python applications and data pipelines (e.g., using Docker, Jenkins, Terraform).
- Hands-on experience with SQL and complex performance tuning.
RequirementsRequired Skills
- Proficiency in Python programming.
- Experience with numpy and pandas libraries.
- Experience in designing and maintaining ETL/ELT data pipelines.
- Strong SQL skills and experience with performance tuning.
- Experience with PySpark for distributed computing.
Preferred Skills
- Experience with AI/ML tools such as Copilot and Claude.
- Experience with DevOps practices for automated deployment.