Role Description:We are seeking a
Senior Data Engineer with strong hands-on experience in building AWS-based data pipelines. The ideal candidate will have deep expertise in Python, AWS Glue, PySpark, and cloud-native data engineering solutions. This role involves designing, developing, and maintaining scalable data pipelines and data warehouse solutions using AWS services.
Key Responsibilities:- Build and maintain AWS-based data pipelines using AWS Glue, PySpark, Python, and AWS Lambda
- Design and develop solutions for loading, organizing, and querying structured and semi-structured data in AWS S3
- Develop ETL processes using AWS Glue ETL and related AWS services
- Design and develop data warehouse applications using common standards and frameworks
- Work with AWS services such as EMR, Redshift, Glue, and S3
- Write and optimize SQL queries for analytical and reporting use cases
- Develop and maintain Python-based data processing code
- Use source code repositories (Git) and follow CI/CD best practices
- Collaborate with cross-functional teams in an Agile delivery environment
- Ensure data quality, performance, and reliability across data pipelines
Essential Skills:- Strong hands-on experience in Python
- Hands-on experience with AWS Glue and AWS Cloud services
- Experience with PySpark for large-scale data processing
- Solid understanding of NoSQL databases
- Strong SQL skills
- Knowledge of AWS Lambda
- Basic knowledge of Java
- Experience with source code repositories (Git)
- Understanding of CI/CD processes
Skills (Mandatory):- Digital : Python
- Digital : Amazon Web Service (AWS) Cloud Computing
- Digital : NoSQL
- Digital : PySpark
Experience Range in Required Skills:- 8-10 years of overall experience in Data Engineering
- Strong hands-on experience with AWS-based data engineering solutions
Preferred / Desirable Skills:- Experience with AWS EMR and Redshift
- Exposure to cloud-based data warehouse architectures
- Experience working in Agile/Scrum teams