Senior Data Engineer

Job Summary:
UST is a mission-driven company that transforms lives through technology. They are seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines and modern data platforms, ensuring high-performance data solutions for analytics and business intelligence in a cloud-native environment.
Responsibilities:
• Design, develop, and maintain scalable ETL/ELT pipelines using Python and PySpark for batch and streaming workloads.
• Build and manage data lake / lakehouse architectures (Bronze/Silver/Gold layers) to support enterprise analytics.
• Optimize Spark jobs for performance, scalability, and cost efficiency using best practices.
• Work with AWS services such as S3, Glue, EMR, Lambda, and Step Functions to integrate and orchestrate data workflows.
• Implement data quality, validation, and monitoring frameworks to ensure data reliability and integrity.
• Collaborate with cross-functional teams (Data, Platform, Application teams) to deliver end-to-end data solutions.
Qualifications:
Required:
• 4+ years of experience in Data Engineering / Software Development.
• Strong proficiency in Python programming (data structures, OOP, design patterns).
• Hands-on experience in PySpark and Spark ecosystem including DataFrames, Spark SQL, and Spark Streaming.
• Experience in building large-scale ETL/ELT pipelines.
• Strong knowledge of data modeling and data lake architecture (Bronze/Silver/Gold).
• Expertise with file formats such as Parquet, Avro, JSON.
• Knowledge of data quality, validation, and governance practices.
• Hands-on experience with AWS Glue (ETL jobs, Data Catalog, workflows).
• Hands-on experience with Amazon EMR (cluster management, scaling, optimization).
• Hands-on experience with Amazon S3 (data lake design, partitioning, lifecycle).
• Hands-on experience with AWS Lambda (serverless integrations).
• Experience with orchestration tools: AWS Step Functions / Airflow (MWAA).
• Monitoring via CloudWatch (logs, metrics, s).
• Strong experience in Spark performance tuning and optimization.
• Handling large datasets with focus on efficiency and cost optimization.
• Experience with CI/CD pipelines (CodePipeline, CodeBuild, CodeDeploy).
• Familiarity with Infrastructure as Code (Terraform/CloudFormation).
• Strong focus on testing and code quality (pytest, unit/integration testing, PEP 8 standards).
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder collaboration abilities.
• Ability to work independently and in Agile environments.
Company:
UST is a Digital Transformations Solutions Provider. Founded in 1999, the company is headquartered in Aliso Viejo, USA, with a team of 10001+ employees. The company is currently Late Stage.