OverviewWe are seeking a Data & Software Engineer works with a small team to build complex data flows for a custom application. Successful candidate will have advanced Python programming skills, familiarity with Java, an understanding of data security, privacy, governance and compliance principles and a demonstrated history of building production data pipelines and ETL workflows at scale. Candidate must have experience:
What will you do?ยท ย Buildingย end-to-end data pipelines leveraging Python
Using orchestration tools to deploy data pipelines, including configuring and updating Spark Jobs
ยท ย Containerizingย and deploying applications in cloud environments like AWS.
ยท ย Workingย with MySQL and PostgreSQL including performance tuning, schema design, and query optimization for complex, analytical workloads.ย
ยท ย Leveragingย industry standard tools for code control (Git, IaaCย control, etc.)
ยท ย Workingย with data catalogs, tracking data lineage ย andย handling a variety of data formats, including Geospatial.
ยท ย Usingย Bash scripting for automation and data processing tasks
ยท ย Integratingย Al/ML services and models
ยท ย Workย with stakeholders to understand data requirements, assess feasibility, and design appropriate solutions with minimal oversight
ยท ย Leverageย strong problem-solving and debugging skills for data quality issues, pipeline failures, and performance bottlenecks
ยท ย Leverageย a background in large-scale data migration or platform modernization efforts
Contribute to data engineering documentation, best practices, and design patterns.
Do you have what it takes?- Active TS/SCI W/ Polygraph required.ย
- Bachelor's degree in Computer Science, Engineering, Finance, or a related technical field, or equivalent practical experience.
Minimum of 5 years' experience with:ย
ยท ย Apacheย Spark & PySpark
ยท ย Advancedย Python skills (including Pandas & NumPy)
ยท ย Docker, Podman
ยท ย AWSย S3, Lambda & Step functions
ยท ย Apacheย Iceberg, Airflow, etc.
ยท ย SQLย (with Trino)
ยท ย NoSQL, DynamoDB
ยท ย Unityย Catalog OSS, Apache Polaris
ยท ย Apacheย Superset
ยท ย Terraformย or CloudFormation
ยท ย OpenLineage
ยท ย H3, PostGIS
Qualifications:
- Active TS/SCI W/ Polygraph required.ย
- Bachelor's degree in Computer Science, Engineering, Finance, or a related technical field, or equivalent practical experience.
Minimum of 5 years' experience with:ย
ยท ย Apacheย Spark & PySpark
ยท ย Advancedย Python skills (including Pandas & NumPy)
ยท ย Docker, Podman
ยท ย AWSย S3, Lambda & Step functions
ยท ย Apacheย Iceberg, Airflow, etc.
ยท ย SQLย (with Trino)
ยท ย NoSQL, DynamoDB
ยท ย Unityย Catalog OSS, Apache Polaris
ยท ย Apacheย Superset
ยท ย Terraformย or CloudFormation
ยท ย OpenLineage
ยท ย H3, PostGIS
Education:UNAVAILABLEEmployment Type: FULL_TIME