Interview : Coding test + Video
Visa : USC, GC, GC EAD, H4, L2
Remote or In-Office (St. Louis, MO)
Description :
- Data Engineer responsible for designing, developing, and maintaining data pipelines and warehousing solutions.
- Key tasks will include API integration, ETL development, data modeling (Star Schema or Snowflake Schema), and supporting Power BI reporting.
- Will primarily work on the OIL Analytics Report Migration project initially, then transition to the Tamarac Data Warehouse project.
- Collaborate with internal project teams to ensure data accuracy, integrity, and structured organization for business intelligence.
Tech stack
Azure Synapse Analytics
- Two separate environments (e.g., Development and Production).
- Handles data warehousing and large-scale analytics workloads.
Azure Data Lake
- Centralized storage layer.
- Supports both structured and unstructured data.
- Scalable foundation for analytics and data integration.
Azure Key Vault
- Manages secrets, encryption keys, and certificates.
- Ensures secure access across both environments.
Azure DevOps
- CI/CD pipelines for automated builds and deployments.
- Manages data pipeline lifecycle and component delivery.
Apache Spark Notebooks
- Deployed in both environments.
- Used for interactive data exploration, transformation, and analytics.
Azure Integration Runtime
- Facilitates secure and scalable data movement.
- Enables transformations across network boundaries within Synapse or Data Factory.
Metastore Data Warehouse
- Centralized metadata repository.
- Maintains schema definitions, and table metadata
ARM Template (Azure Resource Manager)
- Defines and automates infrastructure deployment.
- Enables consistent provisioning of Synapse, Data Lake, Key Vault, and other resources across environments.
