Job Summary:
eTeam is working with US Bank - TOS to find a Cloud Data Platform Engineer. The role involves supporting Azure & AWS Databricks from a DevOps perspective, focusing on cloud infrastructure management, automation, and CI/CD practices.
Responsibilities:
• Implement CI/CD Pipelines: Design and maintain continuous integration and continuous deployment pipelines using tools like Gitlab, Azure DevOps or GitHub Actions for deploying data pipelines and infrastructure changes.
• Infrastructure as Code (IaC): Provision and manage Azure Analytics services (Databricks, etc.) through code using tools such as Terraform, ARM templates, or Bicep to ensure consistency and repeatability.
• Platform Operations: Deploy, manage, and optimize the performance and resource utilization of Azure data services, including monitoring and troubleshooting data pipeline failures and performance issues.
• Automation & Scripting: Automate routine operational tasks and application deployments using scripting languages like Python, PowerShell, or Bash.
• Security & Compliance: Implement security best practices, including identity and access management (IAM), data encryption, and compliance with data governance policies (e.g., GDPR) within the platform.
• Collaboration: Work closely with data engineers, data scientists, and business analysts to translate data requirements into robust technical solutions and foster a DevOps culture within the organization.
Qualifications:
Required:
• Deep knowledge of Microsoft Azure and/or AWS services, specifically, Azure and or AWS Databricks
• Exposure to Azure Synapse Analytics (SQL pools, Spark pools, pipelines)
• Azure Data Factory (pipelines, triggers, data flows)
• Azure Data Lake Storage (ADLS)
• Azure Monitor
• Expertise Gitlab or Azure DevOps (Azure Pipelines, Boards, Repos) or GitHub Actions
• Strong experience with Git and branching strategies
• Proficiency in Terraform, Bicep, or ARM templates
• Experience with Docker and container orchestration (Azure Kubernetes Service - AKS)
• Proficiency in Python, SQL (SQL, query optimization), and scripting languages like PowerShell or Bash for automation and data manipulation tasks
• Basic understanding of Data engineering principles such as ETL/ELT processes, data modeling, data warehousing, and big data concepts
• Experience implementing monitoring, logging, and alerting solutions using tools like Azure Monitor, Log Analytics, and Application Insights
• Excellent problem-solving, analytical, and communication skills to work effectively in fast-paced, agile environments
Preferred:
• Microsoft Certified: Azure Data Engineer Associate (DP-203)
• Microsoft Certified: Azure DevOps Engineer Expert (AZ-400)
Company:
eTeam is a staffing agency that also provides payrolling services. Founded in 1999, the company is headquartered in Somerset, USA, with a team of 501-1000 employees. The company is currently Late Stage.