Job Summary:
Kai is an AI company focused on rebuilding cybersecurity for the machine-speed era. The DevOps Engineer will automate workflows, improve platform reliability, and support data engineering teams to enhance development and deployment practices.
Responsibilities:
• Design, deploy, and operate scalable data platforms and pipelines, primarily on Azure (Databricks, ADF, ADLS)
• Build, manage, and optimize Apache Spark clusters and workloads for batch and streaming data processing across Azure and AWS environments.
• Implement CI/CD pipelines for data engineering code, Spark jobs, and pipeline configurations using Azure DevOps/GitHub Actions
• Automate infrastructure using Infrastructure as Code (Terraform) and manage containerized workloads with Docker and Kubernetes
• Monitor data pipelines and platforms to ensure data reliability, quality, observability, and cost optimization across Azure and AWS data platforms.
• Enforce security, governance, and best practices, collaborating closely with data engineers and platform teams in Azure-first, multi-cloud environments.
Qualifications:
Required:
• 6+ years of professional experience in data engineering, Data DevOps, or Data platform engineering roles
• Proven experience supporting production-grade data platforms in enterprise environments
• Proven ability to design, build, deploy, and maintain scalable data pipelines (ETL/ELT)
• Deep understanding of Apache Spark for batch and streaming workloads
• Experience creating, configuring, and managing Spark clusters, including performance tuning and cost optimization
• Practical experience with at least one major cloud provider: AWS, Azure, or GCP.
• Strong experience using Terraform for infrastructure automation.
• Proven ability to diagnose and resolve system and infrastructure issues.
Preferred:
• Experience deploying and managing Spark workloads on Azure Databricks or Azure Synapse
Company:
Find your people. Find your path. Connect. Explore. Become. Founded in , the company is headquartered in , , with a team of 11-50 employees. The company is currently Early Stage.