ZipRecruiter

Log In

1

Machine Learning Infrastructure Engineer Jobs (NOW HIRING)

next page

Showing results 1-20

All JobsMachine Learning Infrastructure Engineer Jobs

Machine Learning Infrastructure Engineer information

See salary details

$46.5K

$127.1K

$182K

How much do machine learning infrastructure engineer jobs pay per year?

As of Jul 7, 2026, the average yearly pay for machine learning infrastructure engineer in the United States is $127,066.00, according to ZipRecruiter salary data. Most workers in this role earn between $107,500.00 and $141,000.00 per year, depending on experience, location, and employer.

What are some common challenges faced by Machine Learning Infrastructure Engineers, and how can these be addressed on the job?

Machine Learning Infrastructure Engineers often face challenges such as ensuring infrastructure scalability, managing resource allocation, and maintaining system reliability while supporting rapid experimentation by data science teams. Balancing the needs for flexibility in research environments with production-grade stability requires a deep understanding of both engineering best practices and the unique requirements of machine learning workflows. Collaboration with data scientists, clear communication about infrastructure capabilities, and staying current with fast-evolving technologies are key strategies for success. Most companies encourage ongoing learning and provide opportunities to contribute to architecture decisions, which makes this a rewarding environment for problem-solvers and innovators.

What are the key skills and qualifications needed to thrive in the Machine Learning Infrastructure Engineer position, and why are they important?

To thrive as a Machine Learning Infrastructure Engineer, you need a strong background in computer science, cloud computing, distributed systems, and experience with machine learning frameworks, often supported by a degree in a related field. Familiarity with tools such as Docker, Kubernetes, Terraform, as well as cloud platforms like AWS, GCP, or Azure, and certifications in cloud or DevOps technologies are highly valued. Strong problem-solving abilities, effective communication, and collaboration skills help engineers work seamlessly with data scientists and cross-functional teams. These skills are essential to design, implement, and maintain robust, scalable infrastructure that enables efficient machine learning development and deployment.

What is a Machine Learning Infrastructure Engineer job?

A Machine Learning Infrastructure Engineer designs, builds, and maintains the systems that support the development and deployment of machine learning models. This includes managing data pipelines, optimizing model training and inference, and ensuring scalability and reliability in production environments. They work closely with data scientists, ML engineers, and DevOps teams to create efficient workflows and infrastructure. Key technologies often include cloud platforms, containerization, orchestration tools, and distributed computing frameworks.

More about Machine Learning Infrastructure Engineer jobs
What cities are hiring for Machine Learning Infrastructure Engineer jobs? Cities with the most Machine Learning Infrastructure Engineer job openings:
What states have the most Machine Learning Infrastructure Engineer jobs? States with the most job openings for Machine Learning Infrastructure Engineer jobs include:
What job categories do people searching Machine Learning Infrastructure Engineer jobs look for? The top searched job categories for Machine Learning Infrastructure Engineer jobs are:
Machine Learning Infrastructure Engineer jobs near you
Infographic showing various Machine Learning Infrastructure Engineer job openings in the United States as of July 2026, with employment types broken down into 95% Full Time, 2% Part Time, and 3% Contract. Highlights an 87% Physical, 4% Hybrid, and 9% Remote job distribution, with an average salary of $127,066 per year, or $61.1 per hour.
Machine Learning Infrastructure Engineer

Machine Learning Infrastructure Engineer

Astera Labs

San Jose, CA

$140K - $165K/yr

Other

Posted 20 days ago

Job description

Machine Learning Infrastructure Engineer

Location: San Jose, CA
Experience: 1-5 years
Team: Applied AI

The role

We're hiring a Machine Learning Infrastructure Engineer to build the runtime, platform, and operational backbone for modern AI systems. This role is for someone who wants to work on the systems behind the systems: model access layers, routing, serving paths, telemetry, observability, evaluation infrastructure, and the controls needed to make fast-moving AI work reliable in practice.

This is a platform role, but not in the old sense. The work is tightly coupled to how modern AI systems are actually built and used: multiple model providers, agent runtimes, skill and tool layers, inference telemetry, cost-aware routing, AI spend visibility, and governance that is strong enough for real internal adoption.

What you'll do
  • Build and improve internal AI infrastructure for LLM applications, agents, retrieval systems, and model-backed engineering workflows.
  • Own inference deployment paths across managed and self-serve environments, including access control, monitoring, and operational reliability.
  • Build platform layers such as model gateways, routing, runtime integrations, telemetry, and controls for safe execution at scale.
  • Develop AI Ops capabilities across evaluation, release readiness, observability, incident triage, regression detection, and cost monitoring.
  • Build dashboards, tracing, logging, and alerting for production AI systems, including spend and usage visibility across tools and teams.
  • Improve performance and unit economics through routing, caching, batching, failover, and latency/cost optimization.
  • Create reusable APIs, SDKs, and platform abstractions that make AI systems easier to deploy, evaluate, govern, and operate.
What we're looking for
  • 1-5 years of experience in software engineering, ML infrastructure, MLOps, platform engineering, or related backend/infrastructure roles.
  • Strong Python plus strong systems instincts.
  • Experience with AWS or GCP and real production service ownership.
  • Familiarity with inference deployments, model APIs, gateways, serving systems, or runtime infrastructure for LLM/ML workloads.
  • Experience with observability, telemetry, reliability engineering, and incident response.
  • Understanding of eval systems, release workflows, retrieval-backed systems, and debugging non-deterministic AI behavior.
  • Ability to translate messy platform needs into scalable internal infrastructure.
What strong candidates often look like

They have built or operated systems where latency, routing, cost, telemetry, and reliability actually matter. They understand that modern AI infrastructure is not just about getting a model endpoint running. It is about building the runtime, visibility, controls, and developer experience that let an applied AI team move fast without losing quality or trust.

Why this role is interesting

The team is building AI-ready infrastructure in the most literal sense: observability, access control, AI spend tracking, secure managed platforms, skill/tool infrastructure, and telemetry that spans requests, tools, models, and outcomes. If you want to work on the platform layer that makes modern agentic systems possible - and do it in a setting where the downstream users are serious engineers with high expectations - this is that role.

The base pay compensation range for this role is between $140,000 - $165,000

Apply