ZipRecruiter

Log In

1

Llm Training Jobs (NOW HIRING)

OpenTalent

Distributed Training Engineer

Menlo Park, CA · On-site

About the role You will optimize, operate and develop large-scale distributed LLM training systems that power AI scientific research. You will work closely with researchers to bring up, debug, and ...

New

next page

Showing results 1-20

All JobsLlm Training Jobs

Llm Training information

See salary details

$32K

$68.7K

$112K

How much do llm training jobs pay per year?

As of Sep 13, 2026, the average yearly pay for llm training in the United States is $68,682.00, according to ZipRecruiter salary data. Most workers in this role earn between $50,000.00 and $84,500.00 per year, depending on experience, location, and employer.

What is an LLM Training?

An LLM Training job involves developing, fine-tuning, and optimizing large language models (LLMs) to improve their performance and accuracy. This role typically includes data collection, preprocessing, model training, evaluation, and troubleshooting issues related to bias, efficiency, and scalability. Professionals in this field work with machine learning frameworks, large datasets, and computational resources to enhance AI capabilities. They may also collaborate with researchers, engineers, and product teams to deploy models for real-world applications.

What are the key skills and qualifications needed to thrive in the LLM Training position?

To excel in LLM Training, you need a strong background in machine learning, natural language processing (NLP), and computer science, often backed by an advanced degree in a related field. Experience with programming languages such as Python, frameworks like PyTorch or TensorFlow, and familiarity with data annotation tools are essential, along with knowledge of cloud platforms and distributed computing. Strong analytical thinking, effective communication, and the ability to collaborate across interdisciplinary teams set top candidates apart. These skills ensure high-quality model development, efficient project execution, and the ability to adapt to evolving AI technologies.

What types of teams or professionals does an LLM Training specialist typically collaborate with?

Professionals specializing in LLM Training often work closely with data engineers, software developers, domain experts, product managers, and quality assurance analysts. Collaboration is essential for collecting and preprocessing training data, integrating models into products, and ensuring outputs meet business and user requirements. These roles frequently participate in agile project workflows, contribute to cross-functional team meetings, and collaborate on continuous model improvements. Engaging with diverse teams expands your understanding of product goals and helps you deliver robust and reliable language models that align with organizational objectives.

Is it possible to train Llm Training?

Training large language models (LLMs) is possible but requires significant computational resources, expertise in machine learning, and access to large datasets. It typically involves using specialized hardware like GPUs or TPUs and knowledge of frameworks such as TensorFlow or PyTorch. Many organizations opt to fine-tune pre-trained models rather than train from scratch due to the high costs and complexity involved.
More about Llm Training jobs

What cities are hiring for Llm Training jobs?

Cities with the most Llm Training job openings:

What are the most commonly searched types of Llm Training jobs?

The most popular types of Llm Training jobs are:

What states have the most Llm Training jobs?

States with the most job openings for Llm Training jobs include:

What job categories do people searching Llm Training jobs look for?

The top searched job categories for Llm Training jobs are:

What other helpful pages are available for Llm Training?

Other pages related to Llm Training:

Llm Training jobs near you
Infographic showing various Llm Training job openings in the United States as of September 2026, with employment types broken down into 1% As Needed, 79% Full Time, 19% Part Time, and 1% Contract. Highlights an 84% Physical, 1% Hybrid, and 15% Remote job distribution, with an average salary of $68,682 per year, or $33 per hour.

AI/ML Research Engineer, LLM Post-Training & Evaluation

Remote

Innodata Inc.
IT Services • 501 - 1,000 employees
Learn more about Innodata Inc.external

Full-time

Re-posted 9 days ago

Innodata rating

7.5

Company rating: 7.5 out of 10

Based on 6 frontline employees who took The Breakroom Quiz

Job description

Innodata (Nasdaq: INOD) is a global data engineering company. We believe that data and Artificial Intelligence (AI) are inextricably linked. Our mission is to enable the responsible advancement of artificial intelligence by providing the data, evaluation frameworks, and human expertise required to build AI systems that can be trusted at scale. We provide a range of transferable solutions, platforms, and services for Generative AI / AI builders and adopters. In every relationship, we honor our 36+ year legacy delivering the highest quality data and outstanding outcomes for our customers.
Scope of the Role:
Innodata is expanding its team of technical experts in LLM training, post-training, and evaluation systems. As an AI/ML Research Engineer, LLM Training & Evaluation, you will build and optimize the technical foundations that power model improvement for foundation model builders and leading labs.
This role is ideal for someone who has hands-on experience fine-tuning and evaluating large language models (and ideally multimodal models), and who can bridge research and engineering in real-world customer environments. You will work closely with Language Data Scientists, Applied Research Scientists, data engineers, and client technical stakeholders to design and implement robust training/evaluation pipelines using both human-in-the-loop and AI-augmented methods.
The ideal candidate brings a strong computer science / machine learning engineering background, experience with modern LLM post-training workflows, and the ability to engage credibly with technical counterparts at leading AI organizations.
What You'll Own:
As an AI/ML Research Engineer, LLM Training & Evaluation, you will design and implement the pipelines and tooling that connect data, evaluation, and post-training. You will help customers and internal teams move from evaluation findings to measurable model improvements.
Your work may include building fine-tuning workflows (e.g., supervised fine-tuning and preference-based optimization), integrating evaluation harnesses into model development loops, improving experiment reliability and throughput, and supporting advanced evaluation scenarios such as long-context, cross-modal, and dynamic multi-turn interactions.
You will also contribute to Innodata's internal R&D efforts, including benchmark datasets, evaluation frameworks, and reusable infrastructure for model assessment and post-training experimentation. Additional responsibilities include (but are not limited to):
  • Lead or co-lead technically complex ML engineering projects from initial customer discussions through implementation and delivery
  • Design, build, and improve LLM training and post-training pipelines, including data ingestion, preprocessing, fine-tuning, evaluation, and experiment tracking
  • Implement and optimize evaluation systems for LLMs and multimodal models, including offline benchmarks and task-specific test harnesses
  • Integrate human-in-the-loop and AI-augmented evaluation signals into model development workflows
  • Build robust infrastructure and tooling for reproducible experimentation, metrics logging, and regression monitoring
  • Diagnose model behavior and pipeline failures, including data issues, training instability, metric inconsistencies, and evaluation drift
  • Collaborate with Language Data Scientists and Applied Research Scientists to translate evaluation frameworks into executable systems
  • Work closely with customer technical stakeholders to understand goals, constraints, and success criteria; propose and implement technically sound solutions
  • Contribute to internal research and platform development, including benchmark frameworks, evaluation tooling, and post-training workflow improvements
  • Contribute to best practices and standards for LLM training, evaluation, and quality assurance across projects
  • Mentor junior engineers and contribute to technical design reviews, documentation, and engineering rigor across the team

You'll Thrive in This Role If You Have:
  • BS/MS/PhD in Computer Science, Machine Learning, AI, Applied Mathematics, or a related quantitative technical field (MS/PhD preferred)
  • 2-3 years of relevant industry or research engineering experience in ML/AI systems
  • Hands-on experience with LLM training / fine-tuning / post-training, including at least one of:
    • supervised fine-tuning (SFT)
    • preference optimization (e.g., DPO or related methods)
    • RLHF / RLAIF-style workflows
    • task- or domain-adaptation of foundation models
  • Strong programming skills in Python and experience building production-quality ML code
  • Experience with modern ML frameworks (e.g., PyTorch, JAX, TensorFlow) and model libraries/tooling (e.g., Hugging Face ecosystem, vLLM, distributed training stacks)
  • Experience designing and implementing evaluation pipelines for LLM/ML systems, including metrics computation, dataset handling, and experiment comparisons
  • Strong understanding of data pipelines and ML systems engineering, including reproducibility, observability, and debugging
  • Experience with large-scale distributed ML systems and performance optimization for training/evaluation workloads (GPU/accelerator environments preferred)
  • Experience with large-scale data processing and workflow orchestration in support of model training/evaluation
  • Ability to collaborate directly with technical stakeholders including research scientists, ML engineers, data engineers, and customer technical leads
  • Strong written and verbal communication skills, including the ability to explain complex technical tradeoffs to both technical and non-technical audiences

Technical Skills
ML / LLM Engineering
  • Experience training, fine-tuning, and evaluating transformer-based models
  • Understanding of post-training workflows and model iteration loops
  • Familiarity with inference-time considerations (latency, throughput, memory/performance tradeoffs) where relevant to evaluation or deployment

Evaluation & Experimentation
  • Experience implementing automated evaluation pipelines and test harnesses
  • Experience with experiment tracking, versioning, and reproducibility practices
  • Ability to assess metric quality and ensure consistency across model comparisons

Software / Data Engineering
  • Proficiency in Python and strong software engineering fundamentals
  • Experience with data processing pipelines, storage formats, and scalable dataset workflows
  • Familiarity with CI/CD, testing, and engineering quality practices for ML systems

The expected salary range for this position is $80,000 - $175,000 USD per year, based on experience, skills, and qualifications.
Please be aware of recruitment scams involving individuals or organizations falsely claiming to represent employers. Innodata will never ask for payment, banking details, or sensitive personal information during the application process. To learn more on how to recognize job scams, please visit the Federal Trade Commission's guide at https://consumer.ftc.gov/articles/job-scams.
If you believe you've been targeted by a recruitment scam, please report it to Innodata at verifyjoboffer@innodata.com and consider reporting it to the FTC at ReportFraud.ftc.gov.

What Innodata employees say

Hours and flexibility

Workplace

Get the full story on Breakroom

Apply