Responsibilities : • Lead end-to-end performance analysis and optimization of innovative LLM pre-training and post-training workloads on the latest NVIDIA hardware and software platforms. • Drive ...
Responsibilities : • Lead end-to-end performance analysis and optimization of innovative LLM pre-training and post-training workloads on the latest NVIDIA hardware and software platforms. • Drive ...
LLM Dataset Engineer
San Francisco, CA · On-site
$155K - $210K/yr
Post-Training & Alignment Data: Lead the development of high-quality post-training datasets ... Experience building massive LLM training sets from scratch , including raw web crawls (e.g., Common ...
LLM Dataset Engineer
San Francisco, CA · On-site
$155K - $210K/yr
Post-Training & Alignment Data: Lead the development of high-quality post-training datasets ... Experience building massive LLM training sets from scratch , including raw web crawls (e.g., Common ...
... training processes, including the selection of training data and feedback mechanisms. - Stay abreast of the latest developments in cyber security threats, tactics, techniques, and procedures (TTPs ...
... training processes, including the selection of training data and feedback mechanisms. - Stay abreast of the latest developments in cyber security threats, tactics, techniques, and procedures (TTPs ...
... training workflows. This role requires a hands-on deep learning practitioner comfortable with modern frameworks and libraries specific to LLM development. * Enables domain-specific fine-tuning of ...
... training workflows. This role requires a hands-on deep learning practitioner comfortable with modern frameworks and libraries specific to LLM development. * Enables domain-specific fine-tuning of ...
Research Scientist Graduate (Applied Machine Learning - ML System) - 2026 Start (PhD)
San Jose, CA · On-site
Responsibilities : • Responsible for developing and optimizing LLM training&inference&RL framework. • Working closely with model researchers to scale LLM training&RL to the next level. • ...
Research Scientist Graduate (Applied Machine Learning - ML System) - 2026 Start (PhD)
San Jose, CA · On-site
Responsibilities : • Responsible for developing and optimizing LLM training&inference&RL framework. • Working closely with model researchers to scale LLM training&RL to the next level. • ...
Lead end-to-end performance analysis and optimization of innovative LLM pre-training and post-training workloads on the latest NVIDIA hardware and software platforms. * Drive workloads closer to ...
Lead end-to-end performance analysis and optimization of innovative LLM pre-training and post-training workloads on the latest NVIDIA hardware and software platforms. * Drive workloads closer to ...
Lead end-to-end performance analysis and optimization of innovative LLM pre-training and post-training workloads on the latest NVIDIA hardware and software platforms. * Drive workloads closer to ...
Lead end-to-end performance analysis and optimization of innovative LLM pre-training and post-training workloads on the latest NVIDIA hardware and software platforms. * Drive workloads closer to ...
LLM Engineer
San Mateo, CA · On-site
Connect LLM capabilities with Luminary's Physics AI training/evaluation/inference pipelines, physics simulation solvers, mesh tools, and analytics APIs to enable end-to-end automation * Establish ...
LLM Engineer
San Mateo, CA · On-site
Connect LLM capabilities with Luminary's Physics AI training/evaluation/inference pipelines, physics simulation solvers, mesh tools, and analytics APIs to enable end-to-end automation * Establish ...
Research Scientist - Frontier Data - LLM training pipelines, RLHF or RLVR
San Francisco, CA · On-site
$150K - $250K/yr
... LLM training pipelines, RLHF or RLVR, or evaluation methodology. Does not need a PhD but must have the research depth of a strong undergrad or master's researcher Genuine obsession with how data ...
Research Scientist - Frontier Data - LLM training pipelines, RLHF or RLVR
San Francisco, CA · On-site
$150K - $250K/yr
... LLM training pipelines, RLHF or RLVR, or evaluation methodology. Does not need a PhD but must have the research depth of a strong undergrad or master's researcher Genuine obsession with how data ...
NLP Research Scientist
Palo Alto, CA · On-site
Develop synthetic data pipeline/creating real data pipeline for LLM training * Develop effective methods for benchmarking, e.g., LLM-As-Judge * Collaborate closely with hardware engineers, software ...
Quick apply
NLP Research Scientist
Palo Alto, CA · On-site
Develop synthetic data pipeline/creating real data pipeline for LLM training * Develop effective methods for benchmarking, e.g., LLM-As-Judge * Collaborate closely with hardware engineers, software ...
LLM Modeling & Post-Training Engineer About the Opportunity We're partnering with a globally recognized enterprise organization that is making a significant investment in next-generation AI ...
LLM Modeling & Post-Training Engineer About the Opportunity We're partnering with a globally recognized enterprise organization that is making a significant investment in next-generation AI ...
Develop synthetic data pipeline/creating real data pipeline for LLM training * Develop effective methods for benchmarking, e.g., LLM-As-Judge * Collaborate closely with hardware engineers, software ...
Develop synthetic data pipeline/creating real data pipeline for LLM training * Develop effective methods for benchmarking, e.g., LLM-As-Judge * Collaborate closely with hardware engineers, software ...
NLP Research Scientist
Palo Alto, CA · On-site
Develop synthetic data pipeline/creating real data pipeline for LLM training * Develop effective methods for benchmarking, e.g., LLM-As-Judge * Collaborate closely with hardware engineers, software ...
NLP Research Scientist
Palo Alto, CA · On-site
Develop synthetic data pipeline/creating real data pipeline for LLM training * Develop effective methods for benchmarking, e.g., LLM-As-Judge * Collaborate closely with hardware engineers, software ...
Senior AI Researcher, On-Device LLM Efficiency
San Diego, CA · On-site
$159K - $238K/yr
Research and development in the area of LLM inference efficiency algorithms, efficient model architecture design, and/or LLM training * Develop creative solutions with consideration of practical ...
Senior AI Researcher, On-Device LLM Efficiency
San Diego, CA · On-site
$159K - $238K/yr
Research and development in the area of LLM inference efficiency algorithms, efficient model architecture design, and/or LLM training * Develop creative solutions with consideration of practical ...
Python LLM Developer
Irving, TX · On-site
$48.25 - $66.50/hr
Role Python LLM Developer Location: Irving, TX ( day1 onsite, hybrid ) Python LLM Python LLM ... Custom Model Training: Ability to fine-tune and train models on specific datasets, understand ...
Python LLM Developer
Irving, TX · On-site
$48.25 - $66.50/hr
Role Python LLM Developer Location: Irving, TX ( day1 onsite, hybrid ) Python LLM Python LLM ... Custom Model Training: Ability to fine-tune and train models on specific datasets, understand ...
AI/LLM Engineer
New York, NY · On-site
$120K - $130K/yr
Must Have Technical/Functional Skills We are looking for a highly skilled AI/LLM Engineer to design ... Commuter Benefits & Certification & Training Reimbursement. Time Off: Vacation, Time Off, Sick ...
AI/LLM Engineer
New York, NY · On-site
$120K - $130K/yr
Must Have Technical/Functional Skills We are looking for a highly skilled AI/LLM Engineer to design ... Commuter Benefits & Certification & Training Reimbursement. Time Off: Vacation, Time Off, Sick ...
LLM Engineer (GCP Preferred)
Atlanta, GA · Hybrid
$58 - $60/hr
LLM Engineer (GCP Preferred) Work Time Zone: EST Rate: $60/hour on 1099/C2C Location: Atlanta, GA ... Apply techniques such as parameter-efficient fine-tuning , prompt tuning , and adapter training
Quick apply
LLM Engineer (GCP Preferred)
Atlanta, GA · Hybrid
$58 - $60/hr
LLM Engineer (GCP Preferred) Work Time Zone: EST Rate: $60/hour on 1099/C2C Location: Atlanta, GA ... Apply techniques such as parameter-efficient fine-tuning , prompt tuning , and adapter training
Research Scientist - RL environments / METR / LLM/ RLHF or RLVR
San Francisco, CA · On-site
$150K - $200K/yr
... training pipelines Model annotator behavior and run experiments to improve different model ... LLM training pipelines, RLHF or RLVR, or evaluation methodology. Does not need a PhD but must have ...
New
Quick apply
Research Scientist - RL environments / METR / LLM/ RLHF or RLVR
San Francisco, CA · On-site
$150K - $200K/yr
... training pipelines Model annotator behavior and run experiments to improve different model ... LLM training pipelines, RLHF or RLVR, or evaluation methodology. Does not need a PhD but must have ...
New
LLM Inference Deployment Engineer
$180K - $240K/yr
About the Role EnCharge AI is seeking an LLM Inference Deployment Engineer to optimize, deploy, and ... Deploy and optimize LLMs (GPT, LLaMA, Mistral, Falcon, etc.) post-training from libraries like ...
LLM Inference Deployment Engineer
$180K - $240K/yr
About the Role EnCharge AI is seeking an LLM Inference Deployment Engineer to optimize, deploy, and ... Deploy and optimize LLMs (GPT, LLaMA, Mistral, Falcon, etc.) post-training from libraries like ...
AI/LLM Engineer
Springfield, PA · On-site
The AI/LLM Engineer will lead the design and implementation of advanced systems centered on large ... Gap International is a global business management consulting firm that provides executive training ...
AI/LLM Engineer
Springfield, PA · On-site
The AI/LLM Engineer will lead the design and implementation of advanced systems centered on large ... Gap International is a global business management consulting firm that provides executive training ...
Llm Trainer information
See salary details
$19.43 is the 25th percentile. Wages below this are outliers.
$15.63 - $22.60
46% of jobs
The median wage is $23.71 / hr.
$22.60 - $29.57
26% of jobs
$29.57 - $36.54
0% of jobs
$36.54 - $43.51
0% of jobs
$48.74 is the 75th percentile. Wages above this are outliers.
$43.51 - $50.48
4% of jobs
$50.48 - $57.45
13% of jobs
$57.45 - $64.42
0% of jobs
$64.42 - $71.39
0% of jobs
$71.39 - $78.37
3% of jobs
$78.37 - $85.34
4% of jobs
$85.34 - $92.31
4% of jobs
$15
$36
$92
How much do llm trainer jobs pay per hour?
As of Jul 6, 2026, the average hourly pay for llm trainer in the United States is $36.91, according to ZipRecruiter salary data. Most workers in this role earn between $19.23 and $52.88 per hour, depending on experience, location, and employer.
What are some typical responsibilities and challenges faced by an LLM Trainer on a day-to-day basis?
LLM Trainers are responsible for designing and refining training datasets, developing prompts, evaluating model outputs, and working closely with engineers and data scientists to optimize large language models. Common challenges include maintaining data quality, mitigating model biases, and staying up-to-date with rapidly evolving AI research and best practices. You’ll often collaborate with cross-functional teams, communicate findings clearly, and adapt to new tools or methodologies. This dynamic environment offers opportunities for innovation and skill development, making it an excellent fit for those passionate about advancing AI technology.
What are the key skills and qualifications needed to thrive in the Llm Trainer position, and why are they important?
To thrive as an LLM Trainer, you need a deep understanding of natural language processing (NLP), machine learning principles, and data annotation techniques, often supported by a background in computer science or related fields. Familiarity with tools like Python, PyTorch or TensorFlow, data labeling platforms, and version control systems is essential, along with knowledge of prompt engineering and model fine-tuning. Strong analytical thinking, attention to detail, and collaborative communication skills are crucial soft skills for working with cross-functional AI teams. These competencies are important for developing high-quality language models that meet user needs and industry standards.
What is an LLM Trainer job?
An LLM Trainer is responsible for training and fine-tuning large language models (LLMs) to improve their accuracy, efficiency, and relevance for specific applications. This role involves curating and preprocessing training data, designing training methodologies, and evaluating model performance. LLM Trainers work closely with data scientists, engineers, and researchers to optimize models for tasks such as natural language understanding, text generation, and conversational AI. They also ensure ethical AI practices by mitigating biases and refining model outputs.
More about Llm Trainer jobs
What cities are hiring for Llm Trainer jobs? Cities with the most Llm Trainer job openings:
What are the most commonly searched types of Llm Trainer jobs? The most popular types of Llm Trainer jobs are:
What states have the most Llm Trainer jobs? States with the most job openings for Llm Trainer jobs include:
What job categories do people searching Llm Trainer jobs look for? The top searched job categories for Llm Trainer jobs are:
Full-time
Posted 8 days ago
Job description
Job Summary:
NVIDIA is seeking a Principal Engineer to drive the performance of large-scale AI training and post-training workloads across NVIDIA’s full hardware and software stack. The role involves analyzing and optimizing frontier-scale LLM workloads on GPUs and influencing multi-functional decisions to improve performance and efficiency across the AI ecosystem.
Responsibilities:
• Lead end-to-end performance analysis and optimization of innovative LLM pre-training and post-training workloads on the latest NVIDIA hardware and software platforms.
• Drive workloads closer to speed-of-light performance by identifying and removing bottlenecks across compute, memory, communication, scheduling, parallelism strategy, kernel efficiency, framework overhead, and system-level scaling.
• Develop production-quality software, tools, models, benchmarks, and analysis infrastructure that improve training performance, efficiency, and developer velocity across NVIDIA’s AI software stack.
• Build and refine performance models, workload characterizations, and simulation methodologies to guide future GPU, networking, system, and software architecture decisions.
• Serve as a technical authority for AI training performance, partnering closely with teams across GPU architecture, systems, CUDA libraries, compilers, networking, frameworks, product management, and applied AI.
• Translate workload insights into concrete hardware and software recommendations, and advocate for changes that improve performance and efficiency across the AI ecosystem.
• Mentor and provide technical leadership to engineers across the organization, helping establish best practices for large-scale AI performance analysis and optimization.
Qualifications:
Required:
• A MS, or PhD (or equivalent experience) in Computer Science, Electrical Engineering, Computer Engineering, or a related field, with 12+ years of relevant work or research experience.
• Demonstrated principal-level technical impact in one or more of the following areas: large-scale AI training systems, GPU performance optimization, distributed systems, high-performance computing, ML frameworks, compilers/runtimes, or hardware/software co-design.
• Deep hands-on experience analyzing and optimizing performance of large-scale deep learning workloads, especially transformer-based models, LLM pre-training, reinforcement learning, fine-tuning, or other post-training workloads.
• Strong understanding of GPU and AI accelerator architecture from individual accelerators to datacenter-scale systems.
• Experience with distributed training techniques such as data parallelism, tensor parallelism, pipeline parallelism, expert parallelism, sequence parallelism, activation checkpointing, mixed precision training, and communication/computation overlap.
• A strong track record of using profiling, tracing, benchmarking, and performance modeling tools to diagnose complex bottlenecks and drive measurable improvements.
• Excellent communication and technical leadership skills, with the ability to influence architecture and software decisions across multiple teams without relying on direct authority.
Company:
NVIDIA is a computing platform company operating at the intersection of graphics, HPC, and AI. Founded in 1993, the company is headquartered in Santa Clara, USA, with a team of 10001+ employees. The company is currently Late Stage.
NVIDIA is seeking a Principal Engineer to drive the performance of large-scale AI training and post-training workloads across NVIDIA’s full hardware and software stack. The role involves analyzing and optimizing frontier-scale LLM workloads on GPUs and influencing multi-functional decisions to improve performance and efficiency across the AI ecosystem.
Responsibilities:
• Lead end-to-end performance analysis and optimization of innovative LLM pre-training and post-training workloads on the latest NVIDIA hardware and software platforms.
• Drive workloads closer to speed-of-light performance by identifying and removing bottlenecks across compute, memory, communication, scheduling, parallelism strategy, kernel efficiency, framework overhead, and system-level scaling.
• Develop production-quality software, tools, models, benchmarks, and analysis infrastructure that improve training performance, efficiency, and developer velocity across NVIDIA’s AI software stack.
• Build and refine performance models, workload characterizations, and simulation methodologies to guide future GPU, networking, system, and software architecture decisions.
• Serve as a technical authority for AI training performance, partnering closely with teams across GPU architecture, systems, CUDA libraries, compilers, networking, frameworks, product management, and applied AI.
• Translate workload insights into concrete hardware and software recommendations, and advocate for changes that improve performance and efficiency across the AI ecosystem.
• Mentor and provide technical leadership to engineers across the organization, helping establish best practices for large-scale AI performance analysis and optimization.
Qualifications:
Required:
• A MS, or PhD (or equivalent experience) in Computer Science, Electrical Engineering, Computer Engineering, or a related field, with 12+ years of relevant work or research experience.
• Demonstrated principal-level technical impact in one or more of the following areas: large-scale AI training systems, GPU performance optimization, distributed systems, high-performance computing, ML frameworks, compilers/runtimes, or hardware/software co-design.
• Deep hands-on experience analyzing and optimizing performance of large-scale deep learning workloads, especially transformer-based models, LLM pre-training, reinforcement learning, fine-tuning, or other post-training workloads.
• Strong understanding of GPU and AI accelerator architecture from individual accelerators to datacenter-scale systems.
• Experience with distributed training techniques such as data parallelism, tensor parallelism, pipeline parallelism, expert parallelism, sequence parallelism, activation checkpointing, mixed precision training, and communication/computation overlap.
• A strong track record of using profiling, tracing, benchmarking, and performance modeling tools to diagnose complex bottlenecks and drive measurable improvements.
• Excellent communication and technical leadership skills, with the ability to influence architecture and software decisions across multiple teams without relying on direct authority.
Company:
NVIDIA is a computing platform company operating at the intersection of graphics, HPC, and AI. Founded in 1993, the company is headquartered in Santa Clara, USA, with a team of 10001+ employees. The company is currently Late Stage.
About Nvidia
Sourced by ZipRecruiter
NVIDIA has been transforming computer graphics, PC gaming, and accelerated computing for more than 25 years. It's a unique legacy of innovation that's fueled by great technology--and amazing people. Today, we're tapping into the unlimited potential of AI to define the next era of computing. An era in which our GPU acts as the brains of computers, robots, and self-driving cars that can understand the world. Doing what's never been done before takes vision, innovation, and the world's best talent.
Industry
Computer and electronic product manufacturing
Company size
10,000+ Employees
Headquarters location
Santa Clara, CA, US
Year founded
1993