AI/ML Software Engineer At Gallatin, we are rebuilding logistics infrastructure for the national ... Establish scalable MLOps pipelines and real-time inference services to streamline model training ...
AI/ML Software Engineer At Gallatin, we are rebuilding logistics infrastructure for the national ... Establish scalable MLOps pipelines and real-time inference services to streamline model training ...
The Staff Software Engineer for the AI Model Lifecycle team will play a crucial role in building a ... Implement and maintain end-to-end training pipelines for Large Language Models. * RFT and ...
Quick apply
The Staff Software Engineer for the AI Model Lifecycle team will play a crucial role in building a ... Implement and maintain end-to-end training pipelines for Large Language Models. * RFT and ...
Senior Software Engineer, AI Model Lifecycle
San Francisco, CA · On-site
$144K - $190K/yr
The Senior Software Engineer for the AI Model Lifecycle team will play a crucial role in building a ... Implement and maintain end-to-end training pipelines for Large Language Models. * RFT and ...
Senior Software Engineer, AI Model Lifecycle
San Francisco, CA · On-site
$144K - $190K/yr
The Senior Software Engineer for the AI Model Lifecycle team will play a crucial role in building a ... Implement and maintain end-to-end training pipelines for Large Language Models. * RFT and ...
Senior Software Engineer, AI Model Lifecycle
Sunnyvale, CA · On-site
$143K - $189K/yr
The Senior Software Engineer for the AI Model Lifecycle team will play a crucial role in building a ... Implement and maintain end-to-end training pipelines for Large Language Models. * RFT and ...
Quick apply
Senior Software Engineer, AI Model Lifecycle
Sunnyvale, CA · On-site
$143K - $189K/yr
The Senior Software Engineer for the AI Model Lifecycle team will play a crucial role in building a ... Implement and maintain end-to-end training pipelines for Large Language Models. * RFT and ...
The Staff Software Engineer for the AI Model Lifecycle team will play a crucial role in building a ... Implement and maintain end-to-end training pipelines for Large Language Models. * RFT and ...
Quick apply
The Staff Software Engineer for the AI Model Lifecycle team will play a crucial role in building a ... Implement and maintain end-to-end training pipelines for Large Language Models. * RFT and ...
Senior Software Engineer, AI Model Lifecycle
San Francisco, CA · On-site
$144K - $190K/yr
The Senior Software Engineer for the AI Model Lifecycle team will play a crucial role in building a ... Implement and maintain end-to-end training pipelines for Large Language Models. * RFT and ...
Quick apply
Senior Software Engineer, AI Model Lifecycle
San Francisco, CA · On-site
$144K - $190K/yr
The Senior Software Engineer for the AI Model Lifecycle team will play a crucial role in building a ... Implement and maintain end-to-end training pipelines for Large Language Models. * RFT and ...
The Staff Software Engineer for the AI Model Lifecycle team will play a crucial role in building a ... Implement and maintain end-to-end training pipelines for Large Language Models. * RFT and ...
The Staff Software Engineer for the AI Model Lifecycle team will play a crucial role in building a ... Implement and maintain end-to-end training pipelines for Large Language Models. * RFT and ...
Staff Software Engineer - AI Research Infrastructure
New York, NY · On-site
$199K - $270K/yr
Staff Software Engineer - AI Research Infrastructure P-1215 At Databricks, we are obsessed with ... Understand modern ML training and inference workflows (e.g., distributed training, model ...
Staff Software Engineer - AI Research Infrastructure
New York, NY · On-site
$199K - $270K/yr
Staff Software Engineer - AI Research Infrastructure P-1215 At Databricks, we are obsessed with ... Understand modern ML training and inference workflows (e.g., distributed training, model ...
Software Engineer, AI Training and Infrastructure
San Francisco, CA · On-site
$100K - $300K/yr
Position Overview We are looking for a Software Engineer to work at the forefront of developing and optimizing the software infrastructure and tools necessary for training cutting-edge AI models. You ...
Software Engineer, AI Training and Infrastructure
San Francisco, CA · On-site
$100K - $300K/yr
Position Overview We are looking for a Software Engineer to work at the forefront of developing and optimizing the software infrastructure and tools necessary for training cutting-edge AI models. You ...
Staff Software Engineer - AI Research Infrastructure
New York, NY · On-site
$190K - $270K/yr
Staff Software Engineer - AI Research Infrastructure P-1215 At Databricks, we are obsessed with ... Understand modern ML training and inference workflows (e.g., distributed training, model ...
Staff Software Engineer - AI Research Infrastructure
New York, NY · On-site
$190K - $270K/yr
Staff Software Engineer - AI Research Infrastructure P-1215 At Databricks, we are obsessed with ... Understand modern ML training and inference workflows (e.g., distributed training, model ...
Principal Software Engineer (AI/ML Architect-Engineer)
$133K - $179K/yr
UKG Principle Software Engineer, AI/ML At UKG, the work you do matters. The code you ship, the ... model training and deployment. * AI Methodologies: In-depth knowledge of generative AI ...
Principal Software Engineer (AI/ML Architect-Engineer)
$133K - $179K/yr
UKG Principle Software Engineer, AI/ML At UKG, the work you do matters. The code you ship, the ... model training and deployment. * AI Methodologies: In-depth knowledge of generative AI ...
... AI models' alignment to human values. • Diagnose and remediate active incidents on the platform ... Software Engineer (Jr/Mid level) San Francisco, CA $120,000.00-$180,000.00 1 month ago San ...
... AI models' alignment to human values. • Diagnose and remediate active incidents on the platform ... Software Engineer (Jr/Mid level) San Francisco, CA $120,000.00-$180,000.00 1 month ago San ...
Data Engineer & AI/ML Specialist, Mid Level (Remote)
Portland, OR · Remote
$121K - $145K/yr
... model training. • Implement and optimize search indexing (Azure AI Search) for both structured ... Preferred: • Experience in healthcare software development (EHR, EMR, or related domains). • ...
Data Engineer & AI/ML Specialist, Mid Level (Remote)
Portland, OR · Remote
$121K - $145K/yr
... model training. • Implement and optimize search indexing (Azure AI Search) for both structured ... Preferred: • Experience in healthcare software development (EHR, EMR, or related domains). • ...
Software Engineer, AI Compute Infrastructure
Los Angeles, CA · On-site
$186K - $220K/yr
We are seeking a seasoned Software Engineer to build and scale the foundational compute infrastructure that powers our state-of-the-art AI models--from multimodal training data pipelines to high ...
Software Engineer, AI Compute Infrastructure
Los Angeles, CA · On-site
$186K - $220K/yr
We are seeking a seasoned Software Engineer to build and scale the foundational compute infrastructure that powers our state-of-the-art AI models--from multimodal training data pipelines to high ...
Senior Staff Software Engineer, AI Model Lifecycle
San Francisco, CA · On-site
$144K - $190K/yr
The Senior Staff Software Engineer for the AI Model Lifecycle team will manage fine-tuning systems for large foundation models and implement end-to-end training pipelines for Large Language Models ...
Senior Staff Software Engineer, AI Model Lifecycle
San Francisco, CA · On-site
$144K - $190K/yr
The Senior Staff Software Engineer for the AI Model Lifecycle team will manage fine-tuning systems for large foundation models and implement end-to-end training pipelines for Large Language Models ...
Large Model Training Acceleration Engineer
San Jose, CA · On-site
$156K - $316K/yr
... AI model training optimization. - Strong software engineering skills, including proficiency in Python, C++, and CUDA. - Strong proficiency in deep learning frameworks such as PyTorch, Megatron and ...
Large Model Training Acceleration Engineer
San Jose, CA · On-site
$156K - $316K/yr
... AI model training optimization. - Strong software engineering skills, including proficiency in Python, C++, and CUDA. - Strong proficiency in deep learning frameworks such as PyTorch, Megatron and ...
Software Engineer, AI Inference
Pittsburgh, PA · On-site
$100K - $300K/yr
Position Overview We are looking for a Software Engineer to work at the forefront of deploying our cutting-edge AI models, enhancing the performance and capabilities of our embodied systems. You will ...
Software Engineer, AI Inference
Pittsburgh, PA · On-site
$100K - $300K/yr
Position Overview We are looking for a Software Engineer to work at the forefront of deploying our cutting-edge AI models, enhancing the performance and capabilities of our embodied systems. You will ...
Software Engineer, AI Compute Infrastructure
Palo Alto, CA · On-site
$203K - $241K/yr
We are seeking a seasoned Software Engineer to build and scale the foundational compute infrastructure that powers our state-of-the-art AI models--from multimodal training data pipelines to high ...
Software Engineer, AI Compute Infrastructure
Palo Alto, CA · On-site
$203K - $241K/yr
We are seeking a seasoned Software Engineer to build and scale the foundational compute infrastructure that powers our state-of-the-art AI models--from multimodal training data pipelines to high ...
Software Engineer, AI Compute Infrastructure
San Francisco, CA · On-site
$203K - $241K/yr
We are seeking a seasoned Software Engineer to build and scale the foundational compute infrastructure that powers our state-of-the-art AI models--from multimodal training data pipelines to high ...
Software Engineer, AI Compute Infrastructure
San Francisco, CA · On-site
$203K - $241K/yr
We are seeking a seasoned Software Engineer to build and scale the foundational compute infrastructure that powers our state-of-the-art AI models--from multimodal training data pipelines to high ...
Software Engineer, AI Compute Infrastructure
Palo Alto, CA · On-site
$203K - $241K/yr
We are seeking a seasoned Software Engineer to build and scale the foundational compute infrastructure that powers our state-of-the-art AI models--from multimodal training data pipelines to high ...
Software Engineer, AI Compute Infrastructure
Palo Alto, CA · On-site
$203K - $241K/yr
We are seeking a seasoned Software Engineer to build and scale the foundational compute infrastructure that powers our state-of-the-art AI models--from multimodal training data pipelines to high ...
Software Engineer Ai Model Training information
See salary details
$63.5K - $76.4K
2% of jobs
$76.4K - $89.3K
4% of jobs
$89.3K - $102.2K
6% of jobs
$102.2K - $115.1K
8% of jobs
$122.1K is the 25th percentile. Wages below this are outliers.
$115.1K - $128K
7% of jobs
$128K - $141K
18% of jobs
The median wage is $144K / yr.
$141K - $153.9K
16% of jobs
$153.9K - $166.8K
2% of jobs
$172.1K is the 75th percentile. Wages above this are outliers.
$166.8K - $179.7K
26% of jobs
$179.7K - $192.6K
1% of jobs
$192.6K - $205.5K
8% of jobs
$63.5K
$147.5K
$205.5K
How much do software engineer ai model training jobs pay per year?
What are some common challenges faced by Software Engineers specializing in AI model training, and how can they be addressed?
What are the key skills and qualifications needed to thrive as a Software Engineer in AI Model Training, and why are they important?
What is the difference between Software Engineer Ai Model Training vs Data Scientist?
| Aspect | Software Engineer Ai Model Training | Data Scientist |
|---|---|---|
| Required Credentials | Bachelor's in CS, related field; experience with ML frameworks | Bachelor's or higher in CS, statistics, or related field; strong analytical skills |
| Work Environment | Development teams, AI labs, cloud platforms | Data analysis, research environments, business units |
| Employer & Industry Usage | Tech companies, AI startups, research institutions | Tech firms, finance, healthcare, consulting |
While both roles involve working with data and machine learning, Software Engineer Ai Model Training focuses on developing and optimizing AI models through coding and engineering practices. Data Scientists analyze data, build models, and generate insights. The roles often collaborate but differ in their core responsibilities and skill sets.
What does a Software Engineer in AI Model Training do?

Other
Medical, Retirement, PTO
Posted 12 days ago
Job description
At Gallatin, we are rebuilding logistics infrastructure for the national security missions of the United States and allied partners. We build AI systems that determine how logistics decisions are made—not just how they're executed. From factory to foxhole, we operate at the layer where data becomes decisions, and decisions make the advantage.
We're looking for an AI/ML Software Engineer to play a foundational role in developing and deploying AI-powered solutions for our core product. You'll work across the full AI/ML lifecycle—from defining and building models to evaluating and deploying large scale ML pipelines and real-time inference systems—while collaborating with cross-functional teams to deliver impactful, real-world AI applications.
What You'll Do
Build and Deploy AI/ML Models
Own and develop an AI model aligned with our core product needs within the first three months.
Validate AI-driven product features through real-world testing and feedback from warfighters.
Establish scalable MLOps pipelines and real-time inference services to streamline model training, deployment, runtime, and monitoring.
Own high model reliability and uptime by implementing monitoring across your work.
Data Collection & Processing
Identify and integrate structured, unstructured, real-time, and batch data sources.
Work with internal logs, APIs, user interactions, and third-party datasets to improve model training quality.
Continuous Improvement & Best Practices
Stay ahead of AI trends and emerging technologies to improve model performance.
Document and share best practices for AI/ML development.
Contribute to the hiring and mentoring of AI/ML talent as we grow our team.
Teach the wider team about the latest trends and significance in novel approaches in AI.
What We're Looking For
Strong Programming Skills
Expertise in at least one: Python, C++, or C.
ML Framework Proficiency
Hands-on experience with AI/ML training and fine-tuning frameworks, including:
PyTorch, TensorFlow, CUDA, Jupyter Notebooks
Large Language Models (LLMs) and Open-Source Models (Llama, Anthropic, Mistral)
LangChain, Retrieval-Augmented Generation (RAG), Hugging Face, Exo Labs
MLOps & Data Engineering Knowledge
Experience with data processing and pipeline frameworks such as Apache Kafka, Apache Airflow, AWS Kinesis, pandas, and dbt.
Understanding of AI model performance monitoring, data drift detection, and observability tools like Grafana or Kibana.
AI Fundamentals & Security Awareness
Deep understanding of deep neural networks (DNNs), LLMs, over/underfitting, prompt engineering, and LLM security (jailbreaking risks and protections).
Growth Mindset
Passion for continuous learning and staying current with the latest advancements in AI/ML, as well as teaching others
Bonus Points
Experience mentoring and upskilling junior engineers.
Familiarity with legacy systems and working with public sector customers.
Contributions to open-source (AI/ML) projects.
Mission and Identity
We are building the system that enables faster, smarter logistics decisions in contested environments, and we're doing it with a team of seasoned entrepreneurs, operators, and technologists who have built and scaled solutions in this space before. We hold ourselves to an extremely high standard. We value clear thinking, direct communication, and the kind of ownership that doesn't stop until something actually works.
Our mission is to create decision advantage when the stakes are the highest. If we succeed, the system doesn't just run; it gets smarter. We're not building AI for its own sake. We're building it because faster, smarter decisions in the most demanding environments on earth can't wait. If you want to work somewhere the stakes are real and the mission is urgent—you'll fit in here.
Why Gallatin?
The logistics infrastructure that supports America's warfighters and humanitarian disaster responders is overdue for transformation, and we are building it. From defense operations to disaster response, we're solving the hardest problems that keep missions moving when it matters most. Join a team where the mission is the point.
Compensation: Gallatin offers competitive compensation commensurate with experience. Actual compensation may vary based on experience, skills, and location. In addition to base salary, we offer a generous equity grant, full healthcare coverage, 401k, unlimited PTO, and the perks of working in a high-caliber, mission-driven environment.
Gallatin is an equal opportunity employer. We do not discriminate on the basis of race, color, religion, sex, national origin, age, disability, veteran status, sexual orientation, gender identity, or any other characteristic protected by applicable federal, state, or local law.
This position may require the ability to obtain and maintain a U.S. government security clearance. The successful candidate must be able to work in a classified environment when necessary.
We comply with the United States Department of Labor's Pay Transparency provision.