Measurement Learning Evaluation Analyst Jobs (NOW HIRING)

About VTI Aerospace

VTI Aerospace builds AI-powered perception and pilot assist technologies to unlock the future of aviation. With offices in Bozeman, MT and Seattle, WA, we are a team of engineers and technologists from Boeing, Airbus, Aurora, and beyond. We are passionate about pushing the boundaries of autonomy and aviation safety. Our company is dedicated to advancing the field of aviation with cutting-edge solutions

The Role

As a Senior Software Engineer – Evaluation, you will design and implement systems that measure and monitor the performance of our computer vision, automatic speech recognition (ASR), and small language model (SLM) systems. You will develop evaluation methodologies, benchmarking pipelines, and monitoring tools that ensure our AI systems perform reliably in real-world environments. You will help establish the evaluation standards and performance benchmarks that guide the development of all AI systems at the company.

You will work closely with machine learning and data engineering teams to evaluate model performance, identify failure modes, and guide improvements to data collection and model training, helping bring AI-powered aviation tools to market.

This role is ideal for someone who enjoys designing robust evaluation systems and uncovering insights from model performance data, and wants to have a direct impact on the quality and reliability of our AI systems.

What You'll Do

• Define and implement evaluation methodologies for computer vision, ASR, and language systems
• Identify and track key performance indicators (KPIs) that measure system and model effectiveness
• Perform dataset coverage analysis to understand strengths, gaps, and biases in training and evaluation data
• Identify model deficiencies and collaborate with ML engineers to improve training data and model performance
• Build scalable model evaluation pipelines in Python for automated benchmarking and regression testing
• Design and maintain systems for monitoring model performance and drift in production environments
• Architect data models and storage systems for evaluation results using relational and time-series databases
• Build dashboards and reporting tools to visualize model performance and evaluation metrics
• Collaborate with ML and data engineering teams to improve evaluation workflows and data pipelines
• Contribute to ground-truth data pipelines and processes that support reliable evaluation

What We're Looking For

• 5+ years of experience developing production software systems in Python
• Strong experience designing and implementing data processing or analysis pipelines
• Experience building systems for machine learning evaluation, experimentation, or benchmarking
• Experience working with large datasets and building scalable data workflows
• Familiarity with statistical methods, experimental design, and model performance analysis
• Experience building automated pipelines using tools such as Airflow or similar orchestration frameworks
• Experience working with evaluation or experiment tracking tools such as MLflow or similar systems
• Experience designing data models and working with relational databases and time-series data
• Strong collaboration skills and ability to work closely with machine learning and data engineering teams

Bonus Points

• Experience working with computer vision, ASR, or language models in production environments
• Experience using PyTorch or other deep learning frameworks
• Experience evaluating multimodal or large-scale AI systems
• Experience building monitoring systems for ML models in production
• Experience designing dataset analysis or data quality tooling
• Experience building dashboards or monitoring tools using Grafana or similar platforms

Why You'll Love Working Here

• Opportunity to work on meaningful problems in aviation and deliver products to industry at a rapid pace
• Small team with significant impact on company direction
• Collaborative and innovative environment
• Competitive compensation and benefits
• Opportunities for growth and leadership

Location

Work arrangement: On-site

Primary location: Seattle or Bozeman

Compensation & Benefits

Salary range: DOE

Benefits may include:

• Health, dental, and vision insurance
• Retirement plan or 401(k)
• Paid time off and holidays
• Professional development opportunities

Equal Opportunity Statement

VTI is proud to be an equal opportunity employer. We value diverse perspectives and encourage candidates from all backgrounds to apply