Performance Evaluation Jobs (NOW HIRING)

Mission Summary:

The Systems Readiness and Performance team is the crucial bridge between software development and real-world deployment. We are responsible for driving system design, for verifying and validating the autonomy stack, and for defining, measuring, and validating system performance targets. We work closely with stakeholders in autonomy, infrastructure, and operations to build the definitive safety case for the commercial launch of our fully driverless IONIQ 5 robotaxis in Las Vegas.

Rigorous behavioral and system performance evaluation is critical to scaling our service and achieving Motional's long-term goals. We are seeking a Senior Data Scientist to lead initiatives that improve evaluation and testing methodologies, measure the quality and trustworthiness of our evaluation portfolio, and partner with engineering teams to monitor and strengthen the health of the evaluation ecosystem. You will help ensure Motional's performance evaluation is efficient, scientifically rigorous, and aligned with our growth priorities.

In this role, you will lead development of evaluation methodologies and metrics that assess the quality and business relevance of solutions spanning on-road and off-board data. You will influence the evaluation signals software engineers rely on to validate that changes to the autonomy stack deliver intended improvements, conduct deep-dive analyses to understand bottlenecks in current methodologies, and prototype improvements in metrics, sampling strategy, and statistical inference. You will develop deep expertise in how evaluation signals inform launch and release decisions, weigh trade-offs across the evaluation portfolio, and provide actionable insights for designing launch criteria.

If you are a rigorous, collaborative data scientist with a passion for improving how autonomous systems are measured and validated at scale, we encourage you to apply.

What You'll Be Doing:

• Lead the development of evaluation frameworks for the autonomous system, connecting technical problems to rigorous, data-driven approaches for measuring and validating performance.
• Collaborate closely with Functional Safety and Systems Engineering teams to ensure evaluation metrics map effectively to automotive safety standards (e.g., SOTIF, ISO 21448) and launch readiness decisions.
• Ensure evaluation metrics are reliable enough to inform safety cases and launch readiness decisions.
• Monitor the reliability of evaluation metrics and incoming performance data over time, including detecting drift, inconsistencies, and degradation in metric definitions, to ensure the evaluation ecosystem remains accurate and trustworthy.
• Drive our approach to performance analysis using data-backed statistical methods for simulation and on-road data.
• Develop new statistical analysis methods to analyze AV performance data and lead by example in applying them to real problems.
• Partner with triage operators and simulation engineers to turn raw disengagements and identified edge cases into procedural or generative scenarios, feeding them back into the simulation catalog to strengthen test coverage.
• Use fleet and evaluation data to identify edge cases in an automated manner and coverage gaps, and partner with engineering to feed novel scenarios back into the simulation catalog and strengthen test coverage.
• Build confidence in the evaluation framework through data-driven insights and clear communication of findings to technical leaders and stakeholders.
• Establish correlation between on-road and simulation data to improve how we interpret and act on evaluation results.
• Make sense of large datasets to drive insights, solve ambiguous performance questions, and communicate results effectively across teams and upward to leadership.
• Establish a self-service model for developers to understand the impact of their changes.
• Develop new metrics, interpret trends, and investigate anomalies in simulation and on-road data.
• Collaborate with developers to drive action based on these results.
• Serve as an advisor and influence collaborators across multiple teams, promote data-aware decision making, and establish best practices around the use of data.
• Mentor and collaborate with fellow engineers and foster a positive, collaborative work environment.
• Introduce the use of ML methods for performance evaluation where they add rigor and scale.
  

What You Bring:

• 5+ years of industry experience solving complex problems with large datasets, with a track record of framing ambiguous questions into rigorous, data-driven analyses.
• Bachelor's or higher degree in Computer Science, Computer Engineering, Data Science, Robotics, Physics, Mathematics, or a related quantitative field. Master's or PhD preferred.
• Strong problem-solving skills: ability to break down complex performance and evaluation challenges, think logically, and remove bias from how problems are defined and assessed.
• Strong Python and SQL skills, with demonstrated experience using data analysis libraries to work with large, complex datasets.
• Experience applying advanced statistical and ML methods to drive insights from large and complex data sets.
• Demonstrated experience with statistical analysis, hypothesis testing, causal analysis and data analysis.
• Demonstrated ability to work independently with minimal guidance and drive projects from problem definition through to actionable results.
• Proven communication and interpersonal skills, with the ability to explain technical findings clearly to engineering partners and leadership.
• Eager to learn new statistical and ML techniques and demonstrated willingness to teach
  

Bonus Points:

• Experience with adversarial scenario generation and closed-loop simulation environments.
• Experience in autonomous driving or robotics, specifically evaluating sub-systems like Perception, Prediction, or Motion Planning.
• Familiarity with data pipelines and distributed compute (e.g., AWS) to seamlessly collaborate with our Data Engineering and MLOps partners.
• Expertise in Machine Learning and Deep Learning.
• Expertise in modern sequence modeling (e.g., Transformers applied to time-series or trajectory data) and probabilistic ML / uncertainty quantification for distinguishing rare-but-safe behavior from out-of-distribution failures.
• Familiarity with automotive safety standards like ISO 26262 or ISO 21448 (SOTIF).
  

We encourage a hybrid schedule with in-office time at one of our locations in Boston, Pittsburgh, or Las Vegas to support collaboration, or this role can be fully remote.