Live In Ai Validation Jobs in California (NOW HIRING)

Sonatus is looking for an experienced Senior Engineering Manager to build and lead our AI Validation function - the team responsible for how we test, evaluate, and govern the AI models and agentic capabilities embedded in our software-defined vehicle and cloud platforms, as well as our cloud-only AI and LLM-based products. You will deeply understand how AI is developed and deployed across Sonatus's embedded, cloud, and LLM/RAG-driven environments, identify gaps and friction in current validation practices, and turn those insights into scalable test strategy, evaluation frameworks, and governance mechanisms that let Sonatus ship trustworthy AI-driven features at automotive scale. You will lead and grow a high-performing AI validation team and partner closely with engineering, product, and safety stakeholders to make AI quality and safety a competitive advantage.

• Define and drive Sonatus's AI validation strategy across embedded, in-vehicle, cloud-connected, and cloud-native AI systems, identifying gaps in model development, testing, deployment, and governance.
• Lead, hire, mentor, and grow a high-performing AI Validation organization, establishing scalable engineering processes, technical direction, and execution excellence.
• Own the end-to-end validation strategy for AI/ML models, LLMs, RAG pipelines, and agentic AI workflows-from data pipelines and model training through cloud services and in-vehicle deployment.
• Architect and operationalize scalable evaluation frameworks and benchmarking platforms for AI systems, including multi-step agentic workflows, using deterministic metrics, LLM-as-a-Judge methodologies, automated regression testing, and production feedback loops.
• Design and maintain evaluation harnesses for RAG and agentic systems that measure retrieval quality, grounding, citation accuracy, factual consistency, context relevance, safety, latency, reliability, and execution correctness.
• Evaluate, integrate, and optimize open-source and commercial AI validation technologies, driving build-versus-buy decisions for Sonatus's AI quality platform.
• Establish AI governance, Responsible AI practices, model lineage, safety guardrails, and compliance processes appropriate for automotive safety-critical systems and enterprise AI products.
• Act as the quality gatekeeper for AI-enabled releases, partnering with engineering, product, safety, and OEM stakeholders to identify risks, define release criteria, and ensure production readiness.
• Collaborate across engineering teams to define validation strategies for emerging AI capabilities, rapidly prototype new evaluation approaches, and standardize successful practices into reusable frameworks.
• Drive continuous improvement by tracking industry advances in AI evaluation, agentic AI, LLM validation, and RAG systems, translating them into scalable validation capabilities across Sonatus.

• Bachelor's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related field required (MS preferred).
• 10+ years of experience in software or systems engineering-including embedded, cloud, networking, security, or automotive domains-with 3+ years leading high-performing engineering or QA organizations.
• Hands-on experience developing, deploying, testing, or operating AI/ML systems, with strong expertise in modern ML workflows, neural networks, and MLOps.
• Deep understanding of LLMs, RAG architectures, vector databases, embeddings, retrieval optimization, and agentic AI frameworks such as LangGraph or equivalent orchestration platforms.
• Proven experience designing and implementing scalable evaluation frameworks for AI systems, including multi-step agentic workflows, regression testing, benchmarking, and automated quality scoring.
• Strong expertise with hybrid evaluation methodologies combining deterministic validation (citation grounding, structural validation, exact matching) and probabilistic LLM-as-a-Judge techniques (faithfulness, answer relevance, context precision, task completion).
• Practical experience with RAG evaluation frameworks such as RAGAS, including evaluation tuning, embedding optimization, retrieval quality improvement, and production-scale LLM evaluation pipelines.
• Experience validating hallucination, grounding, citation accuracy, bias, fairness, toxicity, and factual consistency in production LLM applications.
• Experience designing systems that verify external knowledge claims and ensure responses are grounded in traceable citations and trusted data sources.
• Strong experience testing cloud-native platforms and cloud-managed embedded products, including end-to-end system validation.
• Experience establishing AI governance, safety, compliance, and Responsible AI practices for enterprise or safety-critical systems.
• Proficiency in Python, Linux, shell scripting, modern test frameworks (PyTest, Playwright, Behave), and engineering productivity tools such as Jenkins and JIRA.
  

• Experience validating AI or agentic systems in safety-critical or regulated industries (automotive, aerospace, medical).
• Track record building and scaling an AI test/evaluation platform or developer experience used by multiple teams (frameworks, reusable components, reference implementations).
• Demonstrated wins moving AI testing practices from ad hoc to standardized, organization-wide adoption, with measurable impact on cycle time, quality, or reliability.
• Experience implementing enterprise-grade AI governance (auditability, monitoring, policy enforcement) in production systems.
• Deep experience evaluating LLM and RAG systems at scale, including agentic workflows, RAGAS-based evaluation, citation verification, hallucination detection, groundedness, faithfulness, answer relevance, tool/task correctness, and automated regression testing across offline and online feedback loops.

Sunnyvale HQ Benefits & Perks Offered:

• Health care plan (Medical, Dental & Vision)
• Flexible and Dependent Care Expense program
• Retirement plan (401k)
• Life Insurance (Basic, Voluntary & AD&D)
• Unlimited paid time off per year, 14+ paid holidays
• Hybrid office work arrangement
• Complimentary lunches, snacks, and beverages during on-site working days
• Wellness benefit allowance
• Phone & Internet reimbursement
• Computer Accessory Allowance