Ai Rater Jobs in California (NOW HIRING)

To get the best candidate experience, please consider applying for a maximum of 3 roles within 12 months to ensure you are not duplicating efforts.

Job Category

Operations

Job Details

Salesforce is the #1 AI CRM, where humans with agents drive customer success together. Here, ambition meets action. Tech meets trust. And innovation isn't a buzzword - it's a way of life. The world of work as we know it is changing and we're looking for Trailblazers who are passionate about bettering business and the world through AI, driving innovation, and keeping Salesforce's core values at the heart of it all.

Ready to level-up your career at the company leading workforce transformation in the agentic era? You're in the right place! Agentforce is the future of AI, and you are the future of Salesforce.

The Director of Help Agent AI & Tooling leads the team accountable for the generative AI capabilities, evaluation tooling, observability, and emerging program initiatives that power Salesforce's Help Agent - a customer-facing Agentforce deployment and one of Salesforce's flagship Customer Zero implementations. This leader sits at the intersection of AI product strategy, data-driven evaluation, and production excellence, and plays a critical role in ensuring the Help Agent performs reliably, improves continuously, and delivers measurable business value.

This leader directly manages a team covering five core areas:

Drive the strategic roadmap for generative AI capabilities within the Help Agent - including retrieval architecture (e.g., Ensemble Retriever), LLM configurations, prompt and planner optimization, and agentic readiness scoring. Partner closely with Tech & Product, and other technology teams to influence the Help Agent roadmap and direct engineering relationships.

Own the end-to-end evaluations tooling strategy for Help Agent performance - including the Testing Center, Agentforce Testing Agents (ATA), and custom evaluation frameworks. Define and track answer quality metrics including completeness, correctness, relevance, faithfulness, clarity, and latency. Manage tooling gaps, coordinate with Tech & Product on product priorities, and expand evaluation capabilities toward greater scale and automation.

Lead the observability program for the Help Agent - encompassing real-time performance monitoring, session tracing, error analysis, failure clustering, and business impact measurement. Serve as the primary stakeholder and voice of the Customer Zero on the Tech & Product Observability product roadmap. Coordinate with Technology Teams to converge on a comprehensive, always-on observability solution that replaces fragmented tooling.

Oversee the Net New Annual Order Value (NNAOV) program, driving pipeline generation through the Help Agent. Partner with the dedicated NNAOV Program Manager to define program goals, manage intake, and report on business outcomes.

Support the creation and delivery of AI Skills via applications like Slackbot, Claude Code and others to support the team's ability to deliver evaluations, insights, and opportunity sourcing for Roadmap development.

• Serve as the AI & Tooling team lead within Digital Success Data & AI, setting team mission, operating model, and priorities
• Act as a Customer Zero steward - representing Salesforce's own use of Agentforce as a reference customer, surfacing product gaps and driving roadmap influence through Shiproom intakes and direct T&P partnerships
• Monitor and manage Help Agent production health, including end-to-end latency, engagement rate, escalation rate, resolution rate, and quality scores - escalating issues and coordinating cross-functional remediation as needed
• Lead cross-functional collaboration with Technology Teams to align on evaluations, retrieval strategies, and tooling development
• Define and mature the AI Evaluations Framework, moving from manual, spreadsheet-based processes toward scalable, automated evaluation at production scale
• Drive the observability roadmap in partnership with Technology Teams, from current OOTB capabilities to a fully centralized, production-grade monitoring platform
• Represent AI & Tooling in leadership forums and communicate progress, risks, and strategic direction clearly to senior stakeholders

• Help Agent resolution rate and quality scores (completeness, correctness, relevance) improve quarter-over-quarter through disciplined evaluation and rapid iteration
• End-to-end latency - Deliver high quality answers as fast as possible for the application.
• A centralized, production-grade observability platform is fully adopted by the team as the single source of truth for agent performance
• The Testing Center is fully onboarded in production, with evaluation coverage scaled well beyond current manual limits
• The NNAOV Program generates measurable pipeline contribution from Help Agent interactions
• Tooling is standardized and integrated across the Data & AI team, eliminating fragmented, ad hoc approaches
• AI & Tooling functions as a center of excellence for generative AI evaluation and tooling practices across Digital Success and the broader Salesforce Customer Zero ecosystem

When you join Salesforce, you'll be limitless in all areas of your life. Our benefits and resources support you to find balance and be your best, and our AI agents accelerate your impact so you can do your best. Together, we'll bring the power of Agentforce to organizations of all sizes and deliver amazing experiences that customers love.

If you need a reasonable accommodation during the application or the recruiting process, please submit a request via this Accommodations Request Form.

Salesforce is an equal opportunity employer and maintains a policy of non-discrimination with all employees and applicants for employment. What does that mean exactly? It means that at Salesforce, we believe in equality for all. And we believe we can lead the path to equality in part by creating a workplace that's inclusive, and free from discrimination. Any employee or potential employee will be assessed on the basis of merit, competence and qualifications - without regard to race, religion, color, national origin, sex, sexual orientation, gender expression or identity, transgender status, age, disability, veteran or marital status, political viewpoint, or other classifications protected by law. This policy applies to current and prospective employees, no matter where they are in their Salesforce employment journey. It also applies to recruiting, hiring, job assignment, compensation, promotion, benefits, training, assessment of job performance, discipline, termination, and everything in between. Recruiting, hiring, and promotion decisions at Salesforce are fair and based on merit. The same goes for compensation, benefits, promotions, transfers, reduction in workforce, recall, training, and education.

In the United States, compensation offered will be determined by factors such as location, job level, job-related knowledge, skills, and experience. Certain roles may be eligible for incentive compensation, equity, and benefits. Salesforce offers a variety of benefits to help you live well including: time off programs, medical, dental, vision, mental health support, paid parental leave, life and disability insurance, 401(k), and an employee stock purchasing program. More details about company benefits can be found at https://www.salesforcebenefits.com. At Salesforce, we believe in equitable compensation practices that reflect the dynamic nature of labor markets across various regions. The typical base salary range for this position is $164,000 - $261,500 annually. The range represents base salary only, and does not include company bonus, incentive for sales roles, equity or benefits, as applicable.