On Call Annotation Specialist Jobs (NOW HIRING)

The Risk & Payments Operations team empowers customers to seamlessly pay and be paid across Meta products by delivering, managing, and optimizing Payments support. As part of Meta's AI transformation of customer support, we are shifting to an AI + Specialized Human Workforce model—where AI handles resolution at scale and human experts drive the strategy, measurement, and continuous improvement behind it.As a Support Design Specialist, you will be the driving force behind AI-led support implementation for Risk & Payments. You will design, build, and deploy AI-powered resolution workflows and quality measurement systems—from LLM judges that evaluate case outcomes at scale, to AI triage agents that automate oncall operations, to conversational AI experiences that resolve customer issues end-to-end. You will combine deep payments domain expertise with hands-on AI tool building to transform how support is delivered.
Support Design Specialist, Business Systems Responsibilities:

• Design and deploy AI first resolution workflows that resolve customer issues at or above human-level quality, defining resolution logic, process design, and signal integration
• Build and operationalize LLM judges for quality measurement, designing evaluation protocols, calibrating against expert reviews, and replacing manual audits with scalable, consistent AI-based measurements
• Perform case labeling and annotation activities to train AI-led support routines, ensuring accurate resolution logic and establishing ground truth for quality measurement
• Define and monitor performance frameworks for AI systems: precision/recall targets, resolution rates, conversation quality guardrails, and bias thresholds to ensure that AI maintains or exceeds human-level outcomes
• Serve as a domain expert and escalation point for complex risk and payment cases, using resolution patterns to continuously improve AI capabilities and identify automation opportunities
• Partner cross-functionally with Product, Engineering, and Data teams to scope experiments, define success criteria, integrate new signals, and scale proven support solutions
• Develop AI-powered operational tools to automate and augment the teams' process and establish replicable playbooks for AI-native workflows, enabling workforce transformation for Risk & Payments

Minimum Qualifications:

• Currently has, or is in the process of obtaining a Bachelor's degree in a directly related field, or equivalent practical experience. Degree must be completed prior to joining Meta
• 3+ years in operations, customer support, or process optimization within a technology environment
• Hands-on experience with AI platforms, prompt engineering, or low-code/no-code AI tooling
• Demonstrated experience in building or deploying AI/LLM-powered tools, automation workflows, or AI agents in an operational setting
• Strong analytical skills with the ability to design evaluation frameworks and translate performance data into system improvements
• Track record of driving measurable operational improvements (efficiency gains, cost savings, quality improvements)
• Demonstrated success in communicating and collaborating with cross-functional partners, including engineering and technical teams
• Willingness to learn emerging technologies and adapt to rapid change

Preferred Qualifications:

• Demonstrated ability to integrate AI tools to optimize/redesign workflows and drive measurable impact (e.g., efficiency gains, quality improvements)
• 3+ years of experience in customer support for payments, risk, trust and safety, or related domains
• Experience with data analysis tools and SQL
• Experience leading AI adoption within non-engineering teams—teaching, enabling, and scaling AI-native practices
• Background in designing conversational AI experiences or chatbot workflows
• Experience adhering to and implementing responsible, ethical AI practices (e.g., risk assessment, bias mitigation, quality and accuracy reviews)
• Demonstrated ongoing AI skill development (e.g., prompt/context engineering, agent orchestration) and staying current with emerging AI technologies
• BA/BS degree

About Meta:
Meta builds technologies that help people connect, find communities, and grow businesses. When Facebook launched in 2004, it changed the way people connect. Apps like Messenger, Instagram and WhatsApp further empowered billions around the world. Now, Meta is moving beyond 2D screens toward immersive experiences like augmented and virtual reality to help build the next evolution in social technology. People who choose to build their careers by building with us at Meta help shape a future that will take us beyond what digital connection makes possible today—beyond the constraints of screens, the limits of distance, and even the rules of physics.