Online User Testing Jobs in Colorado (NOW HIRING)

Launched in 2021, Fanatics Betting and Gaming is the online and retail sports betting subsidiary of Fanatics, a global digital sports platform. The Fanatics Sportsbook is available to 95% of the addressable online sports bettor market in the U.S. Fanatics Casino is currently available online in Michigan, New Jersey, Pennsylvania and West Virginia. Fanatics Betting and Gaming operates twenty-two retail sports betting locations, including the only sportsbook inside an NFL stadium at Northwest Stadium. Fanatics Betting and Gaming is headquartered in New York with offices in Denver, Leeds and Dublin.

• Own the end-to-end performance of FBG's GenAI customer service AI Agent, driving quality, escalation logic, and containment metrics through partnership with Engineering and Product.
• Drive continuous improvement across containment rate, resolution rate, retention rate, and user satisfaction with a data-first approach to prioritization.
• Define and refine conversational flows, intent coverage, fallback handling, and multi-turn experiences in close partnership with Engineering and Product.
• Ensure chatbot responses are accurate, policy-compliant, brand-aligned, and safe, including guardrails for hallucination prevention and confident human handoff thresholds.
• Own knowledge article quality and freshness, including intake workflows and content evaluation processes.

• Own the insights layer for AI Agent performance - using Claude and other AI-assisted tools to query data, investigate root causes, and synthesize patterns across large volumes of conversation and performance data.
• Identify opportunities and surface risks by synthesizing signals from conversation data, user behavior, agent feedback, and operational metrics.
• Translate complex findings into clear, decision-ready narratives for both technical and executive stakeholders.
• Partner with Data Engineering and Analytics teams to ensure the underlying data is reliable and well-organized enough to support fast, accurate AI-assisted analysis.

• Monitor AI Agent performance daily; identify regressions, anomalies, and edge cases before they impact customers.
• Support and help maintain QA frameworks, evaluation datasets, and testing protocols for the AI Agent.
• Own incident response for AI Agent-related issues, including root-cause investigation, corrective actions, and post-mortems.
• Establish and evolve governance processes for model updates, prompt changes, content refreshes, and release readiness.

• Partner with Engineering, Product, Customer Support, Compliance, and Legal to coordinate delivery across the AI Agent roadmap.
• Ensure the AI Agent meets regulatory requirements across gaming, payments, and data privacy, working closely with Compliance and Responsible Gaming teams.
• Communicate status, risks, and outcomes clearly to senior leadership, tailoring messaging to both technical and executive audiences.
• Drive internal enablement and documentation so operational and support teams can effectively work alongside the AI Agent.

• Manage a small, dedicated team responsible for the creation and ongoing management of groundings (knowledge articles, retrieval content, and prompt context) for the AI Agent.
• Lead the team's testing and quality assurance function, including evaluation dataset development, regression testing, golden-answer authoring, and pre/post-release validation.
• Set goals, prioritize the team's roadmap, and develop team members through coaching, feedback, and clear performance expectations.
• Build scalable processes for grounding intake, refresh cadence, and quality scoring so coverage keeps pace with new products, promotions, and regulatory changes.

• Strong analytical instincts and comfort working with data - you don't need to write SQL yourself, but you know how to ask the right questions of data, interpret results, and use tools (including AI assistants like Claude) to get answers quickly.
• Working knowledge of how AI chatbots/LLMs operate conceptually (prompts, knowledge grounding, escalation logic) - enough to have informed conversations with engineers, not to build the system yourself.
• Experience operating a product or system in production, including monitoring performance, spotting regressions, and driving structured, data-informed iteration.
• Proven ability to work cross-functionally across engineering, operations, and compliance in a fast-moving environment.
• Excellent written and verbal communication skills, with the ability to distill complex performance data into clear narratives for both technical and executive audiences.
• Prior people management experience, with a track record of building and developing small teams.

• Experience owning or managing a GenAI-powered chatbot, virtual agent, or similar customer-facing AI product.
• Experience in regulated industries such as online gaming, fintech, payments, or e-commerce.
• Familiarity with SQL or data platforms (Snowflake, BigQuery) is a plus, not a requirement.
• Experience managing teams responsible for knowledge management, content operations, or AI evaluation/QA.

Depending on the role, your interview and onboarding experience may include in-person components, such as onsite interviews or Launching into Better: LIVE-a multi-day cultural immersion in New York City for full-time, non-seasonal hires. These sessions are designed to build connection and bring our culture to life, though specific travel and participation requirements will be confirmed based on your role and location. Your recruiter will provide clear guidance at each stage of the process.

For information about our benefits, please visit https://benefitsatfanatics.com/