Ai Rater Jobs in Draper, UT (NOW HIRING)

RESPONSIBILITIES:
Kforce has a client in SLC that is seeking an AI/Agent Developer & Tester to design, build, and validate LangGraph-based agent workflows within the APEX platform.
Summary:
This role is highly execution-focused, with ownership of testing strategy, quality, and reliability across AI agent systems-from unit-level validation through end-to-end workflows and production monitoring. The AI Agent Developer and Tester will be responsible for ensuring agent accuracy, stability, and scalability in a non-deterministic LLM environment, while also driving comprehensive test coverage across backend workflows and UI experiences.
Responsibilities:
* Develop and implement AI agent workflows using LangGraph / LangChain
* Own end-to-end testing strategy, including:
* Node-level unit testing (isolated components with mocked state)
* Graph-level integration testing (full workflow execution)
* Regression test suite design and maintenance
* Establish LLM testing frameworks, including mocking, stubbing, and deterministic validation strategies for non-deterministic systems
* Manage flaky test mitigation, applying controlled seeds, mocked outputs, or assertion thresholds
* Own regression discipline-ensure every defect is paired with a regression test before release
* Build and maintain E2E test coverage using Playwright across APEX portals, actively identifying and closing coverage gaps
* Implement test coverage reporting and enforce quality standards
* Validate structured outputs (JSON schema enforcement)
* Ensure production observability of agent behavior and performance
* Support and manage cloud infrastructure (Google Cloud Platform preferred) for the agent ecosystem
REQUIREMENTS:
* Hands-on experience with LangGraph and/or LangChain
* Strong Python development skills, including pytest and pytest-asyncio
* Proven experience testing AI/LLM-driven systems, including: Unit testing with mocked state; End-to-end workflow validation
* Experience implementing LLM mocking strategies to control non-deterministic outputs
* Experience with Playwright or similar tools for E2E UI testing
* Familiarity with structured data validation (JSON schema)
* Experience working in cloud environments (Google Cloud Platform preferred, others acceptable)
* Experience with LangSmith, RAGAS, or similar evaluation tools
* Prompt versioning and lifecycle management
* Experience with adversarial/red-team testing of LLM systems
* Experience with token usage and cost instrumentation
* Experience with document processing or OCR pipelines
* Background in financial services or regulated environments
Experience with:
* Google Cloud Platform (Cloud Run)
* Terraform
* Azure DevOps pipelines
The pay range is the lowest to highest compensation we reasonably in good faith believe we would pay at posting for this role. We may ultimately pay more or less than this range. Employee pay is based on factors like relevant education, qualifications, certifications, experience, skills, seniority, location, performance, union contract and business needs. This range may be modified in the future.
We offer comprehensive benefits including medical/dental/vision insurance, HSA, FSA, 401(k), and life, disability & ADD insurance to eligible employees. Salaried personnel receive paid time off. Hourly employees are not eligible for paid time off unless required by law. Hourly employees on a Service Contract Act project are eligible for paid sick leave.
Note: Pay is not considered compensation until it is earned, vested and determinable. The amount and availability of any compensation remains in Kforce's sole discretion unless and until paid and may be modified in its discretion consistent with the law.
This job is not eligible for bonuses, incentives or commissions.
Kforce is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.
By clicking ?Apply Today? you agree to receive calls, AI-generated calls, text messages or emails from Kforce and its affiliates, and service providers. Note that if you choose to communicate with Kforce via text messaging the frequency may vary, and message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You will always have the right to cease communicating via text by using key words such as STOP.