Search Quality Rater Jobs in Bethesda, MD (NOW HIRING)

Job Title: AI QA Analyst Location: 100% Remote Type: Long-Term Contract Compensation: Negotiable, W-2 or C2C Work Model: Remote – offsite Hours: 40 hours a week Security Clearance: No clearance but must pass background check which will include credit check

Ideal Candidate Profile A highly technical analyst who understands both modern software engineering and AI systems. This individual can independently assess whether an AI solution is accurate, secure, explainable, compliant, observable, and ready for production. They serve as the organization's independent quality authority for AI-powered solutions and help establish confidence in enterprise AI deployments before release to production.

Responsibilities • Establish and execute quality, validation, certification, and production readiness processes for AI-powered SDLC and enterprise automation solutions • Develop and implement evaluation frameworks measuring accuracy, relevance, groundedness, completeness, hallucination rates, retrieval quality, recommendation quality, and user satisfaction • Validate Retrieval-Augmented Generation (RAG) solutions utilizing Azure AI Search, Azure AI Foundry, LangChain, LangGraph, and enterprise repositories • Perform architecture reviews and quality assessments for AI solutions deployed on Azure services such as Containers, Functions, Databricks, SQL, Cosmos DB • Evaluate AI orchestration workflows including prompt execution, agent routing, tool calling, guardrails, retrieval pipelines, and human-in-the-loop processes • Validate security, governance, auditability, and compliance controls, including integrations with Entra ID, RBAC, managed identities, Key Vault, and data policies • Create production readiness checklists covering observability, monitoring, resiliency, recoverability, and operational support • Analyze traces, logs, metrics, and telemetry data to identify quality concerns and improvement opportunities • Collaborate with development teams for remediation of quality, security, and performance issues before deployment • Generate certification reports, scorecards, dashboards, and executive summaries for leadership review • Drive continuous improvements to validation methodologies, testing strategies, and evaluation frameworks • Lead certification and production readiness reviews for internal developer portals and platform initiatives

Requirements • Bachelor’s Degree in Computer Science, Software Engineering, Data Science, AI, or related field • 5+ years of experience in software quality engineering, platform engineering, software development, or related roles • 2+ years working directly with Generative AI, LLMs, RAG architectures, AI agents, or AI applications • Strong understanding of AI evaluation methodologies including accuracy testing and hallucination detection • Experience with Azure AI Foundry, Azure OpenAI, LangChain, LangGraph, LangSmith, or similar frameworks • Familiarity with cloud-native architectures and Azure services (Container Apps, Functions, Databricks, SQL, Key Vault, Cosmos DB) • Experience supporting enterprise developer platforms and self-service engineering capabilities • Skilled in designing automated test strategies, regression testing, and certification processes • Strong analytical, troubleshooting, and documentation skills • Excellent stakeholder engagement and communication abilities

Preferred Qualifications • Experience building or validating enterprise AI agents and multi-agent systems • Knowledge of AI observability and evaluation platforms like LangSmith • Familiarity with knowledge retrieval solutions, embeddings, and semantic search • Background in AI governance, Responsible AI, and compliance frameworks • Experience working in regulated industries such as healthcare, finance, or insurance • Proficiency with Azure DevOps, GitHub, and SDLC tooling • Knowledge of identity management concepts including Entra ID and RBAC • Ability to produce quality scorecards, dashboards, and KPI reports

System One, and its subsidiaries including Joulé and Mountain Ltd., are leaders in delivering outsourced services and workforce solutions across North America. We help clients get work done more efficiently and economically, without compromising quality. System One not only serves as a valued partner for our clients, but we offer eligible employees health and welfare benefits coverage options including medical, dental, vision, spending accounts, life insurance, voluntary plans, as well as participation in a 401(k) plan.

System One is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, age, national origin, disability, family care or medical leave status, genetic information, veteran status, marital status, or any other characteristic protected by applicable federal, state, or local law.

#M1 #LI-CB3 Ref: #851-Rockville-S1