Qa Engineer Jobs in Boston, MA (NOW HIRING)

Job Summary:
Third Way Health is dedicated to improving patient experiences in medical practices across the United States through innovative service automation. The Senior QA Engineer will be responsible for developing and maintaining the quality and evaluation infrastructure for the company's AI-powered patient engagement platform, ensuring high standards of quality across various interactions.
Responsibilities:
• Own and extend our multi-layered eval pipeline and verification portfolio: deterministic quality checks on tool calls, risk-factor heuristics, and LLM-graded transcript evaluation.
• Advance our capabilities to evaluate end-to-end system performance (across orchestrated agents, RAG-supported responses, multi-party voice conversations) with modular and auditable verification that is independent of any single model provider.
• Drive improvements to our observability stack to surface eval metrics, detect regressions, and enable data-driven quality decisions across the team.
• Build real-time monitoring and verification loops that catch issues in production interactions as they happen, intervening with context and feeding back for system refinement.
• Partner with ML engineers, product managers, and operations leads to translate real-world failure modes into automated checks, closing the loop between production incidents and eval coverage.
• Build and maintain adversarial and edge-case test suites — including prompt injection resistance, guardrail robustness, and graceful degradation under ambiguous patient inputs.
• Champion 'shift-left' quality practices: embed eval criteria into prompt engineering workflows, define acceptance criteria for new agent behaviors, and make quality a first-class concern in the development cycle.
• Contribute to the design of our QA pipeline orchestration (background processing, Slack notifications, risk assessment persistence) to improve throughput, reliability, and developer experience.
Qualifications:
Required:
• 5+ years of software engineering or test engineering experience, with 3+ years focused on quality infrastructure for AI/ML or data-intensive systems.
• Strong proficiency in Python, particularly for building test frameworks, eval pipelines, and API-level integration tests (e.g., pytest, FastAPI TestClient, Pydantic).
• Demonstrated experience designing evaluation or verification systems for LLM-based applications — with a clear understanding that the model is a generation layer, not the quality layer. Comfort with both deterministic and model-graded assessment approaches, and a point of view on when each is appropriate.
• Familiarity with the architectural tradeoffs of relying on LLM outputs in production — including variance across model versions, prompt sensitivity, and the need for external verification infrastructure that remains stable as underlying models change.
• Experience building extensible, rule-based validation systems (check registries, plugin architectures, or similar patterns) that scale across a growing surface area of features.
• Solid understanding of voice AI or conversational AI systems, including tool-calling patterns, transcript analysis, and interaction-level quality metrics.
• Hands-on experience with observability and metrics instrumentation in production environments.
• Excellent communication skills, with the ability to collaborate effectively across engineering, product, and non-technical stakeholders.
• Strong interest in healthcare innovation and building AI systems that meaningfully improve health outcomes.
Preferred:
• Experience building QA or eval systems in healthcare or regulated environments, with familiarity with standards such as HIPAA, GDPR, or FDA guidance.
• Proven experience leading complex technical initiatives and mentoring junior engineers.
• Experience building or operating systems where quality guarantees live in the verification infrastructure rather than in any single model.
• Familiarity with risk-scoring systems, anomaly detection, or production safety nets for autonomous AI agents.
• Experience with AI safety testing, including adversarial evaluation, jailbreak testing, and bias detection in LLM outputs.
• Hands-on experience with CI/CD pipelines for eval automation (CircleCI, GitHub Actions, or equivalent) and infrastructure-as-code deployment patterns.
• Experience with voice UI testing tools and platforms, with a focus on evaluating speech generation and response quality.
• Knowledge of accessibility testing and inclusive design principles.
Company:
Third Way Health is a healthcare operations partner that provides human-centric solutions to support administrative functions. Founded in 2022, the company is headquartered in Los Angeles, USA, with a team of 501-1000 employees. The company is currently Growth Stage.