Forward Deployed Ai Engineer Jobs in Draper, UT (NOW HIRING)

O.C. Tanner is the global leader in software and services that improve workplace culture through meaningful employee experiences. Our Culture Cloud is a suite of apps designed to enhance the employee experience with strategic recognition, service awards, wellbeing, leadership, and events that help people thrive at work. Our Culture by Design approach provides expert services to organizations looking to create great workplaces.

Our global team of 1,500 people hail from 58 countries and speak 62 languages. As programmers, researchers, designers, client professionals and craftspeople we create the tech, tools and awards that connect employees to purpose at thousands of companies. Join us as we help people all over the world thrive at work.

About the Role

AI is becoming part of the product and platform architecture we need to build, operate, and scale. We are looking for an Applied AI Engineer who can turn AI capability into secure, measurable, governed production systems, not prototypes or demos. This person will help define how O.C. Tanner builds agentic systems that pursue goals, use tools, follow guardrails, recover from failure, and deliver real value inside user workflows.

This role sits at the intersection of software engineering, product experience, AI platform engineering, and responsible AI. You will partner with Product, UX, Design, Architecture, Security, and Engineering to build AI experiences that are useful, understandable, reliable, and safe to operate in production. The right person has hands-on experience building agentic systems with orchestration, tool calling, memory or state, RAG, evaluation, observability, and human-in-the-loop controls.

Responsibilities

• Design, build, deploy, and support production-grade agentic AI systems that operate against explicit goals, constraints, policies, and guardrails.
• Build agent orchestration patterns for multi-step workflows, tool calling, MCP servers, state management, memory, retries, recovery paths, and human-in-the-loop controls.
• Partner closely with Product, UX, Design, Architecture, Security, and Engineering teams to create AI experiences that are useful, understandable, reliable, and aligned with real user workflows.
• Design user-centered AI interactions, including conversational flows, feedback loops, confidence handling, explainability, graceful failure modes, escalation paths, and clear boundaries for autonomous behavior.
• Develop and operate RAG systems that ground model behavior in enterprise knowledge, including ingestion, chunking, embeddings, vector and hybrid retrieval, reranking, retrieval evaluation, and citation or traceability strategies.
• Define and implement evaluation frameworks for AI systems, including offline test sets, regression suites, adversarial testing, groundedness and faithfulness scoring, task completion metrics, and production quality monitoring.
• Instrument agentic systems for observability, including traces of model calls, prompts, tool usage, decisions, retrieved context, latency, cost, errors, and user feedback.
• Establish safeguards for responsible AI use, including prompt injection defense, data access controls, PII protection, bias and toxicity detection, misuse prevention, audit logging, and policy enforcement.
• Optimize model selection, prompts, context windows, caching, routing, inference patterns, latency, throughput, reliability, and cost across production workloads.
• Mentor engineers on applied AI practices, including prompt and context engineering, agent design, RAG, evaluation, safety, observability, and production support.
• Stay current with emerging AI platforms, frameworks, models, and standards.

Our stack

• Python / FastAPI microservices
• LangChain / LangGraph
• GraphQL / REST
• PostgreSQL / Redis
• Kafka
• Kubernetes
• AWS Bedrock
• OpenTelemetry
• Terraform

Required Qualifications

• 5+ years of software engineering experience with strong Python proficiency
• 2+ years building production ML or agentic AI systems
• 1+ years hands-on experience with agentic frameworks (LangGraph, CrewAI, AutoGen, or equivalent)
• Built production AI systems including agents, MCP servers, multi-step reasoning, and multi-turn conversation
• Deployed RAG systems including embedding models, vector databases, hybrid search, and retrieval optimization
• Designed LLM strategies covering tool calling, structured outputs, prompt engineering, and context window management
• Implemented AI safety and evaluation pipelines covering bias detection, PII leakage, faithfulness scoring, toxicity, and prompt injection mitigation
• Optimized models for inference efficiency, latency, and cost management

Strongly Preferred

• Bachelor's degree in Computer Science, Machine Learning, or a related field
• AWS Certified Machine Learning Engineer - Associate or equivalent
• Cloud AI infrastructure management using AWS services and Terraform
• AI observability experience with OpenTelemetry, Langfuse, or equivalent