Contractual Observability Engineer Jobs (NOW HIRING)

Ready to be a Titan?

ServiceTitan runs the businesses behind the trades: jobs, trucks, technicians, equipment, contracts, payments, warranties, compliance obligations, and customer history. That operational context is our advantage. We are building Agent OS to turn that context into safe, observable, production-grade agent work.

Agent OS is the shared runtime, context, memory, action, trust, and evaluation layer behind role-specific AI experiences across Atlas, office, field, voice, mobile, and future product surfaces. This is not a collection of chatbots. It is the platform that lets agents help contractors run their businesses with the right evidence, permissions, approvals, and audit trails.

You will help build the core engineering primitives behind that platform: agent runtime, typed tools, context and memory assembly, trust and approval flows, evaluation infrastructure, and production observability. You are not building one agent for one product surface. You are building the platform that product teams use to build many agents safely.

You will work on a small, senior AI platform team and partner closely with Product, Architecture, Security, Data Platform, Atlas, and domain engineering teams.

What You'll Build

• Agent runtime and workflow execution: Build the runtime for role-specific agents, tool use, delegation, pause/resume, durable checkpoints, retries, and failure recovery. Agents must resume safely without losing state or duplicating side effects.

• Typed tools and action contracts: Build deterministic controls around non-deterministic reasoning: governed reads, proposed writes, precondition checks, business invariants, scoped permissions, idempotency, audit trails, and rollback.

• Context and memory systems: Build tenant-scoped context assembly, retrieval, freshness controls, provenance, transcripts, artifacts, tool results, and replayable evidence. ServiceTitan systems of record stay authoritative; memory provides context and coordination.

• Trust and approval infrastructure: Build human-in-the-loop gates, approval thresholds, reversibility, tenant policy enforcement, and audit history for financial, contractual, dispatch, warranty, and compliance-sensitive workflows.

• Evaluation and observability: Build offline and online evals, scenario libraries, simulation, trajectory review, regression detection, cost and latency telemetry, and autonomy promotion gates.

• Reusable capability platform: Help product teams package prompts, tools, context requirements, policies, evals, rollout controls, ownership, and rollback into governed capabilities for owners, CSRs, dispatchers, technicians, managers, and back-office teams.

• Model and inference architecture: Make practical tradeoffs across latency, cost, quality, structured outputs, caching, fallback behavior, provider choice, and model routing behind a shared platform layer.

What You'll Do

• Design and implement core Agent OS platform services.

• Write production code and review implementation details from other engineers.

• Build reliable APIs, workflows, tools, and services for agent execution.

• Inspect traces, debug failures, and improve production behavior.

• Design evaluation scenarios and regression suites for agent workflows.

• Work through real agent failure modes: stale context, wrong tool calls, missing permissions, unsafe actions, poor retrieval, latency spikes, and cost regressions.

• Partner with domain teams to turn agent use cases into reusable platform patterns.

• Help define platform contracts for tools, actions, approvals, context, memory, evidence, and evaluation.

• Contribute to technical direction while staying grounded in what can ship quickly and safely.

• Communicate clearly with engineers, product managers, architects, security partners, and engineering leadership.

What You'll Bring

• 5+ years of production software engineering experience.

• Strong hands-on coding ability in Python, Java, C#, or another backend language. Python experience is strongly preferred.

• Experience building AI, ML, data, platform, infrastructure, workflow, automation, or developer-platform systems in production.

• Practical understanding of modern LLM application architecture: model gateways, prompt and context assembly, retrieval, tool calling, structured outputs, memory, agent workflows, and human approval patterns.

• Experience with distributed systems, event-driven systems, async workflows, queues, durable execution, or message-driven architectures.

• Strong production-safety instincts for non-deterministic systems: typed contracts, scoped permissions, precondition checks, idempotency, audit trails, rollback, and monitoring.

• Experience designing or operating evaluation systems: behavioral evals, regression suites, scenario tests, trajectory review, simulation, online metrics, or production monitoring.

• Strong data and context instincts: SQL, unstructured data, vector search, metadata, provenance, freshness, source authority, and privacy boundaries.

• Experience with databases, warehouses, or search systems such as PostgreSQL, SQL Server, Snowflake, BigQuery, Elasticsearch, or vector stores.

• Experience building services on public cloud infrastructure such as Azure, AWS, or GCP.

• Good engineering judgment across APIs, reliability, security, observability, and multi-tenant SaaS constraints.

Bonus points

• Experience building or operating agent runtimes, workflow engines, model gateways, ML platforms, evaluation platforms, developer platforms, or internal control planes.

• Experience with LangGraph, LangChain, LlamaIndex, Semantic Kernel, OpenAI Agents SDK, Anthropic tooling, or similar frameworks.

• Experience with MCP, A2A, tool protocols, agent interoperability, or agent-commerce patterns.

• Experience with Kubernetes, Docker, serverless platforms, or cloud-native infrastructure.

• Experience with compliance-sensitive workflows, approval-gated automation, audit trails, policy engines, or governed writes to systems of record.

• Experience in SaaS, vertical software, fintech, ERP, CRM, marketplace, field service, or other domains where software decisions affect real business operations.

• Experience with graph-based data models, knowledge graphs, entity resolution, or cross-domain operational context systems.

Why this role matters

Most AI products fail when the demo becomes a production workflow. The hard problems show up in the platform: context freshness, tool reliability, permissions, evaluation, traceability, rollback, and trust.

That is what this team is building.

At ServiceTitan, agents need to work inside real contractor operations. They need to understand the job, the customer, the technician, the equipment, the agreement, the invoice, the warranty, and the business policy. They need to explain what evidence they used. They need to know when to ask for approval. They need to recover when something fails.

The engineer in this role helps set the technical standard for every AI surface ServiceTitan builds next. This is a high-leverage engineering role for someone who wants to build the platform underneath production agents, not just another agent demo.
Remote Location (US and Canada only)- Candidates based in PST highly preferred.

Be Human With Us:
Being human isn't about checking every box on a list. It's about the experiences we have, people we meet, and the perspectives we share. So, if you have the skills but are hesitant to apply because of your background, apply anyway. We need amazing people like you to help us challenge the conventional and think differently about the problems that we're solving. We're in this together. Come be human, with us.

Use of AI Technology:

We use technology, including automated and AI-assisted tools, to support certain aspects of our recruitment process. These tools are designed to improve efficiency and enhance the candidate experience. AI tools are not used to make hiring decisions; all hiring decisions are made by our hiring teams.

What We Offer:

When you join our team, you're not just accepting a job. You're making a career move. Here's how we'll support you in doing some of the most impactful work of your career:

• Flextime, recognition, and support for autonomous work: Flexible time off with ample learning and development opportunities to continue growing your career. We offer a comprehensive onboarding program, leadership training for Titans at all levels, and other programs and events. Great work is rewarded through Bonusly, peer-nominated awards, and more.
• Holistic health and wellness benefits: Company-paid medical, dental, and vision (with 100% employer paid options and 90% coverage for dependents), FSA and HSA, 401k match, and telehealth options including memberships to One Medical.
• Support for Titans at all stages of life: Parental leave and support, up to $20k in fertility services (i.e. IUI and IVF), surrogacy, and adoption reimbursement, on demand maternity support through Maven Maternity, free breast milk shipping through Maven Milk, pet insurance, legal advisory services, financial planning tools, and more.

At ServiceTitan, we celebrate individuality and uniqueness. We believe that the convergence of fresh perspectives and experiences from all walks of life is what makes our product and culture so great. We strongly encourage people from underrepresented groups to apply. We do not discriminate against employees based on race, color, religion, sex, national origin, gender identity or expression, age, disability, pregnancy (including childbirth, breastfeeding, or related medical condition), genetic information, protected military or veteran status, sexual orientation, or any other characteristic protected by applicable federal, state or local laws.

ServiceTitan is committed to fair and equitable compensation for all of our employees. We thoughtfully consider a wide range of factors when determining individual compensation, which may change over time. We comply with all applicable minimum wage laws.For candidates in the United States, the good faith salary ranges estimate for this role is Zone 1: $179,900 USD - $269,900 USD Applicable for: CA, CT, DC, MD, MA, NJ, NY, VA, and WA Zone 2: $168,200 USD - $252,200 USD Applicable for: All other US locations. International Compensation for candidates residing outside the United States will vary by location and will be discussed during the hiring process. Actual compensation within a range is determined by factors including relevant experience, skill set, qualifications, and performance. In addition to base salary, our total compensation package includes an annual bonus, equity, and a holistic suite of benefits.