Director Observability Jobs in Madison, WI (NOW HIRING)

This position provides strategic and technical leadership for the enterprise AI platform engineering function, setting the technology strategy and reference architecture for how AI is built, deployed, and scaled enterprise-wide. The role is accountable for building and operating shared AI engineering capabilities that enable teams to develop, deploy, monitor, evaluate, and scale AI solutions safely and efficiently. As Director, you will lead teams responsible for AI platform architecture, reusable engineering frameworks, MLOps/LLMOps, AI operations, agentic workflow patterns, LLM pipeline frameworks, observability, evaluation controls, production support patterns, systems-of-record integration, cost optimization, and technical guardrails that support responsible and scalable AI adoption. This role works across Technology and business domains, including Information Security, Governance, Enterprise Architecture, Application Development, Digital Services, Infrastructure, Data Engineering, Product, Operations, Risk, Legal, Compliance, and strategic technology partners. You will also manageor coordinates key AI platform partners, including GCP, AWS, DataDog, ServiceNow, Salesforce and more where applicable.

Position Compensation Range:

$172,000.00 - $294,000.00

Pay Rate Type:

Salary

Compensation may vary based on the job level and your geographic work location. Relocation support is offered for eligible candidates.

Primary Accountabilities

• Lead AI platform architecture and strategy
  • You will define the architecture, standards, and roadmap for shared enterprise AI platform capabilities. This is collaborative with Data & AI Architects.
  • You will also ensure the platform supports scalable, secure, reliable, and governed AI delivery across multiple business domains.
• Balance AI operations, MLOps/LLMOps, and agentic frameworks
  • Lead engineering practices for model and AI-system deployment, monitoring, testing, evaluation, versioning, reliability, and lifecycle management.
  • Balance operational reliability with reusable agentic workflow patterns and LLM pipeline frameworks.
  • Ensure AI systems can be supported and improved after production deployment.
• Contribute to the enterprise AI technology maturity view
  • You will contribute platform, engineering, operations, observability, support, resilience, cost, systems integration, and production-readiness inputs to the enterprise-level shared AI technology maturity view.
  • You will use this maturity view to identify capability gaps, guide investment recommendations, and communicate platform and engineering readiness.
• Build reusable engineering frameworks and capabilities
  • Build and maintain reusable frameworks, components, and patterns that accelerate AI delivery and reduce duplicated engineering effort.
  • Ensure durable reusable capabilities are documented, discoverable, supportable, and governed so they can be leveraged across multiple domains.
• Own platform production support and operational run patterns
  • You will directly own production support for the AI platform and shared AI engineering capabilities, especially L1, L2, and the engineering side of L3 support
  • Establish production run patterns for AI-enabled workflows, including support models, incident paths, escalation patterns, fallback mechanisms, and human handoff design.
  • Partner with customer service, employee support, application support, service management, or other operational channels where AI experiences require context-rich support transitions.
• Establish observability, evaluation, and monitoring capabilities
  • Establish platform capabilities for telemetry, model and system monitoring, traceability, drift or quality signals, evaluation controls, regression checks, and operational visibility.
  • Provide visibility into AI system behavior, performance, usage, and risk signals.
• Embed responsible AI technical guardrails
  • Embed responsible AI technical controls into platform capabilities in alignment with shared enterprise governance expectations.
  • Ensure platform capabilities support auditability, policy adherence, access controls, least-privilege operation, model/agent registration, and responsible AI practices.
• Lead AI cost, capacity, and resilience practices
  • Provide engineering mechanisms for token strategy, usage monitoring, multi-model routing, fallback, caching, capacity planning, inference or compute optimization, and total-cost-of-ownership discipline.
  • Help the enterprise balance speed, performance, reliability, resilience, and cost.
• Support build/buy/partner technical decisions
  • Lead build/buy/partner decisions for AI engineering capabilities, including when to use partner-native agents, managed AI platforms, or internally built orchestration based on data location, control needs, governance, speed, cost, portability, and strategic differentiation.
  • Manage and coordinate key AI Platform partners, including GCP, AWS, DataDog, ServiceNow, and others where applicable.
• Lead people and develop engineering talent
  • Lead teams responsible for MLOps, LLMOps, AI operations, platform engineering, GIS or other assigned platform capabilities, and related AI engineering functions.
  • Build engineering discipline, technical depth, delivery accountability, and collaborative execution across teams, while recognizing the AI space is evolving quickly and required skills will continue to evolve.

Specialized Knowledge & Skills Requirements

• Demonstrated experience leading engineering teams that build production platforms, internal developer platforms, MLOps/LLMOps capabilities, AI operations, or scalable AI/ML systems.
• Experience with AI system architecture, model deployment, agent deployment, monitoring, observability, evaluation, production support, and lifecycle management.
• Demonstrated ability to balance operational reliability with emerging agentic workflow and LLM pipeline frameworks.
• Experience creating reusable frameworks, standards, and platform capabilities that improve delivery across multiple teams.
• Familiarity with GenAI, agentic workflows, LLM pipelines, model orchestration, retrieval-augmented generation patterns, model gateways, systems-of-record integration, and emerging AI platform patterns.
• Demonstrated experience with production support models, including L1/L2 support expectations and engineering-side L3 support.
• Experience with cost, capacity, resilience, usage monitoring, routing, fallback, caching, or FinOps practices for cloud or AI workloads.
• Experience working across Information Security, Enterprise Architecture, Infrastructure, Cloud, Application Development, Digital Services, Data Engineering, Governance, Legal, Risk, Compliance, and business domains.
• Experience managing or coordinating partners such as GCP, AWS, DataDog, or related technology vendors.
• Demonstrated people leadership, technical coaching, prioritization, and talent development skills.

Key Interfaces and Partners

• Applied AI for solution delivery needs, reusable patterns, production enablement, applied feedback loops, L3 enhancement partnership, and solution handoff.
• BI Engineering & Enablement for metric, semantic, metadata, lineage, and knowledge-layer dependencies that support AI workflows.
• Data Engineering for data pipelines, data products, environment dependencies, data availability, and data readiness.
• Information Security, Enterprise Architecture, Infrastructure, Cloud, SRE, Application Development, Digital Services, Privacy, Legal, Compliance, Model Risk, AI Governance, Data Governance, and Procurement/TPRO.
• Product and business-domain teams using AI platform capabilities.
• Finance or FinOps partners for AI cost visibility and optimization.
• Strategic technology partners and platform teams supporting Salesforce, ServiceNow, Guidewire, Workday, GCP, AWS, Microsoft, Google, DataDog, and other AI ecosystem components.

Additional Information

• To ensure a strong start, all employees participate in our New Employee Orientation during their first week. This experience is held in person at our Madison, WI Headquarters or one of our AmFam core locations to help you connect with our mission, meet key team members and build relationships that support your growth. At times, sessions may be delivered virtually based on scheduling and availability.
• Offer to selected candidate will be made contingent on the results of applicable background checks
• Offer to selected candidate is contingent on signing a non-disclosure agreement for proprietary information, trade secrets, and inventions
• Sponsorship will not be considered for this position unless specified in the posting

In this primarily office-based role, you will be expected to spend at least 80% of your time (4+ days per week) working from the office. Candidates should reside within approximately 35-50 miles of one of the following office locations: Madison, WI 53783; or Boston, MA 02110.

#LI-Onsite

We provide benefits that support your physical, emotional, and financial wellbeing. You will have access to comprehensive medical, dental, vision and wellbeing benefits that enable you to take care of your health. We also offer a competitive 401(k) contribution, a pension plan, an annual incentive, 9 paid holidays and a paid time off program (23 days accrued annually for full-time employees). In addition, our student loan repayment program and paid-family leave are available to support our employees and their families. Interns and contingent workers are not eligible for American Family Insurance Group benefits.

We are an equal opportunity employer. It is our policy to comply with all applicable federal, state and local laws pertaining to non-discrimination, non-harassment and equal opportunity. We also consider qualified applicants with criminal histories, consistent with applicable federal, state and local law.

American Family Insurance is committed to the full inclusion of all qualified individuals. If a reasonable accommodation is needed to participate in the job application or interview process, to perform essential job functions, and/or to receive other benefits and privileges of employment, please email AskHR@AmFam.com to request a reasonable accommodation.

#LI-AW1