Senior Validation Engineer Jobs in Decatur, GA (NOW HIRING)

Job Description
Purpose:
The Senior AI Engineer will design, build, and operate the framework and platform capabilities that enable agentic solutions across Floor & Decor - and will build flagship agentic products on that platform to prove it out. The role covers the full delivery lifecycle: planning, designing, configuring, testing, implementing, documenting, and maintaining AI solutions deployed on Microsoft Azure and integrated with the company's operational systems.
The mandate is durable capability, not a single product. Individual agentic solutions will come and go - some will graduate to other teams, some will be retired - but the substrate they are built on is what this team owns: the tool and integration layer, context and memory management, evaluation and observability, guardrails, and the paved road that lets the next agentic workflow ship in weeks rather than quarters.
This is a hands-on senior individual contributor role. This is not a research or data science position. We are looking for an engineer with first-principles command of backend systems who has moved into AI engineering - someone who treats LLMs and agents as components in a well-architected distributed system, and holds them to the same standards of reliability, observability, security, and cost control as any other production dependency. Technical leadership here is exercised through architecture, code quality, and influence rather than through direct reports.
Minimum Eligibility Requirements:

• Engineering Foundation
• 7-10 years of professional software engineering experience, with increasing scope and ownership.
• Deep proficiency in backend engineering in at least one of C#/.NET, Java, Node.js/TypeScript, or Python. We care about first-principles understanding of backend systems, not a specific language - the ability to pick up a new stack quickly matters more than which one you arrived with. Python is a plus, not a requirement.
• Service-based and microservice architectures, RESTful API design, and asynchronous service communication - including API versioning, contract design, error semantics, and backward compatibility.
• Cloud-based serverless microservices in production (Azure Functions, Container Apps, or equivalent).
• Event-driven architecture: queues, pub/sub, idempotency, retries, and dead-letter handling.
• Solid data fundamentals: relational and NoSQL data modeling, query performance, and transactional correctness.
• Demonstrated ownership of code quality - automated testing, code review, and CI/CD as normal practice, not overhead.
• AI Engineering
• Designed and deployed agentic solutions - shipped and operated in production, not prototypes or POCs. We are interested in depth of agent delivery experience; retrieval-augmented generation is one tool in that kit, not the definition of the job.
• Harness engineering - practical command of the scaffolding that surrounds a model and determines whether an agent actually works in production: context construction and management, tool and function-call interfaces, multi-step planning and control flow, memory and state, structured output, guardrails and validation, human-in-the-loop checkpoints, and graceful failure and fallback behavior.
• Azure OpenAI services - LLM APIs, Azure AI Search (vector/index), and associated Azure infrastructure. Azure AI Foundry or AWS Bedrock experience also applies.
• Agent and LLM orchestration frameworks (LangChain, LangGraph, or similar).
• MCP (Model Context Protocol) or other tool-calling/function-calling patterns for LLM-to-system integrations.
• AI-specific failure and risk modes - hallucination, prompt injection, data leakage, non-determinism, runaway tool loops - and concrete techniques for mitigating them.
• Sound judgment on when an LLM or agent is and is not the right tool, and how to bound and validate its output.
• AI-Augmented Development
• Daily working fluency with agentic coding tools (Claude Code, Cursor, GitHub Copilot, Devin, or equivalent). Claude Code is our standard.
• A credible, specific point of view on where these tools accelerate delivery, where they do not, and how to review and test model-generated code responsibly.
• Able to speak to measurable impact on their own or their team's throughput and quality.
• Cloud, Security, and Delivery
• Cloud environments (Azure preferred) including compute, storage, networking, and IAM fundamentals.
• Cloud security fundamentals: identity and access management, secrets management, and network boundaries.
• Agile/scrum methodologies - sprint ceremonies, story estimation, backlog grooming.
• Angular or comparable modern front-end frameworks - working knowledge.
• Clear written and verbal communication with both technical and non-technical partners; comfortable with ambiguity and able to move from a vague business problem to a scoped, specified, shippable increment.

Education
Bachelor's degree in Computer Science, Engineering, or a related field - or equivalent professional experience. Demonstrated capability is weighted above credentials.
Essential Job Functions:
Agentic Platform & Framework Engineering
Design, build, and evolve the shared framework that lets teams across the organization build, deploy, and operate agentic solutions - a paved road covering agent scaffolding, tool and integration interfaces, context and memory management, evaluation, guardrails, and observability.
Turn one-off agent implementations into reusable building blocks: service templates, harness components, MCP server patterns, evaluation harnesses, and deployment pipelines.
Design and maintain MCP integrations with enterprise systems - inventory, merchandising, and the systems that follow - so any agent built on the platform can be grounded in live operational data without rebuilding the plumbing.
Define and document the standards, contracts, and reference architectures that agentic workloads at Floor & Decor are built against.
Make and document architectural decisions for reliability, scalability, security, and observability of AI services deployed in Azure.
Agentic Solution Delivery
Design, build, and operate production agentic solutions end to end - from problem framing through deployment and ongoing operation - using them both to deliver business value and to harden the underlying platform.
Apply the right technique to the problem: tool calling, multi-step planning, retrieval, memory, human-in-the-loop review, or a deterministic service where an agent is the wrong answer.
Build and tune retrieval where retrieval is warranted - chunking strategies, vector indexing, retrieval ranking, and context engineering on Azure AI Search.
Contribute across the stack, including an Angular front end and a Python-based service and LLMOps layer - picking up front-end work to get a feature over the line rather than handing it off.
Support the transition of mature agentic products to partner teams: documentation, runbooks, and knowledge transfer that let a solution outlive its original builders.
Evaluation, Observability & Operations
Establish evaluation and regression testing as a first-class part of the platform - eval sets, LLM-as-judge scoring, task-level success metrics, and regression gates in CI - so changes to prompts, models, tools, or retrieval ship with evidence rather than intuition.
Own evaluation pipelines for retrieval-based components using Ragas, tracking faithfulness, answer relevance, and context precision across releases.
Instrument agentic systems for observability with Langfuse alongside Azure Monitor - tracing, latency, token and cost attribution, tool-call success rates, quality signals, and failure modes - and act on what the telemetry shows.
Manage AI cost and performance: token budgeting, caching, model routing and right-sizing, and latency optimization.
Own production services - on-call participation, incident response, and post-incident follow-through.
Troubleshoot and resolve complex issues in agent behavior, retrieval quality, hallucination, latency, and integration reliability.
Engineering Practice & Collaboration
Apply spec-driven development: turn ambiguous business asks into clear specifications and acceptance criteria before code, and keep specs and implementation in sync.
Set and model the standard for AI-augmented development on the team: effective use of agentic coding tools, plus the review discipline that has to come with it.
Conduct code reviews and mentor peers, fostering a culture of quality and continuous learning - through technical influence rather than direct reporting lines.
Partner with product managers and business stakeholders to identify workflows worth automating, translate them into technical requirements, and help prioritize the backlog.
Act as a technical consultant to other teams adopting the platform - helping them build well on it rather than around it.
Participate in agile ceremonies - sprint planning, retrospectives, and daily stand-ups - as a senior voice on the team.
Communicate technical trade-offs and architectural decisions clearly to both technical and non-technical audiences.
Partner with security, data, and platform teams on data governance, PII handling, prompt injection defense, and responsible use.
Innovation & Quality
Evaluate emerging agent capabilities, tooling, protocols, and Azure OpenAI / AI Foundry updates; recommend and prototype improvements to keep the platform current in a fast-moving field.
Establish and maintain engineering best practices including CI/CD pipelines, infrastructure as code, code quality standards, and security practices for AI workloads.
Continuously reduce the cost and time required to bring the next agentic solution to production.
Nice to Have

• Spec-driven development - specification-first workflows, and using specs to drive AI-assisted implementation.

• Building internal developer platforms, frameworks, or SDKs consumed by other engineering teams.

• Building or publishing MCP servers, not just consuming them.

• LLM and agent evaluation frameworks - Ragas, G-Eval, LLM-as-judge, or agent trajectory evaluation.

• AI observability tooling - Langfuse (our platform) or equivalent tracing and evaluation systems.

• Retrieval infrastructure beyond Azure AI Search - pgvector, Pinecone, Elastic, or hybrid search design.

• Multi-agent orchestration, agent-to-agent protocols, or durable/long-running workflow engines.

• Infrastructure as code - Bicep, Terraform, or ARM.

• Fine-tuning and model adaptation - LoRA/PEFT, distillation, or evaluating fine-tuning against prompting and retrieval alternatives.

• LLMOps tooling such as MLflow, Weights & Biases, or Azure ML.

• Retail systems familiarity - POS, OMS, inventory/merchandising platforms.

• Center of Excellence or innovation-team experience within a larger enterprise.

• Open source contributions, technical writing, or speaking in the AI engineering space.

Our Technology Stack
Layer - Technologies
Front End - Angular, TypeScript
Back End / LLMOps - Python, C#/.NET, Node.js/TypeScript, REST APIs, serverless microservices (Azure Functions, Container Apps)
AI / LLM - Azure OpenAI, Azure AI Foundry, Azure AI Search, agentic
architectures, RAG where warranted
Orchestration & Integration - LangChain / LangGraph, MCP
Evaluation - Ragas, eval sets, LLM-as-judge, CI regression gates
Observability - Langfuse, Azure Monitor, Application Insights
Cloud & DevOps - Microsoft Azure, CI/CD pipelines, IaC (Bicep/Terraform), Agile/Scrum
AI-Augmented Development - Claude Code
Integrations - Enterprise systems - inventory, merchandising, and beyond - via MCP
Work Environment

• This is a hybrid position based at our Atlanta, GA headquarters, with a standard office schedule Monday through Friday during core business hours.

• You will work in a collaborative, open-plan office environment within the IT department, with dedicated space for focused engineering work.

• The role involves regular in-person collaboration with product managers, business stakeholders, and your engineering team.

• Occasional visits to retail store locations may be required to gather associate feedback and observe how the product is used in context.

• Some extended hours may be needed around major releases or on-call rotations for production incidents.

• Standard physical requirements of a professional office environment apply - prolonged sitting, use of a computer workstation, and participation in in-person and video meetings.

Why Join Our AI Center of Excellence

• Build the foundation, not just the feature. You will design the framework that every agentic workflow at Floor & Decor is built on - the decisions you make will shape how this company builds with AI for years.

• Ship production agentic products used by thousands of associates across a national retail footprint, and see the impact of your work in real stores.

• Greenfield scope in a mature company: the platform is being defined now, with real budget, real users, and real operational systems to integrate with.

• Work in an AI-augmented engineering environment where agentic development tooling is the standard, not an experiment.

• Competitive compensation, comprehensive benefits, and a culture that values curiosity and continuous improvement.

• Opportunities to grow into broader AI architecture and engineering leadership as the