Senior Fastapi Developer Jobs in Tennessee (NOW HIRING)

Requisition Id 16695
Overview:
As a Senior AI Architect at Oak Ridge National Laboratory (ORNL), you will operate at the intersection of advanced machine learning, software and systems architecture, and ORNL's enterprise and high-performance computing (HPC) environments. The role is typically less about building individual models and more about designing, integrating, and governing end-to-end AI systems that can be deployed and sustained across research programs and mission operations. This includes architecting solutions that use large language models (LLMs), multimodal and foundation models, and hybrid deployments spanning on-premises HPC resources, controlled networks, and approved cloud services-ensuring these systems meet ORNL's requirements for scale, security, reliability, and scientific reproducibility.
In a practical sense, the Senior AI Architect defines reference architectures for "AI at ORNL," including patterns for data ingestion and curation, training and fine-tuning pipelines, model serving and inference at scale, and integration into scientific workflows (e.g., simulation, experimental facilities, and analysis platforms). You will guide technology selection and implementation for core capabilities such as distributed training and inference, workflow orchestration, GPU/accelerator utilization, model registries and artifact management, vector search and retrieval-augmented generation (RAG), and LLMOps/MLOps practices (CI/CD for models, automated evaluation, and monitoring). You will also establish performance and cost models to decide when workloads should run on HPC versus cloud, and how to engineer systems for throughput, latency, and resource efficiency under real-world constraints.
Because ORNL systems often involve sensitive data, regulated collaborations, and tightly controlled computing environments, this role places strong emphasis on secure-by-design architectures. The Senior AI Architect works closely with cybersecurity and compliance stakeholders to implement robust identity and access management, network segmentation, audit logging, and data-governance controls-supporting needs such as least-privilege access, provenance, and reproducibility. You will also define validation and assurance approaches appropriate for mission use: rigorous benchmarking, red-teaming and prompt-injection testing for LLM applications, model risk assessments, and continuous monitoring for drift and unexpected behavior.
Finally, the Senior AI Architect acts as a technical leader and integrator across the division. You will lead architecture reviews, produce technical roadmaps, and coordinate cross-functional teams of researchers, platform engineers, and application developers to move AI capabilities from prototype to production. This often includes mentoring teams on best practices, standardizing reusable components (shared model-serving stacks, data connectors, evaluation harnesses), and ensuring that AI systems remain maintainable and supportable over time-aligned with ORNL's mission priorities, operational constraints, and evolving research needs.
Basic Qualifications:

• Bachelor's or Master's degree in Computer Science, AI/ML, Software Engineering, or related field
• 7+ years of experience in software architecture, machine learning systems, or distributed systems
• Strong understanding of:
  • Machine learning fundamentals and LLM architectures
  • API-based and local model inference workflows
• Experience using both:
  • Cloud-hosted AI services (e.g., OpenAI, AWS Bedrock, Azure AI, Vertex AI, or similar)
  • Locally deployed models (e.g., via Hugging Face, Ollama, vLLM, or similar)
• Familiarity with Model Context Protocol (MCP) or comparable orchestration/integration patterns
• Proficiency in Python and backend system design
• Experience with containerization and deployment (Docker, Kubernetes)

Preferred Qualifications:

• Experience architecting hybrid AI systems that dynamically route between local and cloud models
• Hands-on experience setting up on-demand local model inference (GPU-backed or CPU-optimized deployments)
• Deep familiarity with LLM system design patterns:
  • Retrieval-Augmented Generation (RAG)
  • Tool use / function calling
  • Agent-based workflows
• Experience with vector databases (e.g., FAISS, Pinecone, Weaviate, Milvus)
• Experience with inference optimization frameworks (vLLM, TensorRT, ONNX Runtime, quantization techniques)
• Knowledge of model serving infrastructure (Ray Serve, KServe, Triton, FastAPI-based services, MLFlow, etc.)
• Experience designing low-latency and high-throughput inference pipelines
• Familiarity with fine-tuning approaches (LoRA, PEFT, instruction tuning)
• Experience integrating AI systems into enterprise applications and data platforms
• Understanding of observability for AI systems (logging, tracing, evaluation metrics, drift detection)
• Experience with GPU infrastructure, scheduling, and cost optimization strategies
• Familiarity with security practices for AI systems, including prompt injection mitigation and data isolation
• Experience working with multimodal models (text, image, geospatial, etc.)

Special Requirements:

• Q clearance with SCI: This position requires the ability to obtain and maintain a Secret Compartmented Information (SCI) clearance from the Department of Energy. As such, this position is a Workplace Substance Abuse (WSAP) testing designated position. WSAP positions require passing a pre-placement drug test and participation in an ongoing random drug testing program. In addition, due the SCI, you may also be subject to random polygraph testing.

For employment at Oak Ridge National Laboratory (ORNL), a Real ID compliant form of identification will be required. Additionally, ORNL is subject to Department of Energy (DOE) access restrictions. All employees must also be able to obtain and maintain a federal Personal Identity Verification (PIV) card as mandated by Homeland Security Presidential Directive 12 (HSPD-12) and Department of Energy (DOE) Order 473.1A, which requires a favorable post-employment background investigation.
To obtain this credential, new employees must successfully complete and pass a Federal Tier 1 background check investigation. This investigation includes a declaration of illegal drug activities, including use, supply, possession, or manufacture within the last year. This includes marijuana and cannabis derivatives, which are still considered illegal under federal law, regardless of state laws.
For foreign national candidates:
If you have not resided in the U.S. for three consecutive years, you are not eligible for the PIV credential and instead will need to obtain a favorable Local Site Specific Only (LSSO) risk determination to maintain employment. Once you meet the three-year residency requirement, you will be required to obtain a PIV credential to maintain employment.
About ORNL:
As a U.S. Department of Energy (DOE) Office of Science national laboratory, ORNL has an impressive 80-year legacy of addressing the nation's most pressing challenges. Our team is made up of over 7,000 dedicated and innovative individuals! Our goal is to create an environment where a variety of perspectives and backgrounds are valued, ensuring ORNL is known as a top choice for employment. These principles are essential for supporting our broader mission to drive scientific breakthroughs and translate them into solutions for energy, environmental, and security challenges facing the nation.
ORNL offers competitive pay and benefits programs to attract and retain individuals who demonstrate exceptional work behaviors. The laboratory provides a range of employee benefits, including medical and retirement plans and flexible work hours, to support the well-being of you and your family. Employee amenities such as on-site fitness, banking, and cafeteria facilities are also available for added convenience.
Other benefits include the following: Prescription Drug Plan, Dental Plan, Vision Plan, 401(k) Retirement Plan, Contributory Pension Plan, Life Insurance, Disability Benefits, Generous Vacation and Holidays, Parental Leave, Legal Insurance with Identity Theft Protection, Employee Assistance Plan, Flexible Spending Accounts, Health Savings Accounts, Wellness Programs, Educational Assistance, Relocation Assistance, and Employee Discounts.
If you have difficulty using the online application system or need an accommodation to apply due to a disability, please email: ORNLRecruiting@ornl.gov.
This position will remain open for a minimum of 5 days after which it will close when a qualified candidate is identified and/or hired.
We accept Word (.doc, .docx), Adobe (unsecured .pdf), Rich Text Format (.rtf), and HTML (.htm, .html) up to 5MB in size. Resumes from third party vendors will not be accepted; these resumes will be deleted and the candidates submitted will not be considered for employment.
ORNL is an equal opportunity employer. All qualified applicants, including individuals with disabilities and protected veterans, are encouraged to apply. UT-Battelle is an E-Verify employer.