Remote Generative Ai Project Manager Jobs in Raleigh, NC

We are seeking a Senior AI Scientist to lead the design, development, and operationalization of evaluation frameworks for Generative AI systems, with a primary focus on Large Language Models (LLMs) and agentic AI solutions.

This role will be responsible for defining and implementing robust methods to assess quality, safety, reliability, and business impact across LLM-powered applications and multi-agent workflows. The position operates within regulated environments such as life sciences, clinical research, and regulatory domains, ensuring that AI systems meet enterprise and compliance standards.

• Design and implement scalable evaluation frameworks for LLMs across use cases including:
  • Question answering, summarization, information extraction, and reasoning
  • Clinical and regulatory document generation (e.g., ICFs, CSRs, protocols)
• Develop both automated and human-in-the-loop evaluation pipelines
• Define, measure, and monitor key performance metrics, including:
  • Accuracy, factuality, faithfulness, and hallucination rate
  • Robustness, consistency, latency, and cost-performance trade-offs
• Build domain-specific benchmarks using real-world clinical, regulatory, and RWD data

• Establish evaluation strategies for agent-based and multi-agent systems
• Measure and analyze:
  • Task completion success rates
  • Planning and reasoning quality
  • Tool usage accuracy
  • Inter-agent coordination and failure patterns
• Develop scenario-based and simulation-driven evaluation environments
• Evaluate orchestration frameworks (e.g., LangGraph, Semantic Kernel, Claude Agents)

• Define evaluation strategies for complete AI pipelines, including:
  • Retrieval-Augmented Generation (RAG) systems
  • Tool-augmented agents
  • Knowledge graph + LLM architectures
• Implement offline and online evaluation mechanisms, such as:
  • A/B testing and canary releases
  • Production monitoring and model drift detection
• Enable observability and traceability using tools such as LangSmith and OpenTelemetry

• Ensure all evaluation practices align with Responsible AI principles and regulatory requirements (e.g., GxP)
• Assess and mitigate risks related to:
  • Bias, fairness, safety, and explainability
  • Data leakage and privacy concerns
• Develop audit-ready evaluation frameworks suitable for regulated environments (e.g., FDA, EMA)

• Build and scale evaluation tooling, including:
  • Automated evaluation pipelines
  • Prompt and version tracking systems
  • Experiment management platforms
• Integrate evaluation frameworks with enterprise AI platforms (e.g., Azure, Databricks, AWS, on-prem GPU environments)

• Collaborate with cross-functional teams including:
  • AI/ML engineers, product teams, and domain scientists
  • Clinical, regulatory, and real-world evidence stakeholders
• Establish enterprise-wide evaluation standards and best practices
• Mentor junior team members and contribute to strategic AI initiatives

• Master's or PhD in Computer Science, Artificial Intelligence, Machine Learning, or a related field
• 5+ years of experience in applied AI/ML, with a strong focus on Generative AI and LLMs
• Demonstrated experience in:
  • LLM evaluation (both automated and human-in-the-loop)
  • Prompt engineering and model behavior analysis
  • Python programming using frameworks such as PyTorch or TensorFlow
• Hands-on experience with:
  • RAG systems, embeddings, and vector databases
  • Agent frameworks (e.g., LangChain, LangGraph, Semantic Kernel)
• Strong understanding of:
  • Evaluation metrics and experimental design
  • Model limitations, failure modes, and debugging techniques

• Experience in life sciences, healthcare, or regulated environments
• Familiarity with:
  • Clinical trial workflows (ICF, CSR, TMF, regulatory submissions)
  • Knowledge graphs and biomedical data systems
• Experience with evaluation tools such as LangSmith, Promptfoo, DeepEval, HELM, or OpenAI Evals
• Exposure to Responsible AI frameworks and regulatory compliance standards

• Standardized evaluation frameworks adopted across AI teams
• Measurable improvements in LLM reliability and agent performance
• Defined quality gates to support production-ready AI deployment decisions
• Strong alignment between AI model performance and business/clinical outcomes