Ai Evaluation Jobs in Raleigh, NC (NOW HIRING)

We're seeking a hands-on AI Product Analyst who's excited to dive deep into building AI systems across multiple ventures. As a core member of our studio team, you'll be the hands-on specialist managing prompt engineering, evaluations, and AI performance optimization across our AI ventures - everything from crafting and testing agent prompts and tools, to building evaluation pipelines, and analyzing what makes our AI products work best.

This role is ideal if you've been building with Cursor, Claude Code, v0, or Replit, and want to be part of a team building multiple AI startups together. You'll spend your days prompt engineering, building with AI tools, and learning how AI products come to life in a fast-paced venture environment. This role is ideal for someone who may not be a trained developer but loves building and is growing in their technical skills along the way.

Sprout AI is pursuing a relational renaissance through venture building. We partner with exceptional entrepreneurs to create AI ventures that deepen authentic human connections in an increasingly digital world. Our founding team combines decades of proven commercial success with a powerful redemptive vision for how entrepreneurs can address the major issues of our time.

You'll quickly become our go-to expert on Braintrust and AI evaluation tools, mastering everything from creating training datasets for model fine-tuning to building comprehensive evaluation frameworks that measure our AI systems' performance. Your days will be spent designing and continuously refining complex prompts while running experiments to optimize our AI agents across multiple ventures.

• Build and maintain evaluation frameworks to measure AI model effectiveness across multiple ventures
• Design and manage prompt testing systems with comprehensive performance tracking
• Craft and iterate on prompt templates through systematic testing and optimization
• Analyze fine-tuning results and optimization strategies including RLHF and other techniques
• Design A/B testing frameworks and automated testing pipelines for different AI configurations
• Partner closely with Product Leads and Founders to translate product requirements into high-performing AI models and agentic systems
• Build reusable evaluation frameworks and testing approaches that accelerate development across the portfolio
• Stay current with AI product best practices and evaluation methodologies while documenting insights and creating playbooks for AI optimization

• 1-3 years of experience working with AI systems, data analysis tools, and basic programming (willingness to learn is key)
• Strong experience with prompt engineering and working with large language models
• Understanding of testing methodologies, performance measurement, and AI evaluation platforms (Arize, Langsmith, Braintrust, etc)
• Eagerness to learn and adapt quickly in a fast-paced venture studio environment with comfort handling ambiguity
• Strong problem-solving skills and ability to work across multiple projects simultaneously and context-switch effectively
• Genuine excitement about using AI to strengthen human relationships and communities, with interest in our redemptive AI thesis
• Collaborative spirit and strong communication skills for working with both technical and non-technical stakeholders
• Bonus: Knowledge of fine-tuning approaches, AI product development background, or previous startup experience

• Work directly with cutting-edge AI technology across multiple real-world applications while learning from experienced founders and product leaders
• Gain exposure to the full venture building process, from ideation to market launch, developing expertise across multiple successful ventures
• Optimize AI systems that genuinely improve people's lives and strengthen communities as part of a mission-driven team
• Shape the future of how AI integrates into human relationships and society, creating technology for human flourishing
• Access ongoing learning resources and professional development with paths for advancement into more senior product roles as our team grows
• Work closely with Product Leads, Founders, and AI Builders in a supportive, collaborative team environment
• Benefit from our partnership with Praxis and access to a network of exceptional entrepreneurs and industry leaders
• Join a culture that values both analytical rigor and human-centered design while building meaningful technology

If you're passionate about AI technology and excited about using it to strengthen human connections, we'd love to hear from you. This role offers a unique opportunity to grow your technical skills while contributing to ventures with lasting impact.