ZipRecruiter

Log In

1

Ai Evaluator Jobs (NOW HIRING)

Mercor

Financial Compliance AI Evaluator

New York, NY · Remote

$80 - $120/hr

Compliance / regulatory response with financial-services AI Evaluator Type: Contract Compensation: $80-$120/hour Location: Remote Role Responsibilities * Evaluate AI-generated artifacts against ...

Weekday AI

Technology AI Evaluation Expert

$60 - $75/hr

Requirements Key Responsibilities Design AI Evaluation Tasks * Create realistic, multi-step benchmark tasks based on professional technology workflows. * Develop challenges using technical ...

Weekday AI

AI Evaluation Specialist

$70/hr

Highlight both strengths and areas for improvement in AI-generated outputs. * Apply consistent judgment across a wide range of evaluation tasks. Maintain Evaluation Quality * Follow detailed project ...

Distyl AI

AI Engineer, Evaluation

New York, NY · On-site

$150K - $250K/yr

AI Evaluation Engineers focus on designing and implementing the evaluation systems that drive this process. They are hands-on engineers who write production Python code, build evaluation pipelines ...

Steampunk

AI Evaluation Scientist

Mclean, VA · On-site

$105K - $145K/yr

The AI Evaluation Scientist will work closely with engineers, data scientists, governance analysts, and product teams to develop evaluation metrics, build test harnesses, analyze model behavior, and ...

Steampunk

AI Evaluation Scientist

Mclean, VA

$105K - $145K/yr

We are looking for an AI Evaluation Scientist to design and execute evaluation processes that ensure our predictive and generative AI systems are accurate, reliable, safe, and aligned with mission ...

Steampunk

AI Evaluation Scientist

Mclean, VA · On-site

$105K - $145K/yr

The AI Evaluation Scientist will work closely with engineers, data scientists, governance analysts, and product teams to develop evaluation metrics, build test harnesses, analyze model behavior, and ...

Steampunk

AI Evaluation Scientist

Mclean, VA · On-site

$105K - $145K/yr

The AI Evaluation Scientist will work closely with engineers, data scientists, governance analysts, and product teams to develop evaluation metrics, build test harnesses, analyze model behavior, and ...

Steampunk

AI Evaluation Scientist

Mclean, VA

$105K - $145K/yr

We are looking for an AI Evaluation Scientist to design and execute evaluation processes that ensure our predictive and generative AI systems are accurate, reliable, safe, and aligned with mission ...

Steampunk

AI Evaluation Scientist

Mclean, VA

$105K - $145K/yr

We are looking for an AI Evaluation Scientist to design and execute evaluation processes that ensure our predictive and generative AI systems are accurate, reliable, safe, and aligned with mission ...

Steampunk

AI Evaluation Scientist

Mclean, VA · On-site

$105K - $145K/yr

The AI Evaluation Scientist will work closely with engineers, data scientists, governance analysts, and product teams to develop evaluation metrics, build test harnesses, analyze model behavior, and ...

Steampunk

AI Evaluation Scientist

Mclean, VA

$105K - $145K/yr

We are looking for an AI Evaluation Scientist to design and execute evaluation processes that ensure our predictive and generative AI systems are accurate, reliable, safe, and aligned with mission ...

Steampunk

AI Evaluation Scientist

Mclean, VA · On-site

$105K - $145K/yr

The AI Evaluation Scientist will work closely with engineers, data scientists, governance analysts, and product teams to develop evaluation metrics, build test harnesses, analyze model behavior, and ...

Steampunk

AI Evaluation Scientist

Mclean, VA · On-site

$105K - $145K/yr

The AI Evaluation Scientist will work closely with engineers, data scientists, governance analysts, and product teams to develop evaluation metrics, build test harnesses, analyze model behavior, and ...

Steampunk

AI Evaluation Scientist

Mclean, VA · On-site

$105K - $145K/yr

The AI Evaluation Scientist will work closely with engineers, data scientists, governance analysts, and product teams to develop evaluation metrics, build test harnesses, analyze model behavior, and ...

Steampunk

AI Evaluation Scientist

Mclean, VA · On-site

$105K - $145K/yr

The AI Evaluation Scientist will work closely with engineers, data scientists, governance analysts, and product teams to develop evaluation metrics, build test harnesses, analyze model behavior, and ...

Steampunk

AI Evaluation Scientist

Mclean, VA · On-site

$105K - $145K/yr

The AI Evaluation Scientist will work closely with engineers, data scientists, governance analysts, and product teams to develop evaluation metrics, build test harnesses, analyze model behavior, and ...

next page

Showing results 1-20

All JobsAi Evaluator Jobs

Ai Evaluator information

See salary details

$29.5K

$65.5K

$106.5K

How much do ai evaluator jobs pay per year?

As of Aug 6, 2026, the average yearly pay for ai evaluator in the United States is $65,471.00, according to ZipRecruiter salary data. Most workers in this role earn between $44,500.00 and $79,500.00 per year, depending on experience, location, and employer.

How much do AI evaluators make?

AI evaluators typically earn between $40,000 and $80,000 annually, depending on experience, location, and the complexity of tasks. Many roles are freelance or part-time, with some positions offering flexible schedules and requiring skills in data analysis and machine learning tools.

What are the key skills and qualifications needed to thrive as an AI evaluator?

To thrive as an AI Evaluator, you need a solid understanding of machine learning concepts, data analysis, and critical thinking, often supported by a degree in computer science, data science, or a related field. Familiarity with annotation tools, model assessment platforms, and sometimes programming languages like Python is highly advantageous. Attention to detail, strong communication skills, and the ability to provide constructive feedback are essential soft skills. These competencies are crucial for accurately assessing AI model outputs and collaborating with development teams to improve system performance.

How to become an AI evaluator?

To become an AI evaluator, candidates typically need a background in computer science, data analysis, or related fields, along with strong analytical skills. Experience with machine learning, natural language processing, or data annotation tools is often required, and some roles may require certification or training in AI or data labeling platforms.

What is an AI evaluator?

An AI Evaluator assesses the performance, accuracy, and relevance of artificial intelligence models, often focusing on language models, search engines, or recommendation systems. They review AI-generated responses, ensuring they align with human expectations and industry standards. Evaluators may follow specific guidelines to rate or provide feedback on outputs, helping improve AI functionality. This role typically requires analytical skills, attention to detail, and proficiency in the target language. Many AI Evaluators work remotely on a contract or freelance basis.

What are some typical challenges an AI evaluator faces in their daily work?

AI Evaluators often encounter complex data or ambiguous responses that require nuanced judgment to assess accurately. Adapting to rapidly evolving guidelines and learning new toolsets as AI projects progress is also common. Routine tasks may include evaluating AI-generated outputs for relevance and bias, documenting errors, and discussing findings with engineers or linguists. While the work is intellectually stimulating, it demands precision and flexibility to ensure models meet high-quality standards.

More about Ai Evaluator jobs
What cities are hiring for Ai Evaluator jobs? Cities with the most Ai Evaluator job openings:
What are the most commonly searched types of Ai Evaluator jobs? The most popular types of Ai Evaluator jobs are:
What states have the most Ai Evaluator jobs? States with the most job openings for Ai Evaluator jobs include:
What job categories do people searching Ai Evaluator jobs look for? The top searched job categories for Ai Evaluator jobs are:
Ai Evaluator jobs near you
Infographic showing various Ai Evaluator job openings in the United States as of August 2026, with employment types broken down into 50% Internship, and 50% Part Time. Highlights an 50% In-person, and 50% Remote job distribution, with an average salary of $65,471 per year, or $31.5 per hour.

Financial Compliance AI Evaluator

Mercor

New York, NY • Remote

$80 - $120/hr

Full-time

Re-posted 13 days ago

Job description

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.

Position: Compliance / regulatory response with financial-services AI Evaluator
Type: Contract
Compensation: $80–$120/hour
Location: Remote

Role Responsibilities

  • Evaluate AI-generated artifacts against domain-specific quality rubrics.
  • Identify factual, aesthetic, and presentation errors.
  • Provide clear, structured written feedback.
  • Apply deep subject-matter expertise to grade outputs.
  • Review and assess AI-generated work products for accuracy, rigor, and domain quality.
  • Work independently and asynchronously to meet deadlines while improving AI model performance.

Qualifications

Must-Have

  • 5+ years of relevant professional experience in Compliance / regulatory response with financial-services AI.
  • Native or professional fluency in English.
  • Highly proficient in Microsoft Office and Google Workspace, especially Slides.

Preferred

  • Advanced degree (Master's or higher) from a reputable institution.

Application Process (Takes 20–30 mins to complete)

  • Upload resume
  • AI interview based on your resume
  • Submit form

Resources & Support

  • For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome
  • For any help or support, reach out to: support@mercor.com

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.