ZipRecruiter

Log In

1

Generative Ai Testing Jobs in New York (NOW HIRING)

Uncountable

Platform Engineer - Generative AI

New York, NY · On-site

$120K - $160K/yr

... tuning/testing behavior) and traditional fullstack development tasks (coding frontends/backends ... Strong interest in Generative AI * Solid computer science and software engineering fundamentals ...

Hearst

Senior Engineer, Generative AI

New York, NY · On-site

$114K - $157K/yr

Design and develop Generative AI applications and features using modern LLM APIs and open-source ... A commitment to automated testing and software development best practices * Strong problem solving ...

Johnson & Johnson

Data and Generative AI Engineer

Raritan, NJ · On-site

$117K - $140K/yr

Design, develop and deliver Generative/Agentic AI solutions that accelerate End-to-End Data ... Ensure data quality and integrity through meticulous testing and validation, applying Intelligent ...

Jj

Data and Generative AI Engineer

Raritan, NJ · On-site

$117K - $140K/yr

Design, develop and deliver Generative/Agentic AI solutions that accelerate End-to-End Data ... Ensure data quality and integrity through meticulous testing and validation, applying Intelligent ...

Jj

Data and Generative AI Engineer

New Brunswick, NJ · On-site

$118K - $141K/yr

Design, develop and deliver Generative/Agentic AI solutions that accelerate End-to-End Data ... Ensure data quality and integrity through meticulous testing and validation, applying Intelligent ...

next page

Showing results 1-20

All JobsGenerative Ai Testing JobsGenerative Ai Testing Jobs in New York

Generative Ai Testing information

What is generative AI testing?

Generative AI Testing refers to the process of evaluating and validating AI systems, particularly those that generate content such as text, images, or code. This type of testing focuses on assessing the accuracy, reliability, fairness, and safety of generative models to ensure they function as intended and avoid producing harmful or biased outputs. Testers use various methods, including automated and manual techniques, to check for issues like hallucinations, inappropriate content, or security vulnerabilities. The goal is to build trust in generative AI systems and ensure they meet quality and ethical standards before deployment.

What are some common challenges faced when testing generative AI models, and how can I prepare to address them in this role?

Testing generative AI models often involves unique challenges such as evaluating the quality and relevance of generated content, detecting bias or inappropriate outputs, and ensuring model consistency across various prompts. You may work closely with data scientists and engineers to create robust evaluation frameworks and develop automated as well as manual testing strategies. Familiarity with prompt engineering, statistical evaluation techniques, and domain-specific knowledge will help you address these challenges effectively. Proactively staying updated on industry best practices and collaborating with cross-functional teams are key to success in this dynamic field.

What are the key skills and qualifications needed to thrive as a generative AI testing specialist, and why are they important?

To thrive as a Generative AI Testing Specialist, you need a robust understanding of machine learning principles, model evaluation techniques, and a background in computer science or a related field. Familiarity with tools such as Python, TensorFlow, PyTorch, and model evaluation frameworks, as well as experience with automated testing platforms, is typically required. Analytical thinking, attention to detail, and strong communication skills help you identify model weaknesses and collaborate effectively with development teams. These skills are crucial to ensure the reliability, safety, and ethical deployment of generative AI solutions.

What is the difference between Generative Ai Testing vs Data Scientist?

AspectGenerative Ai TestingData Scientist
Required CredentialsKnowledge of AI models, testing tools, programming skillsStatistics, programming, data analysis certifications
Work EnvironmentAI development teams, testing labs, tech companiesResearch labs, tech firms, finance, healthcare
Employer & Industry UsageAI product testing, quality assurance in techData analysis, predictive modeling across industries

Generative Ai Testing focuses on evaluating and validating AI-generated content and models, ensuring quality and accuracy. Data Scientists analyze data, build models, and derive insights. While both roles require programming and AI knowledge, Generative Ai Testing emphasizes testing processes, whereas Data Scientists focus on data analysis and model development.

How do I become a Generative AI Testing?

To become a Generative AI Tester, develop skills in machine learning, natural language processing, and programming languages like Python. Gain experience with AI frameworks such as TensorFlow or PyTorch and understand data quality and model evaluation techniques. Relevant certifications and hands-on projects can enhance your qualifications for roles in AI testing environments.

Is Generative AI Testing a good career?

Generative AI Testing is a growing field within AI development, focusing on evaluating the quality and safety of AI-generated content. It requires skills in machine learning, programming, and understanding AI models, often involving tools like Python and TensorFlow. The role offers opportunities in tech companies and research labs, with demand expected to increase as AI applications expand.

What are popular job titles related to Generative Ai Testing jobs in New York?

For Generative Ai Testing jobs in New York, the most frequently searched job titles are:

What job categories do people searching Generative Ai Testing jobs in New York look for?

The top searched job categories for Generative Ai Testing jobs in New York are:

What cities in New York are hiring for Generative Ai Testing jobs?

Cities in New York with the most Generative Ai Testing job openings:

Generative Ai Testing jobs near you
Infographic showing various Generative Ai Testing job openings in New York as of August 2026, with employment types broken down into 1% As Needed, 83% Full Time, 12% Part Time, 3% Contract, and 1% Nights. Highlights an 88% Physical, 3% Hybrid, and 9% Remote job distribution.

Evaluation Engineer (AI Models)

Exl Neo Technologies

Manhattan, NY • On-site

Other

Posted 16 days ago

Job description

Job Description - Evaluation Engineer (AI Models)

Location: New York City, NY / Fort Mill, SC

Role Overview

We are seeking an experienced Evaluation Engineer - AI Models to join our growing AI and Digital Engineering team. The ideal candidate will have a strong background in Quality Engineering and hands-on experience evaluating AI/ML and Generative AI model performance across business and technical use cases.
This role requires a combination of analytical thinking, testing expertise, data-driven evaluation, and strong communication skills to collaborate effectively with engineering, product, and business stakeholders.

Key Responsibilities
  • Design, develop, and execute evaluation strategies for AI/ML and Generative AI models.
  • Validate model outputs for accuracy, relevance, consistency, hallucination detection, bias, safety, and performance.
  • Create automated and manual evaluation frameworks for LLM-based applications and AI systems.
  • Develop test cases, benchmarking approaches, and quality metrics for AI model validation.
  • Work closely with Data Scientists, Product Managers, and Engineering teams to improve model quality and reliability.
  • Analyze model behavior using structured and unstructured datasets.
  • Perform regression testing and continuous validation for model updates and releases.
  • Document evaluation findings, defects, risks, and recommendations clearly for technical and business audiences.
  • Support UAT and production validation activities for AI-enabled products and platforms.
  • Contribute to QA best practices, test automation strategies, and AI quality governance initiatives.

Required Qualifications
  • 10+ years of experience in Quality Assurance / Quality Engineering / Software Testing.
  • 2+ years of hands-on experience in AI model evaluation, Generative AI testing, or ML validation.
  • Strong understanding of AI/ML concepts, LLM behavior, prompt evaluation, and model testing methodologies.
  • Experience with API testing, test automation frameworks, and data validation techniques.
  • Familiarity with evaluation metrics such as precision, recall, accuracy, grounding, relevance, and hallucination detection.
  • Experience testing AI-powered applications, conversational AI, or GenAI platforms.
  • Strong analytical and problem-solving skills.
  • Excellent verbal and written communication skills.
  • Ability to work independently in a remote and cross-functional environment.

Preferred Skills
  • Experience with Python and AI/ML testing tools/frameworks.
  • Exposure to prompt engineering and Retrieval-Augmented Generation (RAG) validation.
  • Knowledge of cloud platforms such as AWS, Azure, or Google Cloud Platform.
  • Experience working in Agile/Scrum environments.
  • Financial Services or Wealth Management domain experience is a plus.

Education

Bachelor s degree in Computer Science, Engineering, Information Systems, or related field preferred.

Apply