Experience with generative AI safety and evaluation practices, including prompt injection testing, jailbreak resilience, hallucination measurement, toxicity scoring, harm scoring, and grounding ...
Experience with generative AI safety and evaluation practices, including prompt injection testing, jailbreak resilience, hallucination measurement, toxicity scoring, harm scoring, and grounding ...
... testing, evaluation, and validation against quality metrics, performance benchmarks, and ... and generative AI models. • Create documentation, user guides, and best practices for internal ...
... testing, evaluation, and validation against quality metrics, performance benchmarks, and ... and generative AI models. • Create documentation, user guides, and best practices for internal ...
... testing, evaluation, and validation against quality metrics, performance benchmarks, and ... and generative AI models. • Create documentation, user guides, and best practices for internal ...
... testing, evaluation, and validation against quality metrics, performance benchmarks, and ... and generative AI models. • Create documentation, user guides, and best practices for internal ...
AI Engineer
Mayfield Heights, OH · Hybrid
The AI Engineer will design, develop, and operationalize Generative AI (GenAI) and Large Language ... Coordinate user acceptance testing, collect structured feedback, and incorporate operational ...
AI Engineer
Mayfield Heights, OH · Hybrid
The AI Engineer will design, develop, and operationalize Generative AI (GenAI) and Large Language ... Coordinate user acceptance testing, collect structured feedback, and incorporate operational ...
Develop, test and deploy complex prompts, prompt chains, workflows, and automation to improve the accuracy, consistency, and usability of generative AI outputs. * Conduct prompt testing and quality ...
Develop, test and deploy complex prompts, prompt chains, workflows, and automation to improve the accuracy, consistency, and usability of generative AI outputs. * Conduct prompt testing and quality ...
Innovation Analyst
Cleveland, OH · On-site
Develop, test and deploy complex prompts, prompt chains, workflows, and automation to improve the accuracy, consistency, and usability of generative AI outputs. * Conduct prompt testing and quality ...
Innovation Analyst
Cleveland, OH · On-site
Develop, test and deploy complex prompts, prompt chains, workflows, and automation to improve the accuracy, consistency, and usability of generative AI outputs. * Conduct prompt testing and quality ...
Develop, test and deploy complex prompts, prompt chains, workflows, and automation to improve the accuracy, consistency, and usability of generative AI outputs. * Conduct prompt testing and quality ...
Develop, test and deploy complex prompts, prompt chains, workflows, and automation to improve the accuracy, consistency, and usability of generative AI outputs. * Conduct prompt testing and quality ...
Develop, test and deploy complex prompts, prompt chains, workflows, and automation to improve the accuracy, consistency, and usability of generative AI outputs. * Conduct prompt testing and quality ...
Develop, test and deploy complex prompts, prompt chains, workflows, and automation to improve the accuracy, consistency, and usability of generative AI outputs. * Conduct prompt testing and quality ...
... generative, agentic, autonomous, embedded, and third-party AI. - Ensure AI use cases are ... Risk Assessment, Monitoring & Issue Management: - Oversee AI testing, monitoring, and metrics ...
... generative, agentic, autonomous, embedded, and third-party AI. - Ensure AI use cases are ... Risk Assessment, Monitoring & Issue Management: - Oversee AI testing, monitoring, and metrics ...
Conduct rigorous model testing, evaluation, and validation against quality metrics, performance ... Background in NLP, conversational AI, or generative AI applications * Open-source contributions or ...
Conduct rigorous model testing, evaluation, and validation against quality metrics, performance ... Background in NLP, conversational AI, or generative AI applications * Open-source contributions or ...
US Tech - AI Engineering Manager
Cleveland, OH · On-site
$73K - $244K/yr
The Opportunity As part of the People Tech & AI team you will lead teams delivering governed Generative AI solutions, designing testing strategies, evaluation frameworks, and governance controls to ...
US Tech - AI Engineering Manager
Cleveland, OH · On-site
$73K - $244K/yr
The Opportunity As part of the People Tech & AI team you will lead teams delivering governed Generative AI solutions, designing testing strategies, evaluation frameworks, and governance controls to ...
US Tech - AI Engineering Manager
Columbus, OH · On-site
$73K - $244K/yr
The Opportunity As part of the People Tech & AI team you will lead teams delivering governed Generative AI solutions, designing testing strategies, evaluation frameworks, and governance controls to ...
US Tech - AI Engineering Manager
Columbus, OH · On-site
$73K - $244K/yr
The Opportunity As part of the People Tech & AI team you will lead teams delivering governed Generative AI solutions, designing testing strategies, evaluation frameworks, and governance controls to ...
US Tech - AI Engineering Manager
Cincinnati, OH · On-site
$73K - $244K/yr
The Opportunity As part of the People Tech & AI team you will lead teams delivering governed Generative AI solutions, designing testing strategies, evaluation frameworks, and governance controls to ...
US Tech - AI Engineering Manager
Cincinnati, OH · On-site
$73K - $244K/yr
The Opportunity As part of the People Tech & AI team you will lead teams delivering governed Generative AI solutions, designing testing strategies, evaluation frameworks, and governance controls to ...
US Tech - AI Engineering Manager
Toledo, OH · On-site
$73K - $244K/yr
The Opportunity As part of the People Tech & AI team you will lead teams delivering governed Generative AI solutions, designing testing strategies, evaluation frameworks, and governance controls to ...
US Tech - AI Engineering Manager
Toledo, OH · On-site
$73K - $244K/yr
The Opportunity As part of the People Tech & AI team you will lead teams delivering governed Generative AI solutions, designing testing strategies, evaluation frameworks, and governance controls to ...
Agentic AI, AI & Data Senior Consultant
Columbus, OH · On-site
$102K - $139K/yr
... testing, training, defining support procedures. Your background in technology will provide the ... 1 year focused on Generative AI, Agentic AI or multi-agent systems * 2+ years of hands-on ...
Agentic AI, AI & Data Senior Consultant
Columbus, OH · On-site
$102K - $139K/yr
... testing, training, defining support procedures. Your background in technology will provide the ... 1 year focused on Generative AI, Agentic AI or multi-agent systems * 2+ years of hands-on ...
... generative AI tools, and intelligent automation to enhance customer and business outcomes. • ... testing, and continuous improvement across the software delivery lifecycle using Azure DevOps. • ...
... generative AI tools, and intelligent automation to enhance customer and business outcomes. • ... testing, and continuous improvement across the software delivery lifecycle using Azure DevOps. • ...
... generative AI tools, and intelligent automation to enhance customer and business outcomes. • ... testing, and continuous improvement across the software delivery lifecycle using Azure DevOps. • ...
... generative AI tools, and intelligent automation to enhance customer and business outcomes. • ... testing, and continuous improvement across the software delivery lifecycle using Azure DevOps. • ...
Data and Analytics - Business Insurance Data Science Executive Director
Columbus, OH · On-site
$204K - $285K/yr
Identify and implement generative AI, agentic AI, and automation opportunities across the business ... Design and implement measurement frameworks and experimentation programs (A/B testing, causal ...
Data and Analytics - Business Insurance Data Science Executive Director
Columbus, OH · On-site
$204K - $285K/yr
Identify and implement generative AI, agentic AI, and automation opportunities across the business ... Design and implement measurement frameworks and experimentation programs (A/B testing, causal ...
Identify and implement generative AI, agentic AI, and automation opportunities across the business ... Design and implement measurement frameworks and experimentation programs (A/B testing, causal ...
Identify and implement generative AI, agentic AI, and automation opportunities across the business ... Design and implement measurement frameworks and experimentation programs (A/B testing, causal ...
Experience with CI/CD pipelines, Automated Testing, Automated Deployments, Agile methodologies ... Experience with Generative AI Guardrails, responsible AI, adversarial attack mitigation, and red ...
Experience with CI/CD pipelines, Automated Testing, Automated Deployments, Agile methodologies ... Experience with Generative AI Guardrails, responsible AI, adversarial attack mitigation, and red ...
Generative Ai Testing information
What is the difference between Generative Ai Testing vs Data Scientist?
| Aspect | Generative Ai Testing | Data Scientist |
|---|---|---|
| Required Credentials | Knowledge of AI models, testing tools, programming skills | Statistics, programming, data analysis certifications |
| Work Environment | AI development teams, testing labs, tech companies | Research labs, tech firms, finance, healthcare |
| Employer & Industry Usage | AI product testing, quality assurance in tech | Data analysis, predictive modeling across industries |
Generative Ai Testing focuses on evaluating and validating AI-generated content and models, ensuring quality and accuracy. Data Scientists analyze data, build models, and derive insights. While both roles require programming and AI knowledge, Generative Ai Testing emphasizes testing processes, whereas Data Scientists focus on data analysis and model development.
What are the key skills and qualifications needed to thrive as a Generative AI Testing Specialist, and why are they important?
What are some common challenges faced when testing generative AI models, and how can I prepare to address them in this role?
What is Generative AI Testing?

Other
Posted 23 days ago
Deloitte rating
8.1
Based on 86 frontline employees who took The Breakroom Quiz
58th of 138 rated financial services
Job description
We are seeking an AI Governance and Privacy Specialist who can operationalize responsible AI in real systems-especially agentic AI and LLM-enabled applications. This role blends governance and privacy expertise with enough software development fluency to create developer-ready guidance, implement controls-as-code patterns, and stand up measurable evaluation and monitoring workflows.
As a Senior Consultant, you will help clients and internal delivery teams move from AI principles to practices: risk tiering, model and agent inventories, technical guardrails, governance workflows integrated into the SDLC, and evidence artifacts suitable for audits and regulators.
Recruiting for this role ends on 12/31/2026.
Work you'll do
As a Senior Consultant, Strategy, Growth and Transformation on the Cyber team, you will be responsible for:
- Designing and implementing AI governance operating models, intake workflows, risk tiering, approvals, documentation standards, exception handling, and audit-ready evidence processes for generative AI and agentic AI deployments.
- Building and maintaining inventories for models, agents, tools, data sources, and integrations, with defined ownership, intended use, risk classification, and change-control requirements.
- Conducting risk assessments across privacy, security, model risk, and misuse scenarios, including prompt injection, sensitive data exposure, excessive agency, and overreliance, and translating findings into implementable mitigations.
- Establishing technical control guidance for teams building agentic AI solutions, including human-in-the-loop patterns, tool access controls, retrieval and grounding practices, logging, monitoring, token and data minimization, and incident response playbooks.
- Integrating governance checkpoints into product and engineering delivery through architecture reviews, release gates, evaluation requirements, documentation automation, evidence capture, dashboards, and cross-functional collaboration with Cybersecurity, Privacy, Legal, Risk, Engineering, and Data Science teams.
A successful candidate would possess these skills:
- Ability to work independently and collaborate as part of a team
- Effective written and verbal communication skills
- Meticulous attention to detail and quality of work product
- Ability to build and sustain professional relationships
- Ability to lead projects or workstreams
- Ability to manage and prioritize multiple tasks in a fast-paced and dynamic environment
- Strong interpersonal skills and professional demeanor
- Ability to meet deadlines
- Ability to provide clear guidance to others
The team
You will join a cross-functional group working at the intersection of cyber, privacy, governance, and emerging AI delivery. The team helps organizations scale AI responsibly by combining governance and engineering patterns so teams can innovate faster without compromising trust.
Qualifications
Required:
- Bachelor's degree or equivalent practical experience.
- 4+ years of experience in AI governance, data privacy, security risk management, compliance and controls, AI product risk, model risk management, or technology risk consulting.
- Experience translating policies and regulatory expectations into operational workflows and artifacts, including intake processes, inventories, decision logs, risk registers, responsibility assignment matrices, playbooks, privacy impact assessments, and data protection impact assessments.
- Experience assessing AI, machine learning, and LLM deployment patterns, including training, retrieval-augmented generation, fine-tuning, tool use, data dependencies, and integration patterns, and defining mitigations for privacy, security, model risk, and misuse.
- Experience prototyping or automating governance workflows using Python or Structured Query Language and working with continuous integration and continuous deployment pipelines and cloud deployment basics.
- Ability to travel 0-50%, on average, based on the work you do and the clients and industries/sectors you serve.
- Limited immigration sponsorship may be available.
Preferred:
- Experience in consulting or a Big 4 environment.
- Experience operationalizing AI governance aligned to the National Institute of Standards and Technology AI Risk Management Framework or ISO/IEC 42001.
- Experience with generative AI safety and evaluation practices, including prompt injection testing, jailbreak resilience, hallucination measurement, toxicity scoring, harm scoring, and grounding effectiveness.
- Experience with governance, workflow, or ticketing platforms, including OneTrust and governance, risk, and compliance systems, and integrating those platforms into engineering delivery processes.
- Certifications such as Certified Information Privacy Professional/United States, Certified Information Privacy Manager, International Association of Privacy Professionals AI Governance Professional, Certified Information Security Manager, or Certified Information Systems Security Professional.
- Experience in cyber or enterprise security environments, including data security, identity, audit logging, secure software development lifecycle practices, human-in-the-loop escalation pathways, exception handling, and automated safety protocols for autonomous systems.
The wage range for this role takes into account the wide range of factors that are considered in making compensation decisions including but not limited to skill sets; experience and training; licensure and certifications; and other business and organizational needs. The disclosed range estimate has not been adjusted for the applicable geographic differential associated with the location at which the position may be filled. At Deloitte, it is not typical for an individual to be hired at or near the top of the range for their role and compensation decisions are dependent on the facts and circumstances of each case. A reasonable estimate of the current range is $105,400 to $207,800.
You may also be eligible to participate in a discretionary annual incentive program, subject to the rules governing the program, whereby an award, if any, depends on various factors, including, without limitation, individual and organizational performance.
#CyberDTP27
We are seeking an AI Governance and Privacy Specialist who can operationalize responsible AI in real systems-especially agentic AI and LLM-enabled applications. This role blends governance and privacy expertise with enough software development fluency to create developer-ready guidance, implement controls-as-code patterns, and stand up measurable evaluation and monitoring workflows.
As a Senior Consultant, you will help clients and internal delivery teams move from AI principles to practices: risk tiering, model and agent inventories, technical guardrails, governance workflows integrated into the SDLC, and evidence artifacts suitable for audits and regulators.
Recruiting for this role ends on 12/31/2026.
Work you'll do
As a Senior Consultant, Strategy, Growth and Transformation on the Cyber team, you will be responsible for:
- Designing and implementing AI governance operating models, intake workflows, risk tiering, approvals, documentation standards, exception handling, and audit-ready evidence processes for generative AI and agentic AI deployments.
- Building and maintaining inventories for models, agents, tools, data sources, and integrations, with defined ownership, intended use, risk classification, and change-control requirements.
- Conducting risk assessments across privacy, security, model risk, and misuse scenarios, including prompt injection, sensitive data exposure, excessive agency, and overreliance, and translating findings into implementable mitigations.
- Establishing technical control guidance for teams building agentic AI solutions, including human-in-the-loop patterns, tool access controls, retrieval and grounding practices, logging, monitoring, token and data minimization, and incident response playbooks.
- Integrating governance checkpoints into product and engineering delivery through architecture reviews, release gates, evaluation requirements, documentation automation, evidence capture, dashboards, and cross-functional collaboration with Cybersecurity, Privacy, Legal, Risk, Engineering, and Data Science teams.
A successful candidate would possess these skills:
- Ability to work independently and collaborate as part of a team
- Effective written and verbal communication skills
- Meticulous attention to detail and quality of work product
- Ability to build and sustain professional relationships
- Ability to lead projects or workstreams
- Ability to manage and prioritize multiple tasks in a fast-paced and dynamic environment
- Strong interpersonal skills and professional demeanor
- Ability to meet deadlines
- Ability to provide clear guidance to others
The team
You will join a cross-functional group working at the intersection of cyber, privacy, governance, and emerging AI delivery. The team helps organizations scale AI responsibly by combining governance and engineering patterns so teams can innovate faster without compromising trust.
Qualifications
Required:
- Bachelor's degree or equivalent practical experience.
- 4+ years of experience in AI governance, data privacy, security risk management, compliance and controls, AI product risk, model risk management, or technology risk consulting.
- Experience translating policies and regulatory expectations into operational workflows and artifacts, including intake processes, inventories, decision logs, risk registers, responsibility assignment matrices, playbooks, privacy impact assessments, and data protection impact assessments.
- Experience assessing AI, machine learning, and LLM deployment patterns, including training, retrieval-augmented generation, fine-tuning, tool use, data dependencies, and integration patterns, and defining mitigations for privacy, security, model risk, and misuse.
- Experience prototyping or automating governance workflows using Python or Structured Query Language and working with continuous integration and continuous deployment pipelines and cloud deployment basics.
- Ability to travel 0-50%, on average, based on the work you do and the clients and industries/sectors you serve.
- Limited immigration sponsorship may be available.
Preferred:
- Experience in consulting or a Big 4 environment.
- Experience operationalizing AI governance aligned to the National Institute of Standards and Technology AI Risk Management Framework or ISO/IEC 42001.
- Experience with generative AI safety and evaluation practices, including prompt injection testing, jailbreak resilience, hallucination measurement, toxicity scoring, harm scoring, and grounding effectiveness.
- Experience with governance, workflow, or ticketing platforms, including OneTrust and governance, risk, and compliance systems, and integrating those platforms into engineering delivery processes.
- Certifications such as Certified Information Privacy Professional/United States, Certified Information Privacy Manager, International Association of Privacy Professionals AI Governance Professional, Certified Information Security Manager, or Certified Information Systems Security Professional.
- Experience in cyber or enterprise security environments, including data security, identity, audit logging, secure software development lifecycle practices, human-in-the-loop escalation pathways, exception handling, and automated safety protocols for autonomous systems.
The wage range for this role takes into account the wide range of factors that are considered in making compensation decisions including but not limited to skill sets; experience and training; licensure and certifications; and other business and organizational needs. The disclosed range estimate has not been adjusted for the applicable geographic differential associated with the location at which the position may be filled. At Deloitte, it is not typical for an individual to be hired at or near the top of the range for their role and compensation decisions are dependent on the facts and circumstances of each case. A reasonable estimate of the current range is $105,400 to $207,800.
You may also be eligible to participate in a discretionary annual incentive program, subject to the rules governing the program, whereby an award, if any, depends on various factors, including, without limitation, individual and organizational performance.
#CyberDTP27