This role leads two tightly coupled but distinct capabilities: (1) governing canonical annotation standards and judgment frameworks, and (2) applying those standards at scale through operational ...
This role leads two tightly coupled but distinct capabilities: (1) governing canonical annotation standards and judgment frameworks, and (2) applying those standards at scale through operational ...
Senior AI Data Operations Analyst
Manhattan, NY · On-site +1
$47 - $51/hr
Execute & Champion Data Annotation: Perform hands-on data annotations and lead larger, cross ... Design and execute qualitative testing using internal tools, keeping human judgment at the center ...
Senior AI Data Operations Analyst
Manhattan, NY · On-site +1
$47 - $51/hr
Execute & Champion Data Annotation: Perform hands-on data annotations and lead larger, cross ... Design and execute qualitative testing using internal tools, keeping human judgment at the center ...
Demonstrated technical judgment in designing or operating annotation systems that support machine learning training, evaluation, or model assessment * Good understanding of annotation systems and ...
Demonstrated technical judgment in designing or operating annotation systems that support machine learning training, evaluation, or model assessment * Good understanding of annotation systems and ...
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
AI/ML Data Contributor
New York, NY · Remote
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Quick apply
AI/ML Data Contributor
New York, NY · Remote
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Quick apply
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
AI/ML Data Contributor
New York, NY · Remote
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Quick apply
AI/ML Data Contributor
New York, NY · Remote
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
... annotation work * Experience writing or editing high-quality written content * Experience comparing multiple outputs and making fine-grained qualitative judgments Application Process (Takes 20-30 ...
Quick apply
... annotation work * Experience writing or editing high-quality written content * Experience comparing multiple outputs and making fine-grained qualitative judgments Application Process (Takes 20-30 ...
Bilingual Evaluator - Odia Specialist
New York, NY · Remote
$15 - $20/hr
... data annotation work . * Experience writing or editing high-quality written content . * Experience comparing multiple outputs and making fine-grained qualitative judgments . Application Process ...
Quick apply
Bilingual Evaluator - Odia Specialist
New York, NY · Remote
$15 - $20/hr
... data annotation work . * Experience writing or editing high-quality written content . * Experience comparing multiple outputs and making fine-grained qualitative judgments . Application Process ...
Bilingual Evaluator - LLM Specialist
New York, NY · Remote
$15 - $20/hr
... annotation work * Experience writing or editing high-quality written content * Experience comparing multiple outputs and making fine-grained qualitative judgments Application Process (Takes 20-30 ...
Quick apply
Bilingual Evaluator - LLM Specialist
New York, NY · Remote
$15 - $20/hr
... annotation work * Experience writing or editing high-quality written content * Experience comparing multiple outputs and making fine-grained qualitative judgments Application Process (Takes 20-30 ...
Arabic Data Labeling Analyst(Speech & Voice )
New York, NY · On-site
$26 - $28/hr
... judgment, attention to detail, and consistency . The work sits at the intersection of language ... Execute high-volume data labeling and annotation tasks across speech and voice datasets * Follow ...
Quick apply
Arabic Data Labeling Analyst(Speech & Voice )
New York, NY · On-site
$26 - $28/hr
... judgment, attention to detail, and consistency . The work sits at the intersection of language ... Execute high-volume data labeling and annotation tasks across speech and voice datasets * Follow ...
Turkish Data Labeling Analyst(Speech & Voice )
New York, NY · On-site
$26 - $28/hr
... judgment, attention to detail, and consistency . The work sits at the intersection of language ... Execute high-volume data labeling and annotation tasks across speech and voice datasets * Follow ...
Quick apply
Turkish Data Labeling Analyst(Speech & Voice )
New York, NY · On-site
$26 - $28/hr
... judgment, attention to detail, and consistency . The work sits at the intersection of language ... Execute high-volume data labeling and annotation tasks across speech and voice datasets * Follow ...
Clinical Informatics Specialist
New York, NY · On-site
$75K - $100K/yr
The Clinical Specialist will play a critical role in data generation, annotation, and evaluation to ... Strong clinical judgment and attention to detail, with the ability to evaluate medication-related ...
Clinical Informatics Specialist
New York, NY · On-site
$75K - $100K/yr
The Clinical Specialist will play a critical role in data generation, annotation, and evaluation to ... Strong clinical judgment and attention to detail, with the ability to evaluate medication-related ...
Product Content Engineer
New York, NY · On-site +1
$162K/yr
... judgment to content quality decisions, we encourage you to apply. Product Content Engineer ... Experience designing and implementing evaluation frameworks, annotation guidelines, or quality ...
Product Content Engineer
New York, NY · On-site +1
$162K/yr
... judgment to content quality decisions, we encourage you to apply. Product Content Engineer ... Experience designing and implementing evaluation frameworks, annotation guidelines, or quality ...
Staff Engineer - Content Intelligence Infrastructure
New York, NY · On-site
$203K - $290K/yr
Our work powers content enrichment, quality analysis, annotation platforms, and scalable ... You have strong technical judgment and can balance scalability, reliability, cost, and product ...
Staff Engineer - Content Intelligence Infrastructure
New York, NY · On-site
$203K - $290K/yr
Our work powers content enrichment, quality analysis, annotation platforms, and scalable ... You have strong technical judgment and can balance scalability, reliability, cost, and product ...
Staff Engineer - Content Intelligence Infrastructure
$203K - $290K/yr
Our work powers content enrichment, quality analysis, annotation platforms, and scalable ... You have strong technical judgment and can balance scalability, reliability, cost, and product ...
Staff Engineer - Content Intelligence Infrastructure
$203K - $290K/yr
Our work powers content enrichment, quality analysis, annotation platforms, and scalable ... You have strong technical judgment and can balance scalability, reliability, cost, and product ...
Driver Behavior Analyst
Manhattan, NY · On-site
$73K - $83K/yr
Behavioral Annotation: Analyze video and sensor data from US roads to label driver intent ... Evaluate "near miss" or ambiguous scenarios to provide expert judgment on what a "Safe Human Driver ...
Driver Behavior Analyst
Manhattan, NY · On-site
$73K - $83K/yr
Behavioral Annotation: Analyze video and sensor data from US roads to label driver intent ... Evaluate "near miss" or ambiguous scenarios to provide expert judgment on what a "Safe Human Driver ...
... judgment into calibration signals the system can act on reliably. * You will architect and build ... Experience building knowledge acquisition workflows for domain-specific AI - annotation interfaces ...
... judgment into calibration signals the system can act on reliably. * You will architect and build ... Experience building knowledge acquisition workflows for domain-specific AI - annotation interfaces ...
Annotation Judge information
What is an Annotation Judge?
An Annotation Judge is a professional who evaluates the quality and accuracy of labeled data, such as text, images, or audio, which has been annotated for use in machine learning and artificial intelligence projects. Their main responsibility is to review, verify, and ensure that the data annotations meet specific guidelines and standards. Annotation Judges play a critical role in improving the reliability of training datasets, which directly impacts the performance of AI systems. They often work closely with data annotators, quality assurance teams, and project managers to maintain high data quality.
What are the key skills and qualifications needed to thrive as an Annotation Judge, and why are they important?
To thrive as an Annotation Judge, you need strong analytical skills, attention to detail, and subject matter expertise relevant to the data being evaluated, usually supported by a degree in a related field. Familiarity with annotation platforms, data labeling tools, and quality assurance systems is typically required. Excellent communication, impartiality, and critical thinking help you provide clear feedback and maintain high annotation standards. These skills are crucial to ensure data accuracy and consistency, which directly impact the performance of machine learning models.
What is the difference between Annotation Judge vs Data Annotator?
| Aspect | Annotation Judge | Data Annotator |
|---|---|---|
| Credentials | Typically requires basic education, sometimes certification in data labeling | Usually requires similar or less formal education, often on-the-job training |
| Work Environment | Office or remote, working with data labeling platforms | Office or remote, performing data labeling tasks |
| Industry Usage | Used across AI, machine learning, and data science projects | Common in AI, machine learning, and data preparation workflows |
| Search & Comparison Intent | Often compared for roles involving data review and quality control | Compared for entry-level data labeling roles |
The main difference between an Annotation Judge and a Data Annotator lies in their roles. Annotation Judges typically review and validate annotations made by Data Annotators, ensuring quality and accuracy. Data Annotators perform the initial labeling of data. Both roles are essential in AI data pipelines, with Annotation Judges focusing on quality control and Data Annotators on data preparation.
What are some common challenges faced by Annotation Judges, and how can they effectively overcome them?
Annotation Judges often face challenges such as maintaining impartiality, handling ambiguous or subjective data, and ensuring high consistency across large volumes of work. To overcome these, it’s essential to follow established guidelines closely, communicate regularly with team members for clarification, and participate in calibration sessions. Staying detail-oriented and seeking feedback can also help maintain accuracy and fairness in their assessments.
What are popular job titles related to Annotation Judge jobs in New York? For Annotation Judge jobs in New York, the most frequently searched job titles are:
What job categories do people searching Annotation Judge jobs in New York look for? The top searched job categories for Annotation Judge jobs in New York are:
What cities in New York are hiring for Annotation Judge jobs? Cities in New York with the most Annotation Judge job openings:
Full-time
Medical, Dental, Vision, Life, Retirement, PTO
Re-posted 16 days ago
Bloomberg rating
9.4
Based on 11 frontline employees who took The Breakroom Quiz
12th of 241 rated software companies
Job description
Team Leader - Annotations Operations and Governance
Location
New York
Business Area
Data
Ref #
10049470
Description & Requirements
Bloomberg runs on data. Our products are fueled by powerful information. We combine data and context to paint the whole picture for our clients, around the clock - from around the world. In Data, we are responsible for delivering this data, news and analytics through innovative technology - quickly and accurately. We apply problem-solving skills to identify innovative workflow efficiencies, and we implement technology solutions to enhance our systems, products and processes - all while providing customer support to our clients.
Our team:
The Bloomberg Data AI group brings innovative AI technologies into Bloomberg's Data organization while contributing deep financial domain expertise to the development of AI-powered products. We partner closely with stakeholders to align AI innovation with Bloomberg's strategic objectives, focusing on optimizing data workflows and elevating the quality, intelligence, and usability of the data that drives our products. Our work amplifies the impact of the Data organization by delivering intelligent data solutions and domain-informed systems that enhance the capabilities and competitiveness of Bloomberg's offerings.
What's the role?
We are seeking a Team Lead to own and scale Bloomberg's annotation function across AI and data initiatives, with responsibility for Annotation Operations and Annotation Standards & Governance. This role leads two tightly coupled but distinct capabilities: (1) governing canonical annotation standards and judgment frameworks, and (2) applying those standards at scale through operational execution, quality control, and continuous improvement.
The Team Lead will operate annotation as a shared service across our Data AI teams. They will ensure that centrally defined standards: schemas, labels, ambiguity frameworks, and calibration rules, are consistently and correctly applied in production through SME-driven workflows, vendor execution, and robust operational controls.
The ideal candidate is a technically grounded, systems-oriented leader who understands how annotated data shapes AI model behavior and evaluation outcomes. You are comfortable operating at scale while exercising strong technical judgment, enforcing standards, interpreting ambiguity, and using metrics to detect drift, diagnose failure modes, and continuously improve data quality.
In this role, you will help build a durable, repeatable annotation capability that produces correct, consistent, and reproducible data over time, supporting training, evaluation, monitoring, and production AI systems.
We'll trust you to:
You'll need to have:
We'd love to see:
Salary Range = 135,000 - 230,000 USD Annual + Benefits + Bonus
The referenced salary range is based on the Company's good faith belief at the time of posting. Actual compensation may vary based on factors such as geographic location, work experience, market conditions, education/training and skill level.
We offer one of the most comprehensive and generous benefits plans available and offer a range of total rewards that may include merit increases, incentive compensation (exempt roles only), paid holidays, paid time off, medical, dental, vision, short and long term disability benefits, 401(k) +match, life insurance, and various wellness programs, among others. The Company does not provide benefits directly to contingent workers/contractors and interns.
Discover what makes Bloomberg unique - watch our podcast series for an inside look at our culture, values, and the people behind our success.
Location
New York
Business Area
Data
Ref #
10049470
Description & Requirements
Bloomberg runs on data. Our products are fueled by powerful information. We combine data and context to paint the whole picture for our clients, around the clock - from around the world. In Data, we are responsible for delivering this data, news and analytics through innovative technology - quickly and accurately. We apply problem-solving skills to identify innovative workflow efficiencies, and we implement technology solutions to enhance our systems, products and processes - all while providing customer support to our clients.
Our team:
The Bloomberg Data AI group brings innovative AI technologies into Bloomberg's Data organization while contributing deep financial domain expertise to the development of AI-powered products. We partner closely with stakeholders to align AI innovation with Bloomberg's strategic objectives, focusing on optimizing data workflows and elevating the quality, intelligence, and usability of the data that drives our products. Our work amplifies the impact of the Data organization by delivering intelligent data solutions and domain-informed systems that enhance the capabilities and competitiveness of Bloomberg's offerings.
What's the role?
We are seeking a Team Lead to own and scale Bloomberg's annotation function across AI and data initiatives, with responsibility for Annotation Operations and Annotation Standards & Governance. This role leads two tightly coupled but distinct capabilities: (1) governing canonical annotation standards and judgment frameworks, and (2) applying those standards at scale through operational execution, quality control, and continuous improvement.
The Team Lead will operate annotation as a shared service across our Data AI teams. They will ensure that centrally defined standards: schemas, labels, ambiguity frameworks, and calibration rules, are consistently and correctly applied in production through SME-driven workflows, vendor execution, and robust operational controls.
The ideal candidate is a technically grounded, systems-oriented leader who understands how annotated data shapes AI model behavior and evaluation outcomes. You are comfortable operating at scale while exercising strong technical judgment, enforcing standards, interpreting ambiguity, and using metrics to detect drift, diagnose failure modes, and continuously improve data quality.
In this role, you will help build a durable, repeatable annotation capability that produces correct, consistent, and reproducible data over time, supporting training, evaluation, monitoring, and production AI systems.
We'll trust you to:
- Lead and develop a team responsible for operating and governing Bloomberg's AI data annotation capability, delivering consistent, high-quality data through strong technical judgment and clear standards.
- Own annotation operations end-to-end, translating schemas and judgment frameworks into scalable workflows with measurable quality.
- Establish governance and quality mechanisms, including calibration, agreement analysis, and drift detection, that ensure consistent interpretation of standards.
- Partner closely with AI, Data, and Platform teams to align annotation outputs with production needs and downstream model requirements.
- Ensure operational strength at scale, including workforce strategy, vendor oversight, capacity planning, and service reliability.
- Drive continuous improvement through metrics, feedback loops, and root-cause analysis.
- Act as steward of judgment integrity, maintaining high agreement and durable decision-making frameworks as models and domains evolve.
You'll need to have:
- Prior people leadership experience, including leading operational or program-focused teams.
- Demonstrated technical judgment in designing or operating annotation systems that support machine learning training, evaluation, or model assessment.
- Strong understanding of annotation systems and quality methodologies, including calibration, agreement modeling, and drift detection.
- Proven experience running large-scale annotation or data operations with vendor and SME workforces.
- Ability to enforce centrally defined standards while maintaining consistency at scale.
- Excellent cross-functional leadership skills and comfort operating in ambiguity-rich environments.
- Strong program leadership capability, with a focus on measurable outcomes and continuous improvement.
- Bachelor's or Master's degree in a relevant field, or equivalent practical experience.
We'd love to see:
- Experience supporting ML training, evaluation, or monitoring pipelines.
- Familiarity with annotation platforms, QA tooling, and data instrumentation.
Salary Range = 135,000 - 230,000 USD Annual + Benefits + Bonus
The referenced salary range is based on the Company's good faith belief at the time of posting. Actual compensation may vary based on factors such as geographic location, work experience, market conditions, education/training and skill level.
We offer one of the most comprehensive and generous benefits plans available and offer a range of total rewards that may include merit increases, incentive compensation (exempt roles only), paid holidays, paid time off, medical, dental, vision, short and long term disability benefits, 401(k) +match, life insurance, and various wellness programs, among others. The Company does not provide benefits directly to contingent workers/contractors and interns.
Discover what makes Bloomberg unique - watch our podcast series for an inside look at our culture, values, and the people behind our success.
What Bloomberg employees say
Pay
Benefits
Hours and flexibility
Workplace
Get the full story on Breakroom
About Bloomberg
Sourced by ZipRecruiter
Bloomberg runs on data. As the Data Management & Analytics team within Engineering, we support our organization's needs around managing data efficiently. The vision of the team is to build solutions that drive data quality, data dictionary, data stewardship, data lineage, reference, and master data management across various data domains (prospect, customer, vendor, material etc.). We partner with business teams across the organization in addressing their data needs and ultimately helping run business operations efficiently and make improved decisions.
Industry
Finance and insurance
Company size
10,000+ Employees
Headquarters location
New York, NY, US
Year founded
1981