Maintain consistent judgment across a high volume of short action segments. * Complete assigned ... Preferred * Experience with video annotation, data labeling, computer vision datasets, or ...
Quick apply
Maintain consistent judgment across a high volume of short action segments. * Complete assigned ... Preferred * Experience with video annotation, data labeling, computer vision datasets, or ...
This role leads two tightly coupled but distinct capabilities: (1) governing canonical annotation standards and judgment frameworks, and (2) applying those standards at scale through operational ...
This role leads two tightly coupled but distinct capabilities: (1) governing canonical annotation standards and judgment frameworks, and (2) applying those standards at scale through operational ...
Demonstrated technical judgment in designing or operating annotation systems that support machine learning training, evaluation, or model assessment * Good understanding of annotation systems and ...
Demonstrated technical judgment in designing or operating annotation systems that support machine learning training, evaluation, or model assessment * Good understanding of annotation systems and ...
Team Leader - Document Research
Manhattan, NY · On-site
$135 - $230/hr
Demonstrated technical judgment in designing or operating annotation systems that support machine learning training, evaluation, or model assessment * Good understanding of annotation systems and ...
Team Leader - Document Research
Manhattan, NY · On-site
$135 - $230/hr
Demonstrated technical judgment in designing or operating annotation systems that support machine learning training, evaluation, or model assessment * Good understanding of annotation systems and ...
Maintain consistent judgment across a high volume of short action segments. * Complete assigned ... Preferred * Experience with video annotation, data labeling, computer vision datasets, or ...
Quick apply
Maintain consistent judgment across a high volume of short action segments. * Complete assigned ... Preferred * Experience with video annotation, data labeling, computer vision datasets, or ...
Manufacturing AI Training Specialist -- Task Creation & Model Evaluation (Contract)
Manhattan, NY · On-site
$90 - $140/hr
About Rise Data Labs Rise Data Labs is a leading AI training and data annotation company that ... professional judgment * Participate in calibration sessions with other experts to align scoring ...
Manufacturing AI Training Specialist -- Task Creation & Model Evaluation (Contract)
Manhattan, NY · On-site
$90 - $140/hr
About Rise Data Labs Rise Data Labs is a leading AI training and data annotation company that ... professional judgment * Participate in calibration sessions with other experts to align scoring ...
Video Data Reviewer - Egocentric
New York, NY · Remote
$20 - $25/hr
Maintain consistent judgment across a high volume of short action segments. * Complete assigned ... Preferred * Experience with video annotation, data labeling, computer vision datasets, or ...
Quick apply
Video Data Reviewer - Egocentric
New York, NY · Remote
$20 - $25/hr
Maintain consistent judgment across a high volume of short action segments. * Complete assigned ... Preferred * Experience with video annotation, data labeling, computer vision datasets, or ...
AI/ML Data Contributor
New York, NY · Remote
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Quick apply
AI/ML Data Contributor
New York, NY · Remote
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
AI/ML Data Contributor
New York, NY · Remote
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Quick apply
AI/ML Data Contributor
New York, NY · Remote
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
AI/ML Data Contributor
New York, NY · On-site
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
Quick apply
AI/ML Data Contributor
New York, NY · On-site
Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...
AI Finance Expert - Remote
New York, NY · Remote
$93K - $116K/yr
... judgment are what matter most. Key Responsibilities * Analyze, review, and edit AI-generated ... Data Annotation * Problem-Solving * Independent Research * Attention to Detail Preferred ...
Quick apply
AI Finance Expert - Remote
New York, NY · Remote
$93K - $116K/yr
... judgment are what matter most. Key Responsibilities * Analyze, review, and edit AI-generated ... Data Annotation * Problem-Solving * Independent Research * Attention to Detail Preferred ...
This is a Human-in-the-Loop (HITL) role where your judgment is the product. You won't just be ... Beyond Annotation * Model Evaluation : Critique AI outputs and provide structured, high-level ...
New
Quick apply
This is a Human-in-the-Loop (HITL) role where your judgment is the product. You won't just be ... Beyond Annotation * Model Evaluation : Critique AI outputs and provide structured, high-level ...
New
Strategic Project Lead
Manhattan, NY · On-site
$150 - $200/hr
Annotation protocols, rubrics, and evaluation designs for creative and multimodal outputs ... You understand how humans judge creative work and design systems around that. * You turn messy work ...
Strategic Project Lead
Manhattan, NY · On-site
$150 - $200/hr
Annotation protocols, rubrics, and evaluation designs for creative and multimodal outputs ... You understand how humans judge creative work and design systems around that. * You turn messy work ...
Senior Data Management Professional - Data Product Owner (Data AI)
New York, NY · On-site
$94K - $118K/yr
We provide evaluation and annotation frameworks connecting natural language processing and human judgment in order to elevate the quality, intelligence, and usability of the data that drives our ...
Senior Data Management Professional - Data Product Owner (Data AI)
New York, NY · On-site
$94K - $118K/yr
We provide evaluation and annotation frameworks connecting natural language processing and human judgment in order to elevate the quality, intelligence, and usability of the data that drives our ...
Nursing Expert - Lexidrug Specialist
New York, NY · On-site
$75 - $110/hr
Comfort judging whether a correct nursing finding was properly escalated and documented Preferred * Prior data annotation, labeling, or AI evaluation experience is a plus Application Process (Takes ...
Quick apply
Nursing Expert - Lexidrug Specialist
New York, NY · On-site
$75 - $110/hr
Comfort judging whether a correct nursing finding was properly escalated and documented Preferred * Prior data annotation, labeling, or AI evaluation experience is a plus Application Process (Takes ...
Research Scientist, Human Data & Evaluation
Manhattan, NY · On-site
$150 - $200/hr
Together, we turn human judgment into the training and evaluation layer that enables AI models and ... Design human-data collection strategies, evaluation protocols, rubrics, annotation schemes, and ...
Research Scientist, Human Data & Evaluation
Manhattan, NY · On-site
$150 - $200/hr
Together, we turn human judgment into the training and evaluation layer that enables AI models and ... Design human-data collection strategies, evaluation protocols, rubrics, annotation schemes, and ...
Paralegal
Manhattan, NY · Remote
Provide structured written feedback on complex legal content to support AI training, data annotation, and quality assurance processes. * Evaluate complex legal materials and apply strong judgment and ...
New
Quick apply
Paralegal
Manhattan, NY · Remote
Provide structured written feedback on complex legal content to support AI training, data annotation, and quality assurance processes. * Evaluate complex legal materials and apply strong judgment and ...
New
Annotation Judge information
What is an annotation judge?
An Annotation Judge is a professional who evaluates the quality and accuracy of labeled data, such as text, images, or audio, which has been annotated for use in machine learning and artificial intelligence projects. Their main responsibility is to review, verify, and ensure that the data annotations meet specific guidelines and standards. Annotation Judges play a critical role in improving the reliability of training datasets, which directly impacts the performance of AI systems. They often work closely with data annotators, quality assurance teams, and project managers to maintain high data quality.
What are the key skills and qualifications needed to thrive as an annotation judge, and why are they important?
To thrive as an Annotation Judge, you need strong analytical skills, attention to detail, and subject matter expertise relevant to the data being evaluated, usually supported by a degree in a related field. Familiarity with annotation platforms, data labeling tools, and quality assurance systems is typically required. Excellent communication, impartiality, and critical thinking help you provide clear feedback and maintain high annotation standards. These skills are crucial to ensure data accuracy and consistency, which directly impact the performance of machine learning models.
What are some common challenges faced by annotation judges, and how can they effectively overcome them?
Annotation Judges often face challenges such as maintaining impartiality, handling ambiguous or subjective data, and ensuring high consistency across large volumes of work. To overcome these, it’s essential to follow established guidelines closely, communicate regularly with team members for clarification, and participate in calibration sessions. Staying detail-oriented and seeking feedback can also help maintain accuracy and fairness in their assessments.
What is the difference between Annotation Judge vs Data Annotator?
| Aspect | Annotation Judge | Data Annotator |
|---|---|---|
| Credentials | Typically requires basic education, sometimes certification in data labeling | Usually requires similar or less formal education, often on-the-job training |
| Work Environment | Office or remote, working with data labeling platforms | Office or remote, performing data labeling tasks |
| Industry Usage | Used across AI, machine learning, and data science projects | Common in AI, machine learning, and data preparation workflows |
| Search & Comparison Intent | Often compared for roles involving data review and quality control | Compared for entry-level data labeling roles |
The main difference between an Annotation Judge and a Data Annotator lies in their roles. Annotation Judges typically review and validate annotations made by Data Annotators, ensuring quality and accuracy. Data Annotators perform the initial labeling of data. Both roles are essential in AI data pipelines, with Annotation Judges focusing on quality control and Data Annotators on data preparation.
What job categories do people searching Annotation Judge jobs in Lodi, NJ look for?
The top searched job categories for Annotation Judge jobs in Lodi, NJ are:
What cities near Lodi, NJ are hiring for Annotation Judge jobs?
Cities near Lodi, NJ with the most Annotation Judge job openings:
$25/hr
Full-time
Posted 13 days ago
Job description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
Position: AI Data Generalist — Egocentric Video Review
Type: Contract
Compensation: $20–$25/hour
Location: Remote
Duration: Approximately 2 weeks
Role Responsibilities
- Review egocentric video data and complete structured quality-review tasks.
- Evaluate action segments within videos for accurate start and end boundaries.
- Adjust or flag incorrect boundaries following provided guidelines.
- Maintain consistent judgment across a high volume of short action segments.
- Complete assigned videos within the project timeline.
Qualifications
Must-Have
- Excellent attention to detail and visual comprehension.
- Ability to distinguish closely related actions and moments in video.
- Comfort with structured, repetitive review work while maintaining accuracy.
- Quick learning and consistent application of detailed annotation guidelines.
- Strong written English communication skills.
Preferred
- Experience with video annotation, data labeling, computer vision datasets, or egocentric video.
Start Date
- Monday morning 17th August (PST)
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome
- For any help or support, reach out to: support@mercor.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.