1

Annotation Judge Jobs in Layton, UT (NOW HIRING)

ASL Linguist

Salt Lake City, UT · On-site

$68K - $113K/yr

The role requires strong ASL linguistic judgment, comfort with annotation tools, and the ability to work independently while collaborating closely with other AI/ML developers. Essential Duties and ...

ASL Linguist

Salt Lake City, UT · On-site

$32.84 - $54.57/hr

The role requires strong ASL linguistic judgment, comfort with annotation tools, and the ability to work independently while collaborating closely with other AI/ML developers. Essential Duties and ...

Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...

Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...

Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...

Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...

Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...

Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...

Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...

Perform AI/ML-related tasks such as data labeling, annotation, and content evaluation * Participate ... These tools assist our recruitment team but do not replace human judgment. Final hiring decisions ...

next page

Showing results 1-20

Annotation Judge information

What is an annotation judge?

An Annotation Judge is a professional who evaluates the quality and accuracy of labeled data, such as text, images, or audio, which has been annotated for use in machine learning and artificial intelligence projects. Their main responsibility is to review, verify, and ensure that the data annotations meet specific guidelines and standards. Annotation Judges play a critical role in improving the reliability of training datasets, which directly impacts the performance of AI systems. They often work closely with data annotators, quality assurance teams, and project managers to maintain high data quality.

What are the key skills and qualifications needed to thrive as an annotation judge, and why are they important?

To thrive as an Annotation Judge, you need strong analytical skills, attention to detail, and subject matter expertise relevant to the data being evaluated, usually supported by a degree in a related field. Familiarity with annotation platforms, data labeling tools, and quality assurance systems is typically required. Excellent communication, impartiality, and critical thinking help you provide clear feedback and maintain high annotation standards. These skills are crucial to ensure data accuracy and consistency, which directly impact the performance of machine learning models.

What are some common challenges faced by annotation judges, and how can they effectively overcome them?

Annotation Judges often face challenges such as maintaining impartiality, handling ambiguous or subjective data, and ensuring high consistency across large volumes of work. To overcome these, it’s essential to follow established guidelines closely, communicate regularly with team members for clarification, and participate in calibration sessions. Staying detail-oriented and seeking feedback can also help maintain accuracy and fairness in their assessments.

What is the difference between Annotation Judge vs Data Annotator?

AspectAnnotation JudgeData Annotator
CredentialsTypically requires basic education, sometimes certification in data labelingUsually requires similar or less formal education, often on-the-job training
Work EnvironmentOffice or remote, working with data labeling platformsOffice or remote, performing data labeling tasks
Industry UsageUsed across AI, machine learning, and data science projectsCommon in AI, machine learning, and data preparation workflows
Search & Comparison IntentOften compared for roles involving data review and quality controlCompared for entry-level data labeling roles

The main difference between an Annotation Judge and a Data Annotator lies in their roles. Annotation Judges typically review and validate annotations made by Data Annotators, ensuring quality and accuracy. Data Annotators perform the initial labeling of data. Both roles are essential in AI data pipelines, with Annotation Judges focusing on quality control and Data Annotators on data preparation.

What cities near Layton, UT are hiring for Annotation Judge jobs?

Cities near Layton, UT with the most Annotation Judge job openings:

Infographic showing various Annotation Judge job openings in Layton, UT as of August 2026, with employment types broken down into 67% Full Time, and 33% Part Time. Highlights an 45% In-person, and 55% Remote job distribution.

ASL Linguist

CaptionCall

Salt Lake City, UT • On-site

$68K - $113K/yr

Full-time

Medical, Dental, Vision, Retirement, PTO

Posted 4 days ago


Job description

Come be a part of our mission and make a meaningful and positive impact with the industry leading provider of language services for the Deaf and hard-of-hearing!

Full time Benefits              

  • Paid Vacation Time and Paid Sick Time and Paid Holidays
  • 401k 6% match with immediate vesting
  • Nationwide Medical Insurance plans and coverage (Medical, Dental/Orthodontia, Vision)
    • TeleDoc
    • HSA company match
    • 3 Medical plan options including a Low Deductible PPO Medical Plan Offering
  • Employee Assistance Program
  • Engaged Employee Resource Groups
  • Outstanding Learning and Career Development Opportunities

Pay Range: Actual pay may vary up or down depending on job-related factors which may include knowledge, skills, experience, and location. In addition, this position may be eligible for incentive compensation.

* Applicants must be legally eligible to work in the United States to be considered. Visa sponsorship is not available for this role *

Job Summary

The ASL Linguist is a native ASL professional who applies deep linguistic and cultural expertise to produce high-quality sign language data for AI/ML applications. Linguists own dictionary entries, annotate complex ASL phenomena, review avatars, and design contractor tasks. The role requires strong ASL linguistic judgment, comfort with annotation tools, and the ability to work independently while collaborating closely with other AI/ML developers.

Essential Duties and Responsibilities

  • Maintain and expand Sorenson's ASL sign dictionary — author new entries, review and update existing entries, document regional and register variants, and resolve disputed entries with the Lead.
  • Annotate ASL video using advanced gloss conventions, including classifier predicates (locative, instrument, semantic, body-part, and element classifiers), non-manual markers, role shift, and prosody.
  • Partner with AI/ML engineering teams to clarify linguistic requirements for training data, review model outputs for linguistic accuracy, and provide feedback on edge cases.
  • Review text-to-sign avatar output for ASL grammatical accuracy, naturalness, and cultural appropriateness; document issues using standardized rubrics.
  • Lead weekly linguistic calibration sessions; own gloss and classifier convention documentation and decision logs.
  • Mentor Sign Language Specialists during cross-functional rotations.
  • Maintain accurate records of annotation work, decisions, and edge cases in team tooling and documentation systems.
  • Other duties as assigned.

Supervisory Responsibility

This position has no supervisory responsibilities.

Travel Requirements

Travel Requirements:  Less than 25%

Education

Minimum 4 Year / Bachelors Degree In ASL Linguistics, Deaf Studies, ASL Studies, Interpreting, Deaf Education, or related fields.

Experience

3 Years of substantive professional ASL work (annotation, linguistics, interpreting, Deaf education, or comparable).

Knowledge, Skills, and Abilities

Required

  • Bilingual in ASL in English, with the ability to produce and quality-assure professional English transcriptions from ASL source content.
  • Deep working knowledge of ASL linguistics — gloss conventions, non-manual markers, classifier predicates, role shift, and ASL grammar.
  • Active and informed participant in the Deaf community, with awareness of regional and generational ASL variation.
  • Experience producing or annotating ASL content for instructional, research, interpreting, or media purposes.
  • Strong written English for documentation, task specifications, and cross-team collaboration.
  • Proficiency with standard productivity and collaboration tools (Microsoft Office, Confluence, JIRA, or equivalent).
  • Comfortable working with computer-based annotation tools and learning new software.
  • Attention to detail, consistency, and a willingness to follow and improve standardized conventions.
  • Ability to work both independently and collaboratively with peer Specialists, the Lead, and engineering teams.
  • Professional attitude, team player, strong interpersonal communication skills, and able to work across research, engineering, and business departments.

Preferred

  • American Sign Language as a primary or native language.
  • Master's degree in ASL Linguistics, Linguistics, Deaf Studies, Interpreting, Deaf Education, or a related field.
  • Prior experience with annotation tools such as ELAN, CVAT, or comparable platforms.
  • Experience working with AI/ML teams, technology companies, or research groups that produce sign language data.
  • Experience reviewing or evaluating signed avatars, animations, or generated ASL content.
  • Experience writing or reviewing annotation guidelines, style guides, or convention documentation.
  • Familiarity with sign language corpora, lexicography, or dictionary projects.
  • Experience mentoring or training other annotators or linguists.
  • Familiarity with PSE, SEE, and other North American signing varieties.

Company Summary

Our Mission…Harnessing the power of language, we connect diverse people and enrich the human experience.

 Our Vision…To provide global language services that expand opportunities, nurture belonging, and empower the world to connect beyond words.

As one of the world’s leading language services providers, Sorenson combines patented technology with human-centric solutions. We strive to increase accessibility and inclusion through communication solutions for all: call captioning and video relay services, over-video and in-person sign language and spoken language interpreting, translation, real-time captioning, and post-production language services. Sorenson’s impact vision and plan extends to enhancing generational wealth and inclusive workplaces for our employees and the communities we serve.

We achieve great things together working “The Sorenson Way” with our employee values: Customer First, Can-Do Attitude, Collective Action, Growth Mindset, Ownership, and Connect Direct.

Equal Employment Opportunity:
Sorenson Communications is an Equal Opportunity, Affirmative Action Employer.