ZipRecruiter

Log In

1

Llm Annotation Jobs in Seattle, WA (NOW HIRING)

Oracle

Senior Applied Scientist

Seattle, WA · On-site

$104K - $142K/yr

Experience developing or calibrating LLM-/VLM-as-a-judge methods against human judgments. * Experience designing human-evaluation programs, including annotation rubrics, sampling plans, gold datasets ...

Amazon

Sr. Software Dev Engineer, SageMaker AI

Seattle, WA · On-site

$139K - $183K/yr

Build and scale systems leveraging auto-labeling and LLM-as-Judge techniques to automatically ... annotation quality assurance, and ground truth generation. 5. Technical Leadership: Mentor ...

Amazon

Sr. Software Dev Engineer, SageMaker AI

Seattle, WA · On-site

$139K - $183K/yr

Build and scale systems leveraging auto-labeling and LLM-as-Judge techniques to automatically ... annotation quality assurance, and ground truth generation. 5. Technical Leadership: Mentor ...

Amazon

Sr. Software Dev Engineer, SageMaker AI

Seattle, WA · On-site

$139K - $183K/yr

Build and scale systems leveraging auto-labeling and LLM-as-Judge techniques to automatically ... annotation quality assurance, and ground truth generation. 5. Technical Leadership: Mentor ...

Amazon

Sr. Software Dev Engineer, SageMaker AI

Seattle, WA · On-site

$139K - $183K/yr

Build and scale systems leveraging auto-labeling and LLM-as-Judge techniques to automatically ... annotation quality assurance, and ground truth generation. 5. Technical Leadership: Mentor ...

Amazon

Sr. Software Dev Engineer, SageMaker AI

Seattle, WA · On-site

$139K - $183K/yr

Build and scale systems leveraging auto-labeling and LLM-as-Judge techniques to automatically ... annotation quality assurance, and ground truth generation. 5. Technical Leadership: Mentor ...

Amazon

Sr. Software Dev Engineer, SageMaker AI

Seattle, WA · On-site

$139K - $183K/yr

Build and scale systems leveraging auto-labeling and LLM-as-Judge techniques to automatically ... annotation quality assurance, and ground truth generation. 5. Technical Leadership: Mentor ...

Amazon

Sr. Software Dev Engineer, SageMaker AI

Seattle, WA · On-site

$139K - $183K/yr

Build and scale systems leveraging auto-labeling and LLM-as-Judge techniques to automatically ... annotation quality assurance, and ground truth generation. 5. Technical Leadership: Mentor ...

Amazon

Sr. Software Dev Engineer, SageMaker AI

Seattle, WA · On-site

$139K - $183K/yr

Build and scale systems leveraging auto-labeling and LLM-as-Judge techniques to automatically ... annotation quality assurance, and ground truth generation. 5. Technical Leadership: Mentor ...

previous pagenext page

Showing results 21-40

All JobsLlm Annotation JobsLlm Annotation Jobs in Seattle, WA

Llm Annotation information

See Seattle, WA salary details

$12.5K

$47.2K

How much do llm annotation jobs pay per year?

As of Aug 23, 2026, the average yearly pay for llm annotation in Seattle, WA is $45,521.00, according to ZipRecruiter salary data. Most workers in this role earn between $45,500.00 and $45,500.00 per year, depending on experience, location, and employer.

What is LLM annotation?

LLM annotation refers to the process of labeling or tagging data specifically for training and evaluating large language models (LLMs) like GPT or BERT. Annotators read text and apply labels, correct errors, or provide feedback to help improve the model's understanding and performance. This work is crucial for supervised learning, as well-annotated datasets help LLMs better recognize patterns, context, and meaning in human language. LLM annotation can involve tasks such as sentiment analysis, named entity recognition, or instruction following. Annotators often use specialized platforms or tools to complete their tasks efficiently and accurately.

What are the key skills and qualifications needed to thrive as an LLM annotation specialist?

To thrive as an LLM Annotation Specialist, you need strong analytical skills, attention to detail, and a background in linguistics, computer science, or a related field. Familiarity with annotation platforms, natural language processing (NLP) tools, and data labeling systems is typically required. Excellent communication, critical thinking, and the ability to follow guidelines precisely are valuable soft skills for this role. These skills ensure high-quality, accurate data annotation, which directly impacts the performance and reliability of large language models.

What are some common challenges faced by LLM annotation specialists, and how can they be addressed?

LLM Annotation specialists often encounter challenges such as interpreting ambiguous language data, maintaining annotation consistency across complex datasets, and keeping up with evolving guidelines. These can be addressed by participating in regular team syncs to clarify guidelines, using annotation tools with built-in quality checks, and collaborating closely with project leads and fellow annotators. Continuous learning and open communication help ensure high-quality, reliable data annotation and support professional growth within the AI and NLP fields.

What is the difference between Llm Annotation vs Data Labeler?

AspectLlm AnnotationData Labeler
Required CredentialsBasic computer skills, sometimes familiarity with AI toolsBasic skills, often on-the-job training
Work EnvironmentRemote or office-based, tech-focusedRemote or on-site, varied industries
Industry UsageAI, machine learning, NLP projectsVarious industries including marketing, healthcare, and tech
Search & Comparison IntentUnderstanding roles in AI data preparationGeneral data labeling tasks

In summary, Llm Annotation involves specialized annotation for large language models, often requiring familiarity with AI tools, while Data Labeler is a broader role focused on labeling data across multiple industries with minimal technical requirements.

How to become an Llm annotator?

To become an LLM annotator, candidates typically need strong language skills, attention to detail, and familiarity with data annotation tools. Many positions require a high school diploma or equivalent, and some companies provide training. Experience with machine learning or natural language processing can be beneficial but is not always necessary.

What are popular job titles related to Llm Annotation jobs in Seattle, WA?

For Llm Annotation jobs in Seattle, WA, the most frequently searched job titles are:

What job categories do people searching Llm Annotation jobs in Seattle, WA look for?

The top searched job categories for Llm Annotation jobs in Seattle, WA are:

What cities near Seattle, WA are hiring for Llm Annotation jobs?

Cities near Seattle, WA with the most Llm Annotation job openings:

Llm Annotation jobs near you

QA Reviewer & Project Coordinator (On-Site - Seattle or Boston)

mpathic

Seattle, WA • On-site

$110K - $125K/yr

Full-time

Posted 13 days ago

Job description

About the Role
We're seeking an experienced QA Reviewer & Project Coordinator. This is a hands-on role that combines expert evaluation and quality assurance with day-to-day project coordination. You'll spend most of your time reviewing work, testing AI systems, and ensuring high-quality outputs while also helping keep projects organized, on schedule, and running smoothly.
Position Overview
This position is ideal for someone who enjoys both deep subject matter expertise and operational execution.
You'll serve as one of the primary reviewers for AI safety work, conducting quality reviews, identifying behavioral edge cases, testing AI models through role play and red teaming, and ensuring consistency across projects. In addition, you'll coordinate portions of active programs by tracking deliverables, supporting staffing and project workflows, communicating with stakeholders, and helping identify operational risks before they become issues.
The ideal candidate enjoys solving ambiguous problems, has exceptional attention to detail, communicates clearly, and thrives in a fast-moving startup environment.
What You'll Do
QA & AI Evaluation
  • Review AI content for accuracy, safety, empathy, and policy compliance.
  • Conduct quality assurance reviews of expert evaluations and annotations.
  • Participate in calibration sessions to ensure reviewer consistency and quality.
  • Help improve QA workflows, reviewer documentation, and operational processes.
  • Role play realistic clinical scenarios with AI systems to evaluate behavior across diverse situations.
  • Perform red teaming to identify failure modes, safety risks, and behavioral edge cases.
  • Develop and refine evaluation rubrics, behavioral taxonomies, personas, and scoring guidelines.
  • Document model inconsistencies, safety concerns, and opportunities for improvement.
  • Collaborate with researchers and engineers to improve AI behavior through structured clinical feedback.
  • Maintain strict confidentiality while working with sensitive clinical content.

Project Coordination
  • Support day-to-day execution of AI safety and evaluation projects.
  • Track project timelines, deliverables, and reviewer assignments.
  • Coordinate review queues and help balance workloads across project teams.
  • Monitor project progress and proactively identify risks, blockers, or quality issues.
  • Maintain project trackers, documentation, and reporting dashboards.
  • Assist with staffing coordination as project needs evolve.
  • Help facilitate project meetings and document action items and follow-up tasks.

Required Qualifications
  • Familiarity with ChatGPT, Claude, Gemini, or other large language models.
  • Excellent written communication and documentation skills.
  • Strong organizational skills with exceptional attention to detail.
  • Comfortable managing multiple priorities simultaneously.
  • Ability to work independently while collaborating effectively across teams.
  • Comfortable working in a fast-paced startup environment with evolving priorities.

Preferred Qualifications
  • Experience reviewing or auditing clinical work for quality.
  • Experience with AI safety, LLM evaluation, prompt engineering, or red teaming.
  • Background in trust & safety, content moderation, or behavioral policy development.
  • Experience developing evaluation rubrics, taxonomies, or annotation guidelines.
  • Familiarity with data annotation or human-in-the-loop evaluation workflows.
  • Clinical experience working with serious mental illness, crisis intervention, or complex behavioral health populations.
  • Project coordination or project management experience.
  • Experience leading calibration sessions or reviewer training.
  • Experience with Google Workspace, Slack, spreadsheets, and project management tools.

You'll Thrive Here If You...
  • Notice details that others miss.
  • Can identify quality issues before they become larger problems.
  • Enjoy both analytical review work and coordinating people and projects.
  • Communicate clearly across clinical and operational teams.
  • Balance independent problem solving with knowing when to escalate.
  • Thrive in ambiguity and rapidly changing environments.
  • Care deeply about building AI systems that are safe, trustworthy, and clinically responsible.
  • Take ownership and consistently follow through.

Additional Requirements
  • Ability to work on-site at our designated office location (Seattle or Boston).
  • Willingness to sign comprehensive confidentiality and NDA agreements.
  • Comfortable working with sensitive mental health and AI safety content.
  • Participation in recurring project meetings, calibration sessions, and operational planning.
  • Availability to support occasional high-priority project deadlines as needed, including potential evening and weekend work.

Apply Even If You Don't Check Every Box
We know great candidates might not fit every bullet on a job description. If this role speaks to you and you're excited to help improve the future of healthcare research and AI safety, we'd love to hear from you.

Apply