ZipRecruiter

Log In

1

Llm Annotation Jobs in New York (NOW HIRING)

ESR Healthcare

LLM Red-Teamer

Manhattan, NY · On-site

LLM Red-Teamer Role Type: Contractor Location: Remote Scope of Work Develop complex, adversarial ... Prior experience in AI human data environments (RLHF, SFT, evaluations, annotation, or prompt ...

Welocalize, Inc.

Data Quality Analyst

New York, NY · On-site

The ideal candidate will have a foundational understanding of machine learning, data annotation ... Deliver detailed reports on findings, including aspects such as utterance quality, LLM evaluation ...

Welocalize, Inc.

Data Quality Analyst

New York, NY · On-site

The ideal candidate will have a foundational understanding of machine learning, data annotation ... Deliver detailed reports on findings, including aspects such as utterance quality, LLM evaluation ...

Triomics

Forward Deployed ML Engineer

New York, NY · On-site

$170K - $190K/yr

Work with the clinical annotation team to build gold-standard datasets and resolve edge cases ... Practical LLM experience: prompt engineering, fine-tuning, RAG, evaluation design * Built ...

HumanSignal

Delivery Lead

New York, NY · Remote

$110K - $140K/yr

... LLM assistance Preferred Qualifications * 1+ year in AI data operations (RLHF, annotation, model evaluation) * STEM background or strong technical fluency * Python & REACT working knowledge

triomics inc.

Forward Deployed ML Engineer

Manhattan, NY · On-site

$100 - $125/hr

Work with the clinical annotation team to build gold‑standard datasets and resolve edge cases ... Built and deployed AI agents or multi‑step LLM pipelines (not just single‑call wrappers) - you ...

Pelago

Medical Director, Clinical AI

New York, NY · Hybrid

$87K - $119K/yr

Oversee clinical operations of the AI arm, including annotation strategy, safety protocols ... LLM capabilities and failure modes in a clinical context * Exceptional communicator - able to ...

TransPerfect

BD Director-AI data Solutions

Manhattan, NY · On-site

$150 - $200/hr

... annotation, and generative AI services--to Fortune 500 leaders. TransPerfect AI offers a premier product suite that addresses the most critical bottlenecks in the AI lifecycle, from LLM fine-tuning ...

Meta

Product Content Engineer

New York, NY

$193K/yr

Drive sprints for LLM model evaluations and auto judge creation in core PCE areas, while devising ... engineering, annotation, and/or content labeling and analysis * Expertise in developing and ...

next page

Showing results 1-20

All JobsLlm Annotation JobsLlm Annotation Jobs in New York

Llm Annotation information

What is LLM annotation?

LLM annotation refers to the process of labeling or tagging data specifically for training and evaluating large language models (LLMs) like GPT or BERT. Annotators read text and apply labels, correct errors, or provide feedback to help improve the model's understanding and performance. This work is crucial for supervised learning, as well-annotated datasets help LLMs better recognize patterns, context, and meaning in human language. LLM annotation can involve tasks such as sentiment analysis, named entity recognition, or instruction following. Annotators often use specialized platforms or tools to complete their tasks efficiently and accurately.

What are the key skills and qualifications needed to thrive as an LLM annotation specialist?

To thrive as an LLM Annotation Specialist, you need strong analytical skills, attention to detail, and a background in linguistics, computer science, or a related field. Familiarity with annotation platforms, natural language processing (NLP) tools, and data labeling systems is typically required. Excellent communication, critical thinking, and the ability to follow guidelines precisely are valuable soft skills for this role. These skills ensure high-quality, accurate data annotation, which directly impacts the performance and reliability of large language models.

What are some common challenges faced by LLM annotation specialists, and how can they be addressed?

LLM Annotation specialists often encounter challenges such as interpreting ambiguous language data, maintaining annotation consistency across complex datasets, and keeping up with evolving guidelines. These can be addressed by participating in regular team syncs to clarify guidelines, using annotation tools with built-in quality checks, and collaborating closely with project leads and fellow annotators. Continuous learning and open communication help ensure high-quality, reliable data annotation and support professional growth within the AI and NLP fields.

What is the difference between Llm Annotation vs Data Labeler?

AspectLlm AnnotationData Labeler
Required CredentialsBasic computer skills, sometimes familiarity with AI toolsBasic skills, often on-the-job training
Work EnvironmentRemote or office-based, tech-focusedRemote or on-site, varied industries
Industry UsageAI, machine learning, NLP projectsVarious industries including marketing, healthcare, and tech
Search & Comparison IntentUnderstanding roles in AI data preparationGeneral data labeling tasks

In summary, Llm Annotation involves specialized annotation for large language models, often requiring familiarity with AI tools, while Data Labeler is a broader role focused on labeling data across multiple industries with minimal technical requirements.

How to become an Llm annotator?

To become an LLM annotator, candidates typically need strong language skills, attention to detail, and familiarity with data annotation tools. Many positions require a high school diploma or equivalent, and some companies provide training. Experience with machine learning or natural language processing can be beneficial but is not always necessary.

What are popular job titles related to Llm Annotation jobs in New York?

For Llm Annotation jobs in New York, the most frequently searched job titles are:

What job categories do people searching Llm Annotation jobs in New York look for?

The top searched job categories for Llm Annotation jobs in New York are:

What cities in New York are hiring for Llm Annotation jobs?

Cities in New York with the most Llm Annotation job openings:

Llm Annotation jobs near you
Infographic showing various Llm Annotation job openings in New York as of August 2026, with employment types broken down into 92% Full Time, 5% Part Time, and 3% Contract. Highlights an 80% Physical, 5% Hybrid, and 15% Remote job distribution.

LLM Red-Teamer

ESR Healthcare

Manhattan, NY • On-site

Other

Re-posted 5 days ago

Job description


LLM Red-Teamer
Role Type: Contractor
Location: Remote
Scope of Work
Develop complex, adversarial multi-turn conversations and task-based scenarios aligned with detailed project specifications.
Author clear, precise evaluation rubrics to rigorously assess model responses against defined behavioral targets.
Iteratively test conversations and tasks against frontier LLMs, escalating difficulty and nuance until the desired quality threshold is achieved.
Deliver comprehensive task packages, including transcripts, target behaviors, binary rubrics, and supporting rationale or evidence.
Validate LLM outputs, documenting model strengths and failure modes relative to the project specification.
Maintain calibration with team leads and quality control contacts as project requirements evolve.
Contribute independently, producing high-quality deliverables at a steady and consistent pace.
Preferred Qualifications
Exceptional written English skills, with clarity, precision, and strong structural organization.
Prior experience in AI human data environments (RLHF, SFT, evaluations, annotation, or prompt engineering).
Deep familiarity with large language models, including the ability to anticipate and identify common failure patterns.
Demonstrated ability to work autonomously, interpreting and executing complex specifications with minimal oversight.
Proven critical thinking and meticulous attention to detail.
Experience designing evaluation items or rubrics is advantageous.
Background in writing-intensive or analysis-centric fields such as research, editorial, technical writing, or quality assurance is a plus.
Compensation Structure
Compensation is output-based; experts are paid per task that meets the project specifications. The time required to complete work may vary depending on the expert's experience and workflow. Minimum submission requirements apply. Experts must submit a minimum of tasks per week.
Start Timeline & Availability
We typically fill roles within 48 hours and are looking for experts ready to jump in right away. If selected, we expect you to start your first tasks within 24-48 hours of completing onboarding.
Working Place:
NYC, New York, United States
Company :
ESR Healthcare

Apply