The project aims to develop an annotation framework for AI risk assessment and establish metrics for data quality in AI risk research, supporting broader work at NIST in assessing and measuring the ...
The project aims to develop an annotation framework for AI risk assessment and establish metrics for data quality in AI risk research, supporting broader work at NIST in assessing and measuring the ...
Vietnamese Language Annotator
Annapolis Junction, MD · On-site
$64K - $113K/yr
Worker Type Regular Summary The Vietnamese Language Annotator will support the research ... Prior experience in data annotation, transcription, translation, or linguistic analysis ...
New
Vietnamese Language Annotator
Annapolis Junction, MD · On-site
$64K - $113K/yr
Worker Type Regular Summary The Vietnamese Language Annotator will support the research ... Prior experience in data annotation, transcription, translation, or linguistic analysis ...
New
Vietnamese Language Annotator with Security Clearance
Annapolis, MD · On-site
$64K - $113K/yr
Worker Type Regular Summary The Vietnamese Language Annotator will support the research ... Prior experience in data annotation, transcription, translation, or linguistic analysis ...
New
Vietnamese Language Annotator with Security Clearance
Annapolis, MD · On-site
$64K - $113K/yr
Worker Type Regular Summary The Vietnamese Language Annotator will support the research ... Prior experience in data annotation, transcription, translation, or linguistic analysis ...
New
We work with some of the top research organizations and facilities in the country including ... annotation, DEG), and Digital Spatial Profiling (annotation, QC, normalization, spatial ...
We work with some of the top research organizations and facilities in the country including ... annotation, DEG), and Digital Spatial Profiling (annotation, QC, normalization, spatial ...
We work with some of the top research organizations and facilities in the country including ... annotation, DEG), and Digital Spatial Profiling (annotation, QC, normalization, spatial ...
Quick apply
We work with some of the top research organizations and facilities in the country including ... annotation, DEG), and Digital Spatial Profiling (annotation, QC, normalization, spatial ...
Data Scientist II
Rockville, MD · On-site
We work with some of the top research organizations and facilities in the country including ... annotation, DEG), and Digital Spatial Profiling (annotation, QC, normalization, spatial ...
Data Scientist II
Rockville, MD · On-site
We work with some of the top research organizations and facilities in the country including ... annotation, DEG), and Digital Spatial Profiling (annotation, QC, normalization, spatial ...
Data Scientist 3
Annapolis Junction, MD · On-site
... annotation of language data with parts of speech information, and improve existing models by ... Research, or Computer Science or a degree in a related field (Computer Information Systems ...
Data Scientist 3
Annapolis Junction, MD · On-site
... annotation of language data with parts of speech information, and improve existing models by ... Research, or Computer Science or a degree in a related field (Computer Information Systems ...
Data Scientist
Fort George G Meade, MD · On-site
... annotation of language data with parts of speech information, and improved existing models by ... Research, or Computer Science or a degree in a related field (Computer Information Systems ...
Data Scientist
Fort George G Meade, MD · On-site
... annotation of language data with parts of speech information, and improved existing models by ... Research, or Computer Science or a degree in a related field (Computer Information Systems ...
Data Scientist
Fort George G Meade, MD · On-site
... annotation of language data with parts of speech information, and improved existing models by ... Research, or Computer Science or a degree in a related field (Computer Information Systems ...
Data Scientist
Fort George G Meade, MD · On-site
... annotation of language data with parts of speech information, and improved existing models by ... Research, or Computer Science or a degree in a related field (Computer Information Systems ...
Data Scientist 3
Annapolis Junction, MD · On-site
$132K - $147K/yr
... annotation of language data with parts of speech information, and improve existing models by ... Research, or Computer Science or a degree in a related field (Computer Information Systems ...
Data Scientist 3
Annapolis Junction, MD · On-site
$132K - $147K/yr
... annotation of language data with parts of speech information, and improve existing models by ... Research, or Computer Science or a degree in a related field (Computer Information Systems ...
Data Scientist 3
Annapolis Junction, MD · On-site
$132K - $147K/yr
... annotation of language data with parts of speech information, and improve existing models by ... Research, or Computer Science or a degree in a related field (Computer Information Systems ...
Quick apply
Data Scientist 3
Annapolis Junction, MD · On-site
$132K - $147K/yr
... annotation of language data with parts of speech information, and improve existing models by ... Research, or Computer Science or a degree in a related field (Computer Information Systems ...
... annotation of language data with parts of speech information, and improve existing models by ... Research, or Computer Science or a degree in a related field (Computer Information Systems ...
... annotation of language data with parts of speech information, and improve existing models by ... Research, or Computer Science or a degree in a related field (Computer Information Systems ...
... annotation of language data with parts of speech information, and improved existing models by ... Research, or Computer Science or a degree in a related field (Computer Information Systems ...
... annotation of language data with parts of speech information, and improved existing models by ... Research, or Computer Science or a degree in a related field (Computer Information Systems ...
Sr. Researcher - AI/ML Enabled Spatiotemporal Imagery
Bethesda, MD · On-site
$105K - $134K/yr
... experience with R&D of machine learning and deep learning algorithms. • One (1) year of ... Ability to develop succinct, comprehensive, understandable documents using complex data from ...
Sr. Researcher - AI/ML Enabled Spatiotemporal Imagery
Bethesda, MD · On-site
$105K - $134K/yr
... experience with R&D of machine learning and deep learning algorithms. • One (1) year of ... Ability to develop succinct, comprehensive, understandable documents using complex data from ...
Coordinate data collection and annotation efforts for supervised training efforts * Design and implement tests for machine learning algorithm effectiveness and performance monitoring * Research and ...
Coordinate data collection and annotation efforts for supervised training efforts * Design and implement tests for machine learning algorithm effectiveness and performance monitoring * Research and ...
We work with some of the top research organizations and facilities in the country including ... Develop benchmark datasets, annotation guidelines, and evaluation pipelines for scientific ...
Quick apply
We work with some of the top research organizations and facilities in the country including ... Develop benchmark datasets, annotation guidelines, and evaluation pipelines for scientific ...
Subject Matter Expert- RxNav
Bethesda, MD · On-site
... data, that scale across institutional boundaries and applications ... Its members translate new research results into operational solutions that can be implemented in ...
Subject Matter Expert- RxNav
Bethesda, MD · On-site
... data, that scale across institutional boundaries and applications ... Its members translate new research results into operational solutions that can be implemented in ...
... experience with R&D of machine learning and deep learning algorithms. • One (1) year of ... Ability to develop succinct, comprehensive, understandable documents using complex data from ...
... experience with R&D of machine learning and deep learning algorithms. • One (1) year of ... Ability to develop succinct, comprehensive, understandable documents using complex data from ...
... data, that scale across institutional boundaries and applications ... Its members translate new research results into operational solutions that can be implemented in ...
... data, that scale across institutional boundaries and applications ... Its members translate new research results into operational solutions that can be implemented in ...
Examine data sources and literary sources for emerging technology related to the Program. Provide ... Two (2) years of experience with R&D of machine learning and deep learning algorithms. One (1) year ...
Examine data sources and literary sources for emerging technology related to the Program. Provide ... Two (2) years of experience with R&D of machine learning and deep learning algorithms. One (1) year ...
Data Annotation Research information
What qualifications do I need for data annotation?
What are some common challenges faced in Data Annotation Research roles, and how can they be addressed?
Does data annotation actually pay?
How hard is it to get hired by data annotation?
What is the difference between Data Annotation Research vs Data Labeling Specialist?
| Aspect | Data Annotation Research | Data Labeling Specialist |
|---|---|---|
| Credentials | Typically requires a background in data science, research methods, or related fields | Often requires basic technical skills and experience with labeling tools |
| Work Environment | Research labs, tech companies, or remote research teams | Data centers, tech companies, or remote labeling teams |
| Industry Usage | Used in AI/ML research, developing annotation methodologies | Used in preparing datasets for machine learning models |
| Search & Comparison Intent | Understanding research-focused roles in data annotation | Looking for practical data labeling jobs |
Data Annotation Research involves exploring new annotation techniques and improving data quality for AI models, often requiring research skills. In contrast, Data Labeling Specialists focus on applying existing labeling tools to annotate datasets efficiently. Both roles are essential in AI development but differ in scope and expertise.
Is data annotation real or fake?
What is data annotation research?
What are the key skills and qualifications needed to thrive as a Data Annotation Researcher, and why are they important?

Full-time
Posted 26 days ago
Johns Hopkins Medicine rating
7.4
Based on 203 frontline employees who took The Breakroom Quiz
227th of 877 rated healthcare providers
Job description
PREP Research Associate
This position is part of the National Institute of Standards and Technology (NIST) Professional Research Experience Program (PREP). NIST recognizes that its research staff may want to collaborate with researchers at academic institutions on specific projects of mutual interest and, therefore, requires those institutions to be recipients of a PREP award. The PREP program involves staff from a wide range of backgrounds conducting scientific research across various fields. Individuals in this position will perform technical work supporting the collaboration's scientific research.
Research Title:
Reliability of Human and LLM Annotations for AI Risk Assessment
The work will entail:
This project focuses on using Large Language Models (LLMs) to provide annotations of evaluation data (a.k.a., LLM as judge), and the design of an Inter-Annotator Agreement study to assess the reliability of both human and LLM annotations. The candidate will explore assessing the indicators of a given AI-related risk, determining how to identify them, and providing annotators with examples to annotate the presence of various risks. The project aims to develop an annotation framework for AI risk assessment and establish metrics for data quality in AI risk research, supporting broader work at NIST in assessing and measuring the validity and reliability of AI-related risks in data annotation.
U.S. Citizen Preferred
Key responsibilities will include but are not limited to:
- Gain familiarity with existing literature on data annotation and LLM as judge
- Understand NIST's role and ongoing efforts in assessing and measuring the validity and reliability of AI-related risks in data annotation
- Contribute to developing an annotation framework for AI risk assessment
- Collaborate effectively with cross-functional and interdisciplinary stakeholders to ensure successful project outcomes
Deliverables
- Contributions to a NIST report that supports ongoing NIST AI evaluation efforts focused on the design of an Inter-Annotator Agreement to assess the reliability of both human and LLM annotations.
Qualifications
- Background in Computer Science, Data Science, or related field.
- Education level: Bachelor's or Graduate Degree
- Strong interest in data annotation and AI risks
- Familiarity with scientific reading and technical writing
Application Instructions
Please upload the following with your application:
• CV/Resume
*Please limit C.V to 3 pages only and ONLY include a valid email address for your contact info. Your resume will not be considered if the following information is included on your CV/resume.
• Self portraits
• Phone number
• Home address/Country
• Citizenship status
• Languages spoken
• Sex/Gender
Privacy Act Statement
Authority: 15 U.S.C. § 278g-1(e)(1) and (e)(3) and 15 U.S.C. § 272(b) and (c)
Purpose: The National Institute for Standards and Technology (NIST) hosts the Professional Research Experience Program (PREP) which is designed to provide valuable laboratory experience and financial assistance to undergraduates, post-bachelor's degree holders, graduate students, master's degree holders, postdocs, and faculty.
PREP is a 5-year cooperative agreement between NIST laboratories and participating PREP Universities to establish a collaborative research relationship between NIST and U.S. institutions of higher education in the following disciplines including (but may not be limited to) biochemistry, biological sciences, chemistry, computer science, engineering, electronics, materials science, mathematics, nanoscale science, neutron science, physical science, physics, and statistics. This collection of information is needed to facilitate administrative functions of the PREP Program.
Routine Uses: NIST will use the information collected to perform the requisite reviews of the applications to determine eligibility, and to meet programmatic requirements. Disclosure of this information is also subject to all the published routine uses as identified in the Privacy Act System of Records Notices: NIST-1: NIST Associates.
Disclosure: Furnishing this information is voluntary. When you submit the form, you are indicating your voluntary consent for NIST to use of the information you submit for the purpose stated.
What Johns Hopkins Medicine employees say
Pay
Benefits
Hours and flexibility
Workplace
Get the full story on Breakroom
About Johns Hopkins Hospital
Sourced by ZipRecruiter
Industry
Health care and social assistance
Company size
10,000+ Employees
Headquarters location
Baltimore, MD, US
Year founded
1889