The PREP program involves staff from a wide range of backgrounds conducting scientific research ... Gain familiarity with existing literature on data annotation and LLM as judge * Understand NIST ...
The PREP program involves staff from a wide range of backgrounds conducting scientific research ... Gain familiarity with existing literature on data annotation and LLM as judge * Understand NIST ...
Data/AI Program Manager
Bethesda, MD · On-site
$98K - $163K/yr
Lead IC-wide data management programs aligned with the IC Data Strategy. * Develop and execute strategic plans to enhance data governance, metadata management, and data sharing across IC elements.
Data/AI Program Manager
Bethesda, MD · On-site
$98K - $163K/yr
Lead IC-wide data management programs aligned with the IC Data Strategy. * Develop and execute strategic plans to enhance data governance, metadata management, and data sharing across IC elements.
Data/AI Program Manager
$98K - $163K/yr
Lead IC-wide data management programs aligned with the IC Data Strategy. * Develop and execute strategic plans to enhance data governance, metadata management, and data sharing across IC elements.
Data/AI Program Manager
$98K - $163K/yr
Lead IC-wide data management programs aligned with the IC Data Strategy. * Develop and execute strategic plans to enhance data governance, metadata management, and data sharing across IC elements.
Program Manager
Potomac, MD · On-site
$120K - $170K/yr
We specialize in data sciences and cloud solutions, application development and maintenance, program management, and health IT operations leveraging experience and trusted talent to solve the federal ...
Quick apply
Program Manager
Potomac, MD · On-site
$120K - $170K/yr
We specialize in data sciences and cloud solutions, application development and maintenance, program management, and health IT operations leveraging experience and trusted talent to solve the federal ...
Program Manager
College Park, MD · On-site
Visit our website at: www.dcca.com Program Manager Candidate must have an active TS/SCI W/ POLYGRAPH Program : DCCA is recruiting candidates for an exciting opportunity to support leading edge ...
Program Manager
College Park, MD · On-site
Visit our website at: www.dcca.com Program Manager Candidate must have an active TS/SCI W/ POLYGRAPH Program : DCCA is recruiting candidates for an exciting opportunity to support leading edge ...
Program Manager - Networking
Baltimore, MD · On-site
$115K - $115K/yr
NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with ... We are currently seeking a Program Manager - Networking to join our team in Baltimore, Maryland (US ...
Program Manager - Networking
Baltimore, MD · On-site
$115K - $115K/yr
NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with ... We are currently seeking a Program Manager - Networking to join our team in Baltimore, Maryland (US ...
Program Manager - Networking
Baltimore, MD · Hybrid
$115K - $115K/yr
NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with ... We are currently seeking a Program Manager - Networking to join our team in Baltimore, Maryland (US ...
Program Manager - Networking
Baltimore, MD · Hybrid
$115K - $115K/yr
NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with ... We are currently seeking a Program Manager - Networking to join our team in Baltimore, Maryland (US ...
Program Manager - Networking
Baltimore, MD · Hybrid
$115K - $115K/yr
NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with ... We are currently seeking a Program Manager - Networking to join our team in Baltimore, Maryland (US ...
Program Manager - Networking
Baltimore, MD · Hybrid
$115K - $115K/yr
NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with ... We are currently seeking a Program Manager - Networking to join our team in Baltimore, Maryland (US ...
Program Manager
Greenbelt, MD · Hybrid
$140K - $190K/yr
Program Management and Leadership: * Overall responsibility for program execution of a $200+M ... Cloud, Earth Science, Data Collection, Modeling, etc; schedule and cost * Coordination across ...
Program Manager
Greenbelt, MD · Hybrid
$140K - $190K/yr
Program Management and Leadership: * Overall responsibility for program execution of a $200+M ... Cloud, Earth Science, Data Collection, Modeling, etc; schedule and cost * Coordination across ...
Program Manager
Greenbelt, MD · On-site
$140K - $190K/yr
Program Management and Leadership: * Overall responsibility for program execution of a $200+M ... Cloud, Earth Science, Data Collection, Modeling, etc; schedule and cost * Coordination across ...
Program Manager
Greenbelt, MD · On-site
$140K - $190K/yr
Program Management and Leadership: * Overall responsibility for program execution of a $200+M ... Cloud, Earth Science, Data Collection, Modeling, etc; schedule and cost * Coordination across ...
Program Manager
Greenbelt, MD · Hybrid
$140K - $190K/yr
Program Management and Leadership: * Overall responsibility for program execution of a $200+M ... Cloud, Earth Science, Data Collection, Modeling, etc; schedule and cost * Coordination across ...
Program Manager
Greenbelt, MD · Hybrid
$140K - $190K/yr
Program Management and Leadership: * Overall responsibility for program execution of a $200+M ... Cloud, Earth Science, Data Collection, Modeling, etc; schedule and cost * Coordination across ...
Program Manager
Fort George G Meade, MD · On-site
Founded in 1997 and headquartered in Washington DC, Expression provides data fusion, data analytics ... Establish and manage program risk, quality assurance, configuration management, and release ...
Program Manager
Fort George G Meade, MD · On-site
Founded in 1997 and headquartered in Washington DC, Expression provides data fusion, data analytics ... Establish and manage program risk, quality assurance, configuration management, and release ...
Program Manager
Suitland, MD · On-site
Program Manager Location: Suitland, MD Clearance Level: TS with ability to obtain an SCI Required ... Data Reports (CSDR), and Cost Data Summary Reports (CDSR) Ensure accurate and timely monthly ...
Program Manager
Suitland, MD · On-site
Program Manager Location: Suitland, MD Clearance Level: TS with ability to obtain an SCI Required ... Data Reports (CSDR), and Cost Data Summary Reports (CDSR) Ensure accurate and timely monthly ...
Program Manager
Suitland, MD · On-site
Program Manager Location: Suitland, MD Clearance Level: TS with ability to obtain an SCI Required ... Data Reports (CSDR), and Cost Data Summary Reports (CDSR) • Ensure accurate and timely monthly ...
Program Manager
Suitland, MD · On-site
Program Manager Location: Suitland, MD Clearance Level: TS with ability to obtain an SCI Required ... Data Reports (CSDR), and Cost Data Summary Reports (CDSR) • Ensure accurate and timely monthly ...
Program Manager
Fort George G Meade, MD · On-site
$140K - $180K/yr
Founded in 1997 and headquartered in Washington DC, Expression provides data fusion, data analytics ... Establish and manage program risk, quality assurance, configuration management, and release ...
Quick apply
Program Manager
Fort George G Meade, MD · On-site
$140K - $180K/yr
Founded in 1997 and headquartered in Washington DC, Expression provides data fusion, data analytics ... Establish and manage program risk, quality assurance, configuration management, and release ...
Founded in 1997 and headquartered in Washington DC, Expression provides data fusion, data analytics ... Establish and manage program risk, quality assurance, configuration management, and release ...
Founded in 1997 and headquartered in Washington DC, Expression provides data fusion, data analytics ... Establish and manage program risk, quality assurance, configuration management, and release ...
Program Manager
Baltimore, MD · On-site
Coordinate resources across multiple interconnected project workstreams (e.g., data conversion ... Program Management - 5 years Progressive experience in managing large scale, complex technical ...
Program Manager
Baltimore, MD · On-site
Coordinate resources across multiple interconnected project workstreams (e.g., data conversion ... Program Management - 5 years Progressive experience in managing large scale, complex technical ...
Program Manager
Suitland, MD · On-site
... Data Reports (CSDR), and Cost Data Summary Reports (CDSR) • Ensure accurate and timely monthly ... program/project management experience leading large, complex DoD/IC IT or Cyber programs • 5+ ...
Program Manager
Suitland, MD · On-site
... Data Reports (CSDR), and Cost Data Summary Reports (CDSR) • Ensure accurate and timely monthly ... program/project management experience leading large, complex DoD/IC IT or Cyber programs • 5+ ...
Program Manager
Bethesda, MD · On-site +1
As the Program Manager, your key responsibilities will include: - Leading the daily operations of ... performance data, implementing corrective actions as necessary to achieve established goals ...
Program Manager
Bethesda, MD · On-site +1
As the Program Manager, your key responsibilities will include: - Leading the daily operations of ... performance data, implementing corrective actions as necessary to achieve established goals ...
Program Manager
Bethesda, MD · On-site +1
As the Program Manager, your key responsibilities will include: - Leading the daily operations of ... performance data, implementing corrective actions as necessary to achieve established goals ...
Program Manager
Bethesda, MD · On-site +1
As the Program Manager, your key responsibilities will include: - Leading the daily operations of ... performance data, implementing corrective actions as necessary to achieve established goals ...
Data Annotation Program Manager information
What are the key skills and qualifications needed to thrive as a data annotation program manager, and why are they important?
To thrive as a Data Annotation Program Manager, you need expertise in project management, data quality assessment, and a solid understanding of machine learning or data annotation processes, typically supported by a relevant degree. Familiarity with annotation platforms, workflow management tools, and data labeling software is essential, along with knowledge of quality assurance frameworks. Strong leadership, problem-solving abilities, and effective communication are crucial soft skills that help manage diverse teams and ensure stakeholder alignment. These skills are important to maintain high data quality, meet project deadlines, and drive successful AI model training initiatives.
How does a data annotation program manager coordinate with cross-functional teams to ensure project success?
A Data Annotation Program Manager regularly collaborates with engineering, data science, and quality assurance teams to align annotation guidelines, project timelines, and quality standards. They often facilitate meetings to clarify requirements, resolve ambiguities in data labeling, and provide feedback on annotation accuracy. This role serves as a bridge between technical teams and annotation staff, ensuring open communication and timely resolution of challenges, which is critical for delivering high-quality datasets essential for machine learning and AI projects.
What is a data annotation program manager?
Data Annotation Program Managers are professionals who oversee and coordinate data labeling projects, ensuring that data used for machine learning and artificial intelligence is accurately tagged and prepared. They manage teams of annotators, set project guidelines, monitor quality, and ensure deadlines are met. Their role is crucial for building high-quality datasets that enable reliable AI model training. Program Managers often collaborate with data scientists, engineers, and stakeholders to define requirements and improve annotation processes.
What are popular job titles related to Data Annotation Program Manager jobs in Maryland? For Data Annotation Program Manager jobs in Maryland, the most frequently searched job titles are:
What job categories do people searching Data Annotation Program Manager jobs in Maryland look for? The top searched job categories for Data Annotation Program Manager jobs in Maryland are:
What cities in Maryland are hiring for Data Annotation Program Manager jobs? Cities in Maryland with the most Data Annotation Program Manager job openings:
Full-time
Re-posted 28 days ago
Johns Hopkins University rating
8.0
Based on 71 frontline employees who took The Breakroom Quiz
183rd of 614 rated colleges and universities
Job description
Description
PREP Research Associate
This position is part of the National Institute of Standards and Technology (NIST) Professional Research Experience Program (PREP). NIST recognizes that its research staff may want to collaborate with researchers at academic institutions on specific projects of mutual interest and, therefore, requires those institutions to be recipients of a PREP award. The PREP program involves staff from a wide range of backgrounds conducting scientific research across various fields. Individuals in this position will perform technical work supporting the collaboration's scientific research.
Research Title:
Reliability of Human and LLM Annotations for AI Risk Assessment
The work will entail:
This project focuses on using Large Language Models (LLMs) to provide annotations of evaluation data (a.k.a., LLM as judge), and the design of an Inter-Annotator Agreement study to assess the reliability of both human and LLM annotations. The candidate will explore assessing the indicators of a given AI-related risk, determining how to identify them, and providing annotators with examples to annotate the presence of various risks. The project aims to develop an annotation framework for AI risk assessment and establish metrics for data quality in AI risk research, supporting broader work at NIST in assessing and measuring the validity and reliability of AI-related risks in data annotation.
U.S. Citizen Preferred
Key responsibilities will include but are not limited to:
Deliverables
Qualifications
Application Instructions
Please upload the following with your application:
• CV/Resume
*Please limit C.V to 3 pages only and ONLY include a valid email address for your contact info. Your resume will not be considered if the following information is included on your CV/resume.
• Self portraits
• Phone number
• Home address/Country
• Citizenship status
• Languages spoken
• Sex/Gender
Privacy Act Statement
Authority: 15 U.S.C. § 278g-1(e)(1) and (e)(3) and 15 U.S.C. § 272(b) and (c)
Purpose: The National Institute for Standards and Technology (NIST) hosts the Professional Research Experience Program (PREP) which is designed to provide valuable laboratory experience and financial assistance to undergraduates, post-bachelor's degree holders, graduate students, master's degree holders, postdocs, and faculty.
PREP is a 5-year cooperative agreement between NIST laboratories and participating PREP Universities to establish a collaborative research relationship between NIST and U.S. institutions of higher education in the following disciplines including (but may not be limited to) biochemistry, biological sciences, chemistry, computer science, engineering, electronics, materials science, mathematics, nanoscale science, neutron science, physical science, physics, and statistics. This collection of information is needed to facilitate administrative functions of the PREP Program.
Routine Uses: NIST will use the information collected to perform the requisite reviews of the applications to determine eligibility, and to meet programmatic requirements. Disclosure of this information is also subject to all the published routine uses as identified in the Privacy Act System of Records Notices: NIST-1: NIST Associates.
Disclosure: Furnishing this information is voluntary. When you submit the form, you are indicating your voluntary consent for NIST to use of the information you submit for the purpose stated.
PREP Research Associate
This position is part of the National Institute of Standards and Technology (NIST) Professional Research Experience Program (PREP). NIST recognizes that its research staff may want to collaborate with researchers at academic institutions on specific projects of mutual interest and, therefore, requires those institutions to be recipients of a PREP award. The PREP program involves staff from a wide range of backgrounds conducting scientific research across various fields. Individuals in this position will perform technical work supporting the collaboration's scientific research.
Research Title:
Reliability of Human and LLM Annotations for AI Risk Assessment
The work will entail:
This project focuses on using Large Language Models (LLMs) to provide annotations of evaluation data (a.k.a., LLM as judge), and the design of an Inter-Annotator Agreement study to assess the reliability of both human and LLM annotations. The candidate will explore assessing the indicators of a given AI-related risk, determining how to identify them, and providing annotators with examples to annotate the presence of various risks. The project aims to develop an annotation framework for AI risk assessment and establish metrics for data quality in AI risk research, supporting broader work at NIST in assessing and measuring the validity and reliability of AI-related risks in data annotation.
U.S. Citizen Preferred
Key responsibilities will include but are not limited to:
- Gain familiarity with existing literature on data annotation and LLM as judge
- Understand NIST's role and ongoing efforts in assessing and measuring the validity and reliability of AI-related risks in data annotation
- Contribute to developing an annotation framework for AI risk assessment
- Collaborate effectively with cross-functional and interdisciplinary stakeholders to ensure successful project outcomes
Deliverables
- Contributions to a NIST report that supports ongoing NIST AI evaluation efforts focused on the design of an Inter-Annotator Agreement to assess the reliability of both human and LLM annotations.
Qualifications
- Background in Computer Science, Data Science, or related field.
- Education level: Bachelor's or Graduate Degree
- Strong interest in data annotation and AI risks
- Familiarity with scientific reading and technical writing
Application Instructions
Please upload the following with your application:
• CV/Resume
*Please limit C.V to 3 pages only and ONLY include a valid email address for your contact info. Your resume will not be considered if the following information is included on your CV/resume.
• Self portraits
• Phone number
• Home address/Country
• Citizenship status
• Languages spoken
• Sex/Gender
Privacy Act Statement
Authority: 15 U.S.C. § 278g-1(e)(1) and (e)(3) and 15 U.S.C. § 272(b) and (c)
Purpose: The National Institute for Standards and Technology (NIST) hosts the Professional Research Experience Program (PREP) which is designed to provide valuable laboratory experience and financial assistance to undergraduates, post-bachelor's degree holders, graduate students, master's degree holders, postdocs, and faculty.
PREP is a 5-year cooperative agreement between NIST laboratories and participating PREP Universities to establish a collaborative research relationship between NIST and U.S. institutions of higher education in the following disciplines including (but may not be limited to) biochemistry, biological sciences, chemistry, computer science, engineering, electronics, materials science, mathematics, nanoscale science, neutron science, physical science, physics, and statistics. This collection of information is needed to facilitate administrative functions of the PREP Program.
Routine Uses: NIST will use the information collected to perform the requisite reviews of the applications to determine eligibility, and to meet programmatic requirements. Disclosure of this information is also subject to all the published routine uses as identified in the Privacy Act System of Records Notices: NIST-1: NIST Associates.
Disclosure: Furnishing this information is voluntary. When you submit the form, you are indicating your voluntary consent for NIST to use of the information you submit for the purpose stated.
What Johns Hopkins University employees say
Pay
Benefits
Hours and flexibility
Workplace
Get the full story on Breakroom
About Johns Hopkins University
Sourced by ZipRecruiter
Gilman believed that teaching and research go hand in hand—that success in one depends on success in the other—and that a modern university must do both well. He also believed that sharing our knowledge and discoveries would help make the world a better place. In 145 years, we haven’t strayed from that vision. This is still a destination for excellent, ambitious scholars and a world leader in teaching and research. Distinguished professors mentor students in the arts and music, humanities, social and natural sciences, engineering, international studies, education, business, and the health professions. Those same faculty members, along with their colleagues at the university’s Applied Physics Laboratory, have made us the nation’s leader in federal research and development funding every year since 1979. That’s a fitting distinction for America’s first research university, a place that has revolutionized higher education in the U.S. and continues to bring knowledge and discoveries to the world.
Industry
Colleges, universities, and professional schools
Company size
10,000+ Employees
Headquarters location
Baltimore, MD, US
Year founded
1876