ZipRecruiter

Log In

1

Ai Rater Jobs in Wapwallopen, PA (NOW HIRING)

Geisinger Health

Director AI Evaluation

Danville, PA · On-site +1

Tracks User Adoption - engagement, override rates, and time-to-action - distinguishing genuine workflow misalignment and alarm fatigue from poor predictive value. * Connects each AI to the Outcome it ...

Outback Steakhouse

Line Cook

Wilkes Barre, PA · On-site

$15 - $19/hr

Text HELP for help or smshelp@paradox.ai. Msg&data rates may apply. Msg freq varies. Text STOP to cancel. For terms, visit paradox.ai/legal/terms-of-use. Privacy policy can be found at paradox.ai ...

Outback Steakhouse

Line Cook

Wilkes Barre, PA · On-site

$15 - $19/hr

Text HELP for help or smshelp@paradox.ai . Msg&data rates may apply. Msg freq varies. Text STOP to cancel. For terms, visit paradox.ai/legal/terms-of-use . Privacy policy can be found at paradox.ai ...

next page

Showing results 1-20

All JobsAi Rater JobsAi Rater Jobs in Wapwallopen, PA

Ai Rater information

What is an AI Rater?

An AI Rater evaluates and provides feedback on artificial intelligence models, typically improving search engines, chatbots, or recommendation systems. They assess the relevance, accuracy, and quality of AI-generated content based on specific guidelines. This role requires strong analytical skills, attention to detail, and familiarity with the subject matter being reviewed. AI Raters often work remotely and on a flexible schedule.

What does an AI Rater do?

A typical day for an AI Rater involves reviewing and evaluating various types of content, such as search engine results, social media posts, advertisements, or chatbot responses, to ensure they meet quality and relevancy standards. You may follow detailed guidelines to rate or annotate content, complete assigned tasks in a web-based platform, and provide feedback to help improve AI performance. Most positions are remote and offer flexible schedules, allowing you to plan your workload around personal commitments. Collaboration is generally limited, as most work is performed independently, but periodic communication with team leads for training or updates is common.

What skills and qualifications are needed to thrive as an AI Rater?

To thrive as an AI Rater, you generally need strong attention to detail, analytical thinking, and proficiency in English, often supported by formal education such as a high school diploma or higher. Familiarity with web browsers, online research, and company-specific rating platforms or guidelines is essential. Excellent time management, adaptability, and effective written communication help individuals excel in this position. These skills and qualities ensure accurate and consistent evaluations of AI-generated content, directly impacting the improvement of artificial intelligence systems.

How to become an AI Rater?

To become an AI Rater, candidates typically need a high school diploma or equivalent, strong language and analytical skills, and the ability to follow detailed guidelines. The role often involves evaluating search engine results or content for accuracy and relevance, using online platforms or specific assessment tools. Prior experience in data annotation or quality assurance can be beneficial, and flexible scheduling is common.

What are popular job titles related to Ai Rater jobs in Wapwallopen, PA?

For Ai Rater jobs in Wapwallopen, PA, the most frequently searched job titles are:

What cities near Wapwallopen, PA are hiring for Ai Rater jobs?

Cities near Wapwallopen, PA with the most Ai Rater job openings:

Ai Rater jobs near you
Infographic showing various Ai Rater job openings in Wapwallopen, PA as of August 2026, with employment types broken down into 64% Full Time, and 36% Part Time. Highlights an 100% Remote job distribution.

Director AI Evaluation

Danville, PA • On-site, Remote

Geisinger Health
Health Care and Social Assistance • 10K+ employees

6.9

Company rating: 6.9 out of 10

Based on 448 frontline employees who took The Breakroom Quiz

453rd of 893 rated healthcare providers

People enjoy working here

Recommended by students

Recommended by parents

Learn more about Geisinger Healthexternal

Full-time

Medical, Dental, Vision

Re-posted 11 days ago

Job description

Location:
Work from home (Pennsylvania)
Shift:
Days (United States of America)
Scheduled Weekly Hours:
40
Worker Type:
Regular
Exemption Status:
Yes
Job Summary:
The Director of AI Evaluation owns how Geisinger defines, proves, and sustains quality across its entire AI portfolio - internally built models and vendor-provided systems alike. This is a hands-on technical leader who also manages the people who do the building and the proving: the data scientists who develop production machine learning and fine-tuned AI systems, and the senior analysts who evaluate them.
Every high-value AI initiative - bought or built - needs a single, credible standard for what constitutes quality, who validates it, and how it stays good in production. The Director sets that standard, leads the team that enforces it, and reports findings to the VP of AI, executive leaders, and the board.
This role is a manager who develops a multidisciplinary team, a technical authority who defines evaluation method across the enterprise, and a quality owner who guides every major AI program toward evidence that withstands scrutiny.
Job Duties:
  • Reports to the VP of AI; directly career-manages the data science line and matrix-manages the Senior Analysts, AI Evaluation.
  • Determines the quality standard for any high-value AI initiative at Geisinger - internally built or vendor-provided - from design through production.
  • Holds bought systems to the same standard as built ones, generating local evidence on whether a tool works here for Geisinger's clinicians and patients rather than accepting vendor aggregate or cherry-picked results.
  • Owns the methodology that holds initiatives to that standard: pre-production validation and live production monitoring.
  • Owns the health of the data science team - attracting and retaining strong technical talent, developing careers, and keeping the bench deep, engaged, and growing - and leads and develops the evaluation team alongside it.
  • Provides hands-on technical guidance to program teams as they design validation studies, equity audits, monitoring plans, and escalation playbooks.
  • Owns the evaluation toolkit and reusable playbooks and templates that let each new program move faster than the last.
  • Translates program-specific failure modes into concrete, measurable production-monitoring metrics; defines what is measured and how, while the AI Platform team builds the backend.
  • Tracks AI System Performance - the single most important accuracy indicator for each system, against thresholds set to clinical tolerance.
  • Tracks User Adoption - engagement, override rates, and time-to-action - distinguishing genuine workflow misalignment and alarm fatigue from poor predictive value.
  • Connects each AI to the Outcome it was deployed to improve (mortality, time-to-treatment, boarding time, denial rate, cost per case) against a pre-launch baseline over a use-case-appropriate horizon, holding both tangible returns and harder-to-quantify value in view.
  • Monitors Equity - the maximum performance gap on the Pillar 1 metric across the subgroups that matter for the initiative, so disparate impact surfaces early.

Work is typically performed in an office environment. Accountable for satisfying all job specific obligations and complying with all organization policies and procedures. The specific statements in this profile are not intended to be all-inclusive. They represent typical elements considered necessary to successfully perform the job.
Position Details:
Required Skills and Qualifications:
  • People-leadership experience - managing, developing, and growing technical staff; building teams, not just leading projects.
  • Strong foundation in experimental design and causal inference, with judgment about which method fits which situation.
  • Hands-on experience designing and running model evaluation studies in real production settings.
  • Experience evaluating LLM or generative AI systems, or comparable experience with complex ML systems where ground truth is ambiguous or noisy.
  • Proven ability to translate ambiguous failure modes into concrete, defensible evaluation designs and monitoring metrics.
  • Strong fluency in Python and SQL; working comfort with modern ML tooling and cloud-native data environments.
  • Experience in evaluating fairness and equity in ML systems.
  • Clear written communication - the role produces evaluation memos and specifications that non-technical decision-makers rely on.
  • Healthcare, clinical, or regulated-industry experience strongly preferred.

Education:
Bachelor's Degree-Related Field of Study (Required)
Experience:
Minimum of 8 years-Related work experience (Required), Minimum of 3 years-Managerial/Supervisory (Required)
Certification(s) and License(s):
Skills:
OUR PURPOSE & VALUES: Everything we do is about caring for our patients, our members, our students, our Geisinger family and our communities.
  • KINDNESS: We strive to treat everyone as we would hope to be treated ourselves.
  • EXCELLENCE: We treasure colleagues who humbly strive for excellence.
  • LEARNING: We share our knowledge with the best and brightest to better prepare the caregivers for tomorrow.
  • INNOVATION: We constantly seek new and better ways to care for our patients, our members, our community, and the nation.
  • SAFETY: We provide a safe environment for our patients and members and the Geisinger family.

We offer healthcare benefits for full time and part time positions from day one, including vision, dental and domestic partners. Perhaps just as important, we encourage an atmosphere of collaboration, cooperation and collegiality.
We know that a diverse workforce with unique experiences and backgrounds makes our team stronger. Our patients, members and community come from a wide variety of backgrounds, and it takes a diverse workforce to make better health easier for all. We are proud to be an affirmative action, equal opportunity employer and all qualified applicants will receive consideration for employment regardless to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or status as a protected veteran.

What Geisinger Health employees say

Pay

Benefits

Hours and flexibility

Workplace

Get the full story on Breakroom

Apply