1

Virtual Ai Rater Jobs in California (NOW HIRING)

Senior AI Engineer

Foster City, CA

$121K - $167K/yr

We run these virtual- and private-label marketplaces in one of the nation's largest media networks ... The hourly rate may be adjusted based on applicant's geographic location. This position is eligible ...

About the Team We deploy Zoom Virtual Agent into live customer environments. Our team bridges Sales ... rate, containment, and customer satisfaction as deployment gates. * Capturing product gaps and ...

next page

Showing results 1-20

Virtual Ai Rater information

What are some common challenges Virtual AI Raters face when evaluating AI-generated content, and how can these be addressed?

Virtual AI Raters often encounter challenges such as maintaining objectivity, adapting to evolving evaluation criteria, and managing repetitive tasks. To address these, it's important to follow standardized guidelines closely, participate in regular calibration sessions with peers or supervisors, and utilize provided tools for efficient task management. Engaging in ongoing training and staying updated on best practices can also help raters ensure consistent and high-quality evaluations.

What are Virtual AI Raters?

Virtual AI Raters are individuals who remotely evaluate and assess the accuracy, relevance, and quality of artificial intelligence outputs, such as search engine results, ads, or chatbot responses. They follow specific guidelines to rate or annotate data, helping improve AI systems' performance. This role typically requires strong attention to detail, familiarity with online search engines, and the ability to provide objective feedback. Virtual AI Raters usually work part-time and have flexible schedules, making it a popular option for those seeking remote work opportunities.

What are the key skills and qualifications needed to thrive as a Virtual AI Rater, and why are they important?

To thrive as a Virtual AI Rater, you generally need strong analytical skills, attention to detail, and proficiency in English or the target language, often supported by a high school diploma or higher. Familiarity with web browsers, online research, and rating platforms or proprietary evaluation systems is typically required. Excellent time management, reliability, and the ability to follow detailed instructions set top performers apart in this role. These skills ensure accurate and consistent feedback, which is crucial for improving AI systems and maintaining quality standards.
What are the most commonly searched types of Ai Rater jobs in California? The most popular types of Ai Rater jobs in California are:
What are popular job titles related to Virtual Ai Rater jobs in California? For Virtual Ai Rater jobs in California, the most frequently searched job titles are:
What job categories do people searching Virtual Ai Rater jobs in California look for? The top searched job categories for Virtual Ai Rater jobs in California are:
What cities in California are hiring for Virtual Ai Rater jobs? Cities in California with the most Virtual Ai Rater job openings:
Senior AI Engineer

$121K - $167K/yr

Full-time

Posted 17 days ago


Job description

Powering Performance Marketplaces in Digital Media

QuinStreet is a pioneer in powering decentralized online marketplaces that match searchers and "research and compare" consumers with brands. We run these virtual- and private-label marketplaces in one of the nation's largest media networks.

Our industry leading segmentation and AI-driven matching technologies help consumers find better solutions and brands faster. They allow brands to target and reach in-market customer prospects with pinpoint segment-by-segment accuracy, and to pay only for performance results.

Our campaign-results-driven matching decision engines and optimization algorithms are built from over 20 years and billions of dollars of online media experience.

We believe in:

  • The direct measurability of digital media.
  • Performance marketing. (We pioneered it.)
  • The advantages of technology.

We bring all this together to deliver truly great results for consumers and brands in the world's biggest channel.

Job Category

We are looking for a Senior AI Developer & Cloud Architect to design, build, and own the AI-powered compliance scraping engine and cloud infrastructure layer for an internal platform monitoring up to 70,000 credit card offer pages per month. This is a hands-on, sole-builder contractor role that sits at the intersection of cloud architecture, AI engineering, and large-scale web scraping, with a clear mandate: deliver a production-grade system that detects compliance violations across issuer offer pages with high accuracy and controlled token costs.

You will do this by architecting the AWS environment from the ground up, building a containerized worker fleet that integrates Playwright rendering with Claude-powered contextual analysis, and defining clean API contracts with the internal team that owns the Laravel control panel.

This is not a managed-PaaS or prototype role. You will be accountable for end-to-end delivery — architecture, build, documentation, and knowledge transfer — owning the full scraping and AI pipeline from URL intake through compliance findings, screenshot evidence, and results delivery back to the portal.

Responsibilities

  • Design and configure the production AWS environment (ECS/Fargate, SQS, API Gateway, RDS PostgreSQL, S3, IAM, CloudWatch) using infrastructure as code (Terraform or CDK).
  • Build a stateless, containerized worker fleet that integrates Playwright-based page rendering, structured rule evaluation, and Claude API analysis.
  • Implement token optimization strategies across the LLM pipeline — prompt engineering, context pruning, caching, model selection, and batching — with measurable cost outcomes.
  • Define and document API contracts, job payload schemas, and database write patterns with the internal Laravel portal team to enable parallel development.
  • Build third-party API ingestion and field-level diff-detection logic that automatically adjusts monitoring rules when product data changes.
  • Handle modern web rendering challenges at scale: JavaScript-heavy SPAs, interstitials, cookie consent overlays, dynamic content, viewport switching, and full-page screenshot capture.
  • Evaluate when LLM analysis is the correct tool versus a classifier or rules-based approach, and design the two-stage rule-engine-plus-AI pipeline accordingly.
  • Build and maintain a unit test suite covering all modules and APIs to ensure uptime and proper functionality.
  • Document every architecture decision, configuration, API contract, and operational procedure continuously — not as a final-week deliverable.
  • Deliver a complete runbook and knowledge transfer to the internal team at engagement close.
  • Operate independently end-to-end while coordinating closely with the internal portal team and reporting directly to the Senior Director, surfacing risks and trade-offs early.

Requirements

  • Production backend Python experience, including async patterns, type hints, packaging, and testing.
  • Direct production experience designing and configuring AWS ECS/Fargate, SQS, API Gateway, RDS (PostgreSQL), S3, IAM, and CloudWatch, with infrastructure as code (Terraform or CDK).
  • Real shipped systems calling the Anthropic Claude API in production, with demonstrated experience in prompt design, structured output, error handling, and cost trade-offs.
  • Demonstrated track record reducing token spend on production LLM workloads, with specific before/after results you can walk through.
  • Working knowledge of other LLM providers sufficient to recommend cheaper or better alternatives for specific tasks.
  • Production Playwright experience at scale, including headless Chromium failure modes, network idle detection, dynamic content handling, viewport switching, and screenshot strategy. Selenium or Puppeteer experience does not substitute.
  • Machine learning fundamentals sufficient to evaluate when LLM analysis is the right tool versus a classifier or rules-based approach, and to reason about evaluation and false-positive rates.
  • Docker and containerization experience, including image optimization, ECR, and stateless worker design.
  • Ability to operate fully independently — no engineering team underneath you — while documenting continuously and coordinating cleanly with an internal team.

Nice to Have

  • Experience with API ingestion and field-level diff-detection systems.
  • Laravel or PHP familiarity, enough to coordinate cleanly on API contracts with the portal team.
  • SOC 2 Type II compliance experience.
  • Salesforce API integration experience.
  • Regulated-industry experience (financial services, healthcare, or insurance).

The expected hourly range for this position is $80/hr - 100/hr. This hourly range is an estimate, and the actual hourly rate may vary based on the Company's compensation practices. The hourly rate may be adjusted based on applicant's geographic location. This position is eligible to participate in the Company's standard employee benefits programs, which currently include health care benefits.

#LI-REMOTE

QuinStreet is an equal opportunity employer. We do not discriminate on the basis of race, color, religion, national origin, pregnancy status, sex, age, marital status, disability, sexual orientation, gender identity or any other characteristics protected by law.

Please see QuinStreet's Employee Privacy Notice here.