Columbus, OH · On-site
$20 - $30/hr
Type: Contract, hourly Location: Remote Hours: 20-30 per week Pay: $20-30/hour, based on experience and language coverage Start: Immediate ABOUT THE ROLE We evaluate how well advanced AI coding ...
New
Columbus, OH · On-site
$20 - $30/hr
Type: Contract, hourly Location: Remote Hours: 20-30 per week Pay: $20-30/hour, based on experience and language coverage Start: Immediate ABOUT THE ROLE We evaluate how well advanced AI coding ...
New
$13.70 - $15.03
2% of jobs
$15.03 - $16.37
7% of jobs
$16.37 - $17.70
10% of jobs
$18.92 is the 25th percentile. Wages below this are outliers.
$17.70 - $19.03
6% of jobs
$19.03 - $20.37
9% of jobs
$20.37 - $21.70
7% of jobs
$21.70 - $23.03
0% of jobs
$23.03 - $24.37
1% of jobs
The median wage is $24.56 / hr.
$24.37 - $25.70
50% of jobs
$25.70 - $27.03
4% of jobs
$27.03 - $28.37
3% of jobs
$13
$22
$28
For Annotator jobs, the most frequently searched job titles are:
The top searched job categories for Annotator jobs are:
Cities with the most Annotator job openings:
States with the most job openings for Annotator jobs include:
The most popular types of Annotator jobs are:
$20 - $30/hr
Full-time
Posted 3 days ago
New
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.