Biddeford, ME · On-site
$20 - $30/hr
Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works ...
Biddeford, ME · On-site
$20 - $30/hr
Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works ...
Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works ...
Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works ...
Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works ...
Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works ...
Lewiston, ME · On-site
$20 - $30/hr
Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works ...
Lewiston, ME · On-site
$20 - $30/hr
Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works ...
Lisbon, ME · On-site
$98K - $135K/yr
... DevOps Modernization. Notable customers include GE, Cisco, Genentech, Electronic Arts, Cox ... Contributions to open-source AI or distributed-systems projects. Why PagerDuty At PagerDuty, AI it ...
Lisbon, ME · On-site
$98K - $135K/yr
... DevOps Modernization. Notable customers include GE, Cisco, Genentech, Electronic Arts, Cox ... Contributions to open-source AI or distributed-systems projects. Why PagerDuty At PagerDuty, AI it ...
Lewiston, ME · Remote
$61.63 - $88.47/hr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Lewiston, ME · Remote
$61.63 - $88.47/hr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Lewiston, ME · On-site
$166K - $191K/yr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Lewiston, ME · On-site
$166K - $191K/yr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Augusta, ME · Remote
$61.63 - $88.47/hr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Augusta, ME · Remote
$61.63 - $88.47/hr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Portland, ME · Remote
$61.63 - $88.47/hr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Portland, ME · Remote
$61.63 - $88.47/hr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Biddeford, ME · On-site
$166K - $191K/yr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Biddeford, ME · On-site
$166K - $191K/yr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Auburn, ME · On-site
$166K - $191K/yr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Auburn, ME · On-site
$166K - $191K/yr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Portland, ME · On-site
$166K - $191K/yr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Portland, ME · On-site
$166K - $191K/yr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Bangor, ME · On-site
$166K - $191K/yr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Bangor, ME · On-site
$166K - $191K/yr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Augusta, ME · On-site
$166K - $191K/yr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Augusta, ME · On-site
$166K - $191K/yr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Auburn, ME · Remote
$61.63 - $88.47/hr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Auburn, ME · Remote
$61.63 - $88.47/hr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
South Portland, ME · On-site
$166K - $191K/yr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
South Portland, ME · On-site
$166K - $191K/yr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Biddeford, ME · Remote
$61.63 - $88.47/hr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Biddeford, ME · Remote
$61.63 - $88.47/hr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
South Portland, ME · Remote
$61.63 - $88.47/hr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
South Portland, ME · Remote
$61.63 - $88.47/hr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Bangor, ME · Remote
$61.63 - $88.47/hr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
Bangor, ME · Remote
$61.63 - $88.47/hr
Work within a small team of mobile engineers to design, build, and maintain the Poe iOS app * Build ... for our job openings. We encourage all candidates who share a passion for growing the world ...
As a Scientific Software Engineer, you will collaboratively shape the machine learning and computer ... Contribute to the publications and open-source releases that come out of these projects. * 20 ...
As a Scientific Software Engineer, you will collaboratively shape the machine learning and computer ... Contribute to the publications and open-source releases that come out of these projects. * 20 ...
Biddeford, ME • On-site
$20 - $30/hr
Full-time
Posted 12 days ago
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
