Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering ... Review -- Take an existing annotation, written by our AI tooling or another person, and confirm ...
Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering ... Review -- Take an existing annotation, written by our AI tooling or another person, and confirm ...
AI Agent Trajectory Annotator and Reviewer
Whitewater, WI · On-site
$20 - $30/hr
Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering ... Review -- Take an existing annotation, written by our AI tooling or another person, and confirm ...
AI Agent Trajectory Annotator and Reviewer
Whitewater, WI · On-site
$20 - $30/hr
Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering ... Review -- Take an existing annotation, written by our AI tooling or another person, and confirm ...
AI Agent Trajectory Annotator and Reviewer
Madison, WI · On-site
$20 - $30/hr
Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering ... Review -- Take an existing annotation, written by our AI tooling or another person, and confirm ...
AI Agent Trajectory Annotator and Reviewer
Madison, WI · On-site
$20 - $30/hr
Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering ... Review -- Take an existing annotation, written by our AI tooling or another person, and confirm ...
Full Stack .NET Developer
Madison, WI · Remote
$50 - $55/hr
Expectations for this position are development of quality code, following coding standards, active participation in code reviews, sharing expertise, respectful communication, transparency, technical ...
Quick apply
Full Stack .NET Developer
Madison, WI · Remote
$50 - $55/hr
Expectations for this position are development of quality code, following coding standards, active participation in code reviews, sharing expertise, respectful communication, transparency, technical ...
Inpatient Medical Coder FT - Up to $5k Sign-on Bonus
Madison, WI · On-site
$32 - $42/hr
Experience with significant level of coding quality review feedback * Experience in computerized encoding and abstracting software What We Offer: * Benefits for Full-Time employees: Medical, Dental ...
Inpatient Medical Coder FT - Up to $5k Sign-on Bonus
Madison, WI · On-site
$32 - $42/hr
Experience with significant level of coding quality review feedback * Experience in computerized encoding and abstracting software What We Offer: * Benefits for Full-Time employees: Medical, Dental ...
NET Web Developer -III
Madison, WI · On-site
Expectations for this position include development of quality code, active participation in code reviews, interest in understanding the business reasons behind coding design/decisions (looking at the ...
Quick apply
NET Web Developer -III
Madison, WI · On-site
Expectations for this position include development of quality code, active participation in code reviews, interest in understanding the business reasons behind coding design/decisions (looking at the ...
.NET Web Developer -II
Madison, WI · Remote
Expectations for this position are development of quality code using TDD, participation in active code reviews, sharing expertise, respectful communication, transparency, technical documentation, and ...
.NET Web Developer -II
Madison, WI · Remote
Expectations for this position are development of quality code using TDD, participation in active code reviews, sharing expertise, respectful communication, transparency, technical documentation, and ...
.NET Web Developer -II
Madison, WI · Remote
Expectations for this position are development of quality code using TDD, participation in active code reviews, sharing expertise, respectful communication, transparency, technical documentation, and ...
.NET Web Developer -II
Madison, WI · Remote
Expectations for this position are development of quality code using TDD, participation in active code reviews, sharing expertise, respectful communication, transparency, technical documentation, and ...
Software Tutor
Madison, WI · Remote
$40/hr
Guides students through requirements analysis, architectural planning, code implementation, unit and integration testing, code review practices, and deployment procedures. Emphasizes writing ...
Software Tutor
Madison, WI · Remote
$40/hr
Guides students through requirements analysis, architectural planning, code implementation, unit and integration testing, code review practices, and deployment procedures. Emphasizes writing ...
Participate in code reviews, provide constructive feedback, and contribute to a positive team culture. * Stay up-to-date with emerging technologies, industry trends, and best practices to ...
Participate in code reviews, provide constructive feedback, and contribute to a positive team culture. * Stay up-to-date with emerging technologies, industry trends, and best practices to ...
Participate in code reviews, provide constructive feedback, and contribute to a positive team culture. * Stay up-to-date with emerging technologies, industry trends, and best practices to ...
Participate in code reviews, provide constructive feedback, and contribute to a positive team culture. * Stay up-to-date with emerging technologies, industry trends, and best practices to ...
Participate in code reviews, provide constructive feedback, and contribute to a positive team culture. * Stay up-to-date with emerging technologies, industry trends, and best practices to ...
Participate in code reviews, provide constructive feedback, and contribute to a positive team culture. * Stay up-to-date with emerging technologies, industry trends, and best practices to ...
Participate in code reviews, provide constructive feedback, and contribute to a positive team culture. * Stay up-to-date with emerging technologies, industry trends, and best practices to ...
Participate in code reviews, provide constructive feedback, and contribute to a positive team culture. * Stay up-to-date with emerging technologies, industry trends, and best practices to ...
Participate actively in code reviews to enforce best practices and patterns * Help troubleshoot and resolve complex technical issues in our distributed systems * Give and solicit feedback on ...
Participate actively in code reviews to enforce best practices and patterns * Help troubleshoot and resolve complex technical issues in our distributed systems * Give and solicit feedback on ...
.Net Developer [ Hybrid]
Madison, WI · On-site
Expectations for this position are development of quality code, participation in active code reviews, sharing expertise, respectful communication, transparency, and technical documentation. Knowledge ...
Quick apply
.Net Developer [ Hybrid]
Madison, WI · On-site
Expectations for this position are development of quality code, participation in active code reviews, sharing expertise, respectful communication, transparency, and technical documentation. Knowledge ...
.NET Developer
Madison, WI · Remote
$45 - $50/hr
Expectations for this position are development of quality code using TDD, participation in active code reviews, sharing expertise, respectful communication, transparency, technical documentation, and ...
Quick apply
.NET Developer
Madison, WI · Remote
$45 - $50/hr
Expectations for this position are development of quality code using TDD, participation in active code reviews, sharing expertise, respectful communication, transparency, technical documentation, and ...
Lead Palantir Application Engineer
$103K - $136K/yr
Review complex designs and code,identifytechnical risks, challenge weak implementation approaches, and ensure applications are built to meet business goals and long-term support expectations.
Lead Palantir Application Engineer
$103K - $136K/yr
Review complex designs and code,identifytechnical risks, challenge weak implementation approaches, and ensure applications are built to meet business goals and long-term support expectations.
ERP Developer lll
Waterloo, WI · On-site
Perform code reviews and enforce development standards, documentation practices, and best practices across the team. * Triage and resolve complex production issues across the ERP ecosystem and ...
ERP Developer lll
Waterloo, WI · On-site
Perform code reviews and enforce development standards, documentation practices, and best practices across the team. * Triage and resolve complex production issues across the ERP ecosystem and ...
ERP Developer lll
Waterloo, WI · On-site
Perform code reviews and enforce development standards, documentation practices, and best practices across the team. * Triage and resolve complex production issues across the ERP ecosystem and ...
ERP Developer lll
Waterloo, WI · On-site
Perform code reviews and enforce development standards, documentation practices, and best practices across the team. * Triage and resolve complex production issues across the ERP ecosystem and ...
Application Architect II
Madison, WI · On-site
We are looking for a team player that promotes sound coding practices, code reviews, documentation, and improves processes leading to cost effectiveness throughout the product development life cycle.
Quick apply
Application Architect II
Madison, WI · On-site
We are looking for a team player that promotes sound coding practices, code reviews, documentation, and improves processes leading to cost effectiveness throughout the product development life cycle.
Coding Reviewer information
What is a coding reviewer?
How does a coding reviewer typically collaborate with software development teams during the review process?
What are the key skills and qualifications needed to thrive as a coding reviewer, and why are they important?
What is the difference between Coding Reviewer vs Medical Coder?
| Aspect | Coding Reviewer | Medical Coder |
|---|---|---|
| Credentials | Certification in coding (e.g., CPC, CCS) | Certification in coding (e.g., CPC, CCS) |
| Work Environment | Reviewing medical codes, often in healthcare settings | Assigning medical codes from patient records |
| Industry Usage | Used in hospitals, clinics, insurance companies | Used in hospitals, clinics, billing companies |
| Primary Role | Review and validate coding accuracy | Assign and input medical codes |
Both Coding Reviewers and Medical Coders require similar certifications and work in healthcare environments. Coding Reviewers focus on verifying the accuracy of codes assigned by others, ensuring compliance and correctness. Medical Coders are responsible for assigning the initial codes based on medical records. While their roles are interconnected, Coding Reviewers primarily audit and validate, whereas Medical Coders handle the coding process itself.
AI Agent Trajectory Annotator and Reviewer
Madison, WI
$20 - $30/hr
Full-time
Posted 12 days ago
Job description
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.