ZipRecruiter

Log In

1

Judge In Jobs Near Me

Judge Group, Inc.

Agentic AI/Java/Python

Columbus, OH · On-site

Strong proficiency in Python and Java, with hands-on experience developing and supporting ... Consistent with Judge's Privacy Policy, information obtained from your consent will not be shared ...

The Judge Group

Sales Director

Westerville, OH · On-site +1

Review and qualify sales opportunities based on The Judge Group criteria and strategy * Attend and actively participate in daily, weekly, and monthly team meetings * Plan and facilitate events to ...

Judge Group, Inc.

Mainframe Engineer

Dublin, OH · On-site

$50 - $60/hr

Participate in code reviews and ensure adherence to coding standards and best practices ... Consistent with Judge's Privacy Policy, information obtained from your consent will not be shared ...

Judge Group, Inc.

Lead Workday Engineer

Columbus, OH · On-site

$75 - $85/hr

... in large-scale, highly integrated enterprise environments. * Knowledge of BIRT reporting. By providing your phone number, you consent to: (1) receive automated text messages and calls from the Judge ...

The Judge Group

Sales Director

Westerville, OH · On-site +1

Review and qualify sales opportunities based on The Judge Group criteria and strategy * Attend and actively participate in daily, weekly, and monthly team meetings * Plan and facilitate events to ...

Judge Group, Inc.

Site Reliability Engineer

Columbus, OH · On-site

$65 - $70/hr

Write secure, high-quality, and maintainable code in at least one modern programming language with ... Consistent with Judge's Privacy Policy, information obtained from your consent will not be shared ...

Judge Group, Inc.

java developer

Columbus, OH · On-site

$55 - $65/hr

No sponsorship or C2C for this role Must be a w2 employee of The Judge Group Sr. Java Developer ... The ideal candidate will have strong expertise in Java, Spring Boot, Microservices, REST APIs, and ...

The Judge Group

Sales Director

Westerville, OH · On-site

Review and qualify sales opportunities based on The Judge Group criteria and strategy * Attend and actively participate in daily, weekly, and monthly team meetings * Plan and facilitate events to ...

Judge Group, Inc.

Full Stack Java developer

Westerville, OH · On-site

$51 - $65.75/hr

Ability to work in large, collaborative teams to achieve organizational goals, and passionate about ... Consistent with Judge's Privacy Policy, information obtained from your consent will not be shared ...

Judge Group, Inc.

Salesforce Developer

Columbus, OH · On-site

$45 - $50/hr

Experience in developing Experience Cloud sites. By providing your phone number, you consent to: (1) receive automated text messages and calls from the Judge Group, Inc. and its affiliates ...

Judge Group, Inc.

Data Architect: III (Senior)

Whitehall, OH · On-site

$65.25 - $87.50/hr

Experience in Financial Services or another regulated industry By providing your phone number, you consent to: (1) receive automated text messages and calls from the Judge Group, Inc. and its ...

Judge Group, Inc.

Automation Engineer (RPA)

Columbus, OH · On-site

$60 - $65/hr

Bachelor's degree in computer science, Information systems or related discipline, or equivalent and ... Consistent with Judge's Privacy Policy, information obtained from your consent will not be shared ...

next page

Showing results 1-20

All JobsJudge In JobsJudge In Jobs Near Me

Judge In information

What cities are hiring for Judge In jobs?

Cities with the most Judge In job openings:

What states have the most Judge In jobs?

States with the most job openings for Judge In jobs include:

Judge In jobs near you
A map of the United States highlighting the number of Judge In job openings by state according to ZipRecruiter. The image is accompanied by a detailed chart listing the number of Judge In job openings in each state, with California having the most at 2 and Hawaii the least at 0.

AI Agent Trajectory Annotator and Reviewer

Columbus, OH

Bespoke Labs
See more jobs at Bespoke Labsexternal

$20 - $30/hr

Full-time

Posted 10 days ago

Job description

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Apply