ZipRecruiter

Log In

Roo
Roo

60 Roo Jobs Hiring Near You

Capstone ISG

Field Property Adjuster

Manhattan, NY · On-site

$70K - $90K/yr

We are seeking a detail-oriented Property Claims Adjuster to independently manage residential and commercial property claims throughout the investigation, evaluation, and resolution process. This ...

Mindrift

Software Engineering Evaluation Specialist

$35/hr

AI coding agent experience - Claude Code, Cursor, Roo Code, or similar, on non-trivial work. You can cite a specific time the AI was confidently wrong and how you caught it. * English - B2+ written.

Unilock

Plant Quality Assurance Technician

Aurora, IL · On-site

$20 - $22/hr

Enter test results and inspection data into ROO.AI software. * Partner with operators and supervisors to identify and implement process improvements. * Lead Quality Daily Management (DMM) meetings ...

Two Circles

Consultant, Partnership Strategy

New York, NY · On-site

$70K - $100K/yr

... ROO. * Provide partners with dynamic measurement and reporting across key metrics, and demonstrate performance of partners across a variety of data types and sources. * Help implement the data ...

Capstone ISG

Field Property Adjuster

Wilmington, DE · On-site

We are seeking a detail-oriented Property Claims Adjuster to independently manage residential and commercial property claims throughout the investigation, evaluation, and resolution process. This ...

Stefanini

IT Specialist II

Aurora, CO · On-site

Stefanini Group is looking for a IT Specialist II for a globally recognized company! For interested applicants, click the apply button or you may reach out Micah Andres at / for faster processing.

previous page

Showing results 41-60

All JobsRoo Jobs

Software Engineering Evaluation Specialist

Mindrift

Manhattan, NY • On-site

$34.44/hr

Other

Posted 9 days ago

Job description

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

About the Role

You’ll design coding tasks that challenge frontier AI coding agents. Each task is a self-contained Docker environment with a broken piece of software; an AI agent attempts the fix; automated tests verify the outcome. Your deliverable is the full task package: broken code, tests, instructions, and a reference solution proving the task is solvable.

Responsibilities
  • Invent a realistic developer scenario — a real bug, a broken ETL, a missing feature — not a toy problem.
  • Build a reproducible Docker environment with pinned dependencies.
  • Write a pytest that verifies outcomes, not specific commands — deterministic, non-flaky, and does not leak the fix.
  • Write an instruction.md that reads like a Jira ticket a developer would receive.
  • Write a reference solve.sh proving the task is solvable.
  • Calibrate difficulty so current state-of-the-art agents solve the task 20–60% of the time.
  • Iterate based on feedback from expert QA reviewers.
  • Later: review other authors’ tasks as a QA reviewer.
Not in scope
  • Data labeling, prompt engineering.
  • Production code to ship — you design problems and verification for AI agents.
  • Leetcode puzzles — scenarios must look like real developer work.
  • Not every candidate task ships — quality over quantity.
Requirements
  • 3+ years of production software development in one backend stack — Python, Go, Node.js, Java, or Rust. Depth in one stack beats breadth.
  • Python + pytest fluency — required regardless of primary stack.
  • Docker authoring — reproducible Dockerfiles, pinned dependencies, multi-stage builds when needed, non-root user.
  • Linux & Bash — comfort debugging inside containers (strace, lsof, journalctl); shell beyond set -euo pipefail.
  • AI coding agent experience — Claude Code, Cursor, Roo Code, or similar, on non-trivial work. You can cite a specific time the AI was confidently wrong and how you caught it.
  • English — B2+ written.
Not a fit
  • Data Science, ML, or Computer Vision engineers without backend-engineering output.
  • Manual QA testers without automation or test authoring.
  • Frontend-only, low-code / no-code, IT Support, or Business Analysts.
  • Engineers who have never written pytest from scratch.
  • Junior, intern, or assistant as the most recent role.
Preferred qualifications
  • Domain depth in Security, System Administration (nginx / systemd / cron), Scientific Computing (NumPy / PyTorch / SciPy), DevOps, or Git internals.
  • Modern Python tooling (uv, poetry, pyproject.toml).
  • Coverage tooling (pytest-cov, coverage.py, gcov, llvm-cov, kcov).
  • Fuzzing or property-based testing (Hypothesis).
  • Prior contribution to agent-evaluation benchmarks or related frameworks.
Process

Apply → Pass qualification (90-minute sample-task screen + short behavioral interview) → Join a project → Complete tasks → Get paid.

Time commitment
  • Onboarding: ~10 hours per first task.
  • Steady state: ~5 hours per task, 2–4 parallel tasks per author.
  • Realistic weekly load: 8–20 hours. Higher volume available for top performers.
  • You choose when and how to contribute; tasks must be submitted by the deadline and meet acceptance criteria.
Compensation:
  • Paid contributions, rates up to $35/hour*.
  • Task-based compensation equivalent to hourly rate, depending on performance and volume.
  • Some projects include incentive payments.

*Rates vary based on expertise, skills assessment, location, project needs, and other factors. Higher rates may be provided to highly specialized experts. Lower rates may apply during onboarding or non-core project phases. Payment details are shared per project.

#J-18808-Ljbffr

Apply