Description Please submit your CV in English and indicate your level of English proficiency. Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on ...
Aitrainer
16 jobs near Columbus, OH
Description Please submit your CV in English and indicate your level of English proficiency. Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on ...
$75/hr
Up to $75/hr Depending on the project Ph.D's , Postdocs Remote Part-time Flexible hours OPT/CPT Eligible Opportunity Overview Handshake is looking for experienced mathematicians and research ...
$75/hr
Up to $75/hr Depending on the project Ph.D's , Postdocs Remote Part-time Flexible hours OPT/CPT Eligible Opportunity Overview Handshake is looking for experienced mathematicians and research ...
Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels the development of the most advanced AI models. Overview A leading AI lab is ...
Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels the development of the most advanced AI models. Overview A leading AI lab is ...
[Vietnamese-US] - Voice Recording Specialist United States AI Trainers - Domain Experts - Data Collection / Independent Contractor - Project Based / Remote Join our team as an Independent Contractor ...
[Vietnamese-US] - Voice Recording Specialist United States AI Trainers - Domain Experts - Data Collection / Independent Contractor - Project Based / Remote Join our team as an Independent Contractor ...
Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels the development of the most advanced AI models. Overview We are hiring a senior ...
Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels the development of the most advanced AI models. Overview We are hiring a senior ...
$50/hr
Remote, United States. Fully Remote. Literary Content Localization & Creative Writing Specialist. Remote freelance Literary Content Localization & Quality Evaluator needed: refine plots, characters ...
$50/hr
Remote, United States. Fully Remote. Literary Content Localization & Creative Writing Specialist. Remote freelance Literary Content Localization & Quality Evaluator needed: refine plots, characters ...
Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels the development of the most advanced AI models. Overview We are hiring a senior ...
Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels the development of the most advanced AI models. Overview We are hiring a senior ...
$100/hr
Up to $100/hr depending on the project. Based on the project. This role is hourly contract work. Handshake is seeking experienced Technical Writers and documentation professionals to support AI ...
$100/hr
Up to $100/hr depending on the project. Based on the project. This role is hourly contract work. Handshake is seeking experienced Technical Writers and documentation professionals to support AI ...
$50/hr
Description Please submit your CV in English and indicate your level of English proficiency. Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on ...
$50/hr
Description Please submit your CV in English and indicate your level of English proficiency. Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on ...
UX Research Analyst (US-Remote) Remote, United States Fully Remote Join a US-remote UX Research Analyst project through May 2027. Build and manage Qualtrics studies, analyze survey results, and help ...
UX Research Analyst (US-Remote) Remote, United States Fully Remote Join a US-remote UX Research Analyst project through May 2027. Build and manage Qualtrics studies, analyze survey results, and help ...
$57.50/hr
Registered Nurse (California, Remote) Remote, United States Fully Remote Work Location California, United States (Remote) Engagement Model Employment; fixed-term/temporary Start Date ASAP Project ...
$57.50/hr
Registered Nurse (California, Remote) Remote, United States Fully Remote Work Location California, United States (Remote) Engagement Model Employment; fixed-term/temporary Start Date ASAP Project ...
$150/hr
STEM PROFESSOR Up to $150/hr Program Overview Handshake is recruiting exceptional STEM professors to join our AI research community. This program brings subject-matter experts together to enhance the ...
$150/hr
STEM PROFESSOR Up to $150/hr Program Overview Handshake is recruiting exceptional STEM professors to join our AI research community. This program brings subject-matter experts together to enhance the ...
Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels the development of the most advanced AI models. Overview A leading AI lab is ...
Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels the development of the most advanced AI models. Overview A leading AI lab is ...
Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels the development of the most advanced AI models. Overview We are hiring a senior ...
Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels the development of the most advanced AI models. Overview We are hiring a senior ...
$45/hr
# Senior Python Data Scraping Engineer (Freelance) at Toloka AI* Part-time* Remote, South Carolina, United States CoApply nowDescription Mindrift is looking for highly skilled Vibecode specialists to ...
$45/hr
# Senior Python Data Scraping Engineer (Freelance) at Toloka AI* Part-time* Remote, South Carolina, United States CoApply nowDescription Mindrift is looking for highly skilled Vibecode specialists to ...
Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels the development of the most advanced AI models. Overview We are hiring a senior ...
Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels the development of the most advanced AI models. Overview We are hiring a senior ...
Software Engineer - AI Agent Evaluation (Remote) at Toloka AI
Manhattan, NY • On-site
aitrainer
$50/hr
Other
Posted 12 days ago
Job description
Description Please submit your CV in English and indicate your level of English proficiency. Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.
What this opportunity involvesYou'll create challenging tasks and evaluation criteria within realistic simulated environments:
- Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
- Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
- Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
- Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust
- Not data labeling
- Not prompt engineering
- Not writing code from scratch - the agent writes most of the code; you guide and evaluate
- 5+ years in software development
- Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
- Experience writing tests (functional, integration)
- English proficiency - B2+
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.
CompensationUp to $50/hr equivalent , depending on level and pace. Tasks are estimated at ~20 hours each; you set your own schedule.