Learning Analytics Remote Jobs in New York (NOW HIRING)

Specialised Full-Time Consulting OpportunityWe are sharing a specialised full-time consulting opportunity for AI evaluation and research professionals with experience identifying failure modes, vulnerabilities, edge cases, and hidden weaknesses in large language models and machine learning systems.This role supports the development of advanced agentic evaluation benchmarks for frontier AI models. Selected professionals will probe complex model behaviour, design challenging multi-step tasks, document reproducible failures, and collaborate with researchers to strengthen benchmark quality across coding, machine learning, experimentation, and technical analysis.Key ResponsibilitiesAdversarial Model EvaluationProbe frontier AI models across coding, machine learning, analysis, and multi-step agentic tasksIdentify subtle errors, vulnerabilities, edge cases, and misleadingly plausible outputsInvestigate situations where models appear capable while reaching incorrect or unsupported conclusionsDesign reproducible experiments to isolate and validate model failure modesBenchmark & Challenge DesignConvert observed model weaknesses into rigorous benchmark tasksDevelop challenges that are technically demanding while remaining fair and objectively assessableDefine clear task requirements, expected outcomes, and evaluation criteriaEnsure tasks require genuine reasoning rather than allowing shortcuts or superficial pattern matchingFailure Analysis & DocumentationDocument findings with clear evidence, methodology, and reproducible stepsExplain why a model failed and which capabilities or assumptions contributed to the errorProduce detailed technical write-ups for researchers and task authorsTrack recurring failure patterns across models, prompts, and evaluation environmentsTask Strengthening & Research CollaborationWork with task authors to close loopholes, grading gaps, and unintended shortcutsReview benchmark tasks for ambiguity, exploitability, and evaluation reliabilityShare insights with researchers and other specialists to improve benchmark coverageParticipate in iterative calibration, peer review, and task-refinement workflowsIdeal ProfileStrong candidates may have:At least 1 year of experience in research, research engineering, security, AI evaluation, or a related technical roleDemonstrated experience identifying vulnerabilities, edge cases, or failure modes in LLMs or ML systemsBackground in red teaming, adversarial testing, security research, benchmark development, or rigorous model evaluationWorking proficiency in Python and GitAbility to develop scripts, probes, and analyses independentlyStrong familiarity with LLM capabilities, limitations, and evaluation techniquesExcellent written communication and technical documentation skillsCreativity, precision, and persistence when working through ambiguous research problemsReliable availability for approximately 35 hours per weekEducational BackgroundA master's degree or PhD in a STEM field is highly relevantEquivalent practical experience in a research-intensive domain involving coding and data analysis may also be consideredAcademic or professional work involving machine learning, computer science, statistics, security, mathematics, or engineering may strengthen an applicationPublications, benchmark contributions, technical research, or impactful open-source work may also be valuableNice to HaveExperience in AI training, model evaluation, or benchmark authoringBackground developing adversarial prompts or red-team evaluation suitesFamiliarity with agentic systems and multi-step tool-use evaluationsExperience assessing coding, ML, or technical-analysis tasksKnowledge of experimental design and reproducibilityExperience developing grading rubrics or automated evaluation methodsFamiliarity with security research or vulnerability assessmentPrior collaboration with AI research or engineering teamsWhy This OpportunityInvestigate where frontier AI models fail across complex technical tasksHelp build stronger and more reliable agentic evaluation benchmarksWork directly with researchers on high-impact AI evaluation challengesApply coding, experimentation, and analytical expertise to open-ended problemsContribute to stronger evaluation standards for advanced AI systemsParticipate in a structured full-time remote role with competitive hourly compensationContract DetailsFull-time W-2 contingent employment opportunityFully remote within the United StatesExpected commitment of approximately 35 hours per weekCompetitive rates between $55–$85 per hour depending on expertise and project scopeIndividual tasks may require one to two days of focused technical workWork may include model probing, benchmark design, failure analysis, technical documentation, and task refinementClose collaboration with research and benchmark-development teamsEngagement scope and duration may evolve according to project requirements and performanceAbout the PlatformThis opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy.