Remote Pcb Test Engineer Jobs in New York (NOW HIRING)

Specialised Full-Time Consulting Opportunity for Experienced Software EngineersWe are sharing a specialised full-time consulting opportunity for experienced software engineers with strong Python development, debugging, version-control, technical documentation, and AI-assisted coding experience.

This role supports the development of advanced agentic evaluation benchmarks for frontier AI models. Selected professionals will design, implement, and review realistic multi-step software engineering tasks that test the capabilities of AI coding agents across Python development, environment setup, tooling, debugging, and technical problem-solving.

Key ResponsibilitiesSoftware Engineering Task DesignCreate realistic, multi-step software engineering challenges based on practical development workflowsDesign technically demanding problems that require implementation, debugging, environment configuration, and analytical reasoningDefine clear requirements, constraints, expected outputs, and acceptance criteriaEnsure tasks assess genuine software engineering capability rather than superficial code generationReference Solution DevelopmentBuild complete and verifiable reference solutions in PythonCreate the supporting setup, dependencies, tests, and validation checks required for each taskWrite clean, readable, and maintainable codeConfirm that solutions run reliably within the intended technical environmentDocument implementation decisions and expected behaviour clearlyAI Coding Agent EvaluationUse AI coding assistants and agent-based development tools within practical engineering workflowsEvaluate how frontier models approach complex coding and debugging tasksIdentify implementation errors, unsupported assumptions, inefficient approaches, and incomplete solutionsAnalyse where AI agents succeed, struggle, or exploit unintended shortcutsDocument failure patterns and provide evidence supporting evaluation conclusionsPeer Review & Task RefinementReview tasks and reference solutions created by other software engineering specialistsAssess clarity, correctness, difficulty, reproducibility, and technical fairnessIdentify ambiguous instructions, hidden assumptions, grading gaps, and environment issuesProvide actionable feedback that improves task quality and benchmark reliabilityCollaborate closely with researchers and fellow task authorsIdeal ProfileStrong candidates may have: At least 1 year of experience in software engineering, research engineering, or a related coding-intensive roleStrong hands-on Python scripting, implementation, and debugging skillsExperience developing clean, readable, and maintainable softwareEveryday fluency with Git, IDEs, repositories, and standard software development workflowsComfort configuring environments, dependencies, tooling, and validation processesStrong technical writing and documentation skillsAbility to work independently through ambiguous and open-ended engineering problemsReliable availability for approximately 35 hours per weekEducational BackgroundAn MSc or PhD in computer science, software engineering, another STEM discipline, or a related technical field is highly relevantEquivalent practical experience in a research-intensive or engineering-intensive role may also be consideredAcademic or professional work involving significant coding, data analysis, or technical experimentation may strengthen an applicationOpen-source contributions, technical projects, publications, or substantial software development work may also be valuableNice to HaveExperience using AI coding assistants, prompt engineering methods, or agent-based workflowsPrevious work in AI training, model evaluation, or benchmark developmentBackground authoring technical tasks, reference solutions, or grading criteriaFamiliarity with automated testing, CI/CD workflows, containers, or reproducible environmentsExperience reviewing code or technical assignments created by other engineersKnowledge of agentic AI systems and multi-step coding evaluationsStrong ability to identify edge cases, unintended shortcuts, and subtle implementation issuesExperience collaborating with AI research or evaluation teamsWhy This OpportunityApply practical software engineering expertise to frontier AI evaluationDesign realistic coding tasks grounded in professional development workflowsHelp researchers understand where advanced AI coding agents succeed and failWork across Python implementation, debugging, environment setup, and benchmark developmentCollaborate closely with AI researchers and experienced software engineersParticipate in a structured full-time remote role with competitive hourly compensationContract DetailsFull-time W-2 contingent employment opportunityFully remote within the United StatesExpected commitment of approximately 35 hours per weekCompetitive rates between $55–$85 per hour depending on expertise and project scopeIndividual tasks may require one to two days of focused engineering workWork may include task design, Python development, reference-solution creation, AI agent evaluation, peer review, and technical documentationEngagement scope and duration may evolve according to project requirements and performanceAbout the PlatformThis opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.

By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy.