Pytorch Developer Jobs in Ontario (NOW HIRING)

Position Overview
As an Applied AI Developer - Agent Evaluation on the Fusion Platform Services team within Product Development and Manufacturing Solutions (PDMS), you'll be part of a team of technologists dedicated to creating cutting-edge AI and generative AI solutions that enhance developer productivity and experience. You'll work closely with AI engineers, software architects, and product engineering teams to build, deploy, and rigorously evaluate intelligent agentic systems while developing MCP (Model Context Protocol)-based tooling that integrates seamlessly with IDEs such as VS Code and Cursor.

• Develop and orchestrate multi-agent AI systems for automated test generation, test execution, and end-to-end development workflow optimization using frameworks such as LangGraph, AutoGen, or the Anthropic Agent SDK (Claude Code).
• Design and implement agentic workflows that coordinate multiple AI agents to autonomously drive test automation across UI, API, integration, and system levels, from test case synthesis to result evaluation, while integrating seamlessly with existing developer tools and MCP-compatible services.
• Build evaluation frameworks and benchmark suites for agentic systems, including comparisons of AI agents against commercial or domain-specific solutions, using tools such as AgentBench and Langfuse.
• Evaluate and optimize MCP servers and tool integrations across agentic pipelines by measuring latency, accuracy, context fidelity, and end-to-end task completion.

• BS/MS in Computer Science, Machine Learning, or a related applied AI field.
• Expertise in Python and ML frameworks (PyTorch, Transformers, scikit-learn).
• Experience with Large Language Models applied to software understanding, test generation, or agentic AI systems.
• Knowledge of AI evaluation methodologies and metrics for agentic task completion and test quality.
• Strong foundation in statistical analysis and experimental design.
• Experience with developer workflow and productivity measurement frameworks.

• Background in software engineering or QA with close collaboration with development teams.
• Familiarity with test automation frameworks (e.g., Playwright, Selenium, Pytest, Appium) and CI/CD pipelines.
• Experience designing evaluation frameworks or benchmark suites for AI agents against commercial or domain-specific solutions.
• Hands-on experience with MCP (Model Context Protocol), including building, evaluating, and optimizing MCP servers and tool integrations.
• Experience using AI evaluation and observability tools such as LangSmith, Langfuse, AgentBench, RAGAS, DeepEval, or similar platforms.
• Experience with Azure AI Foundry/ML or AWS cloud ML platforms.

En tant que Developpeur IA Appliquee - Evaluation d'Agents IA au sein de l'equipe Fusion Platform Services de Product Development and Manufacturing Solutions (PDMS), vous rejoindrez une equipe de technologues dediee a la creation de solutions d'intelligence artificielle et d'IA generative de pointe visant a ameliorer la productivite et l'experience des developpeurs. Vous collaborerez etroitement avec des ingenieurs IA, des architectes logiciels et des equipes de developpement produit afin de concevoir, deployer et evaluer rigoureusement des systemes d'IA agentiques, tout en developpant des outils bases sur le Model Context Protocol (MCP) integres a des environnements tels que VS Code et Cursor.

• Developper et orchestrer des systemes d'IA multi-agents pour la generation automatique de tests, leur execution et l'optimisation des workflows de developpement a l'aide de frameworks tels que LangGraph, AutoGen ou l'Anthropic Agent SDK (Claude Code).
• Concevoir et mettre en uvre des workflows agentiques permettant a plusieurs agents IA de piloter de maniere autonome l'automatisation des tests (UI, API, integration et systeme), depuis la generation des cas de test jusqu'a l'evaluation des resultats, tout en assurant une integration fluide avec les outils de developpement existants et les services compatibles MCP.
• Developper des frameworks d'evaluation et des suites de benchmarks pour les systemes agentiques, notamment afin de comparer les performances des agents IA a des solutions commerciales ou specialisees, a l'aide d'outils tels qu'AgentBench et Langfuse.
• Evaluer et optimiser les serveurs MCP ainsi que les integrations d'outils au sein des pipelines agentiques en mesurant la latence, la precision, la fidelite du contexte et le taux de reussite des taches de bout en bout.

• Diplome de niveau licence ou master en informatique, apprentissage automatique ou dans un domaine connexe de l'IA appliquee.
• Excellente maitrise de Python et des frameworks de machine learning (PyTorch, Transformers, scikit-learn).
• Experience avec les grands modeles de langage (LLM) appliques a la comprehension logicielle, a la generation de tests ou aux systemes d'IA agentiques.
• Connaissance des methodologies et des metriques d'evaluation des systemes d'IA pour mesurer la reussite des taches agentiques et la qualite des tests.
• Solides bases en analyse statistique et en conception d'experiences.
• Experience des outils et methodologies d'amelioration de la productivite des developpeurs.

• Experience en developpement logiciel ou en assurance qualite (QA), avec une collaboration etroite avec les equipes de developpement.
• Maitrise des frameworks d'automatisation des tests (Playwright, Selenium, Pytest, Appium) et des pipelines CI/CD.
• Experience dans la conception de frameworks d'evaluation ou de suites de benchmarks permettant de comparer des agents IA a des solutions commerciales ou specialisees.
• Experience pratique du Model Context Protocol (MCP), notamment dans la creation, l'evaluation et l'optimisation de serveurs MCP et de leurs integrations.
• Experience avec des outils d'evaluation et d'observabilite de l'IA tels que LangSmith, Langfuse, AgentBench, RAGAS, DeepEval ou des plateformes similaires.
• Experience des plateformes cloud d'IA telles qu'Azure AI Foundry/ML ou AWS ML.

Learn More

About Autodesk

Welcome to Autodesk! Amazing things are created every day with our software - from the greenest buildings and cleanest cars to the smartest factories and biggest hit movies. We help innovators turn their ideas into reality, transforming not only how things are made, but what can be made.

We take great pride in our culture here at Autodesk - it's at the core of everything we do. Our culture guides the way we work and treat each other, informs how we connect with customers and partners, and defines how we show up in the world.

When you're an Autodesker, you can do meaningful work that helps build a better world designed and made for all. Ready to shape the world and your future? Join us!

Salary transparency

Belonging
We take pride in cultivating a culture of belonging where everyone can thrive. Learn more here: https://www.autodesk.com/company/global-belonging

Are you an existing contractor or consultant with Autodesk?

Please search for open jobs and apply internally (not on this external site).