QA Lead - AI Systems & Models Testing Quality Assurance Artificial Intelligence Contract Position Contract Montreal, QC AI / ML Testing LLM / RAG / LangChain ABOUT THE ROLE We are seeking an ...
Jay Analytix
36 Jay Analytix Jobs Hiring Near You
QA Lead - AI Systems & Models Testing Quality Assurance Artificial Intelligence Contract Position Contract Montreal, QC AI / ML Testing LLM / RAG / LangChain ABOUT THE ROLE We are seeking an ...
GLOSS Business Analyst / Subject Matter Expert Contract or Full-Time | Hybrid | Toronto, ON | Montral, QC | New York, NY | New Jersey, NJ ABOUT THE ROLE We are seeking a seasoned GLOSS Business ...
Quick apply
GLOSS Business Analyst / Subject Matter Expert Contract or Full-Time | Hybrid | Toronto, ON | Montral, QC | New York, NY | New Jersey, NJ ABOUT THE ROLE We are seeking a seasoned GLOSS Business ...
Job Title: Java Developer - AWS & Capital Markets Location: Toronto, ON (Hybrid) Job Type: Contract Duration: Long-Term Work Authorization: Must be authorized to work in Canada Job Summary We are ...
Job Title: Java Developer - AWS & Capital Markets Location: Toronto, ON (Hybrid) Job Type: Contract Duration: Long-Term Work Authorization: Must be authorized to work in Canada Job Summary We are ...
ICI Impact Accounting Subject Matter Expert Contract or Full-Time | Hybrid | Toronto, ON | Montral, QC | New York, NY | New Jersey, NJ ABOUT THE ROLE We are looking for a highly skilled ICI Impact ...
Quick apply
ICI Impact Accounting Subject Matter Expert Contract or Full-Time | Hybrid | Toronto, ON | Montral, QC | New York, NY | New Jersey, NJ ABOUT THE ROLE We are looking for a highly skilled ICI Impact ...
Quantative Developer
Toronto, ON · On-site
Quantitative Developer Location: Toronto or Montreal (Canada) / New York or New Jersey (USA) Hybrid Employment Type: Contract About the Role We are seeking a Quantitative Developer with strong ...
Quick apply
Quantative Developer
Toronto, ON · On-site
Quantitative Developer Location: Toronto or Montreal (Canada) / New York or New Jersey (USA) Hybrid Employment Type: Contract About the Role We are seeking a Quantitative Developer with strong ...
Only candidates who have work authorization in Canada/USA may apply. Business/Data Analyst - Capital Market Contract Hybrid Toronto / Halifax What is the opportunity? The primary purpose of this ...
Only candidates who have work authorization in Canada/USA may apply. Business/Data Analyst - Capital Market Contract Hybrid Toronto / Halifax What is the opportunity? The primary purpose of this ...
Job Title: Java Developer AWS & Capital Markets Location: Toronto, ON (Hybrid) Job Type: Contract Duration: Long-Term Work Authorization: Must be authorized to work in Canada Job Summary We are ...
Quick apply
Job Title: Java Developer AWS & Capital Markets Location: Toronto, ON (Hybrid) Job Type: Contract Duration: Long-Term Work Authorization: Must be authorized to work in Canada Job Summary We are ...
AI Engineer - Banking Domain
Montreal, QC · On-site
Engagement Contract 12 months, renewable Domain Banking & Financial Services Locations Toronto, ON | Montreal, QC | Vancouver, BC | Calgary, AB Work Model Hybrid Start Immediate ABOUT THE ROLE We are ...
Quick apply
AI Engineer - Banking Domain
Montreal, QC · On-site
Engagement Contract 12 months, renewable Domain Banking & Financial Services Locations Toronto, ON | Montreal, QC | Vancouver, BC | Calgary, AB Work Model Hybrid Start Immediate ABOUT THE ROLE We are ...
Kubernetes Platform Engineer
Halifax, NS · On-site
Experience 7+ Location: Halifax (Hybrid - 3 days office) * Strong understanding of Platform engineering PodsContainers and experience in managing multi cluster environments. * Must have sound ...
Quick apply
Kubernetes Platform Engineer
Halifax, NS · On-site
Experience 7+ Location: Halifax (Hybrid - 3 days office) * Strong understanding of Platform engineering PodsContainers and experience in managing multi cluster environments. * Must have sound ...
Core Java Developer
Halifax, NS · On-site
Develops and maintains backend banking applications using Core Java, focusing on performance, security, and financial transaction processing in a regulated environment.
Quick apply
Core Java Developer
Halifax, NS · On-site
Develops and maintains backend banking applications using Core Java, focusing on performance, security, and financial transaction processing in a regulated environment.
QA Engineer
Halifax, NS · On-site
QA Analyst/Engineer (Manual & Automation) Location: Halifax, Nova Scotia (Hybrid) Employment Type: Contract About the Role We are seeking an experienced QA Analyst/Engineer with expertise in both ...
Quick apply
QA Engineer
Halifax, NS · On-site
QA Analyst/Engineer (Manual & Automation) Location: Halifax, Nova Scotia (Hybrid) Employment Type: Contract About the Role We are seeking an experienced QA Analyst/Engineer with expertise in both ...
We are seeking a Senior Java Full Stack Developer (7+ years) with strong financial services domain knowledge, especially in Capital Markets and Reconciliation systems . The role involves building ...
Quick apply
We are seeking a Senior Java Full Stack Developer (7+ years) with strong financial services domain knowledge, especially in Capital Markets and Reconciliation systems . The role involves building ...
QA Lead - Capital Markets
Montreal, QC · On-site
QA Lead - Capital Markets Location: Montreal, Quebec (OR) Halifax, Noca Scotia (Hybrid) Job Type: Contract Experience Required: 7+ Years Domain: Capital Markets Work Authorization: Candidates must be ...
QA Lead - Capital Markets
Montreal, QC · On-site
QA Lead - Capital Markets Location: Montreal, Quebec (OR) Halifax, Noca Scotia (Hybrid) Job Type: Contract Experience Required: 7+ Years Domain: Capital Markets Work Authorization: Candidates must be ...
We are seeking a Senior Java Full Stack Developer (7+ years) with strong financial services domain knowledge, especially in Capital Markets and Reconciliation systems . The role involves building ...
We are seeking a Senior Java Full Stack Developer (7+ years) with strong financial services domain knowledge, especially in Capital Markets and Reconciliation systems . The role involves building ...
Business Analyst - CTH (Client Analytics & Databricks | Swaps, Derivatives, Reference Data) - Toronto We are seeking a Business Analyst (Contract-to-Hire) with strong experience in capital markets ...
Business Analyst - CTH (Client Analytics & Databricks | Swaps, Derivatives, Reference Data) - Toronto We are seeking a Business Analyst (Contract-to-Hire) with strong experience in capital markets ...
Only candidates who have work authorization in Canada/USA may apply. Business/Data Analyst - Capital Market Contract Hybrid Toronto / Halifax What is the opportunity? The primary purpose of this ...
Quick apply
Only candidates who have work authorization in Canada/USA may apply. Business/Data Analyst - Capital Market Contract Hybrid Toronto / Halifax What is the opportunity? The primary purpose of this ...
Kubernetes Platform Engineer
Halifax, NS · Hybrid
Experience 7+ Location: Halifax (Hybrid - 3 days office) * Strong understanding of Platform engineering PodsContainers and experience in managing multi cluster environments. * Must have sound ...
Kubernetes Platform Engineer
Halifax, NS · Hybrid
Experience 7+ Location: Halifax (Hybrid - 3 days office) * Strong understanding of Platform engineering PodsContainers and experience in managing multi cluster environments. * Must have sound ...
DevOps Engineer - Hybrid
Toronto, ON · On-site
Location: Toronto OR Halifax Type: Contract Domain: Capital Markets *Only candidates with valid work authorization in Canada may apply.* What is the opportunity? As a solution engineer, you will be ...
Quick apply
DevOps Engineer - Hybrid
Toronto, ON · On-site
Location: Toronto OR Halifax Type: Contract Domain: Capital Markets *Only candidates with valid work authorization in Canada may apply.* What is the opportunity? As a solution engineer, you will be ...
Python Developer + Pricing
Montreal, QC · On-site
Role Overview We are seeking a Python Developer with experience in pricing models and banking systems to build, enhance, and support applications used for valuation, risk analysis, and trading. The ...
Quick apply
Python Developer + Pricing
Montreal, QC · On-site
Role Overview We are seeking a Python Developer with experience in pricing models and banking systems to build, enhance, and support applications used for valuation, risk analysis, and trading. The ...
ICI Impact - Accounting Subject Matter Expert
Toronto, ON · On-site +1
ICI Impact - Accounting Subject Matter Expert Contract or Full-Time | Hybrid | Toronto, ON | Montreal, QC | New York, NY | New Jersey, NJ ABOUT THE ROLE We are looking for a highly skilled ICI Impact ...
ICI Impact - Accounting Subject Matter Expert
Toronto, ON · On-site +1
ICI Impact - Accounting Subject Matter Expert Contract or Full-Time | Hybrid | Toronto, ON | Montreal, QC | New York, NY | New Jersey, NJ ABOUT THE ROLE We are looking for a highly skilled ICI Impact ...
Jay Analytix Jobs Information
Contractor
Posted 29 days ago
Job description
QA Lead - AI Systems & Models Testing
Quality Assurance Artificial Intelligence Contract Position
Contract
Montreal, QC
AI / ML Testing
LLM / RAG / LangChain
ABOUT THE ROLE
We are seeking an experienced QA Lead with deep expertise in AI systems testing to join our team on a contract basis in Montreal, Quebec. This role sits at the intersection of quality engineering and artificial intelligence, requiring hands-on proficiency in LLM behavior analysis, RAG pipeline validation, and modern AI orchestration frameworks. You will own the end-to-end test strategy for complex AI products and help define quality standards in a rapidly evolving space.
MUST-HAVE SKILLS
- Proven QA leadership experience designing and executing test strategies for AI/ML systems or LLM-powered applications.
- Strong understanding of LLM internals: tokenization, embeddings, attention mechanisms, and inference behavior to anticipate and diagnose failure modes.
- Hands-on experience with prompt engineering - constructing effective prompts, detecting hallucinations, and evaluating outputs across accuracy, tone, coherence, and bias dimensions.
- Experience testing RAG pipelines and knowledge base integrations, including validation of data quality and retrieval accuracy as they impact model outputs.
- Familiarity with vector database mechanics: similarity search thresholds, embedding drift, near-duplicate documents, and sparse vs. dense embeddings.
- Practical experience with LangChain and/or LangGraph - able to read chain/graph construction code, identify failure points, and write test harnesses.
- Ability to validate MCP (Model Context Protocol) integration points, including tool availability and error-handling scenarios.
- Proficiency applying generative AI evaluation metrics and establishing quality thresholds appropriate for production AI systems.
- Excellent written and verbal communication in English; bilingualism (English/French) is a plus for the Montreal market.
NICE-TO-HAVE SKILLS
- Experience with bias detection and safety testing frameworks for AI systems.
- Exposure to performance and scalability testing of vector databases under high load.
- Familiarity with CI/CD pipelines for ML model deployment and automated regression testing.
- Knowledge of responsible AI principles and AI governance frameworks.
- Contributions to or experience with open-source AI testing or evaluation tooling (e.g., DeepEval, Ragas, PromptFlow).
- Background in data engineering or data quality practices relevant to AI pipeline inputs.
- Cloud platform experience (AWS, Azure, or GCP) in the context of deploying or testing AI workloads.
KEY RESPONSIBILITIES
- Lead design and execution of comprehensive test strategies across AI systems, including prompt evaluation, output quality assessment, and bias/safety analysis.
- Develop and maintain test harnesses for LangChain and LangGraph-based applications; review chain and graph construction code to proactively surface integration risks.
- Validate RAG pipeline integrity - data ingestion, chunking, retrieval accuracy, and embedding consistency - and define edge-case coverage for vector database interactions.
- Establish and track generative AI quality metrics and thresholds; report on model output quality across multiple evaluation dimensions.
- Collaborate with ML engineers, data scientists, and product teams to embed quality practices throughout the AI development lifecycle.
- Document test findings clearly for both technical and non-technical stakeholders.
Contract position based in Montreal, Quebec, Canada On-site / Hybrid
Employment Type: CONTRACTOR