Job Summary:
Cogeco Inc. is a company focused on creating genuine connections and a dynamic work environment. They are seeking a Specialist in QA Test Automation to bridge the gap between strategic vision and technical execution, particularly in the realm of AI-powered solutions.
Responsibilities:
• Embrace the Unpredictable: Implement and refine testing workflows tailored for AI powered solutions where outputs are probabilistic rather than deterministic.
• Build guardrails to identify model drift, prompt regression, hallucination risks, and safety violations before they hit production.
• Own the creation, maintenance, and scaling of high-quality golden datasets used as the source of truth for benchmarking model changes.
• Champion programmatic evaluation tools such as Promptfoo, DeepEval, or Ragas to automate the assessment of prompt variants and model performance.
• Partner closely with the Chapter Lead, QA to translate high-level AI QA frameworks into actionable, daily tasks for technical teams.
• Provide real-time feedback to the Chapter Lead on how the strategy is performing in the trenches, recommending practical pivots based on real-world results.
• Collaborate with DevOps specialists to inject AI evaluation metrics directly into modern CI/CD pipelines, ensuring automated gates handle fluid AI responses.
• Validate data pipelines feeding our models, ensuring that data integrity is maintained from ingestion to inference.
Qualifications:
Required:
• College diploma or Degree in Computer Science, Engineering, or a related discipline (or equivalent practical experience).
• A strong foundation in core QA principles combined with a passion for AI/ML.
• 3+ years of experience in Quality Assurance, Software Engineering, or Data Engineering.
• Experience in Python or JavaScript/TypeScript (essential for configuring evaluation frameworks like Promptfoo and manipulating test data).
• Experience with test automation frameworks, API testing, and code repositories (Git).
• Comfortable analyzing data inputs/outputs and debugging prompts to identify the root causes of unexpected AI behavior.
• Ready to quickly learn, test, and implement new open-source AI evaluation and observability tooling as the ecosystem evolves.
• Thrives in a high-uncertainty environment where standard QA blueprints don’t always apply.
• Excel at taking an established roadmap or framework and running with it, troubleshooting technical blockers independently.
• Able to communicate technical testing gaps clearly to both AI developers and product owners.
• Open to constructive feedback, willing to experiment, and values an environment where mistakes are treated as learning milestones.
Preferred:
• Exposure to Golden Datasets: creating, maintenance, and/or scaling of high-quality golden datasets used as the source of truth for benchmarking model changes.
• Exposure to AI Eval Tooling: Champion programmatic evaluation tools such as Promptfoo, DeepEval, or Ragas to automate the assessment of prompt variants and model performance.
• Experience working with pipeline tools (e.g., Bitbucket pipelines, Google Cloud run, GitHub Actions, GitLab CI, Jenkins).
Company:
Rooted in the communities it serves, Cogeco Inc. Founded in 1957, the company is headquartered in Montreal, CAN, with a team of 1001-5000 employees. The company is currently Late Stage.