Senior Software Engineer In Test Jobs in San Jose, CA

Senior Software Engineer In Test

We are hiring a hands-on Senior Software Engineer in Test to design and operationalize a scalable, automation-first quality framework across our software and AI-driven systems. This role owns test strategy, infrastructure, and execution, ensuring high-confidence releases across APIs, cloud services, data pipelines, and AI/ML components.

The mandate is twofold: (1) build robust, modern testing systems and (2) embed a pragmatic culture of quality that keeps pace with rapid product development.

Key Responsibilities

• Test Architecture & Infrastructure · Design and implement a unified test framework across: · Backend services and APIs · Cloud platforms and distributed systems · Data pipelines and data quality layers · AI application and evaluation systems · Define test environments, mocking/simulation strategies, and synthetic data generation. · Build and maintain CI/CD pipelines. · Integrate testing deeply into CI/CD pipelines with clear gating signals.
• Automated Testing & Tooling · Build and maintain automated test suites: · Unit, integration, system, regression, and performance testing · Develop test orchestration, reporting dashboards, and failure triage workflows. · Ensure tests are deterministic, reproducible, and fast enough for developer iteration.
• Data & Pipeline Validation · Establish validation strategies for data pipelines: · Schema validation, anomaly detection, and data integrity checks · Build automated tests for ETL workflows and downstream system dependencies. · Ensure reproducibility between offline experimentation and production behavior.
• Debugging & Root Cause Analysis · Lead investigation of complex failures across services, data, and AI layers. · Establish structured approaches to failure classification and regression prevention.
• AI/ML Testing & Evaluation · Build continuous evaluation pipelines tied to model releases. · Define acceptance criteria and release gates for AI features (beyond traditional QA). · Develop benchmarking tools for comparing models across datasets and scenarios.
• Quality Culture & Process · Introduce scalable quality practices: · Shift-left testing and testability in design · Definition of done includes validation and observability · Partner with engineering and product to define: · Measurable quality metrics (e.g., defect escape rate, test signal quality) · Release criteria aligned with risk · Balance thoroughness with speed—avoid over-engineering test systems.

Required Qualifications

• 6–10+ years in software engineering, software testing or SDET.
• Strong programming skills (Python required; experience with backend systems preferred).
• Proven track record building test frameworks and automation from scratch.
• Deep understanding of:
• API testing, distributed systems, and cloud architectures
• CI/CD systems (e.g., GitHub Actions, Jenkins)
• Test methodologies (boundary testing, fuzzing, fault injection)
• Experience validating AI/ML systems is a plus, including:
• Model evaluation metrics and tradeoffs
• Dataset validation and management
• Experiment tracking tools (e.g., MLflow, Weights & Biases, or equivalent)
• Experience with LLM or computer vision evaluation is a plus.

Preferred Qualifications

• Experience testing data-intensive systems or analytics platforms.
• Familiarity with data engineering tools and workflows.
• Experience with performance, scalability, and reliability testing.
• Exposure to observability tooling (logs, metrics, tracing) for test validation.
• Experience working in fast-paced product environments with evolving requirements.
• Design and implement evaluation frameworks for AI/ML systems:
• Model performance (accuracy, precision/recall, etc.)
• Robustness, edge cases, and failure modes
• Data quality, drift detection, and dataset versioning

What We're Looking For

· Builder mindset: Creates frameworks and tools, not just test cases. · Systems thinker: Understands interactions across APIs, data, and AI layers. · Pragmatic operator: Applies the right level of rigor for the stage of product maturity. · Quality driver: Elevates engineering standards without becoming a bottleneck. · Hands-on depth: Writes code, debugs systems, and owns outcomes.

Success Metrics (First 6–12 Months)

· Production-grade automated test framework integrated into CI/CD. · Clear, adopted AI evaluation framework used across AI applications releases. · Reduction in escaped defects and regression incidents. · Measurable improvement in data pipeline reliability and validation coverage. · Increased developer confidence in releases without slowing iteration speed.

Why Join Covalent

At Covalent, you'll work alongside world-class scientists and engineers in a dynamic, collaborative environment. We empower our team members to take ownership of their work, innovate constantly, and engage directly with customers, shaping the future of technology.