Job Summary:
InterSources Inc is a company seeking a QA Test Engineer with strong AI experience. The role involves testing APIs for high performance with a primary focus on automation, requiring strong communication and analytical skills.
Responsibilities:
• 3+ years in QA automation or SDET-type work (adjust by level); 1+ year exposure to AI/LLM or ML-driven features is a plus.
• Strong test automation in Python and/or Java/TypeScript.
• We are a platform team, testing APIs for high performance, automation will be primary focus.
• Strong communication and analytical skills.
• Hands-on with frameworks/tools such as: UI: Playwright / Cypress / Selenium and API: pytest + requests, Postman/Newman, REST Assured.
• CI/CD integration: Git, GitHub Actions/Jenkins/GitLab CI, test reporting, gating.
• Test design: equivalence partitioning, boundary testing, risk-based testing, defect triage. AI-Specific Testing Competencies (Key)
• LLM/application behavior testing: validating correctness when outputs are probabilistic.
• Evaluation strategies: golden datasets, scoring rubrics, human-in-the-loop reviews.
• Non-determinism handling: statistical assertions, repeated runs, variance thresholds.
• Prompt and regression management: versioning prompts, detecting prompt drift, replay tests.
• RAG testing (if applicable): retrieval quality (recall/precision), grounding checks, citation validation, doc freshness.
• Safety & quality checks: hallucination detection, toxicity/PII leakage checks, policy compliance tests.
• Ability to create and maintain test datasets (structured + unstructured), including edge cases.
• Familiarity with telemetry for AI systems: - logging prompts/outputs safely, traceability, correlation IDs - tools like OpenTelemetry, ELK/Splunk, Datadog/Grafana (any equivalent)
• Understanding of data privacy constraints (masking/redaction) and secure test data practices.
• Comfortable testing distributed systems: microservices, async workflows, queues/events.
• Basic cloud proficiency (AWS/Azure/GCP) and containerization (Docker, optional Kubernetes). Performance & Reliability Testing (AI-Aware)
• Load/performance testing for inference endpoints (latency, throughput, concurrency).
• Cost-aware testing (token usage, rate limits, fallbacks).
• Resilience tests: retries, circuit breakers, model timeouts, degraded-mode behavior.
• Familiarity with NLP concepts (embeddings, context windows, temperature/top-p).
• Experience with AI tooling: LangChain/LlamaIndex, evaluation tools, model gateways.
• Knowledge of regulatory/security needs relevant to the telecom domain.
• Strong communication —able to explain AI quality issues clearly to product and engineering.
• Comfortable partnering with data science/ML engineers and backend teams.
• Ownership mindset: building reusable test harnesses, improving quality metrics, preventing regressions.
Qualifications:
Required:
• Must have strong AI experience
• Bachelor's in Computer Science, Engineering, Data/Information Systems, or equivalent practical experience.
• 3+ years in QA automation or SDET-type work (adjust by level); 1+ year exposure to AI/LLM or ML-driven features is a plus.
• Strong test automation in Python and/or Java/TypeScript.
• We are a platform team, testing APIs for high performance, automation will be primary focus.
• Strong communication and analytical skills.
• Hands-on with frameworks/tools such as: UI: Playwright / Cypress / Selenium and API: pytest + requests, Postman/Newman, REST Assured
• CI/CD integration: Git, GitHub Actions/Jenkins/GitLab CI, test reporting, gating.
• Test design: equivalence partitioning, boundary testing, risk-based testing, defect triage.
• LLM/application behavior testing: validating correctness when outputs are probabilistic.
• Evaluation strategies: golden datasets, scoring rubrics, human-in-the-loop reviews.
• Non-determinism handling: statistical assertions, repeated runs, variance thresholds.
• Prompt and regression management: versioning prompts, detecting prompt drift, replay tests.
• RAG testing (if applicable): retrieval quality (recall/precision), grounding checks, citation validation, doc freshness.
• Safety & quality checks: hallucination detection, toxicity/PII leakage checks, policy compliance tests.
• Ability to create and maintain test datasets (structured + unstructured), including edge cases.
• Familiarity with telemetry for AI systems: - logging prompts/outputs safely, traceability, correlation IDs - tools like OpenTelemetry, ELK/Splunk, Datadog/Grafana (any equivalent)
• Understanding of data privacy constraints (masking/redaction) and secure test data practices.
• Comfortable testing distributed systems: microservices, async workflows, queues/events.
• Basic cloud proficiency (AWS/Azure/GCP) and containerization (Docker, optional Kubernetes).
• Load/performance testing for inference endpoints (latency, throughput, concurrency).
• Cost-aware testing (token usage, rate limits, fallbacks).
• Resilience tests: retries, circuit breakers, model timeouts, degraded-mode behavior.
Preferred:
• Familiarity with NLP concepts (embeddings, context windows, temperature/top-p).
• Experience with AI tooling: LangChain/LlamaIndex, evaluation tools, model gateways.
• Knowledge of regulatory/security needs relevant to the telecom domain.
• Strong communication —able to explain AI quality issues clearly to product and engineering.
• Comfortable partnering with data science/ML engineers and backend teams.
• Ownership mindset: building reusable test harnesses, improving quality metrics, preventing regressions.
Company:
InterSources Inc. solves operational problems where protection, performance, compliance, AI, and workforce capability must work together. Founded in 2007, the company is headquartered in Fremont, USA, with a team of 501-1000 employees. The company is currently Late Stage.
