AI Testing Interview Preparation for QA Engineers: Non-Deterministic Evaluation at Scale

Short summary

A guide for QA engineers and SDETs transitioning into AI testing roles, teaching why traditional assertions fail for non-deterministic LLM outputs. The post covers interview Q&A patterns, real-world failure scenarios, and how to design evaluation pipelines that catch silent regressions. Practical tools include DeepEval, Promptfoo, and LangSmith; correctness is decomposed into measurable criteria (faithfulness, relevance, safety, format).

• •AI testing requires thinking in distributions, not booleans—same input yields different valid outputs across runs and model versions
• •Silent regressions from model updates are the top production risk; mitigate with version pinning, CI evaluation suites, and metric alerting
• •Decompose correctness into measurable criteria rather than asserting one expected string; use golden datasets and behavioral scoring

Generated with AI, which can make mistakes.