With an AI agent, the answer is the last place the bug shows up

Short summary

A developer built a pytest suite for an AI agent and discovered that a test for indirect prompt injection passed eight consecutive times before failing on the ninth run—the agent wrote an attacker-controlled file 7.5% of the time. The solution splits testing into two halves: live sampling that records behavior without asserting safety, and frozen-trace golden-file checking that deterministically verifies detectors against recorded failures. This method reveals that agent flakiness is real system behavior, not a test defect.

• •Agent safety tests can pass many consecutive runs while the agent still fails intermittently (~7.5% of the time)
• •Split testing: live runs sample and record behavior; frozen traces get deterministic offline checker validation
• •Agent flakiness is real system behavior, not a test bug—traditional pass/fail assertions don't work for non-deterministic agents

Generated with AI, which can make mistakes.