Stratagems #1: Mark Johnson Walked Into an AI Audit. The Benchmark Had Everything Figured Out — Except the Truth.

Short summary

An AI consultant hired to validate a Series B AI testing platform's benchmarks discovers the evaluation dataset is systematically fabricated: 7.9% of 1,247 samples are either exact copies from public defect databases or suspiciously hand-crafted. The narrative teaches a critical audit technique applicable to anyone evaluating AI vendors: production data is inherently messy and noisy; suspiciously clean datasets signal fraud and demand deeper investigation.

• •An AI auditor discovers a Series B company's benchmark dataset contains 7.9% fabricated samples—exact copies or hand-crafted code
• •Key audit heuristic: production data is inherently messy; suspiciously clean data signals fraud
• •Applicable to anyone doing technical due diligence on AI platforms or vendors

Generated with AI, which can make mistakes.