Nothing throws when redaction fails

Short summary

Benchmarking six LLM-based PII redaction approaches reveals that failures are silent: models return plausible-looking strings instead of throwing, garbled HTML still parses, and fail-open gateways leak raw data with HTTP 200. The article prescribes running deterministic and model passes independently over original text, extracting text nodes before redaction, detecting refusal patterns, failing closed, and versioning prompts as deployed compliance artifacts.

• •Silent failures are the core risk: LLM redaction returns well-formed but unredacted text instead of errors
• •Run structural and model passes independently over original text, then reconcile; avoid chaining which corrupts offsets
• •Fail closed on timeouts, log model ID and prompt hash, and never log redacted values — the log becomes a second PII store

Generated with AI, which can make mistakes.