Back to feed
arXiv cs.LG
arXiv cs.LG
7/21/2026
Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels

Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels

Short summary

This paper proposes operator-aware tolerance calibration for tensor kernel correctness tests, replacing hand-picked tolerances with data-driven thresholds mined from 8,076 GPU run results. Calibrated per-(op, dtype) tolerances raise bug-detection recall from 73.2% to 82.4% across LLM-style buggy variants, with only a 1.1 percentage point increase in false positives. The largest tightening was 2,184x for attention_triton fp16, showing current tolerances are far looser than necessary.

  • •Mines error distributions from 8,076 GPU runs to calibrate per-operator tolerances
  • •Raises bug-detection recall from 73.2% to 82.4% with minimal false-positive increase
  • •Largest tightening: 2,184x for attention_triton fp16 kernels

Generated with AI, which can make mistakes.

#research-breakthrough#ai-tools
Read full article at arXiv cs.LG

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs
arXiv cs.CL
LiST: Lipschitz Scaling Training for Robust and Calibrated Neural Networks
arXiv cs.LG
When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning
arXiv cs.CL
Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs
arXiv cs.CL