Back to feed
Dev.to
Dev.to
7/18/2026
I Maintain 23 Benchmark Suites Across My Own Packages. Only 1 of the Serverless Ones Has Real Numbers Yet.

I Maintain 23 Benchmark Suites Across My Own Packages. Only 1 of the Serverless Ones Has Real Numbers Yet.

Short summary

An ESLint plugin maintainer discovered 140 files using APIs that ESLint 10 removes outright, caught by a version-compatibility fixture rather than a blended quality score. The article argues no single metric covers both code quality and runtime compatibility, and introduces context-weighted false-positive scoring across five benchmark tiers ranging from adversarial fixtures to real-world repos. Only one of four serverless benchmark suites currently has real numbers behind it, a gap the author discloses mid-article.

  • •Version-compatibility fixtures catch bugs that quality scores and F1 metrics miss entirely
  • •Context-weighted FP scoring: Arena x10, CWE-Corpus x5, Quality x3, Wild x1, Edge x0.1
  • •Only 1 of 4 serverless benchmark suites has real numbers; author discloses this honestly

Generated with AI, which can make mistakes.

#open-source
Read full article at Dev.to

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

The original title is "I Built What I Benchmark. Here's How I Try Not to Cheat."
Dev.to
The original title is "25% of My Benchmark Verdict Is an Opinion. Here's the Anatomy."
Dev.to
I Tested My Static Analysis Tool Against 4 Real Open-Source Repos. Here's What It Got Wrong.
Dev.to
OpenAI audit finds ~30% of SWE-Bench Pro tasks broken; proposes uncertainty budgets for coding benchmarks
Dev.to