I Tested My Static Analysis Tool Against 4 Real Open-Source Repos. Here's What It Got Wrong.

Short summary

The author tested ArchSetu, a deterministic static analysis tool with no AI, against four real open-source repositories (~5k stars each) to validate findings before launch. Initial runs produced false positives due to gaps in handling monorepo structures, framework conventions like Next.js implicit calls, TypeScript path aliases, and test-runner invocation patterns. Fixing these real-world edge cases proved more valuable than synthetic tests, yielding 76 passing unit tests and a more robust tool.

• •Tested static analysis tool ArchSetu against 4 real open-source repos including Express.js and Next.js
• •False positives stemmed from monorepo handling, framework conventions, TS path aliases, and test-runner patterns
• •Real-world testing revealed more tool gaps than synthetic tests; 18 new regression tests added

Generated with AI, which can make mistakes.