Benchmarking GPT-4o, Claude 3.5 Sonnet, and Llama 3 for Automated Code Auditing & Vulnerability Detection

Short summary

A benchmark comparing GPT-4o, Claude 3.5 Sonnet, and Llama 3 70B on detecting reentrancy and integer overflow vulnerabilities in a Solidity smart contract. Claude 3.5 Sonnet found both vulnerabilities with zero hallucinations but was slightly slower; GPT-4o matched accuracy and excelled at JSON schema adherence but flagged a false critical; Llama 3 ran fastest locally but failed JSON formatting and missed one bug. Overall winner for domain accuracy: Claude 3.5 Sonnet.

• •Claude 3.5 Sonnet detected both vulnerabilities with zero hallucinations—best for high-stakes auditing
• •GPT-4o excelled at structured JSON output and speed but produced a false positive critical warning
• •Llama 3 70B offered lowest latency locally but missed one bug and failed JSON schema adherence

Generated with AI, which can make mistakes.