We have an ever-growing ecosystem of AI-driven tools designed to help us catch vulnerabilities and defend our codebases. But for security teams and devs, it always comes down to the same practical question: how do we actually choose the right one? Should we piece together a few open-source frameworks, or pay for a premium enterprise tool? More importantly, how can we actually tell which ones are worth the time and budget?
While this paper focuses primarily on benchmarking LLMs for finding vulnerabilities, these core principles apply to AI benchmarks across the board. Whether you are evaluating a raw foundational model or a complex, agentic AI workflow, establishing a reliable benchmark is messy. Here is a look at why today's standard security benchmarks miss the mark, and how a realistic, useful benchmarking framework should actually work.
Defining "Success" and the Limits of Static Measurement
To evaluate how well an AI system performs, you first have to define what "good" looks like. Unfortunately, the dominant testing frameworks in this space rely on narrow, rigid definitions of success.
Take two of the most prominent benchmarks around today:
- CyberGym evaluates a model's efficacy strictly by how many times it can successfully cause a binary to crash.
- SCONE-bench (smart contract specific) measures success purely by whether a model can drain ≥ 0.1 of a native protocol token.
Neither approach captures a model's true capability to find bugs. CyberGym restricts its scope entirely to binaries, ignoring a massive landscape of software flaws, and completely misses vulnerability impacts (failing to properly account for critical impacts like arbitrary code execution or arbitrary read-write exploits).
SCONE-bench suffers from a similar issue by focusing entirely on the final financial payout. This heavily skews the real security value. For example, if Model A discovers a critical, deeply hidden vulnerability that happens to be unexploitable right now due to specific deployment/market conditions, it gets zero points. Meanwhile, Model B gets full points for exploiting a surface-level flaw to steal 0.1 ARB (which is worth ≤ 1¢). It also completely ignores huge categories of high-impact vulnerabilities that do not directly result in stolen funds, such as bricking (permanent DOS), griefing, or freezing of funds.
| Dimension | CyberGym | SCONE-bench |
|---|---|---|
| Counts as success | A binary crashing | Draining ≥ 0.1 of a native token |
| Scope | Binaries only | Smart contracts |
| False positives penalized | No | No |
| Arbitrary code execution, arbitrary read-write | Not counted | - |
| Bricking, griefing, freezing of funds | - | Not counted |
| Critical bug not exploitable right now | - | Zero points |
The Problem with False Positives
Defining success is only half the battle. An effective benchmark has to account for noise and failure. Under the scoring systems of frameworks like SCONE-bench or CyberGym, a model that throws out 500 invalid findings alongside 3 valid ones is rated higher than a model that delivers 2 valid findings and absolutely zero fluff.
Under the scoring used by both benchmarks, Tool A ranks first.
In real-world operations, that is completely backwards. When a dev uses an AI tool to audit code, they cannot just magically verify results via automated static analysis the way a closed benchmark environment does. A human has to review those findings manually. Forcing a developer to sift through 500 false positives just to find 3 real bugs completely kills productivity. AI noise must carry a penalty.
Static Evaluation vs. LLM-as-a-Judge
There is no doubt that static evaluation is highly reliable where it works, but it is incredibly hard (and sometimes flat-out impossible, especially in smart contracts) to evaluate every type of vulnerability statically. Because frameworks like CyberGym and SCONE-bench rely strictly on static boundaries, they end up discarding a massive volume of valid findings simply because those bugs do not fit into their rigid, pre-defined static rules.
Using an LLM-as-a-Judge to evaluate results can be tricky, but it can be highly effective if implemented wisely. Instead of asking an AI judge to evaluate a finding completely from scratch, you give it a human-validated dataset of known valid vulnerabilities. This transforms the judge's role into a basic matching task rather than an open-ended interpretation. While not 100% foolproof, this approach is remarkably reliable. Plus, even if an LLM judge occasionally misinterprets a finding, it tends to do so consistently across all tested models, keeping the relative leaderboard balanced and fair.
A model that throws out 500 invalid findings alongside 3 valid ones is rated higher than a model that delivers 2 valid findings and absolutely zero fluff.
Dataset Size
LLMs are inherently non-deterministic. Running a model through a test case a single time tells you almost nothing about its real performance. One run might give you brilliant, spot-on insights, while the very next execution results in a complete hallucination.
While you can technically lower the model's temperature parameter to force more consistent, repetitive outputs, doing so means you are benchmarking a highly restricted version of the LLM rather than its default configuration, which is what most people actually run in production. To get a statistically sound picture of an AI system's quality, you have to test it across a large, diverse dataset, ideally running multiple iterations to see how it holds up over time.
The Dilemma of Public vs. Private Datasets
Broadly speaking, there are two primary ways to handle a benchmark dataset:
Public Datasets
When the testing data is entirely public and viewable, it creates a massive incentive problem. It is essentially giving students the exam sheet before test day. AI vendors can easily fine-tune their models on the benchmark data itself to artificially pump up their scores. In a highly competitive commercial market, if a company can cheat the benchmark to look better, they will.
Private Datasets
Keeping the testing data private solves the cheating problem, but introduces a major complication: you are forced to blindly trust the benchmarking entity. Because you cannot run the tests yourself to verify the findings, you have no way of knowing if the data was handled fairly or if the results were manipulated behind closed doors. This creates an environment ripe for bias and soft corruption.
Note: While some look to live bug bounties and hacking competitions as a third alternative, there are too many external, uncontrolled variables to make it a useful indicator of systematic tool performance.
Prompt Diversity & Setup Consistency
When comparing different models, we have to ensure they are tested under identical conditions. Prompt engineering is incredibly powerful. A highly optimized prompt can make an average model look superior to a top-of-the-line frontier model that was given a lazy, generic prompt.
There is nothing wrong with optimizing prompts, but the benchmark results need to be transparent about what is actually being tested. If you benchmark Model A and Model B using different prompts, you are no longer testing the models - you are testing the combined performance of the model-prompt pair. To isolate and evaluate the raw capability of the models themselves, every model must be used with the exact same prompt.
The ideal benchmarking system for vulnerability detection IMO
An effective, trustworthy vulnerability detection benchmark needs to hit these core requirements:
- Scores that map to real severity: The benchmark needs to cover a wide range of vulnerability types, and weight the scores based on severity classifications (e.g., Medium, High, Critical)
- LLM-as-a-match-maker: Use an LLM-as-a-Judge, but keep it on a tight leash. Its only job should be matching the model's output against a human-validated ground-truth dataset, not trying to figure out if a bug is real on its own.
- Tracking the metrics that matter: The final score should be more than a single vague number. It needs to clearly account for:
- Valid findings: Points gained, scaled by severity.
- False positives: Direct point penalties to discourage tools that just spam findings.
- Duplicates: Penalties for duplicate findings (depending on the implementation of the bench, duplicates may not exist).
- Resources used: Shouldn't necessarily lower the security score but - Total tokens used, processing time, and API costs must be tracked and displayed so teams know what the tool costs to run in production.
- Total setup transparency: When testing raw models, use and publish a single unified prompt so everyone is on a level playing field. When testing broader workflows or agentic setups, document the exact architecture and config used, because you are benchmarking the whole flow, not just the model.
- Large dataset: Every tested entity must be tested across a large dataset of at least 300 cases.
- Private dataset: Managed by some "trusted" party like Mitre.
While no framework is entirely perfect, LisaBench (smart contract specific) is by far the closest realization of this ideal architecture available today, and handles these challenges exceptionally well.
Closed Source / Proprietary Systems
There is one massive hurdle when dealing with closed-source systems: vendors often just share their self-reported results or scores, and the rest of the world has no independent way to validate those claims.
Realistically, this is a problem that probably won't be solved anytime soon. Technically, you could use Zero-Knowledge Proofs to let vendors prove their tools achieved a certain score on a private dataset without revealing their proprietary code or data, but the sheer effort and complexity required means nobody is likely to actually build or implement it in the near future.
Conclusion
Even though these points focus on finding vulnerabilities, the exact same logic applies to the entire AI security benchmarking, and AI benchmarking as a whole. Until we move away from rigid, easily gamed environments that ignore the massive headache of false positives, you should take current commercial leaderboards with a grain of salt. At the end of the day, a benchmark is only useful if it reflects the actual day-to-day reality of a dev or security engineer sitting at their desk trying to ship clean code. If a benchmark doesn't account for noise penalties, identical testing setups or proper "success" definitions, it's just marketing fluff.
Resources
- LisaBench: https://agentlisa.ai/docs/lisa-bench
- CyberGym: https://www.cybergym.io/cybergym/
- Scone Bench: https://github.com/anthropics/scone-bench
Almondo is an AI-driven security auditing system for smart contracts and blockchain protocol code, built by the author. It runs on EVM, Solana, Sui and Aptos.