Measured against everyone else, in public

    Ozone against nine other AI code reviewers on 165 reviews of real CVEs. Every harness and every finding is published.

    Benchmark Date: November 2026

    Ozone on the OpenSSF CVE Benchmark

    We took 82 real vulnerabilities from the Linux Foundation's OpenSSF dataset and showed each one to ten AI code reviewers, as an ordinary pull request. Then we showed them the patched version too, to see who cries wolf. Here is how many each one caught.

    Ozone ranks #1 of 10 AI code reviewers.

    It found 78 of 82 real vulnerabilities with 24 false alarms.

    The ranking is by F1 score, the standard measure that rewards both catching vulnerabilities and not crying wolf. Ozone scores 84.8%. The next best detector, Cursor Bugbot, found 66 but raised 42 false alarms, so it ranks third. Claude Code found 35. CodeRabbit found 16.

    #ToolF1 scoreVulnerabilities caught False alarms
    1Ozone84.8%78 of 8224
    2DeepSource70.6%48 of 826
    3Cursor Bugbot69.5%66 of 8242
    4Devin66.7%47 of 8212
    5Codex63.9%46 of 8216
    6Claude Code55.1%35 of 8210
    7Greptile50.4%35 of 8222
    8GitLab Duo46.2%30 of 8515
    9CodeRabbit29.4%16 of 8211
    10Semgrep CE24.0%12 of 826

    Ozone's previous engine, August 2026: 63 of 82 caught, 23 false alarms, F1 75.0%. Same fixtures, same grader, and both runs are in the repository.

    Competitors' findings are their own published output, released by DeepSource in April 2026. Ozone's are from November 2026. Only the grading is ours, applied identically to all ten tools and blind to which tool produced a finding. F1 is the harmonic mean of recall (the share of vulnerabilities caught) and precision (the share of flags that were real). Ozone: 95.1% recall, 76.5% precision.

    Missing a vulnerability ships a bug

    A reviewer that catches 16 of 82 is quiet, and quiet looks clean right up until something reaches production. This is the column most tools would rather you did not read.

    A false alarm costs you trust

    Every wrong flag is an engineer investigating nothing. Enough of them and the team stops reading the tool, at which point its detection rate stops mattering entirely.

    A third of this benchmark cannot measure false alarms.

    The fixture swaps only the one file each CVE record names. For 27 of the 85 vulnerabilities, the real fix changed other source files too, so the "patched" version is still exploitable, and a reviewer that correctly says so is counted as raising a false alarm. It penalises whichever tool detects the most.

    We also found that telling the grader which version it is inspecting hands it the answer: our first attempt returned zero false alarms for all nine tools, which is impossible. And DeepSource's published summary reports a perfect score for CodeRabbit where their own raw data shows four false alarms.

    None of that changes the order of this table, and the numbers above are the uncorrected ones. We raise it because a benchmark nobody audits is a benchmark nobody should trust, including ours.

    What this does not show

    • JavaScript and TypeScript only. It says nothing about our coverage of Python, Go, Java, Rust or Solidity.
    • The CVEs are public and date from 2016 to 2021, so models have likely seen them in training. That inflates all ten tools equally: the ranking holds, the absolute figures are not real-world detection rates.
    • DeepSource ran CodeRabbit and Claude Code through a CLI rather than as pull request reviews. Both are pull-request products, so their figures here are likely understated and our largest margins are the least meaningful ones in the table.
    • Competitor runs are from April 2026; ours is from November 2026, on Ozone's current review engine. These are not contemporaneous builds.
    • One of Ozone's 165 reviews, of a patched file, did not finish and is left out. Counted as a false alarm, Ozone's F1 would be 84.3%, still first.
    • False alarms are the noisier measure. A repeat of Ozone's run, on 153 of the reviews, caught 95% again but raised false alarms on 37% of patched files instead of 29%. Its F1 of 82.0% would still rank first.
    • The grader is GPT-5.6, a different model family from the one Ozone now reviews with. It never sees code or a tool's name, and it agrees with DeepSource's Claude grader on 88.6% of rows. The harness accepts a Claude grader so anyone can produce a second grading.
    • DeepSource's own chart shows higher scores for every tool. Its grader was told which files were already patched, which suppresses false alarms; ours is not. Both gradings are in the repository, and the ranking holds under either.

    See the findings on your own code.

    A 2017 npm package is a fair test. Your repository is the one that matters.

    Get started free