How large is the gap between AI benchmark claims and independent reruns?

Eligible pairs: 0. Excluded candidates: 1.

How is the Hype Gap Index calculated?

For each eligible model–benchmark pair: vendor score minus independent score, in percentage points. The index is the median of those differences. Positive values mean a lower independent score; negative values mean a higher independent score. A zero value requires actual matched evidence.

Require the same model version, benchmark version, task split, harness, tool access, compute budget and reasoning effort. Both scores must be on a 0–100 higher-is-better scale, source-linked and independently reviewed. One observation per model–benchmark pair avoids duplicate weighting. Corrections create a new edition; method changes receive a new method version.

Why is Astra’s 99.9% versus 62.7% excluded?

Correction, September 26: the source is ARC-AGI-3; our previous page incorrectly labeled it ARC-AGI-2. ARC Prize reported different harness/effort configurations, rather than a failed like-for-like replication. Subtracting them would confound configuration with claim accuracy.

Inspect ARC Prize’s report ↗

What does this index not establish?

It does not measure dishonesty, overall intelligence, safety, investment returns or all AI claims. Coverage is selected, scores have uncertainty, and percentage-point gaps across benchmarks are not equal units of difficulty. Small samples and changes in sample composition can move the median.

Which candidates were excluded?

• Astra · ARC-AGI-3: Both scores are reported by ARC Prize under different harness and reasoning-effort configurations. This is not a matched vendor claim versus independent rerun.

What is the running record?

Download the series and exclusions