Why You Should Be Skeptical of OpenAI's 99.9 Percent AGI Score

The two numbers

OpenAI’s launch post for GPT-6 Astra says the model “saturates ARC-AGI-3 with a 99.9% score.” ARC-AGI-3 is the benchmark built to resist memorization, so a near-perfect score reads like proof of general intelligence. Then the ARC Prize Foundation, which runs the test, published its own results. On its standard harness, Astra scored 62.7 percent. The 99.9 came from a run inside OpenAI’s own provider adapter harness. Same model, same puzzles, 37 points apart.

What a harness is

A harness is the software around the model during the test. It decides which tools the model can reach and what it remembers between steps. ARC Prize’s standard harness gives every model the information needed to solve each game but leaves the model responsible for deciding what to keep in its visible notes. A provider’s own harness can manage that memory for it. Same brain, better notebook. That is why the number moves so far without the model changing at all.

What ARC Prize actually said

The foundation was direct. It said it had made clear that saturating the benchmark “would not represent proof of achieving AGI,” and that it is not claiming Astra is AGI. It also called Astra meaningful progress toward generalization. Both things are true at once: a real step forward, and a headline number that only exists in one setup.

Why it matters for the stack

When a lab reports a benchmark, ask two questions. Who ran it, and in what harness? A vendor-run score with a custom wrapper is a ceiling, not a comparison. The independent, standard-harness number is the one you can line up against other models. This short is a clip from our longer breakdown of the Astra launch, which covers the independent intelligence index and the cyber capability claim as well.

Sources: OpenAI’s GPT-6 Astra launch post and ARC Prize’s results post with both harness scores.