Frontier Models Below 1%, Humans Solve Everything

On March 25, 2026, the ARC Prize Foundation launched ARC-AGI-3 with a $2 million grand prize and a blunt finding: every frontier AI model scored below 1% on the new benchmark, while all 135 environments were solved by humans with no prior knowledge and no instructions. GPT-5.4 reached 0.26%, Claude Opus 4.6 managed 0.25%, Gemini 3.1 Pro Preview hit 0.37%, and Grok-4.20 scored exactly 0.00%.

That gap — between near-zero AI performance and perfect human performance — is the point. ARC-AGI-3 is not another coding benchmark where months of RLHF can push a model from 72% to 76%. It is testing something different, and the Milestone 1 prize results, announced in early July, offer the clearest picture yet of what that something is.

What Makes ARC-AGI-3 Different

Previous ARC-AGI benchmarks presented static visual puzzles: transform this grid by identifying the underlying rule. ARC-AGI-3 is fully interactive. Each environment is a novel, video-game-like world with its own hidden mechanics. The agent must explore, figure out how the world works, discover what winning looks like, and carry that understanding forward across increasingly difficult levels.

The benchmark scores using RHAE (Relative Human Action Efficiency, pronounced “ray”). It measures how many actions the model needs compared to a human baseline, normalized per game. A score of 1.0 means the agent is as efficient as a human; a score of 0.01 means it takes 100x as many actions. Scores below 0.1 — where every frontier model sits — indicate the model is largely flailing.

Crucially, the environments cannot be memorized. A hidden hold-out set ensures no amount of training data leakage can substitute for on-the-fly learning. This is why frontier models, which excel at pattern-matching against their training distribution, struggle so dramatically here. The task requires building a mental model of a system you have never seen, and current deep learning architectures are genuinely bad at that.

The Three Winners: What Actually Worked

ARC Prize introduced milestone prizes this year to incentivize open-source submissions mid-competition. The first milestone (June 30 deadline) distributed $37,500 across three teams. Their approaches are worth studying closely because they represent the current ceiling of what works — and two of the three are surprisingly simple.

1st Place: Tufa Labs “The Duck” — Agent Writes Code in a Live REPL

The winning submission, built by Tufa Labs and running Qwen 3.6 27B FP8 locally, scored 1.21% RHAE. It works by encoding the game state as Python variables and giving the model access to a live REPL. The model reasons, calls helper functions, runs code, takes an action, sees the result, and repeats. It perceives the board through multiple representations simultaneously: a rendered image, the raw ASCII grid, and a segmentation tool for zooming into regions.

Two design decisions stand out. First, the team uses “infinite play via eviction” — the oldest messages are continuously dropped to keep the context window manageable without ever stopping. Second, and counterintuitively, hand-crafted tools hurt the model. Tufa Labs tried building specialized helpers and found that letting the model improvise its own tools through the REPL worked better. The harness is intentionally minimal; the model does the reasoning.

This is the only winner using an agent-writes-code approach. The other two took a simpler route.

2nd Place: Reki — Vision-LLM Picks a JSON Action

Reki runs Gemma-4-31B locally and operates at a much lower level of abstraction: each turn it renders recent game frames as labeled images, feeds them to the model, and asks for a single JSON object describing what changed, a short plan, and the next one to four actions. It also maintains a running reflection memory refreshed every ten steps.

A small numpy heuristic — no GPU required — makes exploratory clicks prefer small, rare-colored, button-like shapes rather than random pixels. A “dead-signature” mechanism detects when clicking a type of object never changes anything and stops wasting clicks on it for the rest of the level. These rules are togglable, which made ablation easy: the team could directly measure the impact of each change.

3rd Place: “forge” — Same Approach, More Framework

The third-place submission by Md Boktiar Mahbub Murad uses the same vision-LLM-picks-JSON-action architecture as Reki, also on Gemma-4-31B, but wraps it in a configurable framework called “forge.” It adds a generator that proposes multiple candidate actions, an arbiter that picks between them, and a confidence prompt that forces safe or reversible moves when the model is uncertain. The interesting detail: the top-scoring run turned all of that extra machinery off. The baseline worked better.

The Schema Harness and the 99% Asterisk

On the ARC-AGI-3 public set, a research harness called Schema — developed by a team at Carnegie Mellon — reports 98.98% RHAE using Claude Opus 4.8 and Fable 5. The approach has frontier models act like physicists: they write each game’s mechanics as an executable program, verify it against recorded history, and plan inside it using search.

That number deserves scrutiny. The 99% result is self-reported, not yet verified by ARC Prize. More importantly, it applies only to the 25 public games — not the semi-private evaluation set used for official scoring, where frontier models currently score far lower. ARC Prize explicitly uses a hidden hold-out set to prevent overfitting, so public-set performance is not a reliable proxy for the real benchmark. Schema is an interesting research direction — jointly solving state grounding and mechanism discovery in a single editable program — but for now, the Milestone 1 results remain the only verified data on what actually works.

What This Tells Us About the AI Capability Gap

The ARC-AGI-3 results point to a specific limitation in current models: they are excellent at retrieving and recombining patterns from training data, but weak at building new causal models of unfamiliar systems through interaction. This is not a benchmark designed to be solved by scale alone — Grok-4.20 at 0.00% and the $2 million grand prize still unclaimed make that clear.

Equally interesting is what the winning solutions have in common. All three use small, openly available models running locally: Qwen 3.6 27B and Gemma-4-31B. None of them use a frontier API. The 1st-place team actively found that adding more scaffolding hurt. The benchmark is not being won by throwing more compute at it — it is being won by building the right feedback loop between a model and the environment.

That is a meaningful signal for anyone building AI agents in production. Raw model intelligence matters less than the design of the observation-action loop. The agents that perform best observe the consequences of their actions, update their working model of the situation, and choose next steps accordingly. That is how humans approach novel problems — and it is exactly what ARC-AGI-3 is measuring.

The second milestone closes September 30. Watch what Milestone 2 winners do differently. If a new approach breaks past the 5% barrier on the verified hold-out set, it will be worth paying close attention to. If scores stay clustered around 1-2%, that is its own finding: interactive reasoning at human efficiency remains genuinely unsolved, and the most capable AI systems on the planet cannot come close.

Further Reading

• ARC Prize 2026: ARC-AGI-3 Milestone Prize #1 — Full write-ups from each winning team, including Kaggle notebooks and video walkthroughs from the Tufa Labs team.
• ARC-AGI-3: Frontier Models Below 1% — Good technical overview of the benchmark design and why frontier model performance collapsed at launch.
• LLM Leaderboards Are Breaking. Here’s What’s Next. — Our earlier analysis of why standard benchmarks are losing their predictive value — ARC-AGI-3 is the clearest counter-example yet.