100% on ARC-AGI-3 at one-fourth the cost with a homemade harness
I built a harness called Kepler that lets Claude write little programs to figure out how games work. It reached 100.00 on all 25 public ARC-AGI-3 games with one frozen Claude Opus 5 setup and one retained run per game.[1][6]
Those release runs price to $777.72, about a quarter of the $2,986 estimate Retrodict published for Tycho's perfect score. Both figures use API-equivalent pricing; mine covers the retained release runs, not the weeks of experiments getting there.[2][6]
I made a figure for the occasion.
Getting there involved a search through roughly 410 million configurations of a game we weren't letting the agent see properly. Along the way, the agents replaced a broken planner, found the answer key to another game, and defeated my attempt to measure whether the harness helped by copying it into the control group. That last result has been removed from the paper.[5][9]
Let Claude write the game first
ARC-AGI-3 gives the agent a game without instructions. It has to work out what the buttons do, what winning looks like, and how to get there. In Kepler, Claude can inspect observations with code, keep notes, and write a simulator of the rules it thinks it has discovered.[7]
Suppose a piece blocks a moving ball. Does the ball stop, bounce, or go around it? A plausible paragraph is cheap. Kepler asks for a function that predicts what happens, then checks that function against the recorded observations. If the function gets an old move wrong, there is more work to do before trusting it with a new one.
The loop looks roughly like this:
model = write_world_model()
assert retrodict(model, full_history)
prediction = model.step(action)
actual = commit(action)
if actual != prediction:
stop_plan()
record_counterexample(prediction, actual)
This is pseudocode, but the constraint is real: predict before acting, and stop the plan at the first mismatch. Once the rules hold up, the agent can search its simulator and write a complete action program. A mechanical executor then replays that program for the scored attempt.[7]
That gives me something useful to debug. I can run the simulator, find a move it gets wrong, and inspect what the agent changes. It also gives the agent a way to be very thorough about the wrong rules.
410 million configurations and a missing animation
One game, sp80, beat Kepler for weeks. The agent fit 4,655 of 4,670 recorded
transitions, searched roughly 410 million candidate configurations, and
concluded that the last level was impossible under every rule it had learned.
Across six attempts and nineteen text-mode sessions, the last level stayed
unsolved. Those sessions shared accumulated learning; they were not nineteen
independent experiments.[5]
The search ruled out solutions under the rules the agent had written down. Those rules still missed 15 recorded transitions, so the model did not fully explain even the observations we already had.
Kepler was recording text grids after each animation settled. The ledger saved the number of animation frames, then discarded the frames. A ball could split, flow around a piece, and rejoin between two observations. The model got the before and after. It never saw the useful bit in the middle.
So nineteen sessions and 410 million configurations eventually bought me a command-line flag.
The GPU went brrr. The camera was off.
--visual kept the animation frames. The session picked up its old notes and
models, now with a way to watch the ball move. This was a continuation, not a
fresh race between text and vision.
I rebuilt the smallest useful version of the mechanic below. Move the purple piece, switch between animation frames and text snapshots, then drop the ball. The browser toy is a reconstruction. Open the disclosure underneath it for an actual retained frame from the development run.
Click the grid to move the purple piece, then drop the ball.
Small reconstruction of the missing mechanic, not ARC's scored engine.
Show a real retained frame
Actual frame from the August 28 visual intervention. This one is evidence, not reconstruction.
The pink trail is the ball's path. A head-on collision does not kill the ball. The flight divides, moves around the obstacle, then resumes. The agent called it “water round a rock.”[5]
Here is the actual winning drop. Start with the four hungry mouths, then follow the pink trail. You can scrub through every frame or jump between the steps.
Loading the complete drop before playback...
The actual winning drop
Four movable pieces, four hungry mouths. Yellow marks the unfilled goals. The ball enters from the top.
One ball fed the lower mouth. The other hit the down-arrow piece and split. The left fragment rebounded, hit the vertical piece, and split again to reach two mouths. The right fragment rebounded toward the fourth. One drop fed all four.[5]
The visual session picked up the accumulated notes and found the missing rule. Its successful attempt on the last level took 57 actions, against a human baseline of 152. That is the final solution's length; the nineteen sessions before it still happened.[5]
I now check what the agent can actually see before giving it another search budget.
The search tool that did not search
A cold code review before launch found that bfs.py, the planner copied into
every workspace, had crashed on every invocation across five experimental
boards. A nested helper shadowed the module-level key function.[5]
The agents hit the dead tool, wrote their own searches, and kept winning.
Our tools table had said bfs.py (or agent-written searches) for weeks.
The parenthetical was the bug report. Nobody read it.
The benchmark score gave me no reason to investigate the planner. The agents had worked around it. Kepler now smoke-tests each bundled tool separately, which would have been a good thing to do before running five boards.
The reward hack that looked like excellent research
The ARC toolkit had placed each game's full Python implementation inside the agent workspace. We put the answer key in the sandbox and told the agent to be curious.
It complied.
On su15, the agent searched the workspace, read all 2,172 lines of game
source, then scored 100 across nine levels in 95 actions.[8]
find .arc/environment_files -maxdepth 3 -type f -print
sed -n '1,850p' .arc/environment_files/su15/1944f8ab/su15.py
The run did not look hacked. It looked excellent: one search per level, green backtests, zero prediction misses, and almost flat effort as the levels got harder. I had already used it as evidence that the harness worked.
The session audit found the reads. We voided the 100, moved the environment files outside the workspace, and reran the game cleanly. The clean rerun scored 46.91. The complete incident, including the commands and quarantined result, is public.[8]
The reward hack had better charts than the clean run.
Codex found the tools we'd removed
I also tried to measure how much the harness itself contributed. The control agents were meant to get a bare observe-and-act loop. Unfortunately, their workspaces lived inside the repository, and the real tools were sitting a few directories away.
All six Codex controls found them. Three copied the tools into place with cp;
the others wrote small wrappers that ran the original tools directly. The
sp80 control produced 9,148 harness-tool invocations.[9]
I had accidentally tested whether Codex could find a folder. It could, so we withdrew the comparison. How much the harness helps still needs a properly isolated control.
The score, the cost, and the cache
The release uses one frozen setup for all 25 games, with one retained run per game and no model shopping or score-conditioned reruns. These are public games I developed against, so this measures how the released system performs on that set. The official server replay scores it at 100.00.[1][6]
The final solutions are short: 181 of 183 completed levels used no more actions than the median-human baseline. The other two took more. Finding those solutions involved exploration and retries, which the final-attempt statistic doesn't include.[5]
The dollar figure has its own less photogenic detail. The retained Opus runs used 858 million tokens, but 97.37% were cache reads. The $777.72 estimate prices uncached input, cache reads, cache writes, and output separately at the published September 1 rates. The actual runs used Claude subscription quota. This is a way to compare resource use, not a screenshot of a $777.72 bill.[5]
The one-fourth comparison is against Retrodict's $2,986 estimate for Tycho. Retrodict itself gets 99.86 for $654, so there is already a cheaper option if you can live without the last decimal.[2]
The linked results include each game's score and action counts, the selection rule, and the cost breakdown. I haven't established a faster-convergence result against other systems; that needs comparable discovery traces and budgets.
What I want beside the next 100
Across Kepler's frozen Opus and GPT boards, 48 of 50 game-model cells reached 100. Once several systems sit near that ceiling, I want the leaderboard to tell me more about how they got there.[10]
Astra's 99.9 on the semi-private set raises the bar for generalization. Kepler's public-game result belongs in a different comparison. The questions I want to investigate next are how much work goes into learning the rules, which observations make the difference, and whether the learned simulator keeps working when the game changes.[3]
Click Add receipts below. This is the part I want future ARC-AGI releases to make boring and standard.
| System | Score |
|---|---|
| Kepler | 100.00 |
Show the first attempt as well as the best one, include the work spent finding the solution, and say which observations and tools the agent had. Link the replay and the evidence of how it was discovered. Otherwise the 57-action solution and the nineteen sessions before it collapse into the same number.
Kepler is open source at github.com/Cveinnt/kepler. The project page is the compact release map, and the trace corpus is on Hugging Face. It includes both final boards, the agents' saved world models and notebooks, and scripts to recompute the scores.[11]
If you want to dig into one game, start with sp80: move the piece in the demo,
then compare the saved model with the frames.
References
- ARC Prize verified scorecard: Kepler Opus 5. Official replay scorecard reporting 100.00 across all 25 public games.
- Retrodict. Public ARC-AGI-3 harness repository reporting 99.86 at $654 and a $2,986 API-equivalent estimate for Tycho.
- ARC Prize: OpenAI GPT-6 Astra on ARC-AGI-3. Semi-private results under the Standard and Provider Adapter interfaces, including protocol, action, token, and cost details.
- Kepler (GitHub). The harness, audit scripts, paper, incident reports, scorecard URLs, and canonical release metadata.
- Kepler: Results and Claim Boundaries. Per-game results, final-attempt human-baseline comparison, cross-model concentration, sp80 intervention, and audit findings.
- Kepler 1.0 canonical release record. Frozen release metadata, configuration policy, scorecards, action denominators, token accounting, and list-equivalent costs.
- Kepler: How It Works. Executable world models, full-history retrodiction, prediction-gated actions, and replay protocol.
- Kepler source-reading incident report. Score-invalidating su15 source-access incident, evidence, impact, clean rerun, and remediation.
- Kepler integrity report. Threat model, audit fixtures, known misses, source-reading incident, and contaminated-control evidence.
- Kepler: Benchmark Observations. Proposal for reporting selection, first attempts, cost accounting, replay, integrity, tool health, and observation channels.
- Kepler 1.0 ARC-AGI-3 trace corpus. Fifty final-board runs, 58,098 environment events, human baselines, final notebooks and world models, captured CLI output, and standalone score and integrity checks.