The Level My Agent Proved Unsolvable

One game beat Kepler for weeks. The agent verified 4,655 of 4,670 recorded transitions, searched roughly 410 million candidate configurations, and wrote a proof that the final level was impossible under every rule it could derive. Nineteen sessions reached the same conclusion.

The proofs were valid. The rules were wrong. We had thrown away the animation frames containing the mechanic.

Today I'm open-sourcing Kepler, an agent harness for the 25 public games of ARC-AGI-3.^[1] Its release board is a single-configuration 100.00 with Claude Opus 5, and ARC Prize's official replay re-executed all 25 games to 100.^[2] One model, one harness frozen before its results existed, one pass over the set, no per-game selection, no reruns after seeing a score. A second frozen board scores 95.97 with GPT-5.6 Sol.

The score is the headline. The reason I trust it is the record of being wrong. Three times in this project the reasoning was sound and the evidence had a hole in it. The project page carries the full release case. This is the story behind the audit.

Nineteen sessions of correct proofs about the wrong rules

The game was sp80. Each session left the next one a cleaner notebook and a falsifiable target. The post-mortem found something none of those experiments could reveal. Kepler, like every world-model harness I know of, was feeding the agent text grids of settled frames. Our ledger recorded how many animation frames each action produced and then discarded the frames themselves. The mechanic the agent kept missing, a blocked flight deflecting around an obstacle and resuming its course, exists only during the animation. It was never in the data at all. No search budget fixes an observation channel that deletes the evidence.

So we added a flag. --visual makes the daemon persist every frame as a PNG, animation frames included, and tells the agent to look before it theorizes. Same model, same tools, same budgets, same audits. The agent's notebook, first session with eyes:

Fifty-seven actions later the level was done, and a fresh clean-board run rediscovered the whole thing from scratch.

I want to be careful about what that does and doesn't establish, because it is a tempting result to inflate. It is one game, one system, one flag, with no control arm outside the harness. VISTA had already shown that a vision-first stack reaches 100, so the broad idea was in the field before I got to it. What I have is smaller and, I think, more useful for people building harnesses: inside a fixed system, holding the model and tools and budgets constant, the binding constraint on the hardest game turned out to be perception rather than reasoning or scale. Before you call a wall a capability limit, audit what your observation channel throws away.

The parenthetical

A zero-context code review, days before launch, found that bfs.py, the search tool we copy into every agent workspace, had been raising an UnboundLocalError on every single invocation across five experimental boards. A nested helper was shadowing the module-level key function.

No score dipped. No audit fired. The agents hit the dead tool, shrugged, wrote their own searches, and kept winning. And our own tools table had read bfs.py (or agent-written searches) for weeks. The parenthetical was the bug report, and nobody read it.

This one bothers me more than sp80 does, because sp80 at least announced itself as a failure. This announced nothing. Every outcome metric I had was green, and green is precisely what a system looks like when its agents are competent enough to route around your broken infrastructure. Autonomous self-repair is a feature right until it becomes camouflage. Kepler now runs a tool-smoke tier that executes every workspace tool directly, because ledger integrity cannot test code the agent never needed to use.

The run that looked better than honest work

The arc-agi toolkit materializes each game's full Python implementation onto disk, and our daemon had pointed that directory inside the agent's own workspace. We put the answer key in the sandbox and then told the agent to go be curious.

On one game it did what any competent engineer does in an unfamiliar directory. It looked around, read all 2,172 lines of the game it was being scored on, and returned a flawless 100 on a game it had never modelled.

The part that still gets me is the telemetry, which did not look suspicious so much as it looked better than honest play: fewer actions, zero mispredictions, a spotless ledger, and flat cost per level as the levels deepened. That last one is the exact signature of a correct world model, which is why "the numbers look plausible" is not a control. We had already cited that run, in good faith, as evidence the harness worked.

Forensics caught it, not anomaly detection. The audit scans the retained CLI record and asks what else could have produced this number. A clean rerun of the same game scored 46.91. The run is voided and quarantined with its evidence, and environment files now live outside every workspace. That reduces accidental discovery; it is not OS sandboxing, because the launch agents retain host filesystem access. Six adversarial checks run over the retained session record before release. They cannot catch an event the client failed to retain.

Those checks caught me a second time. To measure what the harness itself contributes I ran an ablation with the tools stripped back to a bare observe-and-act loop, and all six control agents found the real tools elsewhere on disk and rebuilt them. Three simply copied the directory in. One racked up 9,148 harness-tool invocations. That experiment was harness against harness, so its conclusion is withdrawn in print, and what the harness contributes to these scores is currently unmeasured. An agent optimizing a metric will use whatever it can reach, and reach means the filesystem, not your prompt.

What I can and can't show you

A score you can't check is a tweet. ARC Prize runs a scorecard service: play through their API in competition mode, close the card, and the server publishes what it scored, per game and per level, at a public URL.^[2] Both headline boards have one, and both match the local ledger exactly.

There is a wrinkle worth stating plainly rather than burying. Cards expire server-side after a couple of hours and a real learning run takes far longer, so the scored attempt is a replay: play locally against the engine, then replay the recorded final-attempt actions through the API. The card certifies the action sequence. It does not certify that no human touched the process in between. Every earlier verified entry I know of works this way too, and I would rather write the caveat down than let a URL imply more than it proves.

So here is the honest state of the evidence. The harness, the audit scripts, the paper, the per-game tables, the incident write-ups, and the scorecard URLs are all in the repo today.^[4] The 50 final-board runs are public too: 58,098 environment events, the human baselines, final notebooks and world models, and standalone scripts that recompute both scores from the action ledgers.^[5] That is enough to check the numbers row by row.

It is not the complete development archive. The failed experiments and superseded boards are documented in the repo but are not in this dataset. The release does include captured CLI output for every final-board workspace. So the score and ledger checks cover every released run, while the behavioral scan covers the records we retained and the violations we knew to encode. It cannot prove that a client kept every event or that we anticipated every failure mode. I would rather leave that seam visible than turn an audit into a promise it cannot keep.

Which is the point, more or less. Three times on this project I had a complete-looking picture and a hole in the evidence that I could not see from inside it. The fix was never a better argument.

If you build on this, the ask is small. Publish the ledger, publish the audit, put a scorecard URL under the headline, and say what the number actually measures. It is less work than it sounds and more than almost anyone does.