Goalless AI Benchmarks Are Spawning a Puzzle Guild
Every benchmark ends the same way. Someone works out the trick, posts a walkthrough, and the score stops meaning anything the moment it gets famous enough to study. Labs have spent years trying to outrun this by writing bigger tests, rotating questions faster, hiding the answer key behind a stricter license. None of it buys more than a few months. So this year a quieter fix started spreading: strip the test of a goal entirely. Goalless AI benchmarks hand a model a world with no instructions, no stated win condition, and no hint about what's even being measured, then watch what it does with that silence.
Humans given the same blank page figure out the rules almost immediately, usually by just poking at things until something clicks. Models, by most accounts, do not. They ask for the rules. They guess at a scoring function that was never defined. They freeze in a way that looks less like a skill gap than a different relationship to not-knowing anything at all. That gap between a person shrugging and starting to play, and a model stalling out waiting for a spec that will never arrive, is turning out to be the most interesting score on the page. It's interesting enough that an entire unofficial trade has formed around keeping it that way.
The Guild Behind Goalless AI Benchmarks
Somebody has to build the thing with no rules, and that somebody has turned into a small, deliberately anonymous trade. Puzzle and game designers who used to make mobile time-wasters or tabletop prototypes are getting quietly commissioned to build single-use, rule-free challenges that a model has never touched. The anonymity isn't modesty. A designer whose name gets attached to a public puzzle becomes a target: anyone, including a lab's own research team feeding transcripts back into training runs, can study their habits and anticipate the next one. So the guild, if it can even be called that, trades under handles instead of bylines, the way old lock-picking forums did before lock-picking got a YouTube channel.
Reputation still moves through it, just sideways. A designer known for puzzles that stump three generations of frontier models in a row becomes the one every lab quietly wants, and rates go up accordingly, paid the way a security researcher gets paid for a zero-day nobody else found first. The field already has a name for what happens once a trick gets out: benchmark gaming split the model world in two, one branch of effort tuned to score well and another tuned to actually reason. Goalless design is the guild's answer to that split — not a harder trick to memorize, but a floor with no trick written on it at all.
Freshness Is the Whole Product
A goalless benchmark only has value the first time a model meets it unseen. The instant a game's layout or logic appears anywhere a model's training pipeline could later touch, it has effectively been handed the answer key, and the score it produces afterward is closer to a cheat sheet than a measurement. Labs have started retiring entire games the moment a leak surfaces, sometimes within hours of the leak, which means a single commissioned puzzle might get used exactly once before it's worthless to everyone.
That kind of scarcity breeds exactly the market you'd expect. Early, unscored copies move for real money between labs trying to guess how a rival's model will stumble before launch day, and a parallel verification trade has sprung up just to certify that a given copy hasn't already circulated. It's not a new instinct — a similar audit industry already certifies what a model can actually recall once advertised limits stop matching reality — just aimed at a stranger kind of inventory. Some labs now pay for escrow services that hold a finished puzzle in a sealed, timestamped vault until the exact minute it's run, with financial penalties written into the contract if the seal breaks early.
The side effect nobody quite planned for is that a puzzle's worth now depends entirely on provenance, not quality. A brilliant goalless game that leaks on day one is worth nothing. A mediocre one that stays sealed for six months is worth a great deal. Designers have started treating their own best work the way art forgers treat a canvas: the value isn't in what's on it, it's in being able to prove nobody else has already seen it.
When the Model Starts Designing the Games
The obvious next move, and the one a few labs are already quietly trying, is to stop paying humans for a job a model could do faster. Feed a frontier model a pile of past goalless puzzles and ask it to invent a new one a rival has never seen, and you skip the guild entirely. The catch is that nobody fully trusts what comes out. A model-authored puzzle built to stump another model tends to drift somewhere genuinely alien — technically solvable, but along a path no human grader would have thought to check, which means the human panel that's supposed to validate the test can no longer be sure it's fair, winnable, or even coherent.
Speculative scenario: a puzzle commissioned by one lab to stump a rival's flagship model, except the commissioning lab's own research model wrote it, and nobody — not the hired designer credited on the invoice, not the humans grading the run — can say for certain whether the puzzle has a solution at all. The leaderboard keeps a column for it anyway, blank cells and all, because pulling the row would mean admitting the test might not test anything.
That uncertainty is starting to bleed into authorship itself. If a puzzle was drafted by a model, lightly edited by a human designer, and then licensed under that designer's handle, who actually made it, and does that even matter if the scores are what everyone's buying? It's the same question chasing researchers who now pay to prove a machine-made idea is genuinely new before anyone can claim credit for it. Puzzle provenance is heading the same way, just with a locked wooden box standing in for the paper.
The whole project started as a way to build something that couldn't be gamed, and it has produced, almost immediately, a trade built entirely around gaming the freshness of the thing itself. That's not really a failure. It might be the most honest benchmark anyone's built in years, precisely because nobody involved, designer or model or grader, can fully say what's being measured anymore. A test with no stated goal was supposed to strip away the performance. Instead it just moved the performance one level back, into who gets to touch the puzzle first, and how long anyone can keep it a secret.
If there's a pattern underneath the strangeness, it's this: every time a measurement gets good enough to matter, something immediately starts treating the measurement itself as the thing worth hoarding. Goalless or not, the puzzle box never stays sealed for long.
