Self-challenging language models: endless verifiable problems at the edge of ability

Self-play with executable task verification. Early results show improved task generation; external benchmark gains have not yet been measured.

The problem with datasets

Reinforcement learning on problems with checkable answers works well: code that passes tests, math with a verifiable result, puzzles that a program can grade. Many such setups start from a supplied dataset of problems, reference answers, or a judge model to score the output. Datasets run out, and most of what is in them is either too easy for the model or out of its reach.

The question behind this project: can a model train itself using nothing but an environment it can run code in? No task dataset, no gold answers, no judge model.

The idea

The model plays against itself. Two roles, A and B, are played by the same model, and their interests are opposed. A writes a task. B tries to beat it. Everything runs and is measured in a sandbox, a terminal playground where programs can be executed safely, with a pretrained model, role instructions, and sandbox rules supplied at the start. No task dataset or reference solutions are added during this training loop.

The training loop starts with a pretrained model and an executable sandbox. Rewards are computed by running code, without an LLM judge; this does not guarantee that the tasks teach transferable skills.

Why it can keep going

The two roles have contradicting interests. A is rewarded for defeating B. B is rewarded for defeating A, and B is given a larger budget to do it.
Transparency exposes reward logic. B can inspect how A is rewarded and try to exploit weaknesses. This creates pressure against reward gaming, but does not prove that every exploit will be found.
A has to certify its own task. If A cannot show that its task is informative, it is heavily penalized. The certification is checked by running code, never by a model's opinion. The intended incentive is for A to propose valid tasks that B cannot solve within its budget. Certification alone does not establish that a task is useful or free of exploits.
The intended pressure is adaptive. If B improves, A must find harder problems to keep winning. Whether that produces sustained progress rather than collapse or repetitive tasks is an empirical question.

What I do not know yet

By design, both roles chase the frontier, because that is where the reward is. It should get harder over time, though I have not yet shown that. What I cannot say is whether it is useful: whether the problems at that frontier line up with what people mean by useful, or whether the two roles find a frontier that is hard but beside the point. That is the real open question of this project.

So the game itself is not the test. The test is whether the model gets better at things it never trained on: held-out and public benchmarks.

What the trajectories show

I archived every task that A proposed in several runs, with 27B and 9B models, including one run that collapsed. Here is what is supported, and what is not.

A learns to write tasks that can be measured

At the start of a run, most of what A writes cannot be run and measured by the sandbox. By the end, about half can, in both the 27B and the 9B runs, while malformed output stays below 4%.

The share of A's proposals that the sandbox could actually measure, by fifth of each run. The two 27B lines come from different runs. One of the two 9B runs collapsed and is shown in red.

Some of the tasks A wrote

Here is the typical task from five stretches of that same 27B run, picked by a fixed rule (the middle one in each 25-step window by how hard B found it), not by eye. Every one is a classic hard optimization problem with a numerical score, written by the model itself, with nothing taken from a dataset.

Step	Problem	What A asked for
25	Quadratic assignment	Place 10 machines on 10 floors so the total pairwise transport cost is as small as possible.
64	Max-cut	Split the nodes of a 40-node graph into two groups so that as many edges as possible cross between them.
83	Makespan scheduling	Assign 50 jobs to 5 identical machines so that the last machine finishes as early as possible.
110	Facility dispersion	From 90 candidate sites in a unit square, choose sites that are as far apart from each other as possible.
133	Max coverage	Choose a limited number of sets so that together they cover as many elements as possible.

These are small instances and the kinds of problem change from window to window. So this is not a clean ladder of rising difficulty, and I do not claim it is. It shows the model producing a stream of different, runnable, scored problems on its own.

What is not shown

B improving is not shown. No external benchmark has been run on these runs yet. That is the test that matters.
Failure is possible. One 9B run collapsed, with 81% malformed output by the end. I include it because dropping it would be dishonest.

Where it stands: limited by compute

For now, the main limitation may be compute.

The model that shows the pattern is expensive. Only the 27B model shows the task evolution I was looking for. Because I target the full context, it needs at least 4 L40S GPUs for a LoRA configuration, or 2 H200 GPUs for full-parameter tuning. Even with my best engineering effort (and a lot of help from AI), I have rarely managed to get a run trained properly.
The models I can afford need too many steps. Smaller models spend their training just learning to write a valid task template, so the game never gets to the interesting part. The first chart above shows the early part of this: most of what A writes cannot be measured at first.

I am sharing the write-up now because the question is interesting and the early results are real. It may be worth another try for someone with a strong starting model and more compute.