AutoRLBench: An Agent Benchmark for RL Diagnosis and Repair
A thousand-mile dike can collapse from an ant hole. — Han Feizi, Yu Lao
✨ A Tsinghua University team presents AutoRLBench, a unified "exam in RL failure diagnosis and repair": each task is a real reinforcement learning run that went wrong, with code, configs, logs, and metrics preserved exactly as they were, testing whether an agent can find the root cause and fix it.
✨ In the first leaderboard, ten frontier models hold a joint consultation on crashed RL training runs: the top score is only 40%, with glaring gaps between tiers.
✨ The initial benchmark and live leaderboard are now open, and we invite the community to contribute real-world RL failure cases.
From the RSI Vision Back to Reality
Using AI to develop and improve AI (AI for AI) is the key loop toward recursive self-improvement (RSI) — a high ground that frontier labs and top research institutions are all racing to stake out. Agents are already deeply embedded in day-to-day R&D, yet current progress makes one thing clear: there is still a long way to go before real AI for AI.
Today's AI-for-AI evaluations mostly take a black-box view of "completing a task end to end." But for AI to truly iterate on its own, an end-to-end "total score" is not enough; we must break the R&D pipeline down into fine-grained stages so we can pinpoint where a model's real weaknesses lie.
So we chose reinforcement learning (RL) — the most "mystical" part of large-model R&D, with its diverse failure modes and enormous policy space — as the core touchstone. A tiny distribution shift, a hidden reward hack, or a low-level train-inference mismatch can all make an RL run collapse silently, without a single error.
We present AutoRLBench — the first fine-grained agent benchmark focused on fault diagnosis and repair across the entire RL pipeline. What we test is not "can you get training to run," but "when RL training collapses, can an agent notice the problem, locate the root cause, and fix it."
Call for Contribution: Building a High-Standard RL "Mistake Notebook" Together
If everyday RL training is live combat, AutoRLBench is the community's notebook of real mistakes.
No single team can ever exhaust the complex ways training can go wrong. Building a fine-grained RL diagnosis-and-repair benchmark therefore takes a community, so we are issuing an open call: researchers and developers, come build AutoRLBench with us!
🛠️ If you have a crashed RL run on hand, don't let all that burned compute sink without a trace. Send us that bizarre curve that puzzled you for a week, along with the scene of the accident, and let it become the next standard exam question in the benchmark — contribute two valid tasks and you become a co-author of the AutoRLBench paper!
🚀 And if you're building AI-for-AI systems, you're welcome to test the waters on the current benchmark and see whether your agent can pass.
Participating institutions — researchers and developers are invited to build AutoRLBench together!
Inclusion Criteria: What Makes a Qualifying RL "Mistake"?
AutoRLBench's core moat is extreme data quality. To make sure the benchmark reflects production-grade debugging, we enforce strict quality control on data provenance, case depth, and grading mechanism.
01 100% Drawn from Real Production Incidents, Covering Six Fault Domains
Every task must come from real production training: the workspace preserves the code, configuration, logs, and monitoring metrics exactly as they were when training crashed, paired with a hidden answer key that clearly defines the root cause and the criteria for it. By where the fault sits in the RL pipeline, tasks cover six fault domains:
configuration: mismatched hyperparameters or protocols, such as a bad learning rate or an unbalanced entropy coefficient.
reward: biased reward signals, such as an unconditional length bonus that only teaches the model to write longer, not more correctly.
data: unsuitable data, such as a training set that is too hard and a validation set that is too easy.
environment: external environment and tool failures, such as a retrieval tool that fails silently and quietly drags training down.
algorithm: bugs in core algorithm logic, such as a flawed GRPO implementation that turns every group into a single sample.
infrastructure: low-level system coordination failures, such as a train-inference mismatch that crashes training outright.
02 No Explicit Errors — Reconstructing the Real Process of Debugging by Inference
A valuable RL exam question is rarely one that throws an explicit error and exits. It is a hidden fault that can only be traced through deep reasoning across multiple signals, peeling the onion layer by layer. Here are two typical examples:
The first is a reward question. The nominal reward climbs from +0.82 all the way to +3.84, shooting past its own scale. At the same time, the truncation rate rises from 0.8% to 82.5% and response length grows from 4.1k to 15.3k tokens — while actual accuracy falls from about 29% to about 8%. The model has learned to be long, not correct.
The second is a data question. Training reward creeps from about 0.008 to 0.15, yet validation accuracy ends at 0.0380 — below where it started. The root cause is a training set that is too hard and a validation set that is too easy. The difficulty here is that you have to actively compare the difficulty composition of training and validation, not just stare at metrics.
03 Verifiable Through a Closed-Loop Sandbox
Every accepted task must support a three-stage evaluation inside an isolated container: Prepare spins up the container and mounts the model and data; the Agent receives only the task instructions and a writable copy of the workspace, and investigates and modifies it independently; finally a Verifier steps in to close the loop and grade.
Level 1, Issue evaluation (the evidence chain): checks not only whether the conclusion is right, but whether the root cause is pinned to a specific piece of code or location in the data, and whether the chain of evidence is complete.
Level 2, Solution evaluation (the rerun decides): the agent must actually edit code and tune parameters, after which the system really launches and reruns the training, grading directly against thresholds on the core metrics.
The First Leaderboard: Top Score Only About 40%, With a Clear Gap
In the just-launched first round of AutoRLBench, ten leading frontier models from China and abroad (including Claude, Grok, GPT, GLM, Qwen, Kimi, Gemini, and DeepSeek) sat the exam together. The top score is only about 40%, with a striking gap between tiers!
The road to AI for AI needs agents that can write training code — but even more, it needs the system-level ability to understand failure and repair it. Together with the community, we want to turn every real training stumble into a building block for agent evolution.