Stop making agents ’guess and check’: deep thinking at test time helps machines truly learn to explore hidden rules
Latest research from the Tsinghua NLP team: how to let agents explore a new environment in real time like humans — reasoning about rules and breaking the trial-and-error loop
In today's fast-evolving world of large language models, having agents interact with environments and complete complex tasks is no longer novel. Agents appear in mobile web browsing, text games, and simulated embodied AI scenarios. Yet a key problem remains thorny: an agent may excel in a "familiar" environment, but once it enters a new environment with "hidden rules," it easily falls into a cycle of blind trial and error, hits wall after wall, and ultimately fails.
➤ Paper link
🔗 https://arxiv.org/abs/2605.24828
Existing methods mostly rely on knowledge acquired during training or offline experience, but when executing tasks in a new environment, training-time environment experience often cannot be reused. In the ALFWorld environment below, for example, the agent does not know the hidden rule that "you must stand directly in front of an object to interact with it," so it keeps generating similar actions; it does not know the hidden rule that "you cannot hold two objects at once," so it keeps trying to pick up a second pepper shaker, and the task times out and fails.
Figure: An agent interacting with the environment in ALFWorld
The latest research from Tsinghua University's Natural Language Processing Lab (TsinghuaNLP) — the test-time exploration (TTExplore) framework — targets exactly this pain point. It gives agents a kind of "deep thinking at test time" ability, letting them stop during task execution to analyze and reason about the "hidden rules" behind the environment, so they can truly learn to explore like a player who keeps gaining experience.
Not just acting, but stopping to think deeply
When people solve problems, they usually do not rush into trial and error; they alternate between "fast action" and "slow thinking." The essence of the TTExplore framework is to explicitly bring this cognitive pattern into agents.
The framework has two core roles:
Actor: responsible for fast, intuitive action, akin to "System 1."
Thinker: a specially trained 7B-parameter model, Exp-Thinker, responsible for slow, deep reasoning, akin to "System 2."
During task execution, after the actor takes a certain number of steps, the thinker is activated. It reviews the entire interaction history, summarizes current progress, and — more importantly — infers the latent environment rules that led to failure, then proposes a revised action plan.
Figure: In the TTExplore framework, the Thinker intervenes periodically to analyze history, infer rules, and revise plans, guiding the Actor's actions.
This is like a new employee (the actor) operating an unfamiliar machine. After pressing the wrong buttons several times, an observer (the thinker) walks over, looks at the operation log, and says: "Pressing the red button won't help — this model requires flipping the green switch on the left to unlock it first." With that hidden rule, the new employee can move forward.
Training a model that thinks: inferring thought quality from final outcomes
Training such a "thinker" role with reinforcement learning is not easy. The biggest challenge is: how do you evaluate whether a "thought" is good? We cannot directly score a "thought."
To solve this, the core ideas are:
Indirect reward mapping: use final task completion as an indirect reward signal for the "thought."
Single-node focus: keep only one thinking node in each training trajectory, so the final success or failure reward is clearly attributed to that one "thought," greatly improving training stability and efficiency.
Through this approach, the team successfully trained a dedicated thinker model with only 7 billion parameters — Exp-Thinker. It is small, but its "deep thinking" ability is very strong.
Results: significant gains, even surpassing much larger models
The research team rigorously tested TTExplore + Exp-Thinker in five textual embodied task environments and found:
Significant improvement over base models: after connecting Exp-Thinker to base models such as Qwen2.5-7B and LLaMA3-8B, average performance rose by 14 and 19 percentage points, respectively. On some tasks, base-model performance was even raised to the level of specialized fine-tuned agents.
A boost for seasoned performers: even when the actor model is already well trained for a specific task (Qwen2.5-Actor reaches a 97.76% success rate on ALFWorld), Exp-Thinker still delivers an average 8-percentage-point gain on unseen new tasks (BabyAI). This shows that "deep exploration" and conventional "task training" capabilities are complementary.
Breaking the trial-and-error loop: quantitative analysis shows that with Exp-Thinker, the agent's action and observation diversity increases significantly and repetition drops sharply. It successfully escapes the "local trial-and-error loop" and begins meaningful exploration.
Figure: With Exp-Thinker (green line), the agent's action and observation diversity rises significantly, repetitive behavior decreases, and exploration improves.
Takeaway: from "brute force" to "structural breakthroughs"
The core contribution of TTExplore is not a higher benchmark score, but a new way of thinking: improving agent capability does not depend only on more environments and data; it can also start from improving the agent's cognitive structure and decision process.
In the past, we cared more about "how to make agents do more things." TTExplore asks: when it hits an obstacle, can it stop and "think" like a human, analyze the cause, and proactively adjust its strategy?
When agents begin to explicitly reason about "environment rules" and dynamically revise their internal model at test time, the interaction between agent and environment changes. It is no longer a blind "action-feedback" loop, but observation, hypothesis, and verification — more like a scientist than a dice roll.
Although the paper focuses on textual embodied tasks, its core mechanism — decoupling "exploring hidden rules" from "executing tasks" and training a dedicated "thinking module" with reinforcement learning — has broad potential. Any scenario where agents must make decisions in dynamic, unknown environments with discoverable rules, such as robot control, game AI, and GUI operation, could benefit.
TTExplore is more than a methodological improvement; it is a reminder: as agents move into a more open and complex real world, we must not only teach them "how to do things," but also give them the wisdom to "think clearly before trying." That may be one step toward more general and robust agents.