As small models get stronger, is the cloud becoming a strategist? Tsinghua and other teams systematically study Agent device-cloud collaboration
When AI Agents handle email, organize files, and schedule meetings locally, should the local small model do the work itself, or hand the task to a cloud large model? Researchers from Northeastern University, Tsinghua University, and other institutions introduce EC²-Bench. Based on 128 executable long-horizon Agent tasks, it systematically compares four device-cloud collaboration strategies on the trade-offs among task performance, compute cost, and privacy exposure: a 9B on-device model reaches about 73% of the pure-cloud task completion score using only about 15% of the pure-cloud invocation cost; the stronger the device side, the more the performance gap converges, yet the privacy exposure gap remains huge; and the most effective collaboration mode flips from "the cloud doing the work" to "the cloud acting as strategist."
This article was completed by researchers from Northeastern University, Tsinghua University, and other institutions.
As AI Agents begin handling email, organizing files, and scheduling meetings locally, a question follows: should the local small model do the work itself, or hand the task to a cloud large model?
Intuitively, cloud models are more capable, so letting them participate more in execution should complete tasks better. But a recent study finds it isn't that simple.
For weaker models, having an external Router decide when to let the cloud take over difficult steps works best; but when the on-device model gets stronger, the most effective collaboration mode flips — the cloud no longer does the work but provides advice, and the on-device Agent executes itself.
More interestingly, as on-device capability grows, performance differences between collaboration strategies shrink, yet the privacy exposure gap remains huge: the maximum task completion score difference is only 3.67 points, while the sensitive-information exposure ratio differs by as much as 66.23 percentage points.
To systematically study these phenomena, the team introduces EC²-Bench, based on 128 executable long-horizon Agent tasks, exploring the trade-offs of different device-cloud collaboration strategies among task performance, compute cost, and privacy exposure.
🎯 Key finding: device-cloud collaboration can do more with less — a 9B on-device model, using only about 15% of the pure-cloud invocation cost, reaches about 73% of the pure-cloud task completion score, significantly surpassing standalone on-device execution.
🤔 Surprising finding: performance gaps converge, privacy gaps remain significant — as on-device capability grows, task performance across strategies gets closer, yet sensitive-information exposure can still differ by tens of percentage points.
💡 Key takeaway: from "the cloud doing the work" to "the cloud as strategist" — weak Agents need the cloud to help execute directly, while strong Agents are better off deciding for themselves when to ask for help and how to use cloud advice.
Long-Horizon Edge-Cloud Agent Collaboration: collaboration trace, Workspace environment, four collaboration strategies, and three research questions, using "scheduling a project meeting" as an example (RQ1 collaboration effectiveness / RQ2 impact of on-device capability / RQ3 evolution of collaboration modes)
➤ Paper
🔗 https://arxiv.org/abs/2610.17816
➤ Project page
🔗 https://passionate11.github.io/ec2-bench-project-page/
➤ Open-source code
🔗 https://github.com/OpenBMB/EC2-Bench
Background: Agent device-cloud collaboration is no longer just "large model or small model"
In the past, discussions of device-cloud collaboration were usually simple: the user enters a query, and the system decides whether to send it to the local small model or the cloud large model.
Local processing is cheap and keeps data more controllable, but capability is limited; cloud models are stronger but bring extra invocation costs and privacy exposure risk. Designing a sufficiently smart Router to decide who handles each request seems to solve the problem.
But Agents differ fundamentally from ordinary Q&A: an Agent doesn't finish after one answer; it continuously interacts with the environment and completes the task step by step.
Take scheduling a meeting: the Agent may need to read emails, check multiple people's calendars, find free time, coordinate conflicts, and finally create the meeting and send invitations.
In this process, some steps are easy for a local model, while others need stronger planning and reasoning. The cloud may join multiple times mid-execution, and each participation can affect subsequent execution.
Worse, the Agent may have read customer materials, email content, or internal documents in the first few steps. When it asks the cloud for help, that information may also be sent along with the execution context.
So the real question isn't just "whether to call the cloud," but "when to call it, what to have the cloud do, and how much cost and privacy to pay for it."
These new complexities make device-cloud collaboration in long-horizon Agent scenarios unpredictable, and we still lack a systematic understanding of the underlying patterns. To this end, the team studies three core questions, building a systematic understanding of Agent device-cloud collaboration from collaboration effectiveness, the impact of on-device capability, and the evolution of collaboration modes:
RQ1: Is device-cloud collaboration still effective in long-horizon Agent tasks?
RQ2: How does stronger on-device capability affect collaboration outcomes?
RQ3: Does the most effective collaboration mode change as on-device capability improves?
How we study it: four device-cloud collaboration modes compete on the same Agent tasks
The team first set up two single-side execution baselines, then compared four representative device-cloud collaboration strategies.
First, Query Routing: choose the executor before the task starts.
The Router makes one decision; the entire task goes either local or cloud. Simple and direct, but it can't adjust dynamically during execution.
Second, Per-step Routing: re-decide who executes at every step.
The Router judges at each step whether local or cloud executes. Steps local can handle stay local, difficult ones go to the cloud — more flexible than one-shot routing.
Third, Edge-Controlled Delegation: the local Agent decides when to call the cloud to take over.
No longer does an external Router give orders; the Agent asks for help proactively. The cloud takes over the current step, then hands control back to the local side.
Fourth, Edge-Controlled Advisory: the local Agent does the work, the cloud offers ideas.
The Agent always holds execution authority; when facing a hard problem it consults the cloud for plans, suggestions, or corrections, then performs the actual operations itself.
The core difference among the four comes down to two questions: who decides when to ask for help? And does the cloud step in to do the work, or stay behind the scenes as a strategist?
Two single-side execution baselines (Edge-only / Cloud-only) and four collaboration strategies: Query Routing and Per-step Routing are decided by an external Router; Edge-Ctrl. Cloud Delegation and Edge-Ctrl. Cloud Advisory are decided by the on-device Agent itself
With different collaboration strategies in place, the next need is an evaluation environment that compares them fairly. The team built a unified execution and evaluation environment based on OpenClaw, assembling 128 executable Agent tasks across six categories: daily office work, information retrieval, security and privacy, development and operations, data analysis, and automation. These aren't simple Q&A problems. The Agent must actually call tools, operate files, modify the environment, and be judged by the final execution result.
EC²-Bench's evaluation doesn't only ask whether the task was completed; it examines three dimensions at once:
Task performance: how much did the Agent actually accomplish?
Resource consumption: how much did cloud calls cost, and how much local compute was used?
Privacy exposure: how much sensitive information was sent to the cloud during execution?
To quantify privacy exposure, the team also annotated fine-grained sensitive information for 100 of the tasks, covering 3,385 sensitive information units across 12 categories of sensitive data including personal information and organizational secrets.
In other words, EC²-Bench doesn't just look at who does the job better; it also tracks how much the Agent spent and how much information it revealed to the cloud to get the job done.
EC²-Bench main results (cloud model: GLM-5.3)
Result 1: device-cloud collaboration does more with less — 15% of cloud cost unlocks 73% of pure-cloud performance
The first question: in complex long-horizon Agent tasks, is device-cloud collaboration really effective?
With Qwen3.5-9B as the on-device model and GLM-5.3 as the cloud model, pure on-device execution scores only 39.12; with a reasonable device-cloud collaboration strategy, the score rises to 60.46, a relative improvement of about 55%, reaching 73% of the pure-cloud solution.
More importantly, this performance requires only $8.73 in cloud invocation cost, about 15% of pure-cloud execution ($56.42).
This shows the "do more with less" potential of device-cloud collaboration: you don't need the cloud to handle the entire task; leveraging cloud capability at key moments can significantly improve the on-device Agent's task performance. Further analysis finds that some collaboration strategies can even surpass the linear interpolation reference line between pure on-device and pure-cloud solutions on the performance-cost and performance-privacy trade-offs.
The value of device-cloud collaboration isn't just reducing large-model calls; it's letting large and small models complement each other, using less cloud investment to unlock stronger task execution.
Trade-offs of device-cloud execution modes. (a) Cloud cost–task completion; (b) privacy exposure–task completion. Green is Qwen3.5-2B, blue is Qwen3.5-9B; dashed lines are linear interpolation between on-device and cloud; upper left is better
Result 2: stronger on-device models narrow performance gaps, but privacy gaps persist
As model intelligence density keeps rising, the model size needed for the same capability keeps shrinking. Tasks that require a large model today may be handled by a lighter on-device model tomorrow. So, when future on-device small models gradually gain today's large-model capabilities, does device-cloud collaboration still matter?
To simulate the trend of continuously improving on-device capability, the team tested multiple configurations from Qwen3.5-2B and 9B to higher-capability models.
First, as on-device capability grows, device-cloud collaboration's task performance approaches pure-cloud execution. On the higher-capability Qwen3.7-Plus, the best collaboration scheme reaches a task completion score of 76.84, close to the pure-cloud solution's 82.70.
More interestingly, performance gaps between collaboration strategies also shrink significantly. On the weaker 2B model, the four strategies differ markedly; by Qwen3.7-Plus, the maximum task completion score difference is only 3.67 points.
Looking only at task performance, as on-device models get stronger, the strategies seem increasingly hard to separate.
But when the team turned to privacy, the results were completely different.
In the Qwen3.7-Plus experiments, the four strategies' task performance is already very close, yet the sensitive-information exposure ratio differs by as much as 66.23 percentage points!
More notably, the best-performing strategy is exactly the one with the most privacy exposure. Cloud advice helps the on-device Agent complete tasks better, but frequent consultation and passing execution context can also expose more sensitive information to the cloud.
Better performance doesn't necessarily mean better privacy.
This finding suggests that as on-device capability improves, the focus of device-cloud collaboration research may gradually shift from purely pursuing task performance to controlling cost and privacy exposure better at similar performance levels.
(a) Task completion and (b) privacy exposure of each collaboration strategy under different on-device models. Dots are means, bars are ranges across strategies, red dashed line is pure-cloud GLM-5.3
Result 3: from the cloud doing the work to the cloud as strategist
Earlier experiments found that as on-device capability improves, task performance across strategies converges. But this raises a deeper question: what collaboration mode suits on-device Agents of different capabilities?
Comparing the four strategies across on-device capability levels, the team found an interesting phenomenon: the most effective collaboration mode shifts as on-device capability improves.
For the weaker 2B model, Per-step Routing — where an external Router decides when to call the cloud and has the cloud execute directly — performs best, with a task completion score of 39.97; letting the on-device Agent decide when to delegate to the cloud scores only 15.86.
The reason isn't hard to understand: when a small model's own capability is limited, judging when it needs help is itself difficult. Here, having an external Router decide for it and letting the cloud take over hard steps is often more effective.
But as on-device models get stronger, this relationship starts to reverse.
Starting from the 9B model, Cloud Advisory — where the on-device Agent decides when to ask for help and the cloud only offers advice — becomes the best-performing strategy, with a task completion score of 60.46, exceeding the other three. The trend holds on stronger on-device models.
This suggests that as on-device execution capability improves, the cloud doesn't necessarily need to take over hard steps directly; timely guidance may be more effective. Weak Agents need the cloud to do the work; strong Agents are better suited to letting the cloud act as strategist.
But why are stronger on-device models better at this autonomous collaboration? What capabilities does a model actually need to make good use of the cloud as a strategist?
Collaboration modes shift with on-device capability — from external routing to edge-controlled delegation to edge-led cloud advisory
Why the reversal? The key is learning to ask for help and to take advice
Why are stronger on-device models better suited to letting the cloud provide advice rather than take over tasks directly?
The team analyzed this from two angles.
First, knowing when to ask the cloud for help.
The team designed a random-invocation control experiment: keeping the cloud invocation ratio the same, they compared the Agent's autonomous help-seeking with random cloud calls.
The results show that the stronger the on-device model, the more obvious the advantage of choosing when to ask for help autonomously. This suggests strong models aren't only better at executing tasks independently; they also better understand when to seek help at critical moments.
Second, whether the model can actually use the cloud's advice.
Good cloud plans don't mean the on-device Agent can execute them correctly. Especially for weak models, understanding advice and turning it into action is itself a threshold.
To this end, the team performed instruction-following training on the 2B model. The results show that as the model's ability to understand and follow instructions improves, its performance in using cloud advice to complete tasks also improves markedly.
Together, the two experiments reveal two key capabilities behind autonomous collaboration: knowing when to ask for help, and being able to make real use of the advice received.
(a) Task completion of edge-side autonomous invocation vs. random cloud invocation; (b) task completion of Qwen3.5-2B under the cloud-advice setting after instruction-following training (x-axis: GRPO training steps)
Final thoughts: device-cloud collaboration expands the capability boundary of on-device Agents
EC²-Bench shows that reasonable device-cloud collaboration can not only do more with less cloud investment, but its most effective collaboration mode also evolves as on-device model capability improves.
These findings offer new ideas for the future of on-device Agents: improving on-device capability needn't be limited to strengthening the model's own reasoning and execution; device-cloud collaboration can also be built as a core capability.
Let Agents learn to judge when they need help, how to seek cloud support, and how to turn cloud advice into action — using cloud large-model capability more effectively while balancing cost and privacy.
The capability boundary of future on-device Agents shouldn't be determined only by the model's own ability; it can be continuously expanded through efficient device-cloud collaboration.
The EC²-Bench paper, code, and evaluation resources are now public — we welcome you to read and discuss!
For more experimental details, see the original paper and project page.
🛰 EC²-Bench: turning the cloud from doer into strategist
In one sentence:
We introduce EC²-Bench, based on 128 executable long-horizon Agent tasks, jointly examining task performance, cloud cost, and privacy exposure, and systematically comparing four device-cloud collaboration strategies; a 9B on-device model reaches 73% of the pure-cloud score with about 15% of the cost, and we find that the most effective collaboration flips from "the cloud doing the work" to "the cloud as strategist" as on-device capability grows.
The takeaways:
Do more with less: 15% of cloud invocation cost unlocks about 73% of the pure-cloud task completion score
Performance converges, privacy diverges: the stronger the device side, the narrower the score gap (down to 3.67 points), while the privacy exposure gap remains 66.23 percentage points
Collaboration mode reverses: weak Agents need the cloud to do the work; strong Agents are better off with the cloud as strategist
Two key capabilities: knowing when to ask for help, and being able to make real use of the advice received
➤ Paper
🔗 https://arxiv.org/abs/2610.17816
➤ Project Page
🔗 https://passionate11.github.io/ec2-bench-project-page/
➤ Code
🔗 https://github.com/OpenBMB/EC2-Bench
If you're also following Edge-Cloud Agent Collaboration / On-device Agent / Agent Benchmark, follow our work and try EC²-Bench.
As the edge agent grows stronger, the cloud shifts from doing the task for it to advising on how to do it itself.