AgentCPM-GUI is an open-source device-side LLM agent jointly developed by THUNLP, Renmin University of China, and ModelBest for autonomous smartphone task execution.
Technical highlights
：Different from general multi-modal models, AgentCPM-GUI is specifically pre-trained for Chinese Android application interfaces, covering local applications such as Amap, Dianping, Bilibili, and Xiaohongshu. This in-depth optimization of vertical scenes makes it more advantageous in understanding and controlling Chinese applications, and solves the problem of "acclimatization" of general models in the recognition of Chinese interface elements.
：GUI Grounding: Accurately identify and locate common controls such as buttons, input boxes, labels, and icons.
：OCR Grounding: Not only can it recognize the text content on the screen, but it can also perform corresponding operations based on the text description.
：By designing reward functions in three dimensions: action format reward, action type reward, and action parameter reward, the model is guided to independently generate a high-quality Chain-of-Thought process. Compared with traditional supervised fine-tuning, this reinforcement learning method allows the model to continuously optimize the strategy during execution and significantly improves the success rate of task execution.
：Compressing the action representation into JSON format with an average of only 9.7 tokens, this design brings three major practical benefits:
：Reduce inference costs: Reduce computational and memory requirements.
：End-side deployment friendly: more suitable for running locally on mobile devices.
：Enhanced privacy and security: Reduce cloud interactions and sensitive operations can be completed locally.
：The model can not only understand single-step instructions, but also automatically split complex tasks into executable step sequences and coordinate them between different applications. This closed-loop capability from "user intention" to "multi-application collaborative execution" is its core competitiveness as an intelligent assistant.