Scalable RL: ModelBest Open-Sources Meshy, the Post-Training Framework Behind MiniCPM5-2B, with Four Innovations That Raise On-Device Intelligence Density
In the past, the reinforcement learning stage of a large model looked like an assembly line: the model generated data, the system evaluated the results, and the parameters were updated, with every step proceeding in order and in sync.
But as AI moves from single-turn Q&A toward executing complex tasks, large models must not only keep generating, learning, and making multi-turn tool calls, but also interact with external environments and handle data from different stages and versions at the same time. The once clear-cut, fixed training pipeline has therefore become far more dynamic and complex.
Under these conditions, centralized control architectures such as Single-Controller began to fall short. All data converges on a single controller before being dispatched again, which easily causes transfer congestion; if one step is slow, the whole chain has to wait, and efficiency drops sharply; as tasks grow more complex, managing state across versions and stages becomes a muddle. The flexibility and scalability of the entire system are constrained.
To address this, ModelBest and OpenBMB have open-sourced Meshy, a self-developed reinforcement learning framework. Through a service-oriented architecture, it achieves a structural innovation that removes the dependence on a central controller and on Ray, splitting roles such as inference, training, and reward computation into independently running services coordinated through a unified data queue, TransferQueue. It supports synchronous training, asynchronous reinforcement learning, streaming training, online distillation, and other modes.
This architecture reduces unnecessary data communication and scheduling overhead, making large-model reinforcement learning more flexible and efficient and further raising intelligence density.
Compared with traditional reinforcement learning frameworks, Meshy introduces four major technical innovations.
Roles as Services, Composed on Demand Like Building Blocks
In Meshy, every role — inference, training, rollout, and so on — is defined as an independent service. Each handles its own complete business logic and cooperates with other services through the data queue contract TransferQueue. This design is equivalent to splitting the formerly centrally controlled "big factory" into multiple "independent workshops," each with its own job.
In practice, developers define the entire training pipeline in Meshy with a single recipe file made up of two parts of configuration.
ServiceGroup describes a set of service instances with the same role, including a stable ID, a typed config, the number of replicas, and the number of GPUs each replica uses.
ColocationRing describes which services are colocated on the same set of GPUs, as well as the scheduling mode and order among them.
The biggest benefit of this design is flexibility. Developers can freely combine different roles like building blocks: just select the participating roles through ServiceGroup, define data transfer between roles with TransferQueue, and specify the colocation mode with ColocationRing to implement a variety of training schemes.
This separation keeps algorithmic workflows and system implementation from interfering with each other. To add a new role, you write the service and allocate resources, and the existing inference, generation, and training services need no changes at all. This makes it easier for the RL system to gain new capabilities and provides a unified foundation for building synchronous, asynchronous, streaming, and distillation training pipelines.
Deterministic Topology Derivation for Simpler Startup
Once services are split apart, a practical question comes first: which machine should each service run on, and which GPUs and ports should it use. A common approach is to use a registry or configuration file to record and distribute this information, but that adds components the system must maintain.
Meshy takes a simpler approach: computing the topology with a pure function.
Put simply, the system starts in a pretraining-style SPMD manner. A single torchrun command launches an ignitor management process on every GPU; this process does no training computation — it only derives the topology, starts the services assigned to its card, and manages their running state.
Once all ignitors have started, they first obtain the global GPU list and learn which host each card is on and its index. Each ignitor then independently executes the same deterministic pure function based on the same recipe (the service configuration scheme) and GPU list, and the function computes the complete deployment information for every system. Ports are also generated by fixed rules.
Because all processes use the same information and the same computation rules, as long as the service configuration and GPU environment stay the same, every process computes exactly the same result. Processes therefore need not pass service locations to one another — each simply starts the services assigned to it.
The biggest benefit of this design is that it is lightweight. The system needs no extra registry, and deployment relationships are clear and traceable. The same configuration yields consistent results in the same GPU environment, which makes reproduction and debugging easier. Problems such as insufficient GPUs, port conflicts, or configuration mismatches surface at startup rather than halfway through training.
Data-Driven Collaboration for More Efficient Services
One of Meshy's biggest innovations is its redesign of how services collaborate. It lets different services run independently around the same data system and advance on their own. That system is TransferQueue (TQ), on which both data transfer and flow control are built.
TQ is an open-source data transfer system that can be thought of as several fixed-capacity "shared data tables." Each table is called a partition, each sample corresponds to a row, each row contains multiple named data columns, and each piece of data contains multiple data items that can be written separately. Different services can add their own contributions around the same piece of data; a downstream service only needs to confirm that all the data items it needs are ready, then it can read them and start computing, and write the results back to TQ for subsequent services.
Data collaboration: start work once everything is in place. Each service declares in advance which data it needs, queries periodically, reads on demand, computes locally, and writes results back to TQ. For example, a training service that needs teacher-model scores simply adds the score column to its read conditions; once the scores are written, the samples become readable automatically, with no need to send a "scoring complete" notification. Each service maintains its own read records without contention, and the terminal service cleans up samples after use to keep storage from growing indefinitely.
Flow synchronization: coordinate the rhythm with simple signals. For control information that must be passed explicitly, Meshy uses a separate data partition in TQ. For example, in synchronous RL training, the Trainer writes a gate signal containing the latest weight version to the gen-gate partition after each training step. Rollout checks that partition before each dataset read to obtain the current weight version and decide how many samples to read this time, thereby pacing generation against training.
In short, once the data is ready, downstream services start automatically; when the rhythm needs to be unified, a separate signal coordinates it. Large blocks of data no longer pass through a central node, and services can run independently while staying in step. TQ completely separates data movement from flow control, achieving efficient cooperation among services with a lightweight, traceable mechanism.
Multiple Roles Sharing GPUs, Letting Limited Compute Flow Flexibly
In reinforcement learning training, the training engine and the inference engine are usually placed on the same set of GPUs, and the system switches between training and inference as needed to maximize compute utilization.
But when more roles such as a Teacher are added to the pipeline, GPU switching becomes more complex. Past frameworks typically required developers to write resource handover and VRAM switching logic separately for each role combination, and any change to the training recipe could require re-adaptation.
To address this, Meshy designed a token-ring-based mechanism for multi-role colocation. Whoever needs the GPU requests the token, and only after obtaining it can it start computing. When the task finishes, the service automatically releases the VRAM it occupies and passes the token to the next role. If no other role requests it for the moment, the GPU goes to the inference service by default, keeping resources working continuously.
Take a distillation scenario in which inference, the teacher model, and training run together. The inference service normally generates data. When the teacher model needs to score, or the training service needs to update parameters, they can temporarily take over the GPU. When their tasks finish, the GPU is returned to the inference service.
Throughout this process, TransferQueue carries the request and grant signals. Developers only need to specify in the configuration which roles share GPUs and which token ring they belong to, and implement a unified VRAM release and restore interface for each service, to enable multi-role collaboration.
With this design, Meshy does not need a separate set of adaptation code for every training scenario. Limited compute can be flexibly allocated among multiple roles, giving developers a simpler workflow and more freedom of combination.
Based on these four mechanisms, Meshy drives each service autonomously through data-readiness states, passing samples directly from producers to downstream consumers so that generation, scoring, and training proceed in parallel, reducing the overhead of routing data through a central node. When a new role is added, the system mainly bears that role's own computation and processing cost instead of simultaneously adding to the central scheduling burden, improving overall resource utilization.
The significance of this shift goes beyond cost reduction: it provides more flexible, scalable infrastructure for agent training. Moving from centralized scheduling to autonomous collaboration, Meshy lets limited compute support more roles, more complex interactions, and higher-frequency training iterations, supporting the continuous improvement of agent capabilities.
Currently, MiniCPM5-2B, ModelBest's new-generation high-intelligence-density on-device language foundation model trained with the Meshy framework, has been officially open-sourced. The model ranks first among open-source models under 4B parameters worldwide on the Artificial Analysis composite intelligence index, and with a 2B parameter scale it has begun to show general on-device agent capabilities. For more architecture design and implementation details, see the official technical blog.
➤ Meshy framework open-source links
GitHub:
🔗 https://github.com/OpenBMB/Meshy
Blog:
🔗 https://maydomain.notion.site/meshy-blog-zh