SimpleMemVLA: native visual context lets robots remember the past

2026-09-18

➤ Paper

🔗 https://arxiv.org/abs/2609.05533

➤ Code

🔗 https://github.com/OpenBMB/SimpleMemVLA

➤ Model

🔗 https://huggingface.co/collections/yinchenghust/simplememvla

No dedicated memory module, just timestamped visual history as context — what can a robot do? SimpleMemVLA's answer: within the scope compared in the paper, it reaches SOTA overall on four memory benchmarks and, without any parameter updates, exhibits visual in-context learning. It is also the technology behind the context memory and streaming memory in MiniCPM-RobotManip.

The idea is simple: let the robot use what happened before to understand what to do now. Where a covered object is, how many times an action has been completed, how far a task has progressed — all of it can be grounded in raw visual history.

Why is this possible now? In the paper's setup, 60 seconds of history takes only about 5.6k tokens, while the backbone supports a 262k-token context. Minutes of history fit directly into the model without first compressing them into a separate memory representation.

The work is a collaboration led by ModelBest with Huazhong University of Science and Technology, Renmin University of China, Tsinghua University, Peking University, Beijing Academy of Artificial Intelligence (BAAI) and Beijing Zhongguancun Academy.

Native visual context helps robots remember more accurately on long tasks

From occlusion and counting to long-horizon interaction, SimpleMemVLA achieves the best overall results on four memory benchmarks while keeping strong performance on two general manipulation evaluations.

These tasks test more than whether the robot can read the current frame. They ask whether it can remember clues that have disappeared, tell repeated events apart, and keep tracking task progress.

Figure 1 | Four memory benchmarks and two general manipulation evaluations. Each benchmark is compared separately; gray bars are reference results and are not ranked.

In long-horizon interaction, where task progress must be tracked continuously, this memory pays off clearly. RoboMemArena has 26 tasks with trajectories averaging over 1,000 control steps. With a 126-second history window, SimpleMemVLA reaches 63.6% success on all-stage tasks, 17.4 percentage points above the previous strongest model. Counting tasks rise from 31.4% to 71.4%, and occlusion tasks from 39.1% to 64.3%.

The advantage comes not only from the backbone but from the model still reading raw visual history directly when it decides.

Whether a detail matters often becomes clear only later in a task. Filtering, compressing or overwriting history early can discard information the future needs. The paper calls this the "write-time commitment": at the moment of storing a memory, you must already decide what to keep.

SimpleMemVLA keeps historical visual information, temporal order and timestamps in context, so the model understands the past together with the current task rather than acting on a fixed summary prepared in advance.

Figure 2 | Native visual context versus dedicated memory mechanisms. The figure shows the specific implementations tested in the paper.

Concretely, past frames follow the video format used in vision-language model pre-training, ordered in time and annotated with plain-text timestamps; the current wrist view is fed in separately. The model then generates a one-sentence current subtask, such as "lift the lid on the red block."

The context hidden state formed while generating that sentence is passed to the action head along with token embeddings and the robot's proprioceptive state to produce the next action chunk. Ablations show that history shapes behavior mainly through this hidden state, not just through the literal words of the subtask.

Figure 3 | SimpleMemVLA architecture and streaming inference pipeline

To separate the backbone's ability from the memory mechanism, the researchers fix the training data, backbone, subtask supervision, action head and optimizer, and change only the memory mechanism. On RoboMME:

Native context

88.3%

Retrieval

31.5%

Token compression

22.6%

Recurrent state

20.6%

Figure 4 | Task-level success rates for different memory mechanisms under the same model and training setup.

These results support a key conclusion: on the tasks tested, letting the model interpret still-readable history in light of the current task beats deciding in advance which information is worth keeping.

Further ablations show that key frames and temporal order are indispensable. In the RMBench block-covering task, a 30-second window still contains the covering event from about 25 seconds earlier, and the model succeeds; shrink the window to 15 seconds and the key frame falls out, and the task fails. The button-counting task degrades steadily as the history window shrinks. Keep only the current frame, or shuffle frame order, and both tasks fail.

Figure 5 | Ablations on native context. (a)(b) test the history window, frame order and timestamps; (c)(d) test how the subtask hidden state carries history to the action head.

Visual in-context learning: adapt policy to visual context without parameter updates

SimpleMemVLA can do more than use past events: it adjusts current behavior to newly supplied visual context.

The researchers splice a segment from a different execution trajectory into the existing history: change where the block was covered and the model turns to the new location; change the color cue from magenta to red and it picks the red object; replace the opening action of a demonstration sequence and the robot changes its next move accordingly.

None of these spliced histories appear in training, and no parameters are updated at inference. The model still infers task state from the new visual context and acts on it. This is the visual in-context learning the paper observes.

Figure 6 | Masking or replacing history segments to test how the model acts on the past

The behavioral changes are targeted: altering key history changes decisions, altering irrelevant history does not. Masking the frame that shows where a block was covered makes the model choose the wrong lid; masking a completed pick-and-place event makes it undercount by one and execute again. Masking an irrelevant segment of equal length leaves the output unchanged.

On the tasks tested, native context both carries memory and lets new visual information directly shape behavior.

Streaming inference optimization keeps minutes-long memory in step with real-time control

Long history need not mean long waits. By reusing computation over history, SimpleMemVLA cuts decision latency with 60 seconds of history from 1.02 s to 0.68 s, below the 0.96 s real-time budget measured on the tasks.

The result comes from a single H100 with bf16 and batch size 1. The implementation is straightforward: consecutive decisions share most of their history, so while the robot executes the current action, the system precomputes the history prefix the next decision will reuse and stores its key-value cache. At the next decision it processes only the new content and generates the subtask.

This exact streaming pipeline produces the same subtask text as full recomputation, but overlaps shared-history computation with action execution to cut the wait at decision time.

Figure 7 | Shared-history computation overlaps with action execution, shortening the wait before the next decision

In a longer-context latency test, about 45 minutes of history maps to 245k tokens: full recomputation takes 32.1 s while the streaming decision path cuts it to 1.18 s. That shows the payoff of computation reuse; at extreme context lengths, background prefill and cache overhead still grow with history.

Real-robot demonstration

Across three autonomous runs with different layouts, SimpleMemVLA uses visual history from before the blocks were covered to lift the lids in red → green → blue order.

The task runs on a real dual-arm robot: it first covers three colored blocks with identical black lids, then lifts them in a specified color order. The blocks' colors are no longer visible in the current view, so the robot must rely on earlier experience to know what is under each lid.

Layout 1: lifting lids by color

Layout 2: lifting lids by color

Layout 3: lifting lids by color

Across the three runs, the lid-lifting path follows the color positions. The third run uses a middle → far → near order, showing that the robot does not scan in a fixed direction but uses the color and position relations from history to complete the task.

Outlook: from remembering the past to learning from experience

SimpleMemVLA uses native context as memory, and visual in-context learning emerges along the way.

Next, the team will add richer context: egocentric human data, robot demonstration videos and histories of autonomous execution. By fully exploiting the model's in-context learning, context will not only help the robot remember the past but become a key source for learning and adaptation.