OpenSQZ Glass Harness × MiniCPM-o: more controllable, customizable and fluid on-device omni-modal models in the first-person world

2026-10-20

➤ Paper link

🔗 https://aclanthology.org/2026.acl-demo.82/

➤ Open-source code repository

🔗 https://github.com/OpenSQZ/OpenGlass

As AI glasses move from one-off "take a photo, ask a question, get an answer" interactions toward continuous first-person interaction, the system no longer faces only the question of whether a model can understand an image. Users may change intent mid-broadcast, switching from ordinary chat to specific tasks such as finding an object or reading text; different tasks need different context and may require specialized capabilities such as object detection and OCR. At that point, no matter how capable a single model is, it needs a runtime layer to manage when it speaks, when it stops, when it starts over, and which set of capabilities the current task should activate.

At WAIC 2026 this July, the OpenSQZ team at Shanghai Qi Zhi Institute presented OpenSQZ Glass, built on ModelBest's MiniCPM-o 4.5, closing a real-time omni-modal interaction loop between low-cost first-person glasses and an on-device computing device. After WAIC, the team further developed OpenSQZ Omni Harness to make on-device omni-modal models more controllable, customizable and fluid inside a glasses system: it does not replace MiniCPM-o, but adds broadcast control, Session management, Skill routing and a vision sidecar alongside the model, moving on-device omni-modal AI glasses from "able to interact continuously" toward "able to be controlled, customized and extended."

OpenSQZ Glass uses MiniCPM-o 4.5 as its on-device omni-modal model foundation, folding visual understanding, speech understanding and speech generation into a unified end-to-end model. Its full-duplex multimodal streaming interaction lets the model generate text and speech while continuously receiving real-time video and audio, achieving "seeing, listening and speaking at the same time," and perceiving user interruptions and environmental changes during interaction. For proactive perception and interaction, the model can initiate reminders or comments based on continuous understanding of the live scene, without waiting for the user to ask a new question each time. Combined with local deployment, these capabilities provide the basis for continuous environmental understanding, real-time narration and natural voice interaction on AI glasses. OpenSQZ Omni Harness then further manages broadcast timing, conversation boundaries and specialized tasks on top of this, so model capabilities better fit actual first-person use.

In a typical WAIC scenario demo, OpenSQZ Glass showed MiniCPM-o continuously receiving the glasses' view, speaking while watching, and proactively reminding based on scene changes.

See and speak | Continuously receiving the glasses' view and responding by voice

Proactive reminders | Proactively raising reminders as the scene the user is focused on changes.

From a WAIC prototype to a reproducible UbiComp/ISWC research artifact

The OpenSQZ Glass shown at WAIC 2026 uses a "separated sensing and compute" architecture: the glasses capture first-person images and audio, while a nearby PC or edge host runs MiniCPM-o for model inference and speech generation. With the model and related services deployed locally, data travels over the local network and does not need to be uploaded to the cloud.

OpenSQZ Glass further packaged its 3D-printed hardware and local interaction pipeline into OmniGlass-Edge, which was accepted to UbiComp/ISWC 2026 Posters & Demos. The system uses an ESP32-S3 and an OV5640-AF as the sensing front end; the 3D-printed frame leaves room for the camera, mainboard, power supply and cabling, keeps an externally accessible Type-C port, and reserves expansion positions for future sensors such as ToF, infrared and IMU.

OmniGlass-Edge emphasizes "printable, assemblable, reproducible and further modifiable." The repository already provides editable frame files, print bed layouts, a bill of materials and build tutorials in Chinese and English, and integrates capture firmware, a local host bridge and backend runtime code. Developers can swap hardware components on the existing pipeline, change the model backend, or test new interaction mechanisms.

This research artifact provides both a reproducible hardware base and support for recording and observing the local interaction process. OpenSQZ Omni Harness builds on it to further manage tasks, sessions and output, so different interaction mechanisms can keep iterating on the same platform.

Figure 1 | Evolution from the WAIC prototype and OmniGlass-Edge to OpenSQZ Omni Harness with MiniCPM-o 4.5

Architecture upgrade: adding a Harness layer beyond the on-device model

MiniCPM-o gives AI glasses unified understanding and generation across vision, speech and language. But model capability does not automatically equal complete wearable interaction capability. While the model is broadcasting, the Runtime needs to respond to the user's pause and resume commands; when ordinary chat switches to finding an object or reading text, the system needs to establish a new conversation boundary so it does not carry over the old task's context; and connecting lightweight vision models such as YOLO also requires a separate scheduling and extension interface. For tasks built on existing model capabilities, developers can extend through configuration and task-specific prompts without retraining the model or rewriting the entire main program.

OpenSQZ Omni Harness is designed as a runtime control layer between the model and the first-person device, responsible for recognizing and executing these operations. MiniCPM-o handles understanding, reasoning and generation, while the Harness manages task routing, session lifecycle and output control. In terms of implementation responsibilities, this layer has five parts:

Speech Controller: executes pause and resume, manages pending audio output, and prevents the old session's answers from continuing to play during a task switch.

Session Controller: ends the old session, waits for backend cleanup confirmation, and establishes a session with the new task prompt, isolating context between tasks.

Intent / Skill Router: obtains speech transcripts through local FunASR, matches configurable commands, and extracts task parameters such as the target object for a find-object task.

Skill Loader: loads the specialized System Prompt for the corresponding session, specifying the task goal, information of interest and answering style.

Vision extension interface: supports connecting independent vision components. The current YOLO ONNX sidecar uses shadow observation mode to record detection results; the ESP32 input pipeline also has image quality filtering and PaddleOCR-based text orientation detection.

In a typical task switch, the user's audio enters both the model input pipeline and local ASR at the same time. After the Harness matches a task command, it stops the old session from outputting, completes old session cleanup, and then initializes a new session with the corresponding Skill's prompt. This process keeps the loaded model in place and completes the task switch through a hot session reset. The current YOLO sidecar is used only for observation and logging, and is not yet connected to automatic answering or broadcasting.

Device onboarding can also share this local PC backend. Besides self-built ESP32 glasses, OpenSQZ Glass already supports commercial smart glasses, for example connecting Rokid glasses through developer mode. Developers can choose a starting point based on the devices they have and use the same task and interaction control.

Figure 2 | Overall architecture of OpenSQZ Omni Harness

More controllable, more fluid: letting the model pause, resume and start over

MiniCPM-o's full-duplex capability lets the model continuously receive vision and audio and respond, but "the model can perceive a user interruption" and "the Runtime can explicitly control speech that has already been generated" are not the same thing. For a glasses system, broadcast control needs to become an independent, observable runtime capability.

In the current implementation based on MiniCPM-o 4.5, OpenSQZ Omni Harness can already control the model's own TTS. When the user or the system issues a command, the Harness can pause the answer being played, freeing the broadcast window for a new task, a system notice or a specialized model result, and later resume from the original audio position. Users do not have to wait for a long answer to finish before hearing earlier content again after a task is done.

Beyond controlling speech, the Harness also has to actually clean up the context left by the previous task. The current MiniCPM-o 4.5 pipeline distinguishes three recovery modes: (1) "Soft Session Reset": only lifts output blocking, keeping the original Session and KV; (2) "Cold Session Reset": fully rebuilds the model pipeline; (3) "Hot Session Reset": changes the Session ID, clears the KV, and performs lightweight cleanup through duplex_cleanup without reloading the entire model. OpenSQZ Glass Harness chooses the hot reset approach, which establishes a clear boundary for each task. When the user moves from ordinary chat to finding an object, the system creates a new find-object Session and loads a specialized System Prompt; when the user wants to return to chatting with omni, the Harness can establish a new general Session through the relevant command. Task start, end and recovery thus become checkable Runtime events rather than a soft prompt constraint.

Here is the fluid first-person demo of OpenSQZ Omni Harness:

This article includes video content. Please view it in the original WeChat post.

More customizable: anyone can write their own Skill for the glasses

A pair of AI glasses built for real users should not offer only the few functions developers hard-coded in advance, but more customization options: some people want to find objects, some want to read medicine boxes or shipping labels, and others may need exhibit guides, equipment inspections or product recognition. Different scenarios need different triggers, answering styles and safety boundaries, but this does not necessarily require retraining the model.

OpenSQZ Omni Harness organizes specialized tasks as configurable Skills: reading text, finding objects, describing scenes, indoor wayfinding and ordinary conversation each use a corresponding task prompt that specifies the model's goal, focus and answering style; both trigger commands and prompts are open to developers.

Skill configuration is written uniformly in YAML, with specialized System Prompts maintained in separate files. The configuration links trigger rules, task parameters and session behavior; a find-object task, for example, needs both command recognition and extraction of the object name the user specifies. The relevant files are organized as follows:

extensions/assistive_harness/├── config/│ └── skills.example.yaml # skill declarations and runtime config example└── prompts/ ├── find_object_zh.txt # find-object task System Prompt ├── read_text_zh.txt # text-reading System Prompt └── ...```

For example, when the user says "help me find the water cup," "start reading text" or "back to ordinary chat," local ASR first transcribes the speech, and the Harness then matches the task in the command table. The Session Controller clears the old context, the Skill Loader writes the new System Prompt into the new Session; if the Skill is bound to a specialized model, the CV Sidecar Controller starts the sidecar as well.

The point of this mechanism is effectiveness, a low barrier and explainability. In our own testing, we found that task-specific system prompts or skills can significantly improve system performance. Developers can also directly see which words trigger a task, what constraints the model is under, when external capabilities are invoked, and how to exit. For key operations such as stopping a broadcast, starting over and returning to ordinary chat, deterministic keywords are also easier to debug and reproduce than relying entirely on the model's free interpretation.

As a result, developers can copy a Skill directory, change the command words and System Prompt, and form a task combination that suits them. Skills that do not need an external model require no retraining of MiniCPM-o; when a new CV model or sensor is needed, an Adapter can be added under a unified interface without rewriting the entire interaction pipeline.

More secure: an extensible vision sidecar for specialized tasks

MiniCPM-o handles scene understanding and natural interaction in a unified way, but a glasses system still needs to address two concrete questions: whether the image sent to the model is clear, and whether the current task needs specialized vision components to supply extra information. OpenSQZ Glass combines image quality filtering with a vision sidecar interface to handle input quality and task-specific visual information separately.

The first step is adapting to how glasses capture images and improving what reaches the model. The camera moves with the head, and turning, walking or looking at objects at different distances can bring shaking, motion blur and focus changes. MiniCPM-o receives input by the second, so the system uses the multiple candidate frames within that second, has the PC side judge quality, and works with the ESP32 camera's autofocus control to prioritize clearer frames for the model.

Image filtering also introduces a lightweight PaddleOCR-based text orientation classifier to determine the direction of text in the frame. Quality judgments and orientation information make the filtering process traceable and give subsequent specialized components a basis for processing frames. The demo below shows candidate frame selection, quality judgment and rejection.

Figure 3 | Image funnel demo: quality judgment of candidate frames

After input processing, the vision sidecar provides another checkable information channel for specialized tasks. Information such as object locations and text regions can be output as structured results by dedicated components. The Harness reserves CV/OCR interfaces so developers can connect capabilities such as object detection and text recognition per task, recording the result source, confidence and location.

Image filtering and the vision sidecar can work together along these interfaces: the former improves the input image, the latter supplies the specialized results a task needs; when later components need image quality, text orientation or detection confidence, the runtime can design processing conditions and failure feedback accordingly, and log the corresponding frames and results for verification and review.

One continuous interaction, from chat to a specialized task and back

For example, the user is asking the glasses to describe the environment and MiniCPM-o has started answering. The user says "pause"; the Harness pauses the model's response but keeps the current session and context, and the model keeps listening. The user can continue expressing needs and resume the conversation in the original session with a resume command.

When the user instead says "help me find the water cup," local FunASR transcribes the speech, the Harness recognizes the find-object command and extracts the target, performs a Hot Session Reset, loads the find_object Skill's System Prompt and establishes a new find-object session. The model looks for the specified object based on the current view and the task prompt; if the Skill has a vision sidecar configured, the system can also run the corresponding observation and record the specialized component's results.

The user then asks to read the text on the cup. The Harness switches sessions again, enters the read_text Skill, and MiniCPM-o performs the task under the text-reading prompt. When that is done, the user says "back to ordinary chat," and the system clears the specialized task context and re-establishes a general session.

Throughout this process, the model does not need to be fully reloaded, and the user does not need to open menus and select functions layer by layer. Ordinary chat, finding objects and reading text all run on the same device and local backend, with each task using its own session and specialized prompt. For the user, it is one continuous voice interaction; for the developer, every switch has clear trigger rules, task parameters, session state and logs.

This also maps to the three goals of this upgrade: (1) managing task switches through session and interaction control, (2) customizing tasks through open Skills, and (3) handling input quality and specialized visual information through the image funnel and vision extension interfaces. Resuming after a pause continues the original context, while switching tasks establishes a new context; the two kinds of operation run separately under the same control mechanism.

Figure 4 | Task modes and duplex interaction control. Pause and resume keep the same session; when switching tasks, the old session context is replaced by the new task context

About the joint team and the next steps for open source

The OpenSQZ open-source platform at Shanghai Qi Zhi Institute works to build an open, reproducible and sustainably evolving technology ecosystem. OpenSQZ Glass is the platform's open research project for first-person on-device omni-modal interaction, and its hardware, runtime, experimental tools and research materials will continue to be updated in the public repository.

The code, example Skills, 3D-printed hardware and reproduction documentation for OpenSQZ Omni Harness will continue to be maintained together in the OpenSQZ Glass repository. The official release is planned to include Harness usage instructions, editable command words and System Prompt examples, the CV Sidecar interface, verified upstream version information, and logs and demo materials for reviewing the task switching process.

From "separated sensing and compute" to the reproducible 3D-printed hardware of OmniGlass-Edge and then to OpenSQZ Omni Harness, OpenSQZ Glass aims to offer more than a pair of AI glasses with preset functions: it is an open on-device platform that can continuously take in new models, new tasks and new sensors. MiniCPM-o 4.5 provides a new-generation on-device omni-modal intelligence foundation, and the Harness gives that intelligence clearer control boundaries and ways to extend in the first-person world.

As AI's interaction entry point moves from screens into the real environment, what determines the experience is not only how smart the model's answers are, but whether the system can stop in time, switch to the right task, use the right tools and give controlled feedback when uncertain. OpenSQZ Glass will keep advancing along this path: making on-device omni-modal models not only see, hear and speak, but also be controlled, customized and extended together by the open-source community.

➤ OpenSQZ Glass open-source repository

🔗 https://github.com/OpenSQZ/OpenGlass