OpenSQZ Glass: bringing on-device full-duplex omni-modal models into the first-person wearable world
As AI interaction moves beyond screens and chat boxes into cameras, microphones, and the physical world, first-person wearable devices are becoming a key foothold for multimodal intelligence. OpenSQZ Glass is a locally running, real-time multimodal AI glasses reference implementation built by the open-source platform team at the Shanghai Qi Zhi Institute. Aimed at researchers and developers, it is designed for first-person perception and interaction. The system adopts a "perception-compute separation" architecture: the lightweight glasses handle image and audio capture, while a local device runs omni-modal model inference — exploring a viable path for real-time first-person visual assistance while balancing privacy, latency, cost, and reproducibility. The project provides complete 3D-printed glasses files and assembly tutorials, open-sourcing both hardware and software to lower the barrier to reproduction and use.
Why build OpenSQZ Glass?
The AI glasses race has heated up quickly in recent years. In China, ByteDance, Alibaba, Xiaomi, RayNeo, and Leqi have all launched smart glasses one after another; abroad, Google's Gemini-powered AI glasses are competing head-on with Meta Ray-Ban. Market data confirms the surge: global shipments in Q1 2026 grew 130% year over year, full-year 2026 sales are expected to reach 20 million units, and the market is projected to expand to USD 5.6 billion over the same period (data from a report by market research firm Smart Analytics Global (SAG) 🔗 https://my.feishu.cn/docx/ZIVkdxJVFoIAICxS1nic5verneg). The industry widely believes that 2028 may bring the "iPhone moment."
Yet beneath the hype, a fundamental problem remains unsolved: the dominant interaction paradigm of today's AI glasses is structurally misaligned with how the real world works. Most products still operate as a discrete loop — "the user takes a photo and asks a question, then waits for the model's answer." But the real world is fluid: viewpoints shift, scenes change, and events worth noticing can happen at any moment. A static photo cannot represent a continuously evolving scene, and every wait interrupts the user's attention and rhythm of action. This misalignment stems from a set of interlocking structural constraints:
Cloud inference brings latency and privacy pressure. In first-person assistance, response latency directly determines the usability threshold. A wait of several seconds is enough, in many scenarios, for the context to change and the information to go stale. At the same time, first-person images naturally contain sensitive information — passersby's faces, work environments, home spaces, computer screens. Uploading such data to the cloud by default is not an ideal choice.
Interaction is fragmented and static. Although most systems incorporate vision and audio, they still operate in an isolated "photo → question → wait for answer" flow. That hardly meets the continuous needs of real assistance scenarios. A first-person scene is not a still image but a continuously changing stream of environment, and both user and system need to stay in interaction as it changes.
Wearables are constrained by compute, power, and weight. Concentrating large-model inference, heat dissipation, and battery in the glasses themselves sacrifices wearing comfort and battery life. Products trade off among "compute, power, and lightness," leaving smart glasses perpetually "barely usable" rather than "genuinely good."
The research community lacks a complete, reproducible platform. Commercial AI glasses are expensive — often over a thousand yuan — and their hardware and software stacks are closed. Model choice, inference pipeline, prompt orchestration, and evaluation methods are not fully open. Researchers find it hard to run experiments and iterate on top of them.
OpenSQZ Glass's approach is to build an open-source AI glasses reference implementation around local omni-modal inference, full-duplex interaction, low-cost hardware, and reproducible 3D printing, so that researchers and developers can build first-person real-time visual assistance systems with readily available low-cost hardware, an auditable software pipeline, and a local-first deployment model.
Architecture: separating perception from compute
The core design principle of OpenSQZ Glass is simple: the glasses only need to do perception well; they do not need to run a large model. To that end, we adopted a "perception-compute separation" system architecture. The glasses stay lightweight and handle only streaming capture of first-person data; the actual multimodal inference runs on a local computer or edge device. Data travels over local Wi-Fi, is processed by a locally running omni-modal model that unifies vision, audio, and text, and results are fed back to the user through streaming speech.
The perception-compute separation architecture gives OpenSQZ Glass the following advantages:
It significantly reduces the burden on the glasses. With no heavy inference on the front end, there is no need to pay a weight penalty for high-compute chips, complex cooling, and large batteries, which makes it easier to control power consumption, heat, and wearing burden.
It keeps system latency and stability within local boundaries. Compared with cloud solutions that depend on public network links, local inference delivers lower and more stable response latency, better suited to real-time first-person assistance.
It protects privacy. First-person data never needs to be uploaded to a remote server; it is processed directly within the local network and local device, giving users stronger control over where data flows and how it is used.
The decoupled architecture upgrades well. Glasses hardware stays relatively stable, while host-side inference capability, model versions, and toolchains can keep evolving. This separation is especially suited to a research platform: it lets the system iterate as models and algorithms advance without frequently redoing hardware.
With this division of labor, the glasses take charge of "seeing and hearing" and the local device takes charge of "understanding and answering," forming an interaction loop of looking, asking, and answering; the model can handle interruptions and being cut off the way a person would, enabling smoother two-way conversation.
Technical highlights: from image Q&A to full-duplex omni-modal interaction
The perception-compute separation resolves the contradiction at the hardware and system level — the glasses capture continuously, the host performs multimodal inference. But if interaction remains a discrete, turn-based "user takes a photo → system answers" loop, AI glasses will still struggle to fit into the real world. The real world does not wait for the system to finish answering before changing: a user may ask a question while walking, a new target or risk may appear in the environment, and the user may cut in, correct themselves, or switch tasks at any point during the answer. The traditional half-duplex interaction mode cannot keep up with this rhythm.
The core of the problem is that AI glasses need more than stronger single-turn Q&A; they need continuous presence — sustained perception and responsiveness. ModelBest's MiniCPM-o 4.5 offers a key answer: with only 9B parameters, it is the industry's first end-to-end full-duplex omni-modal model, reaching omni-modal SOTA in visual understanding, document parsing, speech understanding and generation, and breaking the traditional "input → output" paradigm.
Built on MiniCPM-o 4.5, OpenSQZ Glass pushes interaction from discrete turns to parallel streams along a continuous timeline, bringing visual, audio, and text streams onto a unified timeline for streaming processing. It achieves genuine "seeing, listening, and speaking proactively," maintaining continuous awareness of the environment at all times — missing no key detail and without being distracted by redundant information. This gives OpenSQZ Glass three capabilities that are indispensable in real environments:
Continuous perception: the system does not "take a look" only when the user asks; it continuously receives first-person input and understands scene changes and context continuity.
Continuous response: the system does not detach from the current environment while responding; it dynamically adjusts its answer based on new visual and audio information.
Interruptibility: when the user changes intent or more urgent information appears, the system can promptly stop its current output and switch to the new task — especially critical for assistance systems in open environments.
The essence of full-duplex technology is turning interaction from "discrete turns" into a continuous "perceive-decide-act" stream. The model keeps receiving new visual and audio input while producing speech; when the user cuts in, the system folds the new input into its state judgment and decides anew whether to keep listening or switch tasks. By integrating MiniCPM-o 4.5, OpenSQZ Glass closes the loop of "listening, seeing, speaking, and deciding at the same time," evolving AI glasses from a Q&A tool into a collaborator that is always present.
System implementation: low-cost hardware and a local multimodal pipeline
At the hardware level, OpenSQZ Glass takes a pragmatic route: building the system from off-the-shelf, low-cost components. The front end uses an ESP32-S3 + OV5640 as the sensing unit — what we value is not peak single-point performance but overall reproducibility: controllable cost, a mature open-source ecosystem, stable Wi-Fi transmission, and abundant community resources, so developers can quickly build prototypes and modify them. Together with a small lithium battery and 3D-printed structural parts, the entire front-end assembly costs about 150 yuan. The premise is that the front end only handles capture and transmission and takes part in no model inference, so the hardware can be light enough without trading off compute, power, and weight.
On the compute side, a commercial edge device (laptop, tablet, or phone) serves as the inference host for local omni-modal inference. This approach offers a mature debugging environment, flexible model swapping, and a low development barrier. It also makes OpenSQZ Glass's software architecture inherently hardware-compatible: the front-end perception layer and the back-end inference layer communicate through a standardized data interface, so any front-end device that can output stable first-person images and audio streams can plug seamlessly into the OpenSQZ Glass inference pipeline.
We have already completed full-pipeline adaptation on Rokid glasses and validated it on real hardware. The OpenSQZ Glass software stack is designed for cross-device compatibility: the ESP32-S3 firmware layer wraps a unified Wi-Fi transport protocol, and the data reception and scheduling logic on the inference host is decoupled from specific hardware. Adapting a new pair of glasses requires only interfacing with the camera and microphone capture APIs; the core inference pipeline needs no changes. That means the system's dependence on front-end hardware is reduced to a minimum.
Going forward, we will progressively adapt more commercial smart glasses, and we welcome hardware vendors to get in touch to advance a standardized integration process together.
For researchers, the value of this design is clear: it is not tied to any specific hardware, so the front-end device can be chosen freely; even when hardware is upgraded or replaced, the software stack and experiment scripts migrate smoothly. More importantly, the fully open-source code and auditable logs not only help the system "run" but let researchers quickly locate root causes when problems arise — whether the issue is Wi-Fi jitter, resolution policy, state-switching logic, or audio playback stutter, it can be traced to a specific link instead of guessed at through a black box.
Test results and key findings: 0.99 s median latency, 97.5% of requests completed within 2 s
In a real Wi-Fi environment, we ran end-to-end latency tests on OpenSQZ Glass based on ModelBest's MiniCPM-V 4.5 — from the moment the user stops speaking to audio output, covering capture, preprocessing, VLM inference, and TTS synthesis across the full pipeline. MiniCPM-V 4.5 was chosen as the test baseline because its standard turn-based "input → output" interaction allows precise quantification of the system's physical latency, providing repeatable, comparable numbers for engineering optimization. The measurements show:
Resolution tuning delivers a 2.3× speedup. After resizing 1280×720 images to 896×504, VLM inference time to first byte dropped from 1222 ms to 528 ms, a 2.3× speedup, with image-quality loss all but imperceptible in real assistance scenarios. Median end-to-end latency thus crossed the one-second threshold — 993 ms.
Cloud solutions offer no advantage in real network conditions. Qwen-VL-Max had a median latency above 2 s, with only 46.7% of conversational responses under 2 s; Gemini 2.5 Flash had a median latency of 4.9 s and a P95 above 11 s, with the under-2-second pass rate dropping to zero. OpenSQZ Glass, by contrast, had a P95 of just 1.78 s, staying stable even when the network fluctuates.
The gap between <1 s and >2 s is the difference between "real-time presence" and "remote assistant." Two seconds is the psychological limit of human tolerance for interactive response. In visual assistance scenarios, when you are walking, reading a street sign, or looking for a product, every wait interrupts attention. A 993 ms response makes the AI feel "right there with you," while a cloud solution's wait of more than 2 s turns interaction into an asynchronous task.
Wi-Fi end-to-end latency comparison. All systems used an ESP32 Wi-Fi camera for capture, paired with the local TTS engine pyttsx3. Latency is in milliseconds. Results come from the ACL 2026 System Demonstrations paper (🔗 https://my.feishu.cn/docx/ZIVkdxJVFoIAICxS1nic5verneg).
The measurements show that OpenSQZ Glass, with 150-yuan glasses hardware plus a local consumer GPU, delivers real-time visual assistance that is faster, more stable, and more private than cloud large models — 0.99 s median latency, 1.78 s P95, and 97.5% of requests completed within 2 s.
Application scenarios
In human conversation, pauses, thinking, background noise, and overlapping speech are normal. For AI to truly fit into this environment, it must break out of the mechanical "question-and-answer" mode. OpenSQZ Glass aims to evolve AI from a "tool waiting to be woken" into "a collaborator that is always present." It shows distinctive interaction capabilities in three typical scenarios:
Proactive reminders: through continuous perception and event memory, the system registers user instructions as trigger conditions, shifting from "passive response" to "proactive alert." For example, when the user says "remind me when you hear clapping," the system detects that event in the continuously running audio stream and triggers a reminder as soon as it matches.
Seeing and telling: as the user walks and looks around wearing the glasses, the system recognizes and streams descriptions of objects, layout, and changes in view in real time, turning the dynamic environment into a coherent spoken narrative so the user can "walk, look, and listen" their way through the world.
Interrupting at any time: the user can follow up or change topics at any point during an answer. For example, while the model is introducing Shanghai, the user suddenly asks, "Is Pudong New Area in Shanghai?" The system instantly stops its current reply, smoothly stops speaking, switches to listening, and handles the new question. The whole process takes seconds, making interruption a natural way to clarify information.
This article includes video content. Please view it in the original WeChat post.
The video shows the full-duplex omni-modal interaction experience OpenSQZ Glass delivers with MiniCPM-o 4.5.
Conclusion
OpenSQZ Glass has closed the real-time loop between vision and speech, completing the full chain from first-person input to local multimodal understanding to streaming speech feedback. Two papers built around this open-source system have been accepted to ACL 2026 System Demonstrations and ACM UbiComp/ISWC 2026, respectively.
The ACL 2026 work validated the feasibility of the perception-compute separation architecture in terms of low latency, stability, and reproducibility; the UbiComp/ISWC 2026 demo paper focuses further on the wearable hardware form, presenting a complete loop with a 3D-printed ESP32-S3/OV5640 glasses front end, local omni-modal model inference, and real-time speech interaction, covering first-person tasks such as finding objects, reading text, describing scenes, and checking view quality. The project will progressively open-source CAD/STL files, bills of materials, firmware configurations, assembly documentation, runtime scripts, and demo logs, providing a reproducible reference implementation for wearable Edge-Omni system research and lowering the barrier to reproduction and secondary development.
It should be noted that OpenSQZ Glass is a research reference platform, not a safety-certified navigation device. Before dedicated wearable AI chips mature, we have validated a viable technical path through the separated architecture and full-duplex interaction.
If AI's interaction entry point is destined to move from the screen into the real world, OpenSQZ Glass hopes to be an open starting point on that path.
➤ GitHub
🔗 https://github.com/OpenSQZ/OpenGlass
➤ ACL 2026 System Demonstrations paper
🔗 https://aclanthology.org/2026.acl-demo.82/
➤ UbiComp/ISWC 2026 demo paper
《OmniGlass-Edge: A 3D-Printed, Real-Time Edge-Omni Smart Glasses Demo for First-Person Visual Assistance》
🔗 The accepted version will be updated once it is available on the ACM Digital Library
About the OpenSQZ open-source platform: OpenSQZ is an open-source ecosystem hub built by the Shanghai Qi Zhi Institute, dedicated to fostering an active community for hosted collaboration, shared innovation, and continuous evolution. Here, research ideas can be incubated quickly and industry needs can be addressed in one place. All projects on the platform use Apache/MIT open-source licenses; through a well-developed community operation mechanism and professional expertise, it helps hosted projects build open-source communities with industry influence. Its governance charter, technical infrastructure, and operations system provide every open-source team with full-cycle support from "0 → 1" to "1 → N."
Projects on the OpenSQZ open-source platform include:
➤ AutoMathText-V2 dataset and related models
1M+ downloads! AutoMathText-V2: a 2.46-trillion-token math "textbook" built to train the next SOTA model
➤ AI safety Jailbreak Foundry
28 minutes to turn a jailbreak paper into runnable code — this open-source tool is changing AI safety research
➤ OpenSQZ Glass
Bringing on-device full-duplex omni-modal models into the first-person wearable world
🔗 https://github.com/OpenSQZ/OpenGlass
➤ MiniCPM-V CookBook
One click to cover individuals, enterprises, and researchers, unlocking on-device inference across all scenarios
🔗 https://github.com/OpenSQZ/MiniCPM-V-CookBook
➤ MegatronApp
Breaking the black box of large-model training: MegatronApp open-sources trillion-parameter "visible and controllable" training