Spark Talk | VoxWeft Pitfall Log: A Development Diary of an On-Device Simultaneous Interpretation Project

2026-10-16

In the on-device AI ecosystem, what really brings technology to life is not grand narratives but individual developers — students, independent developers, or people in the industry who "understand both the technology and the pain points." Using ModelBest's on-device models, they have made surprising attempts in fields they know well.

This episode's Spark is an independent developer. Standing on the shoulders of VoxCPM2, using this 2B-parameter multilingual TTS model as the speech generation core, he built a locally running real-time simultaneous interpretation system on Apple Silicon — VoxWeft.

In other words, a single Apple computer can complete the whole process from "hearing" to "speaking" locally. All speech recognition, translation, and speech synthesis run on your own device.

VoxWeft chains microphone input, speech segmentation, ASR, translation, streaming TTS, and playback into one complete pipeline. VoxCPM2 supports 30 languages, 48kHz output, voice cloning, and voice design, and runs via MLX with 4-bit quantization. On an M3 Ultra, the first-sound latency of streaming TTS after warm-up can be as low as 124–168 milliseconds; on the M2 development machine, after system resource optimization, the end-to-end latency from speech segment submission to translated speech output is about 2.5 seconds.

This article includes video content. Please view it in the original WeChat post.

Now VoxWeft's code, development log, and related configuration are all open-sourced.

But today we won't just talk about how impressive VoxWeft is — let's open the DEVLOG he wrote.

01.

First, a story no one noticed

In September 2026, while VoxWeft was running at an exhibition, it harbored a problem that was never discovered.

It was only during the code review after the project was open-sourced that the developer found: the ASR bridge layer had been decoding WAV file headers as audio data.

Each audio segment came in RIFF WAV format, but the bridge parsed the entire file directly as int16, turning the 44-byte file header into 22 audio samples with amplitudes up to ±0.98. That is, the start of every ASR input was mixed with about 1.4 milliseconds of high-amplitude noise. This problem affected ASR input in all languages during the exhibition.

Strangely, speech recognition still worked. The developer explains in the log that the model has some robustness to this kind of brief noise, which is exactly why the problem never drew attention. It was only truly found during the code review after open-sourcing.

02.

A "counterintuitive" development log

What really makes this development log valuable is how it is written.

The developer set himself a rule: every entry is written as "symptom → root cause → fix → cost." More unusually, he treats "falsified hypotheses" and "unexplained phenomena" as formal content. The log has a dedicated section called "Open - measured but not yet explained."

In his words: recording the paths that don't work is as important as recording the fixes, because it keeps those who come later from walking into the same dead ends.

03.

Adding a lock brought the whole system to a halt

Initially, to avoid conflicts between simultaneous TTS (text-to-speech) and ASR (speech recognition) inference, the developer came up with an intuitive fix: add a lock and let them queue up.

As a result, the system froze instead. The bridge process was alive, CPU at about 50%, /health returned normally, /ready kept timing out, and the log produced zero output. Neither TTS nor ASR responded.

The problem was that the streaming TTS handler is an async generator. It acquires the lock, yields an audio chunk, and then stays suspended while still holding the lock as the client reads. When an ASR request arrives, it waits forever on the same lock. The event loop has nothing to run.

"This bug was misdiagnosed for a day," the developer says. "Because the deadlock happened to be triggered by a request that arrived during a streaming session, we thought 'some piece of Chinese audio is toxic.'"

The fix: just delete the lock. The single-threaded executor already provides mutual exclusion, and await executor.submit(...) releases control between chunks.

Cost: the problem was misdiagnosed for a full day. Because the deadlock happened while processing a certain piece of Chinese audio, the developer once thought the audio itself was triggering the exception.

04.

Two threads constructing the same MLX model at once

Symptom: the process disappeared, exit code -9. launchd restarted it, and it disappeared again when the next warm-up request arrived.

The cause is that VoxWeft warms up the model in the background at startup, but uvicorn has already begun accepting requests by then. If a real request arrives during warm-up, both paths enter the lazy-loading code at the same time and concurrently construct two MLX model instances. MLX is not thread-safe here and cannot withstand two threads constructing the model at once — the process vanishes outright with exit code -9.

The original guard was asyncio.Lock. But asyncio.Lock is per-event-loop and has no effect across threads.

The fix is straightforward: switch to threading.Lock, with warm-up and requests going through the same ensure*_locked() logic, ensuring only one thread can create the model at any time.

05.

VoxCPM2's temperature

is not a sampling temperature

People who work on TTS have an instinct: lower the temperature to reduce variance.

On VoxCPM2, at t = 0.01, generation never emits a stop signal and runs all the way to the 80-second limit, with 80% of frames silent.

Root cause: VoxCPM2's temperature is not a softmax sampling temperature; it scales the initial noise of the flow-matching ODE (z = randn(...) * temperature). As it approaches zero, the CFM has no randomness to work with and does not converge.

Fix: keep the official default of 1.0.

The log has a line worth remembering for anyone tuning model parameters: "This knob looks like it treats jitter, but it only makes things worse."

06.

7 samples almost led to a wrong conclusion

NLLB-1.3B produced repetitions on short segments. Someone suggested: the model is too small, switch to 3.3B.

First test: 7 manually written samples, 1 repetition. Conclusion: "It's not about model size, it's about decoding parameters."

Looked reasonable. But the sample was too small.

Second test: 28 real ASR production audio segments. Both 1.3B and 3.3B had 0/28 repetitions.

The repeated output people saw was actually another bug — a configuration ordering error that fed text in the wrong language into the translator.

"A test with 7 samples only tells you that a hypothesis is not obviously wrong. That's all."

07.

Deleting one "redundant" line of code crashed the demo

stopFlag = false. Looked like leftover code. Deleted.

Result: the player task exited immediately, and the generation counter advanced by 2-3 instead of 1 per keypress. The demo blew up.

That false was a reset. Delete it, and the flag retains the previous true; the task sees "already stopped" and exits directly.

"If a flag reset looks redundant, find who reads it first, then delete."

08.

Leave the pitfalls you've stepped in behind you

These entries are all things the developer actually encountered on the M2 / M3 Ultra, and that is the scarcest quality of this log.

When you read "one temperature parameter almost destroyed the entire TTS generation," you recall being burned by some intuition that "should have worked"; when you read "7 samples led to a wrong conclusion," you recall review meetings that were still confident with insufficient data; when you read "deleting one seemingly redundant line of code crashed the demo," you recall your own git revert history.

A cloud model is only a chat window away from the user, but an on-device model has to fight through checkpoint after checkpoint in the real world.

During development, the author also found that warmup_patches=2, inherited from the reference implementation, actually introduced 470 milliseconds of extra latency and caused the very pop it was meant to eliminate.

"We inherited a default value and wrote a set of documentation for an unmeasured mechanism. Measurement took only 20 minutes." What he means is: many "best practices" have never been verified, and many "well-known" pitfalls turn out to be backwards only after you step in them.

And only those who have actually done the work know this. The developer chose to record them checkpoint by checkpoint and then open-source them, so that those who come later don't have to walk the same dead ends again.

This is probably the meaning of Spark — not how fast you run yourself, but turning the pitfalls you've stepped in into signposts for those who come after.

In addition, the open-source project also includes an AGENT-SETUP.md. After users have an agent clone the project, they can follow it to start the project step by step: a correct use case can be written by any LLM; but failure conditions can only come from actual testing.