1. Opening: a one-sentence request, delivered in 68 minutes

At the moment I made the request, I didn't know what LoRA was, I didn't know how many modalities MiniCPM-o 4.5 had, and I didn't know ms-swift existed.

My exact words were:

I've got a Spark on hand, and I want to fine-tune https://huggingface.co/openbmb/MiniCPM-o-4_5 so that it becomes Alice and we can video-call her. Let's discuss the technical plan for this fine-tuning.

68 minutes later, I had: a technical plan with a narrowed scope, 1,000 quality-checked training samples, a training script ready to execute, and a source-level format verification report. Throughout the whole process, I only answered 6 rounds of questions and never read a single line of MiniCPM documentation.

This worked not because the model is especially smart, but because OpenBMB put the key facts where an agent can reach them. This article records the complete process — which retrieval paths the agent took, which key decisions it made, which pitfalls it hit, and how it ultimately went from a one-sentence request to a real video call.

Mermaid

flowchart LR
    A["一句话需求 / 零知识起步"] --> B["检索四跳 / 30次搜索 + 源码核验"]
    B --> C["方案锁定 / 68分钟交付"]
    C --> D["数据生成 / 1000条 + 质检"]
    D --> E["Smoke测试 / 2次失败 + 1次成功"]
    E --> F["正式训练 / V1到V5迭代"]
    F --> G["行为回归 / 指标与行为分开验收"]
    G --> H["视频电话 / 端到端交付"]

Figure: The end-to-end pipeline of agent-driven fine-tuning — from a one-sentence request to end-to-end delivery. There is no single "magic prompt" in between; instead, there is a gated evidence loop.

---

2. The retrieval chain: which hops the agent actually took

This is the most informative part of the article. Below is the actual sequence of actions reconstructed from the session logs:

Plain Text

02:48:36  我提出需求
02:48:49  第一次检索发出（13 秒后）
02:56:52  给出第一版经检索的结论 + 第二轮提问
03:15:31  方案锁定，进入执行
03:18:02  git clone ms-swift 源码做事实核验
03:23:04  开始批量生成 1000 条训练数据
03:56:43  交付

Over the whole process, the agent issued 30 retrieval actions against public materials, plus 1 source-code clone and about 10 targeted source-code reads. They fall into four hops:

Hop 1: The Hugging Face model card

The first thing the agent did was open the model card and search for the keywords "Training" and "Voice Cloning".

The model card directly answered three decisive questions:

1. The model natively supports full-duplex voice interaction, interruption, and voice cloning from reference audio — which means "sounding like Alice" might not need training at all.
2. The model does not generate a video avatar — which means the visuals in the "video call" have to be solved separately and are outside the scope of fine-tuning.
3. The model card states support for LLaMA-Factory and SWIFT — which pointed the direction for the next hop.

Point 1 alone cut what I had assumed was the "training goal" in half. The credit goes to the Voice Cloning example in the model card: it isn't just an adjective-style line saying "supports voice cloning", but a concrete description from which the agent could parse the factual structure "input = reference audio, output = cloned timbre".

---

Hop 2: The official repository and technical report

The agent then searched OpenBMB's GitHub repositories, found MiniCPM-o-Demo, and opened the HTML version of the technical report on arXiv.

Three outcomes:

• MiniCPM-o-Demo became the product architecture directly. It already had a mobile frontend, HTTPS/WSS, full duplex, proactive observation, custom system prompts and a reference-audio interface. The agent's judgment was "don't build a separate real-time communication protocol" — the work saved by this decision alone was larger than the entire fine-tuning effort.
• The technical report provided a prior for the training paradigm. The report's two-stage structure of "large-scale instruction data adaptation + small-scale high-quality data refinement" was mapped directly onto our "synthetic data for coverage + a small amount of manual spot-checking to tighten the persona".
• The report also provided a disqualifying fact: the speech decoder and the vision system are tightly coupled with the LLM's hidden states. This directly determined that the LoRA later on had to be conservative and had to be regression-tested.

This hop also exposed the first gap: as of 2026-07-24, OpenBMB's official fine-tuning tutorial did not yet fully cover 4.5. The official fine-tuning entry points the agent found were organized around earlier versions, and it could not get 4.5's training parameters from them. So it did the right thing — instead of guessing, it moved on to the third hop.

---

Hop 3: Public documentation of ecosystem frameworks

The agent searched the documentation and repositories of several frameworks, including ms-swift, LLaMA-Factory and align-anything, for MiniCPM-o-4_5 support.

Key finding: the model card says "supports SWIFT", but doesn't say which version or after which commit. It was only by digging up PR #9147 that the agent confirmed "4.5 support had just landed on the main branch, and that PR explicitly disables the TTS template".

This one piece of information rewrote the plan outright: the two voice-over clips would not go into the LoRA. The public evidence only supported "audio input", not "training full-duplex speech output". In the same round of retrieval, no support entry for 4.5 was found on the LLaMA-Factory side at the time.

If the model card had said "supported in SWIFT >= x.y.z or after commit abc; training the TTS head is not yet supported", this hop could have been skipped, with no risk of misjudgment.

---

Hop 4: Clone the source — treat docs as leads and code as facts

This was the most critical hop. The agent didn't stop at "readthedocs says it's supported"; it cloned the ms-swift source directly and made targeted reads of core files such as model registration, template definitions and argument parsing.

The most valuable snippet obtained from that single clone:

Python

ModelGroup(
    [ Model('OpenBMB/MiniCPM-o-4_5', 'openbmb/MiniCPM-o-4_5') ],
    template=TemplateType.minicpmo4_5,
    requires=['timm', 'transformers==4.51.3', 'decord', 'soundfile', 'minicpmo-utils==1.0.6'],
),

The registration entry itself is the dependency list. Version pins, the template name and modality tags are all in one place, as machine-readable structured literals rather than natural language scattered across README paragraphs. With this snippet in hand, the agent completed its environment planning — the most textbook example of "documentation that can be invoked", and it isn't even in the docs directory.

---

3. Four key moments: how retrieval directly rewrote the plan

This section answers "what's the actual point of reading the docs". Each item below is an actual change to the plan that happened after a specific retrieval.

Moment 1: The training goal cut in half

Before: I thought we needed to train Alice's voice into the model.

Trigger: The Voice Cloning section of the HF model card + ms-swift PR #9147 disabling the TTS template.

After: The two 30-second voice-over clips are used only as reference audio at inference time, with an A/B test to pick one. Training TTS on 60 seconds of material would have been extremely prone to overfitting anyway.

Significance: Material preparation for phase one dropped from "needs professional studio-grade corpus" to "existing material can be used directly".

---

Moment 2: All visual training data dropped

Before: At one point the agent suggested adding 300–500 image-text/video samples.

Trigger: I asked back, "We're only training the persona — can't we just keep the original model's reaction abilities?" After going back to check the technical report, the agent admitted it had conflated "training visual capabilities" with "accepting visual capabilities".

After: Phase-one training material is 100% plain text. Looking at objects, looking at the screen, judging when to speak, leaving/coming back — all of these keep the original model's capabilities and serve only as acceptance criteria, not training items.

Significance: Data preparation cost dropped from "recording material + uploading footage" to zero, and privacy risk dropped to zero as well.

This one deserves a separate remark: the agent will make scope-expanding mistakes, but the public technical report allows such a mistake to be falsified within minutes. If the technical report weren't public, this round of expansion would very likely have turned into two weeks of wasted work.

---

Moment 3: LoRA target modules changed from target_modules to target_regex

Before: --target_modules q_proj v_proj

Trigger: Reading the actual matching logic in tuner_args.py and get_target_modules().

After: --target_regex '^llm\\..*\\.(q_proj|v_proj)$'

Reason: MiniCPM-o is an omnimodal model, and its vision tower and audio tower also contain modules named q_proj / v_proj. Matching by name would silently attach LoRA to the vision/audio encoders too — training still runs and the loss still drops, but what you're training is no longer what you think it is.

This was the only pitfall in this hands-on test that "the docs gave no hint of at all and could only be found by reading the source". The failure is completely silent; you might not discover the degraded visual capabilities until the acceptance stage, and by then you wouldn't suspect this one line of parameters.

---

Moment 4: The empty think prefix of the hybrid thinking template

MiniCPM-o 4.5 uses the Qwen3 hybrid thinking template. During training, the framework automatically adds an empty non-thinking prefix; if you don't handle its loss, SFT on data without chains of thought will erode the model's reasoning ability.

Implementation: Don't embed think tags in the data, keeping Alice's replies clean; on the training side, use --loss_scale to ignore the loss on the empty think prefix.

---

Appendix: A training config you can copy directly

Below is the final training command. The core parameters all come from the four key decisions above — target_regex to avoid silently attaching to the vision tower, freeze_vit/freeze_aligner to train only the persona, and loss_scale to ignore the empty think prefix:

Bash

swift sft \
  --model openbmb/MiniCPM-o-4_5 \
  --use_hf true \
  --tuner_type lora \
  --dataset "${ROOT_DIR}/train.jsonl" \
  --val_dataset "${ROOT_DIR}/validation.jsonl" \
  --torch_dtype bfloat16 \
  --num_train_epochs 1 \
  --per_device_train_batch_size 1 \
  --per_device_eval_batch_size 1 \
  --gradient_accumulation_steps 16 \
  --learning_rate 5e-5 \
  --lora_rank 8 \
  --lora_alpha 16 \
  --lora_dropout 0.05 \
  --target_regex '^llm\..*\.(q_proj|v_proj)$' \
  --freeze_vit true \
  --freeze_aligner true \
  --gradient_checkpointing true \
  --max_length 2048 \
  --loss_scale default+ignore_empty_think \
  --eval_steps 25 --save_steps 25 --save_total_limit 3 \
  --logging_steps 5 --warmup_ratio 0.05 \
  --dataloader_num_workers 4 \
  --output_dir "${ROOT_DIR}/output/alice-persona-r8"

The data uses the standard ms-swift JSONL format, with one dialogue sample per entry:

JSON

{
  "messages": [
    {"role": "system", "content": "<人物卡>"},
    {"role": "user", "content": "<用户输入>"},
    {"role": "assistant", "content": "<模型回复>", "loss": true}
  ]
}

The system prompt (character card) is the soul of the training. Our character card is about 400 Chinese characters long, and its core constraints include:

Plain Text

你是 Alice，一个有独立判断的技术伙伴。

- 说普通话，默认 1~3 句话，不啰嗦
- 称呼对方「老板」，但不是客服语气
- 不用 emoji，不说套话
- 有自己的观点，可以反驳
- 不确定就说不确定，不假装执行了工具
- 不自称 AI 或语言模型
- 有边界感，不越权操作

---

4. From smoke test to V5: a failure-driven iteration loop

The 68 minutes of the planning phase solved "how it should be trained in theory". Once connected to the Spark, the questions became a different set: had the weights for the specified version been fully downloaded, could the ARM64 environment install everything, did the framework really support this template, would this command crash on the first batch of data, and did the trained LoRA actually update the intended modules.

Pin the inputs first, then decide whether to burn compute on full training

Codex didn't just throw the generated script onto the Spark. It downgraded every conclusion in the plan back into a hypothesis to be verified, and then re-checked them one by one along the chain of evidence:

• Model files: Pinned the revision, verified the SHA-256 of all 54 files one by one, and confirmed a parameter count of 9.37 billion. This step guards against the hardest class of problems to debug: discovering halfway through training that an old revision has crept into the cache.
• Data audit: Recounted every sample, confirmed a maximum sample length of 599 tokens, and tightened max_length from 2048 to 768; confirmed there were no exact duplicates and no scenario leakage across splits.
• Resource profile: Confirmed the target machine is aarch64 with an NVIDIA GB10 GPU and about 119 GiB of available unified memory. Created an isolated environment so as never to pollute the main application environment.

---

How a 10-step smoke test overturned two priors

Before full training, Codex didn't run all 842 samples first. Instead, it compressed the goal into five things: "the model loads, the template encodes, LoRA injects, backpropagation completes, and the checkpoint actually contains updates". This smoke test failed twice in a row, and both failures were more valuable than a lucky first pass.

First failure: the dependencies crashed before the model even loaded. librosa still depends on pkg_resources, while the newer setuptools in the environment had removed that entry point. The fix was to pin a compatible setuptools version, not to start replacing model code.

Second failure: plain text doesn't mean you can turn off the audio tower. The prior assumption was: the data is 100% plain text, so init_audio=false can be set. Testing showed that MiniCPM-o 4.5's template still constructs dummy audio and passes it through the corresponding forward path. Without initializing the audio module at all, even plain-text training fails.

The third smoke test succeeded. The successful config used Rank 8 on the LLM's q/v projections; the 10-step training took 109 seconds with a peak memory of 18.34 GiB. Codex didn't treat exit code 0 as acceptance complete — it went on to open the checkpoint and inspect 144 tensors, confirming that all 72 LoRA-B tensors were non-zero, with no NaN, OOM or missing CUDA kernels.

The deliverable of the smoke test isn't a low loss, but a set of runtime facts proven to hold.

---

V1 to V5: a data loop driven by failure samples

Full training didn't end as simply as "run one version each at Rank 8 and Rank 16 and pick the one with lower loss". The actual process went through five versions, with each version's behavioral failures turned into targeted data for the next.

Version	Data and config	Fixed comparison set (loss / acc)	Core improvement
V1	Original 842, R8 q/v, 2 epochs	2.701 / 0.416	Baseline version; persona boundaries didn't hold
V2	+108 calibration prompts, each repeated 2 times	2.638 / 0.426	Filled in behavioral dimensions, fixed some fake-action issues
V3	+calibration prompts 4 times each, R16 q/k/v/o, new persona	2.538 / 0.439	Expanded LoRA expressiveness; best language metrics
V4	Continued training from V3, 172 correction items, 1 epoch	Not a final candidate	Good-looking loss but behavioral regression wasn't good enough
V5	Continued training again from V3, 172 items, 3 epochs	2.593 / 0.435	Most stable behavior; ckpt-63 was the final choice

The iteration logic was the same for every version: not looking at how much the loss dropped, but at which category the failure samples belonged to, then adding data, adjusting configs and re-running regression in a targeted way. V1's failures were split into eight categories with 108 calibration prompts; V3's tail problems were expanded into 172 training items across 43 topics; for V5, instead of continuing to look at the original 79 items, 12 new critical prompts were created specifically to test high-risk behaviors.

---

Why we didn't pick V3, which had the best language metrics

If the only rule were lowest loss, the answer would be V3. But loss measures the probability of target tokens; it doesn't know that saying "the email has been sent" without any tools is a serious error, nor that "roll it out gradually first" is no substitute for a real backup.

Choosing V5 checkpoint-63 in the end was a clear product trade-off: accept slightly worse language-modeling metrics than V3 in exchange for more stable real-world action truthfulness, risk judgment, contextual honesty and relationship boundaries.

---

5. From checkpoint to a real video call

After the model passed text and voice unit tests, Codex didn't stop at the command line. The delivery goal was restored to the original sentence: let the user video-call Alice from a mobile browser.

The final pipeline follows the division of responsibilities in the official Demo:

Plain Text

手机浏览器
  -> HTTPS / WSS Gateway
  -> Worker
  -> PyTorch Backend
  -> MiniCPM-o 4.5 + Alice LoRA + 参考音频

Deployment was first verified on the LAN, then exposed over HTTPS through a configured domain, so demo users wouldn't have to install certificates manually or change browser security settings. Backend acceptance wasn't just checking that the port was alive, but confirming that: the LoRA was actually loaded, phone camera frames entered the vision encoder, silence after 10 speech units could trigger a reply, and streaming speech was generated.

The next few rounds of user feedback effectively formed the most valuable end-to-end tests:

What the user saw	Root cause	Fix
Stuck on Preparing	The mobile page was missing the ttsEnabled DOM element, causing a null reference during initialization	Filled in the state dependency and re-verified
Could talk, but speech was choppy	Each 1-second audio chunk took 1.47–1.62 seconds to compute, so production was slower than playback	While listening, the camera dropped to one frame every 3 seconds; frame sending paused while speaking
Playback over the public internet still jittered	Uneven intervals between network and generation chunks	Server-side chunk time reduced to the 0.8-second range; playback pre-buffer increased to 600ms
After one user disconnected, the next one got stuck	A disconnect before queue_done didn't release the Worker	Added release logic and regression tests
"Device not found" after restart	The browser had saved an exact deviceId that was no longer valid	Default to the system microphone; added device fallback for the camera

The page also converged from an engineering debug console into a simple "Alice Phone": hiding model, sampling, TTS and other settings ordinary users don't need to understand, and keeping only answer, hang up and essential status. In the end, demo users didn't need to import certificates, choose a backend or fill in a model path.

Finally, the user asked to stop the service. The public tunnel, Gateway, Worker and Backend have all been shut down. Shutting down the service is part of delivery too — you can't leave an experimental service running unattended on the public internet.

---

6. Six reusable methodologies for agent-driven fine-tuning

Mermaid

mindmap
  root((Agent 驱动微调
六条方法论))
    1. 先锁事实再调参数
      模型 revision + SHA
      依赖版本锁
      模板名固定
    2. 把 smoke 当闸门
      加载编码反向保存
      checkpoint 权重验证
    3. 失败样本变数据分类
      八类失败维度
      校准题与纠偏题
    4. 指标与行为分开验收
      loss 看总体退化
      高风险提示定交付
    5. 训练与产品解耦
      人格音色视觉通信
      哪层坏了修哪层
    6. 完成在用户设备上
      命令行不等于可用
      端到端工程修复

Figure: Six reusable methodologies for agent-driven fine-tuning — general principles distilled from this experiment that can be transferred to other fine-tuning projects.

Looking back at the whole process, what Codex did wasn't one long "automatic training" run, but a gated evidence loop:

Plain Text

公开资料找方向
  -> 源码确认真实接口
  -> 固定模型、数据与环境
  -> 最小 smoke 证明运行链路
  -> 正式训练
  -> 用失败样本生成定向数据
  -> 新提示回归，而不是只看旧 loss
  -> 放进真实产品链路
  -> 根据用户可感知故障继续修复

Six of these points can be transferred directly to other fine-tuning projects:

1. Lock down facts first, then tune parameters. If the revision, hashes, template and dependencies aren't pinned, no training comparison can be trusted. In this experiment, the model revision was pinned to a specific commit and all 54 files had their SHA-256 verified one by one — this step guards against the hardest class of problems to debug.
2. Treat the smoke test as a gate. It must cover loading, encoding, backpropagation, saving and weight updates — not just running --help. The two smoke-test failures overturned two priors (dependency compatibility, and whether plain text allows turning off the audio tower); each was worth more than a lucky pass.
3. Failure samples must become data categories. Every bad answer in V1 was classified into regressable dimensions such as memory, action and judgment, rather than ending with a human tweaking one answer. 108 calibration prompts, 43 correction topics — every round of iteration was structured, not a matter of luck.
4. Accept metrics and behavior separately. Loss is responsible for detecting overall degradation; high-risk prompts decide whether it can ship. In the end we didn't pick V3, which had the best language metrics, but V5, whose behavior was more stable — a trade-off that can only be made with separate acceptance.
5. Decouple training modules from product capabilities. Persona, timbre, vision and communication each have their own source; whichever layer breaks, fix only that layer. To change the voice, just replace the reference audio without retraining the persona LoRA; to change the persona, train only on text without polluting the acoustic modules.
6. Real completion happens on the user's device. A checkpoint that can run inference doesn't mean the phone can connect; connecting doesn't mean the audio is continuous or that it still works after the device restarts. Between the command line and the real user experience, there are at least five more rounds of engineering fixes to do. This also explains why agents need retrievable open-source assets. Without the model card, the first step would have trained the wrong target; without the framework source, the plain-text audio path and LoRA matching would have been guesswork; without real tests on the Spark and on a phone, every plan would just be a syntactically correct document.

7. Honest boundaries

This section is written out clearly because "the agent says it's done" doesn't mean it's done.

1. No TTS fine-tuning was done. Less than a minute of usable speech was used only for zero-shot cloning. This article cannot claim on that basis to have solved few-shot timbre training.
2. The original 79 test items have become a fixed comparison set. The V3 and V5 numbers are suitable for comparing candidates within the same project and should not be packaged as an independent public benchmark.
3. V5 is the best after behavioral trade-offs, not the best on every metric. If the task looked only at text likelihood, V3 would be better; the Alice product puts truthfulness and boundaries at a higher priority.
4. Omnimodal regression is product-grade, not paper-grade. The camera, voice triggering, streaming TTS and the real phone pipeline have been verified, but no large-scale, cross-device, long-duration statistics on vision and interruption have been produced.
5. The data is still primarily synthetic dialogue. There are rule-based quality checks, model review and round-by-round corrections based on real failures, but these cannot replace long-term feedback from real human use.
6. Dependency conclusions are time-anchored. This article records the state of the source code, wheels and hardware from 2026-07-25 to 2026-08-05; re-run the smoke test after upstream updates.
7. The public demo has been shut down. The current 502 is the result of an intentional shutdown, not an online product that is still being maintained.

From a request by someone who didn't understand LoRA, to a loadable 58.5 MiB adapter, to an Alice call that actually connected on a phone, there was no single "magic prompt" in between. What really made it work: public materials let the agent find its direction, source code let it falsify its own assumptions, the smoke test kept errors at minimal cost, failure samples drove the data to keep growing, and the real user experience provided the final acceptance criteria.