MiniCPM-RobotTrack

MiniCPM-Robot Embodied Series

MiniCPM-RobotTrack provides a public pipeline from Habitat rollouts to training data, visual feature caching and launching fine-tuning, supporting three types of target-tracking tasks: STT, DT and AT.

Raw Trajectories

Each Habitat rollout should keep its video, per-step simulation state and episode result. Per-step records can include fields such as robot velocity, collision state, target distance and target position in the image.

Generating Training Trajectories

Bash

cd MiniCPM-RobotTrack
python tools/make_tracking_data.py \
  --input_root sim_data/raw_sample \
  --output_root sim_data/train/stt \
  --only_success \
  --history 31 \
  --horizon 8 \
  --dt 0.1 \
  --incremental

For full training, place the STT, DT and AT raw rollouts in sim_data/raw/<task> respectively, and run the same data conversion pipeline for each task.

Pre-caching Visual Features

Bash

for task in stt dt at; do
  python tools/precompute_features.py \
    --json "sim_data/train/${task}/jsonl" \
    --data-root "sim_data/train/${task}" \
    --cache-root "sim_data/train/${task}/vision_cache"
done

The processed data directory contains frames/, jsonl/ and vision_cache/.

Launching Fine-tuning

Bash

bash scripts/train.sh

Data paths, batch size, learning rate and number of epochs can be adjusted in scripts/train.sh.

Data Quality Recommendations

• Filter out abnormal trajectories, wrong actions and invalid interactions.
• Keep task splits, data versions and conversion parameters so experiments are traceable.
• Use methods such as DAgger to supplement long-tail failure samples such as target crossings, sharp turns, brief occlusions and multi-person crossings.
• Keep training and evaluation data strictly separated to avoid scene leakage.

HM3D, MP3D, visual foundation models and other third-party resources each have their own licenses. Before training, confirm the permitted scope of use for the data, models and simulation resources.

Related resources

Topic	Link
Official training code	Official training code
Sample data	Sample data