Aranya is a plant expedition app built for the Build Small Hackathon: two fine-tuned MiniCPM-V 4.6 models (~1.3B parameters each) identify plants or diagnose their health from a single photo, and the result is narrated to you as a spoken story.

The Vision: Plant Care as an Expedition

Here's a stat that caught my attention: the average plant parent has killed at least seven plants. Not from neglect — from confusion. Overwatering, misidentification, missed early signs of disease. There are 86 million households in the US alone with plants on their windowsills, and most of them are winging it.

The question I set out to answer: can a 1.3-billion-parameter vision-language model, fine-tuned on community knowledge, deliver genuinely useful plant identification and health diagnosis? And can I wrap it in an experience that doesn't feel like homework? The answer, it turns out, is yes. With caveats.

Most plant care apps feel clinical. You scan a leaf, you get a paragraph of text, you close the app. The retention problem isn't information — it's engagement. What if every time you point your phone at a plant, you're not "scanning" it — you're discovering it? What if you're a Wildkeeper on an expedition, and each plant you encounter adds to your journal?

Key UX decisions that followed from the metaphor:

• Narrated responses via text-to-speech, so you don't read about a plant — you hear its story
• A discovery journal that grows as you explore: species counter, rescue counter, your personal botanical record
• Storytelling tabs after each identification: folklore and legends, "plant superpowers" (medicinal/ecological uses), fun facts, care tips
• A community leaderboard, because adventure is better with others on the trail

Data: 310K Training Conversations from Community Knowledge

The backbone of Aranya's intelligence isn't a massive pretrained model. It's a fine-tuned specialist trained on real-world plant discussions.

Sourcing and Scale

I collected approximately 806,000 image posts from online plant care communities. Each post typically contained a photo of the plant, the owner's description of the problem or question, and community responses (often detailed diagnoses from experienced growers). The signal-to-noise ratio in raw community data is poor, so getting to the useful subset required a multi-stage pipeline.

The Classification Layer

• Stage 1 — Rule-based classification: a keyword scoring system with community-specific priors bucketed posts into five categories: disease/stress, species identification, showcase, general care questions, and other.
• Stage 2 — ML refinement: a TF-IDF + Logistic Regression model trained on the high-confidence rule-based labels re-classified the ambiguous posts. Only predictions above 0.60 probability got re-labeled.

Final breakdown: 239K disease/stress posts and 304K species identification posts.

Synthetic Training Data via Gemini Batch API

Raw community threads are messy, multi-party and unstructured, so they had to be distilled into clean user/assistant training pairs. I used Gemini 2.5 Flash Lite on Vertex AI's Batch Prediction API — a larger model distilling messy community knowledge into clean training pairs for a smaller model. This is synthetic data generation, but grounded in real-world examples. The key instruction: the assistant response must reference what's visible in the image and not just regurgitate textbook information.

Two parallel batch jobs:

• Disease diagnosis: 288K requests
• Plant identification: 250K requests

Each batch file held 50,000 requests, submitted with temperature=0.4 for consistency. The Batch API's 50% cost discount made processing at this scale feasible. Results: 228K disease training records and 105K plant-ID training records (92% acceptance rate; the rest filtered as irrelevant or malformed).

The full pipeline:

1. Community plant posts (~806K)
2. Classify (rules + ML) → 239K disease + 304K plant_id posts
3. Gemini Batch API (Vertex AI) → 310K+ multimodal training conversations
4. ms-swift LoRA on Modal (L40S) → two fine-tuned adapters
5. Merge + GGUF quantize → F16 + Q8_0 weights on the Hugging Face Hub

Fine-Tuning: LoRA on MiniCPM-V 4.6

Why MiniCPM-V 4.6

I needed a vision-language model that stays well under the hackathon's 32B parameter limit, handles multimodal input natively (image + text in, text out), has a permissive license, and actually works well enough at small scale to be worth fine-tuning. MiniCPM-V 4.6 checked every box. At 1.32B parameters it's genuinely small and fits on a single consumer GPU, but punches above its weight on vision-language tasks thanks to a solid architecture (vision tower + language model + aligner).

The LoRA Configuration

Training was orchestrated through ms-swift v4.2.3 with PEFT as the LoRA backend. All runs executed on Modal with L40S GPUs.

Parameter	Value
LoRA Rank	16
LoRA Alpha	32 (scaling ratio 2x)
Dropout	0.05
Target Modules	all linear layers
Vision Tower	unfrozen (LoRA applied)
Aligner	frozen
Trainable Parameters	~20.3M (1.54% of model)

The interesting decision here is vision-inclusive LoRA. Most practitioners freeze the vision encoder during fine-tuning. I didn't: plant photographs look nothing like the web images MiniCPM-V was pretrained on. Community posts are often poorly lit, blurry, close-up shots of diseased leaves taken with phone cameras. Freezing just the aligner preserved the learned vision-language alignment while letting both the vision encoder and the language model specialize.

Training Runs

	Plant Disease	Plant ID
Dataset size	~210K records	~100K records
Epochs	1	2
Steps	1,641	1,576
Learning rate	2e-6	2e-6
LR schedule	cosine, 3% warmup	cosine, 3% warmup
Effective batch	128 (8 x 16 grad accum)	128
Time	26.7 hours	22.6 hours
Final train loss	0.79	1.54
Final eval loss	1.40	1.31
Final token acc	63.1%	66.7%

The learning rate (2e-6) is deliberately conservative. With a 1.3B model and 200K+ samples, there's enough data that you don't need aggressive learning, and you really don't want to blow past the good loss basin early on.

Results

Evaluated on 300 held-out test samples per task, comparing the base MiniCPM-V 4.6 against the fine-tuned models:

Metric	Disease: Base → Fine-tuned	Plant ID: Base → Fine-tuned
BLEU	2.52 → 3.87 (+53.9%)	1.69 → 2.30 (+36.3%)
ROUGE-1	0.298 → 0.357 (+19.6%)	0.240 → 0.237
ROUGE-2	0.063 → 0.089 (+40.8%)	0.053 → 0.064 (+21.3%)
ROUGE-L	0.153 → 0.163 (+6.7%)	0.144 → 0.140
MoverScore	0.531 → 0.539	0.519 → 0.520

The disease model showed stronger gains across the board, which makes sense: disease diagnosis requires specific vocabulary ("root rot", "fungal infection", "overwatering damage") that the base model simply hadn't seen enough of. The plant-ID model improved BLEU and ROUGE-2 (phrasal precision) while ROUGE-1 stayed flat — it got better at constructing correct multi-word identifications without necessarily changing its unigram vocabulary.

Merging and Quantization

After training, I merged the LoRA adapters back into the base weights on Modal:

Bash

swift export \
  --model openbmb/MiniCPM-V-4.6 \
  --adapter checkpoint-1641 \
  --merge_lora true \
  --output_dir merged-checkpoint-1641

Then converted to GGUF using llama.cpp's conversion tool:

• F16: full precision for ZeroGPU (where VRAM isn't the constraint)
• Q8_0: 8-bit quantized for local deployment
• Q4_K_M: aggressive quantization for truly resource-constrained scenarios
• mmproj: the vision projector, exported separately

Published models: ankurkaul17/minicpm-v-4.6-pdd-gguf (disease/health) and ankurkaul17/minicpm-v-4.6-pid-gguf (identification).

The App: Stitching It Together

Architecture

Aranya runs as a Gradio Space on Hugging Face, but it doesn't look like a Gradio app. gr.Server acts as a headless FastAPI host that handles deployment, the ZeroGPU lifecycle and OAuth, while a completely custom vanilla HTML/CSS/JS frontend is served on top.

1. The browser uploads an image, chooses a mode and calls POST /aranya/run
2. The server preprocesses the image (EXIF transpose, center crop, resize to 1024px) and passes it as a base64 data URI alongside the text prompt
3. Inference runs the MiniCPM-V GGUF via llama-cpp-python with MTMDChatHandler; ZeroGPU provides the GPU with no standing reservation
4. Pocket TTS (~100M parameters) streams speech on CPU
5. Text and audio flow back to the browser as an NDJSON stream

The Streaming Pipeline

The LLM generates text token by token, the TTS needs sentence-length chunks to sound natural, and the frontend needs to sync text appearance with audio playback. The solution is parallel producers feeding a single NDJSON stream:

• The LLM streams text deltas into a buffer
• A "speakable segmenter" extracts natural phrase boundaries (min 32 chars, max 350 — balancing TTS latency against prosody)
• Each segment goes to Pocket TTS, which produces PCM audio
• Both text deltas and audio chunks flow as NDJSON events to the client
• The frontend trickles text at 80ms/character until audio playback catches up, then syncs the reveal rate to the audio stream via the Web Audio API

It's a bit of an asyncio circus — queues, producers, consumers, backpressure — but the result feels seamless: you see words appearing as you hear them spoken.

One gnarly bug: llama-cpp-python's multimodal handling has a quirk where hybrid models wipe image embeddings on state reset. I had to monkey-patch Llama.generate() to preserve the multimodal context across the generation loop — not something you'll find in the docs, just hours of debugging why the model was not seeing the images.

Off the Grid

No external APIs are called during core inference. The plant identification model, the disease diagnosis model and the TTS engine all run within the Space. The only outbound calls are for follow-up content (fun facts, mythology, etc.), which uses DuckDuckGo search + the base MiniCPM-V as a utility model. Frontend assets are vendored too, with images inlined as base64; only marked.js comes from a CDN for markdown rendering.

The SQLite Backbone

Every discovery, rescue and leaderboard entry is persisted in SQLite with WAL mode, which handles concurrent reads during streaming, automatic backups and corruption recovery. On Hugging Face Spaces the /data persistent volume can get into weird states after cold starts, so I added stale WAL sidecar cleanup and auto-recovery.

Making Plant Care Interesting: The Storytelling Angle

Traditional plant care apps treat information delivery as the product. The problem: nobody retains information presented that way. People remember stories. When you identify a plant in Aranya, you get a spoken explanation of what the model sees, how it arrived at the identification, and why this plant is interesting.

After identification, four storytelling tabs extend the engagement, each generated on demand by the utility model with web search context:

• Did You Know? — surprising facts that make you look at the plant differently
• Keep It Happy — care tips framed as a relationship ("your Monstera likes humidity because...")
• Stories & Legends — mythology, folklore and cultural significance across civilizations
• Plant Superpowers — medicinal uses, ecological roles, culinary applications

The gamification layer is light by design. You're a "Wildkeeper". Each scan is either a "Discovery" (identification) or a "Rescue" (health diagnosis). Your stats grow and you can see where you stand on the leaderboard, but there's no XP grinding and no artificial gates.

I think this is an underexplored design space. We spend so much time squeezing another point of BLEU out of a fine-tune when the UX framing might matter more for actual user retention. A slightly worse model wrapped in a compelling narrative will outperform a better model behind a clinical interface — at least for consumer apps.

Limitations

• Small VLM ceiling: MiniCPM-V 4.6 is a compact model and may struggle with rare diseases, ambiguous symptoms or visually similar species
• Training data bias: fine-tuned on community plant posts; accuracy varies by species popularity and image quality
• Cold start latency: ZeroGPU and model loading on HF Spaces can delay the first request
• TTS quality: narration depends on the cloning sample; Pocket TTS may produce artifacts on unusual plant names
• Streaming edge cases: text-audio sync can drift on very long or very short responses

Links

Thanks to Hugging Face, OpenAI, Modal and OpenBMB for the compute, credits and models that made this possible.

• Live demo: Hugging Face Space
• Demo video: YouTube
• Blog post: Aranya: Teaching a Small Model to See Plants and Tell Their Stories
• Code: allthatido/sounth_pddm
• Author: Ankur Kaul