Viva Mais AI is a local-first WhatsApp copilot for the Viva Mais travel agency. The owner sells airline tickets over WhatsApp and drowns in the follow-up: re-typing itineraries, checking who paid, chasing documents, and answering late at night. Viva Mais AI turns one raw WhatsApp export into a CRM dashboard, running entirely on small local models so client data never leaves.

It was built for the Hugging Face × Gradio Build Small Hackathon (Backyard AI track).

Demo

• Demo video: final-demo.mp4
• Walkthrough video on X: @amphetamarina
• Write-up: the Viva Mais AI build-small-hackathon blog post

How it works

The user uploads a raw WhatsApp conversation export (a zip with its media). Every image and PDF becomes one typed artifact — quote, reservation, boarding pass, payment proof, invoice, accommodation, identity document, or an explicit "uncategorized" with its full transcription preserved — and voice notes are transcribed with whisper. From those artifacts the app assembles a CRM-style dashboard: identity, reservations, payments, sales-pipeline stage, and the next action to take.

No export handy? A built-in synthetic demo loads a small, fictional WhatsApp conversation and runs the whole pipeline on made-up data: a quote, a booking, a payment, and a boarding pass.

Stack

• Models: MiniCPM-V 4.6 transcribes and classifies each image; MiniCPM5-1B fine-tuned on PT-BR WhatsApp conversations extracts the typed fields and answers dashboard Q&A; whisper-small fine-tuned on Common Voice Portuguese transcribes PT-BR voice notes; pypdf reads PDF text. Structured extraction is orchestrated with DSPy over the in-process models. All small, in-process, no external APIs. Every PT-BR prompt, glossary and demo is accent-faithful (orçamento, sessão, cartão), because the fine-tunes were trained on correctly accented text rather than ASCII-folded approximations.
• RAG (NVIDIA Nemotron): dashboard Q&A is grounded by hybrid retrieval. nvidia/llama-nemotron-embed-vl-1b-v2 embeds the corpus multimodally — each receipt or ticket as its image plus OCR text in one vector, plus customer fact sheets and sliding conversation windows — at the end of the upload pipeline, and embeds each question at ask time. Semantic top-k chunks are combined with deterministic domain matching (customers, passengers, reservation codes, and transaction ids named in the question always contribute their full facts), so the 1B answer model reads a small, precise context instead of the whole conversation.
• Runtime: auto-detected at startup. On ZeroGPU everything runs through torch inside @spaces.GPU functions; on a real CUDA GPU vision runs through transformers and Q&A through llama.cpp with full offload; on CPU everything runs through llama.cpp GGUF Q4 and faster-whisper int8. Modal is the dev bench for data curation, evals, and LoRA fine-tuning only — never a dependency of the deployed Space.
• Frontend: no stock Gradio components anywhere. The UI is a hand-built hypermedia app — gr.Server keeps Gradio's queue, streaming and ZeroGPU plumbing while plain FastAPI routes serve server-rendered HTML driven by HTMX + hyperscript (vendored, no build step, no CDN). The design is a travel agency desk: artifacts render as boarding-pass ticket cards with tear-off stubs and CSS barcodes, KPIs as a split-flap departure board, the sales pipeline as a flight route with a plane en route, the conversation as a pixel-faithful WhatsApp replay, model traces as a control-tower radar terminal, and the robot-cowfish mascot reacts to the pipeline through pure-CSS sprite animation. Each streamed processing tick is one HTML chunk whose hx-swap-oob fragments update five regions at once; the browser drives the ZeroGPU resume loop through the Gradio JS client. To the author's knowledge this is the first HTMX frontend on gr.Server.
• Architecture: Clean Architecture (domain, application, adapters, frameworks).
• Tooling: uv + ruff + mypy for local development and CI; the Space installs from requirements.txt.

Fine-tuned models

Every model runs in-process and is well under the hackathon size cap — the largest is 1.3B. Each fine-tune is published with a model card and starts from an OpenBMB base.

Role in the app	Fine-tune (published)	Base model
Image OCR + artifact classification	minicpmv4.6-vivamais-v1 · GGUF · LoRA	openbmb/MiniCPM-V-4_6
Field extraction + dashboard Q&A	minicpm5-1b-vivamais-v4 · GGUF	openbmb/MiniCPM5-1B · GGUF
PT-BR voice notes	faster-whisper-small-pt-cv11-v5	openai/whisper-small + Common Voice 11 PT
Q&A retrieval (used as-is)	—	nvidia/llama-nemotron-embed-vl-1b-v2

Published, redacted training data:

• marinarosa/vivamais-synthetic-ptbr — synthetic, accent-correct PT-BR document corpus (no client PII).
• marinarosa/minicpm5-vivamais-text-sft-v4 — the text SFT mix.

Training on Modal

All fine-tuning, data curation and evaluation ran on Modal GPUs in the owner's own account, kept entirely off the deployed Space. The training loop itself was directed with OpenAI Codex (GPT-5.5, x-high reasoning) — the long, fiddly, unforgiving part of the build — with every run written to a dated log (finetune/*/reports/*_runlog.md), which is what keeps the Field Notes write-up honest.

• Vision (MiniCPM-V 4.6, LoRA rank 64, A10G). Synthetic-primary data (~2.2k accent-correct rendered documents) plus teacher–student distillation: teacher prefeitura-rio/Rio-2.5-Open-VL (a BR-PT VLM) labels on A10G, and a Qwen2.5-VL-32B judge on A100-80GB filters rows for accent fidelity and schema plausibility before training. An accent-sensitive eval harness and the Q4_K_M GGUF export both run as Modal pipelines. Held-out gains over the base: artifact-kind accuracy 0.006 → 0.58, BR-format field accuracy 0.81 → 0.88.
• Text (MiniCPM5-1B, QLoRA, A100). A PT-BR mix of public corpus (GigaVerbo) and synthetic WhatsApp Q&A, trained with Unsloth, then merged, GGUF-exported and scored — prepare, train, merge and eval are each a Modal step over a shared data volume, with privacy and leakage gates between them.
• Voice (whisper-small). Fine-tuned on Common Voice 11 Portuguese and exported to the faster-whisper / CTranslate2 format the Space loads.

Privacy is non-negotiable: the teacher and judge run only in the owner's Modal account, identity documents are excluded from every dataset and never published, and no raw export, audio or PII leaves the machine.

Build documents

• USE_CASES.md: the product, as prioritized Given/When/Then use cases.
• AGENTS.md: the engineering contract (TDD, BDD, Clean Code, Clean Architecture, atomic commits, tooling, project specifics).
• TASKS_PROMPTS.md: the from-scratch implementation spec, one precise prompt per task for a coding agent.
• ARTICLE.md: the Field Notes write-up — how it was built with coding agents, and what fine-tuning small models actually taught.

Quickstart

Bash

uv sync          # install deps (dev group included)
make check       # ruff + mypy + pytest (unit and Gherkin acceptance, offline)
make run         # launch the Space with the mock backend, no model needed

Links

• Hugging Face Space: build-small-hackathon/vivamais
• Write-up: Viva Mais AI

Built by marinarosa.