StackChan-LocalAI — a StackChan desk robot running entirely on local AI

A commercial M5Stack CoreS3 "StackChan" desk robot, rewired to run entirely on your own local machine instead of the tenclass/xiaozhi cloud — plus firmware mods that make it far more expressive. No cloud AI, fully reversible.

Throughout these docs, local-ai and LOCAL_AI_IP are placeholders for the host name and LAN IP of whatever machine you run the server on. Substitute your own.

Brain = local-ai's GPU vLLM (Qwen2.5-3B-Instruct) · Ears = local FunASR/SenseVoice · Voice = local Kokoro TTS · all offline.

On the model size: we'd happily run a bigger, smarter brain — the ceiling here is hardware, not ambition. This runs on a low-VRAM AMD GPU (ROCm, not CUDA), so a compact 3B fits and stays fast for live voice. With more VRAM (or an Nvidia card) you can drop in a larger model — just point LLM.LocalVLLM.model_name at it in data/.config.yaml; nothing else changes.

How it fits together

 StackChan (CoreS3, wifi "<your-wifi>")                     local-ai (LOCAL_AI_IP)
 ┌────────────────────────────┐   websocket   ┌──────────────────────────────────┐
 │ modified firmware (v1.4.3)  │◀────8090─────▶│ xiaozhi-esp32-server (docker)     │
 │  · rich decorated faces     │               │  ├ VAD  SileroVAD                 │
 │  · mood backgrounds         │   OTA 8003    │  ├ ASR  FunASR / SenseVoice       │
 │  · actions (dance, nod, …)  │◀──config──────│  ├ LLM  vLLM Qwen2.5-3B (:8000)   │
 │  · MiniJoyC over ESP-NOW    │               │  └ TTS  Kokoro (:8881)            │
 └────────────────────────────┘               └──────────────────────────────────┘
        ▲ NVS ota_url override points the device at local-ai (no reflash needed)

Two independent layers, either reversible on its own:

1. Redirect (no reflash): an ota_url value written into device NVS makes it fetch its server config from local-ai and connect over WebSocket. See device-nvs/.
2. Firmware mods (app-flash): the open-source m5stack/StackChan firmware, modified for rich AI-driven faces, on-demand expressions/actions, a dance, mood-tinted backgrounds, and MiniJoyC-during-AI joystick control. See firmware-mod/.

Why xiaozhi?

Because it's the device's native language, not a choice we made. The stock StackChan firmware is built on xiaozhi-esp32 — out of the box it already talks to the xiaozhi cloud over WebSocket. xiaozhi-esp32-server is the open-source, self-hostable twin of that cloud: a swappable VAD→ASR→LLM→TTS pipeline. So instead of rewriting and reflashing firmware, we run that server locally, plug in local providers (SenseVoice, your vLLM, Kokoro), and just repoint the device's ota_url at our box. The device thinks it's still calling the cloud. See xiaozhi-server/README.md for the redirect details.

Repository map

Path	What
xiaozhi-server/	the local backend (docker-compose): server config, English prompt, patches/textUtils.py (emotion→face), tests/
firmware-mod/	the firmware changes as *.patch files + build/flash/verify scripts + README
device-nvs/	the NVS ota_url redirect (scripts + README)
backup/	how to take & restore the full 16 MB flash backup (the .bin itself is git-ignored / local-only — it embeds Wi-Fi creds)

The firmware source tree lives outside this repo at ~/stackchan-fw/StackChan (built with ESP-IDF 5.5.4 at ~/esp/esp-idf); this repo carries the patches.

Setup — build your own

1. Back up the device first (reversibility): read the full flash and keep it local — see backup/.
2. Backend (a Linux box — "local-ai" here; a GPU helps for the LLM):

   cd xiaozhi-server
   cp data/.config.yaml.example data/.config.yaml     # then set LOCAL_AI_IP + YOUR_VLLM_API_KEY
   # download the ASR model per models/SenseVoiceSmall/README.md
   docker compose up -d

   You provide the LLM (any OpenAI-compatible endpoint — we run vLLM with Qwen2.5-3B); the compose wires up local FunASR (STT) + Kokoro (TTS).
3. Point the StackChan at your server: write the ota_url into device NVS (or use the device's Wi-Fi portal) — see device-nvs/. No reflash needed for this step.
4. (Optional) Firmware mods — rich faces, voice actions, MiniJoyC control: build the patched firmware and app-flash it — see firmware-mod/.

Config with real values (data/.config.yaml), the flash backup, NVS images, and the ASR model are all git-ignored — this repo ships sanitized skeletons (*.example, download READMEs) only.

Common tasks

# backend
cd xiaozhi-server && docker compose up -d          # start the local AI
docker compose logs -f xiaozhi-esp32-server        # watch (grep '[face] emotion=' for faces)

# checks (offline, no server needed)
python3 xiaozhi-server/tests/test_emotion_inference.py   # server↔firmware emotion contract
python3 xiaozhi-server/tests/check_config.py             # .config.yaml sanity
firmware-mod/verify-patches.sh                            # patches match the source tree

# firmware
firmware-mod/build-and-flash.sh --flash            # build + app-flash (NVS preserved)

# full restore to stock (reversible)
esptool --port /dev/ttyACM0 --no-stub write-flash 0x0 backup/stackchan_full_16MB_backup.bin

What she can do

Chats in English; shows decorated emotion faces (heart, blush, anger mark, tears, dizzy) with mood-tinted backgrounds (and "change your background to blue" via set_background); voice actions: dance, celebrate, nod, shake_head, look_around, nap/wake_up, set_face, plus head/LED/camera/reminders/volume; and the MiniJoyC joystick can steer her head while she talks (ESP-NOW on the router's channel).

See firmware-mod/README.md for the firmware-side details.

License

This project is released under the MIT License — do what you like, no warranty.

The firmware changes here are distributed as *.patch files against the upstream open-source StackChan firmware (see firmware-mod/); that upstream code remains under its own license and is not redistributed in this repo. Third-party components (the xiaozhi-esp32-server backend image, the SenseVoice ASR model, Kokoro TTS, and any LLM you supply) are each governed by their own licenses — check them before use.

Disclaimer

Independent, unofficial project — not affiliated with, endorsed by, or sponsored by M5Stack, the Stack-chan project/authors, or any other trademark holder. "M5Stack", "CoreS3", "StackChan" / "Stack-chan", and other names are trademarks of their respective owners and are used here only to describe hardware compatibility.

This modifies commercial hardware and reflashes firmware. You do it at your own risk — flashing, NVS edits, and hardware changes can brick a device or void warranties. Always take the full-flash backup first (see backup/); everything here is designed to be reversible, but no guarantees are made. Provided "as is", without warranty of any kind.