A 100% browser-native anime companion — local LLM + on-device STT + EMAGE motion + Edge-TTS, zero backend
English • 简体中文
Important
Notice: This repository is a browser-native AI companion research prototype. It explores 100% client-side AI stacks (WebLLM / WebGPU / ONNX Runtime Web) and Cloudflare Workers edge TTS. It is not recommended for direct commercial production deployment.
Project XiaoChun (小蠢) is a fully browser-native anime companion. The character renders through @pixiv/three-vrm with MToon NPR shading inside an immersive linework outdoor scene; all AI inference runs in your browser tab — no Python backend, no server GPUs.
Entry is a 2D MAD preload into 3D — the camera pushes in from a distant 3.3× view while the overlay dissolves. You talk to her (ChatBar mic dictation via on-device SenseVoice, or type). She thinks (WebLLM MiniCPM5 2B q4f16_1 on WebGPU, falls back to Qwen2.5 0.5B; thinking optional), speaks (Edge-TTS over a Cloudflare Workers WebSocket), and moves with her (EMAGE ONNX in a Dedicated Web Worker). If the model is not ready yet, typed messages queue instead of getting dropped. The chat-bar menu can switch models from WebLLM's prebuilt list, or connect any custom OpenAI-compatible provider (Ollama / LM Studio / vLLM / LocalAI / cloud) via the in-app config dialog — credentials stay AES-GCM encrypted in IndexedDB.
The UI is fully SSR-hydrated multi-language (zh-CN / en / ja) via TanStack Start + i18next, and is mobile-first responsive (iOS HIG 44 pt / Material 48 dp touch targets).

Entry: idol-debut style loading screen, then a dimension-break into the 3D stage

3D stage: MToon NPR character, sunny beach scene, head bubble, and chat bar

Built-in outfits, all swappable at runtime
Individual outfits
![]() Dinner Dress |
![]() Techwear |
![]() Cheongsam |
![]() Bikini |
![]() Maid |
![]() Swimsuit |
![]() Bandage Wrap |
![]() Office Lady |
![]() Wedding Dress |
In addition to running directly in the browser, Project XiaoChun provides a cross-platform native desktop client (built with Tauri 2.0). It supports both Full-Stage 3D Mode and Frameless Transparent Desktop Pet Mode with minimal memory overhead and instant startup.
We offer three installation methods (ranked from easiest to most advanced):
Open your terminal (macOS Terminal / Linux Shell / Windows Git-Bash) and run:
curl -fsSL https://raw.githubusercontent.com/FireTable/project-xiaochun/main/scripts/install.sh | bash🤖 Autonomous AI Agent Friendly: Coding agents (such as Claude Code, OpenClaw, Cursor) can run this command unattended to install XiaoChun. See Agent Skill Specification.
If you are on macOS with Homebrew installed, manage installation and future updates via Homebrew Tap:
brew tap FireTable/project-xiaochun https://github.com/FireTable/project-xiaochun
brew install --cask project-xiaochunTo upgrade in the future:
brew upgrade --cask project-xiaochunThe desktop app can also check GitHub Releases in-app (top-bar ⋯ → Check for updates). Homebrew and in-app updates are both valid.
You can download installer packages directly from the official Releases page:
👉 Go to GitHub Releases for All Downloads
| Platform / Architecture | Package | Description | Quick Download |
|---|---|---|---|
| 🍏 macOS (Apple Silicon) | .dmg |
For Apple Silicon (M1 / M2 / M3 / M4) Macs | Download DMG (ARM64) |
| 🍏 macOS (Intel) | .dmg |
For Intel-based Macs | Download DMG (x64) |
| 🪟 Windows (x64) | .msi / .exe |
For Windows 10 / 11 (64-bit) | Download Windows Installer |
| 🐧 Linux (x64) | .deb / .AppImage |
For Ubuntu / Debian and mainstream distros | Download Linux Package |
- macOS ("App is damaged and can't be opened" or unverified developer):
Because this open-source project is not notarized with Apple's paid developer certificate, macOS Gatekeeper flags browser-downloaded files with quarantine attributes (in macOS 15+, the "Open Anyway" button was even removed from System Settings).
Fix:
- Open the downloaded DMG and drag
Project XiaoChunto Applications; - Open Terminal and execute:
xattr -cr "/Applications/Project XiaoChun.app" - Double-click the app again to launch immediately!
- Open the downloaded DMG and drag
- Windows (SmartScreen "Windows protected your PC"): Click "More info", then click "Run anyway" at the bottom right.
- Linux (AppImage permission):
If using
.AppImage, runchmod +x <filename>.AppImage, then execute normally.
- Frameless Transparent Desktop Pet (桌宠): Floats borderless and chromeless directly over the desktop environment with 100% transparent operating system compositing.
- 60Hz Real-Time Canvas Alpha Bitmask Click-Through (macOS & Windows): 16ms high-frequency thread evaluates true character alpha pixels using 1-ns bitwise shift tests. Clicks seamlessly pass through transparent pixels to desktop apps beneath, while retaining solid interaction on the character and UI. Sampling pipeline is optimized with 32-bit dirty checks and byte packing.
- Screen-Boundary Snap-Back Clamp (Scheme A): On corner resize or dragging release, window position and bounds automatically snap back within the active display's
work_area(respecting taskbars and docks), preventing XiaoChun from getting lost offscreen. LoadingOverlay natively supports window dragging. - Persistent Window Coordinates & Freeform Dragging: Built-in
tauri-plugin-window-stateautomatically restores window position and size. Four corner handles enable fluid resizing without canvas flickering, alongside dragging anywhere on the character or loading overlay. - Cross-Process Protocol (
xiaochun://): Custom OS URL Scheme enabling external scripts, AI agents, browsers, and CLI tools to wake up XiaoChun and triggerspeakTextwith full body motion and lip-syncing. - In-app updates: Official Tauri updater reads
latest.jsonfrom GitHub Releases, prompts after the splash, and replaces the whole app (not a model-file patch). Details:docs/HYBRID_DESKTOP_APP.md.
- In-World Holographic Guides: Replaces flat 2D sliders with embedded 3D controls —
TurnGuide3D(horizontal yaw ring with responsive light beacon),PitchGuide3D(elevation arc), andCameraYGuide3D(floating 3D height rail). - Localized Mouse Wind Aerodynamics: Cursor velocity induces real-time 3D wind drag impulses (
windForce.ts), causing hair ribbons, clothing fabrics, and accessories to flutter dynamically. - Pure Quaternion Chain & Biomechanical Fixes: Eradicates head rotation snap and non-uniform shearing distortions through pure affine coordinates and quaternion interpolation.
- WebGL Context Lost / Restored Resilience: Registers
webglcontextlostandwebglcontextrestoredevent hooks to gracefully suspend render loops and reconstruct renderer/texture states upon GPU reset or system wake from sleep. - Lightweight Audio Pipeline & Anti-Aliased Resampling: EMAGE downsampling incorporates area-weighted box filtering to eliminate high-frequency aliasing; pre-decoding utilizes pure software
OfflineAudioContextto avoid occupying OS audio device output handles; Tone.js named imports minimize initial bundle footprint. - VRM 1.0 & VRM 0.x Dual-Standard Modular Pipeline: Powered by
@pixiv/three-vrmwith MToon NPR shading. The core engine (VRMEngine) is decoupled into specialized subsystems, natively accommodating both modern VRM 1.0 and classic VRM 0.x avatars:- Transparent Pipeline Translation: All upstream motion generators (idle, EMAGE, VRMA clips, locomotion stepping, gaze) author poses strictly in the VRM 1.0 standard coordinate space.
PoseBufferautomatically handles bone axis inversions and coordinate mapping under the hood, ensuring VRM 0.0 avatars work out-of-the-box with zero joint inversions. - Dual-Theme Linework Outdoor Scene (
LineworkWorld): Pure procedural code generating a minimalist wireframe outdoor world (sun with 12 radial rays, 19 stylized skyline buildings, ground grid, and 5 tree archetypes); natively supportslinework-lightandlinework-darkdual themes linked to the global color mode — 100% white-on-white / black-on-white, zero external textures. - Studio 3-Channel Lighting (
StudioLighting): Streamlined directional sunlight (dir 1.10 with 2048 PCFSoft shadow camera), hemisphere sky/ground light (hemi 0.80), and back-rim fill light (fill 0.65), delivering clean and crisp anime contours without redundant lighting overhead. - MToon Material Manager (
VRMMaterialManager): Automatic mesh semantic categorization (skin / hair / eyes / clothing); dynamic fragment shader injection with uniformuMatSaturationfor live color calibration; tuned hair saturation (default 1.50) for rich anime highlights. - Companion Gaze Controller (
GazeController): Companion eye & head tracking, physiological yaw/pitch safety clamping, micro-saccades, and thinking head sways. - 3D Head Bubble Tracker (
BubbleTracker): 3D world-to-screen 2D projection with a 1.5px dead-zone filter, writing directly to DOM transforms to bypass 60~120 FPS React re-renders.
- Transparent Pipeline Translation: All upstream motion generators (idle, EMAGE, VRMA clips, locomotion stepping, gaze) author poses strictly in the VRM 1.0 standard coordinate space.
- Runtime VRM Upload: Top-bar upload button accepting arbitrary standard VRM 1.0 or VRM 0.x avatars with automated full-pipeline motion and ground-anchor adaptation.
- Zero-Frame T-Pose Pop-in & Seamless Hot Reloading: Captures a millisecond-level snapshot of bone rotations, expression weights, gaze coordinates, and playback timestamps before swapping. The new VRM is built in the background, pose-restored in memory before scene mounting, and atomically swapped within a microtask — guaranteeing 0 frames of T-pose glitching.
- Delta Patch Distribution (
.vrmbase+.vrmaddon): Eliminates bloated multi-megabyte model downloads by stripping shared body geometry into a compact base modelxiaochun_base.vrmbase(5.9 MB, oxipng texture-compressed); 9 distinct wardrobe sets (Dinner Dress, Techwear, Cheongsam, Bikini, Maid, Swimsuit, Office Lady, Wedding Dress, Bandage Wrap) are distributed as incremental4 MB each, 13 MB for grand wedding gown)..vrmaddonpatches (1 - Web Worker WASM bspatch Assembly: Reconstructs the complete VRM in a Dedicated Web Worker using WASM bspatch with zero main-thread frame drops.
- Two-Tier IndexedDB Cache: Base models and patched VRMs are locally cached in IndexedDB (
xiaochun-vrm-cache) for instant subsequent swaps. Detailed indocs/OUTFIT_SWAP.mdanddocs/VRM_BUILD_WORKFLOW.md.
- UnrealBloomPass Anime Halos: Renders soft bloom glow (default strength 0.35, radius 0.65, threshold 0.75) for luminous character highlights.
- Background Luminance Bypass: Solves over-exposure in bright linework scenes by selectively applying bloom to character hair and clothing while rejecting high-luminance background linework.
- ToneMapping Capsule Grid: Integrates 5 film curve mappings (ACESFilmic, Neutral, Cineon, Linear, None) with real-time exposure tuning.
- Full-Spectrum Color Grading (BC/HS): A unified single-pass fullscreen shader controlling Brightness, Contrast, Saturation, and Hue Shift. Full technical specifications in
docs/POSTFX.md.
- 28-Parameter Orthogonal Morphing: Complete live anatomical customization across head, neck, shoulders, torso thickness, waist, belly, hips, buttocks, bust, arms, fingers, thighs, calves, and feet. Shoulder width defaults to 110% (1.10) with an expanded range up to 250% (2.50) to support diverse proportions from slender anime figures to tailored jacket silhouettes. UI groups the 28 sliders into 7 body regions inside sub-cards with a search filter.
- Anatomical Boundary Locking: Converts symmetrical center-out bone scaling into directional expansion (e.g., torso thickness expands posteriorly, locking the anterior chest and abdomen flat; buttocks volume expands backwards without pelvic protrusion).
-
Procedural Abdominal Morphing (
belly): Deforms 687 front abdominal wall vertices with smooth cosine falloff ($0.70\times \sim 1.80\times$ ), completely decoupled fromSpinebones — zero impact on lumbar thickness or spinal curvature! - Knee-Foot Ground Anchor Alignment: Broadens saddlebag hips while automatically neutralizing lateral femur shifts at the knee, keeping legs strictly plumb and feet planted together.
- Shear-Free Head Composition: Isolates the Head world matrix from non-uniform neck scaling, preventing eye flattening or hair slant.
- Dynamic Crown Height Measurement: Live millimeter-accurate character height dynamically derived from vertex meshes across shoes, FootIK sink, and morph slider changes. Displayed in the drawer header chip and the optional on-canvas floating ruler (visible while the drawer is open on non-mobile).
-
Technical Standard: Full mathematical specifications, formulas, and parameters matrix available in
docs/BONE_MORPH.md.
-
Zero-Friction Any-Motion Ingestion (
playMotion): Ingest VRMA URLs, raw ArrayBuffers, orTHREE.AnimationClips through a single call; automatically performs humanoid retargeting, hips normalization, and supports whole-body (all) or upper-body (upperBody) masking. -
Quintic Smootherstep Inbetweening: Based on
$6t^5 - 15t^4 + 10t^3$ curves over a unified 0.75s window with strictly continuous velocity and acceleration. Detailed indocs/MOTION_PIPELINE.md. -
Biomechanical FootIK & Ground Anchoring: Two-bone analytical IK during EMAGE speech; plants from the previous motion's feet (
footIkIdlePlant), yields completely to BodyTurn strides, then recaptures after the step. Shoe-off sink 4.6cm/3.9cm. Detailed indocs/FOOT_IK.md. -
Procedural Locomotion Stepping & Gaze: 4-phase stepping state machine with spring yaw tracking and bio-saccades. Detailed in
docs/BODY_TURN_AND_GAZE.md. -
Layered Pose Evaluation Graph:
-
Layer 0 (Base):
NaturalIdleSystemprocedural breathing, 8-figure pelvic postural sway, and relaxed biomechanical finger curling; -
Layer 1 (Main Action):
VRMAthinking loops,EMAGEstreaming gestures, and universal clips crossfading smoothly; -
Layer 2 (Locomotion):
BodyTurnSystemprocedural stepping onLEGS_MASKonly (no hip rotation, no spine yaw); -
Compose:
composeLayeredSmoothafter FootIK + Gaze so ground plant and look-at are follow targets, thencommitToVRM.
-
Layer 0 (Base):
-
Architecture Rules & Lifecycle Order: Full 10-step render loop lifecycle and developer anti-patterns detailed in
docs/ARCHITECTURE_AND_RULES.md. -
Loading Screen: layout, visual layers, motion/performance rules and copy for the splash are documented in
docs/LOADING_OVERLAY.md. -
Technical Docs Center: Complete architecture sitemap and agent navigation available in
docs/README.md. -
Tauri vs Web vs
/embeddifferences: side-by-side table (window / background / click-through, gestures, UI, camera, persistence, loading, security, build & test) with code locations and maintenance rules indocs/PLATFORM_DIFFERENCES.md.
-
LLM (WebLLM, default) —
@mlc-ai/web-llmMiniCPM5 2B (q4f16_1) on WebGPU (fallback Qwen2.5 0.5B). Lightweight and responsive on mobile and low-VRAM devices; chat-bar menu supports live model switching and thinking mode toggles; response language adaptively mirrors the user's prompt. - LLM (Custom OpenAI-Compatible Providers, optional) — Connect any OpenAI-compatible HTTP service via the in-app config dialog: Ollama / LM Studio / vLLM / LocalAI / cloud (OpenAI, DeepSeek, Qwen API …). Provider profiles are AES-GCM encrypted in IndexedDB; the active provider is one click away from switching. When a custom provider is active, WebLLM is not preloaded — saves 1-2 GB VRAM on local and avoids wasting bandwidth on a model you won't use.
-
Unified Provider Factory (
chatWorkflow.runChat) — Same-shaperunChat(opts) → stringcontract for both WebLLM and custom providers; the dispatcher picks one per request via aChatProviderregistry. Adding a new provider = drop in a descriptor. -
User-Customizable System Prompt, Memory Turns & Head Bubble — Open the chat-bar menu → 对话设置 / Chat Settings to override the character system prompt (free-form text, falls back to default when empty/equal), tune the conversation memory-turn count (1–50, default = device-recommended), and toggle the head chat bubble (
APP_CONFIG.chat.showHeadBubble, default on). When off,HeadBubbledoes not render; ChatDirector /bubbleTrackerevents still fire. Overrides persist in IndexedDB (xiaochun-user-settings); turn bounds live inAPP_CONFIG.memory.userTurnsMin/Max. -
Motion — EMAGE full-body co-speech motion (ONNX Runtime Web) in a Dedicated Web Worker: wasm execution provider + INT8 (
useInt8); does not use WebGPU (model tensors include int64). Temporal Gaussian smoothing and natural idle blends. Streaming windows T=64 with per-windowmotion_chunkfor TTFA; A/V hold until audible TTS; hop/seam tunables underAPP_CONFIG.emage.motion(advanceFrames60..64). Seedocs/EMAGE_MODEL.md. The shipped model set is a slimmed, INT8-quantized and graph-optimized build (download 191.35 → 71.59 MB, no retraining), produced bypackages/emage-onnx; seedocs/PERFORMANCE.md. -
TTS — Edge-TTS 晓伊 (XiaoyiNeural, zh-CN, +10 Hz) via a hand-rolled native WebSocket client in
src/lib/edge-tts-core.ts(no third-party TTS SDK); emoji stripped before speech. -
Vocal Post-Processing & 3D Spatial Acoustics — Tone.js 15.x + Web Audio DSP pipeline (
src/director/vocalProcessor.ts): low-cut highpass filter, 3-band EQ, broadcast dynamics compressor, subtle room reverb, bit-exact pure dry bypass routing, and 3D camera distance acoustics (near-field$<2.2\text{m}$ proximity boost,$2.2\sim 8\text{m}$ flat conversation, and$>8\text{m}$ smooth distance attenuation). Documented indocs/TTS.md. -
STT — SenseVoice Small int8 (zh/en/ja/ko/yue) in a Dedicated Worker: energy VAD auto-segments utterances, recognizes on-device, inserts at the ChatBar caret. CDN weights under
cdn.firetable.tech/xiaochun/stt/…-2024-07-17. Seedocs/STT.md. - LLM + TTS + EMAGE orchestrated by the chat director on the main thread at 60 FPS. LLM may use WebGPU (WebLLM); EMAGE stays on wasm/INT8.
-
Smart Speech Chunking: Eliminates long-text generation wait bottlenecks by segmenting speech into natural 30~60 character clauses split at semantic punctuation (
.!?\nor natural comma pauses).xc.sayhas no character cap; this slicer is the limit. -
Speech handoff: A newer
say/speakText/speakAudiofades the line still playing (about 0.4s, skipped when the gain is already 0) and then starts the new one. The previous pipeline exits on its utterance id, so the two do not play together.stop()still cuts immediately. - Zero-Latency Concurrent TTS Prefetching: Downloads audio for all chunks concurrently via non-blocking network I/O, flattening TTS latency to 0ms.
-
Dual-Condition Pre-buffering: Balances chunk ratio (
$\lceil N / 3 \rceil$ ) with an upper-bound cap (max 2 chunks,812s of audio). 1~2 segments start almost instantly; long paragraphs begin playback as soon as 2 chunks are ready while subsequent motions stream in the background. -
Continuous Latent Autoregressive Seed Carryover: The Dedicated Web Worker retains the 4-frame latent seed (
continueFromPrevious) across chunks, making multi-chunk generation mathematically identical to a single long-run autoregressive inference. -
Streaming EMAGE (
motion_chunk): Worker steps T=64 audio/motion windows and posts transferablemotion_chunkafter each successfulrunStepso gestures start before the full utterance finishes (TTFA). First chunk is buffered until TTSAudioContext.start(releaseMotionForAudio) so motion never leads audible audio. -
P0b isolation + wasm threads: Document responses set COOP
same-origin+ COEPcredentialless(src/server.ts, Vite preview/dev,public/_headers) socrossOriginIsolatedenables SharedArrayBuffer and ORT wasm multi-thread when the browser isolates; scratch buffers are reused across windows. -
P0c / E1+E2 hop & seams: Chunk seams and PCM hop are driven by
APP_CONFIG.emage.motion—advanceFrames(60..64),chunkSeamMaxFrames,seamJumpThreshold/seamJumpFramesScale, pose micro-fade knobs. - Physiological Angular Velocity Limiting Transition: Biomechanical angular velocity limits (arms 2.2 rad/s, neck/head 1.6 rad/s, torso 1.2 rad/s) and critical spring damping ensure natural, snap-free transitions.
-
Adaptive Conversational Idle (
SpeakIdleSystem): Characters adaptively respond to the current gesture during inter-segment pauses — high gestures hover with breathing buoyancy and gentle micro-settling (>1.5s); fingers flex along the anatomical Z-axis; awareness gaze drifts and micro-nods eliminate frozen mannequins. -
Live Pipeline Console Table Tracker: Real-time
console.tablemonitors chunk TTS, EMAGE inference, playback progression, and transition modes. -
Head Bubble Progress Indicator: A pulsing progress pill (
🟢 1 / 5) in the bubble status bar cleanly displays segment progress without intruding on dialogue text.
- Text-Only Transfer — Open the chat-bar menu → 跨设备同步 / Cross-Device Sync → pick what to sync (custom providers + active model / thinking mode / chat settings) → an AES-GCM-256 encrypted blob (
xs:v1:iv.ct.key) is generated and put in a copyable text block. The key travels inside the ciphertext — paste once on the other device, no separate key channel needed. - Provider Config Sync — Custom OpenAI-compatible providers (including their AES-encrypted API keys) round-trip across devices via
listProvidersDecrypted()on the sender side andsaveProvider()re-encryption on the receiver side, using each device's local salt. - Preview-Before-Apply — The receiving device decrypts locally, shows exactly what will change (active service / N providers / thinking on-off / chat settings including head-bubble on-off), and only commits after the user clicks 确认导入. Import writes
userSettingsthroughsaveUserSettingsso the in-memory cache andHeadBubblesubscribers update immediately. - Pure Client-Side Architecture — Pure client-side execution with no third-party telemetry and no backend coordination.
- 100% Local Privacy (IndexedDB): Powered by browser-native IndexedDB (
xiaochun-memorydatabase). Dialogue turns, personal preferences, and recalled facts stay entirely on the client device — zero telemetry or chat logs sent to any server. - 3-Tier Memory Architecture:
- Short-Term Turns: Automatically buffers the most recent conversational rounds (default 6 turns, clipped to 180 chars per turn via
APP_CONFIG.memory.shortTermTurns), maintaining dialogue coherence. - Entity Profile Extraction: Multilingual (zh / en / ja) regex smart extraction captures user names/nicknames (e.g., "my name is...", "call me...", "我叫...", "叫我...", "私は..."), likes, dislikes, and entity facts with deduplication.
- Long-Term Notes & n-gram Retrieval: Archives conversational summaries (up to 80 notes); retrieves the most contextually relevant Top-K (default 4) past memories using an n-gram similarity scoring algorithm (
gramScore).
- Short-Term Turns: Automatically buffers the most recent conversational rounds (default 6 turns, clipped to 180 chars per turn via
- Non-blocking Async Persistence & Dynamic Injection:
- Async Commit: Executes
rememberTurn()in the background upon turn completion, never stuttering UI animations or audio playback. - Contextual Prompt Injection:
recallForChat()swiftly retrieves entities and relevant notes, injecting localized memory headers (appendMemoryToSystem) directly into the LLM system prompt so XiaoChun naturally remembers your identity and prior topics while prioritizing current chatter.
- Async Commit: Executes
- TanStack Start full-stack SSR with cookie-based language hydration — no client-side language flash.
- 3 languages shipped: 简体中文 / English / 日本語.
- System prompt — XiaoChun is a companion; she replies in the language of this user message, not the UI language.
- shadcn-style dropdown in the top-right for language switching (Radix UI primitive).
- iOS HIG 44 pt / Material 48 dp touch targets throughout.
- Bottom chat input ≥ 40 px with 16 px text (prevents iOS Safari auto-zoom on focus).
- Tw + tailwindcss-animate micro-interactions; liquid-glass aesthetic.
- Top-right action bar — upload VRM, language, GitHub; the debug drawer only appears on localhost.
- Chat bar — queue-while-loading, model/thinking menu, send.
- Floating speech bubble writes
transformon the DOM with a deadzone, so React does not re-render at 60 FPS. - Mobile — preload stickers stay; the chat bar clears the home-indicator inset.
- Debug drawer (
isDev(): Vite DEV or browser loopback, not Tauri production) — 8 sections in fixed order (🎭 预设表情 → 🎥 镜头设置 → 🎨 画面色彩 → ✨ 骨骼体型 → 🧩 模型部位 → 💡 灯光通道 → 🔮 画面后期 → ⚡ EMAGE Perf). Each section owns its own React state, has its own reset, and shows a "modified" dot only when the user has changed something since mount. Schema-driven render: add a section = one entry inSECTIONS+ one in the componentREGISTRY.PostFxSectionmanages bloom intensity/radius/threshold, tone mapping modes, and color grading matrix;EmagePerfSectionsurfaces wasm_env / isolation / numThreads / stage timings for P0b acceptance screenshots. Seedocs/ARCHITECTURE_AND_RULES.md§3 for the full primitive breakdown. - ChatBar test-speak dropdown (dev) — spring / 蜀道难 presets with live segment counts via
splitIntoSpeechChunks, for end-to-end TTS+EMAGE streaming checks without typing. - Per-frame slider drag: sliders split per-tick (engine) from commit-time (state +
localStorage) so a 28-sliderBoneMorphSectiondoesn't re-render its siblings on every pointer event. Display numbers next to each slider follow the thumb in real time via imperativetextContentwrites fromSliderWithAnchors'liveValueRefmechanism — React reconciliation is bypassed entirely. - Camera section: FOV slider (with hover
ⓘtooltip listing 20°/30°/45°/60° reference values),📷min /🔭max camera distance sliders (mouse wheel + pinch zoom range), and the body-turn toggle. Default push-in distance is computed from FOV viadefaultShotExtentso the framing stays consistent at 15°/20°/60° (no more "long-lens crops the face"). - Bone morph section: 28 sliders grouped into 7 body regions (整体 / 头颈 / 躯干 / 臀部 / 胸部 / 上肢 / 下肢), each as a sub-card; shoulder width defaults to 110% with a maximum range of 250%; search filter hides empty regions.
- Wardrobe section: 3-state per-part rendering —
穿(toggle on) /未穿(toggle off, line-through) /未装配(dashed disabled div, no checkbox). "Equipped" comes fromvrmEngine.materialManager.partMaterials[id]?.length, not the user's visibility toggle. Note: For full wardrobe changes, use the top navigation bar's outfit selector. - Cloudflare Workers (
src/server.ts) — handles full-stack TanStack Start SSR alongside native WebSocket streaming for Edge-TTS; also applies COOP/COEPcredentiallesson HTML/SSR document responses (Workers+Assets ignorespublic/_headersfor documents) so EMAGE ORT wasm can use SAB threads when isolated. - Vite dev middleware (
vite/localApiPlugin.ts) — local development powered by Miniflare runtime for 100% dev/prod parity. ForwardsGET /api/ttsto yourTTS_PROXY_URL(e.g., the deployed Worker) when set, else opens a local native WebSocket to Edge-TTS (src/lib/edge-tts-core.ts). - Single source of truth:
src/config.tsconsolidates lighting / camera / expressions / saturation / LLM / R2 model config / 28 bone morph params.
- EMAGE download 191.35 → 71.59 MB (brotli per file 161.3 → 57.7 MB), with no retraining: slim step (face branch removed, seed computed in JS), INT8 Conv weights, cross-attention layers 0–3 removed, offline graph optimization. Node wasm window 548 → 274 ms (1 thread) and 165 → 79 ms (4 threads); session create in Chrome about 196 → 66 ms (offline graph optimization). Quality is checked at logits/argmax level only (top-1 about 92.5% vs FP32); phones, Safari and Firefox are not measured.
- Parallel model download with pipelined session create (session create stays serial); models load from
cdn.firetable.tech/xiaochun/emage/into theemage-models-v2cache, old caches are purged. @mlc-ai/web-llmis lazy-loaded: the sharedvrmEnginechunk shrinks 6.95 → 1.01 MB and/embedno longer requests web-llm unless chat / eager preload is used.- Opt-in multi-threaded EMAGE in iframes: SDK option
crossOriginIsolated(default off; the host needs COOPsame-origin+ COEPcredentialless/require-corp).
Numbers, methods, what was tried and rejected, and what is not measured: docs/PERFORMANCE.md (Chinese).
Put XiaoChun on your own site with a few lines: a lazy, origin-checked <iframe>, a framework-free SDK, a <xiaochun-avatar> Web Component, React bindings, and speakAudio() for feeding your own audio. Pass audible: false when the host page plays the sound and XiaoChun should only move and lip-sync. For that audio, utterance start fires when the clock starts, after the thinking pose and the first motion window. playbackRate (0.25–3) and volume (0–1) scale host audio and its motion clock; setPlaybackRate() / setVolume() change them while it plays. TTS is unchanged. Since 0.1.35, hostClock: true skips the thinking pose, and syncClock({ time, playing, playbackRate?, seek? }) keeps motion and lips on the page player's clock through pause, seek and buffering (supportsHostClock() tells whether the embed supports it).
npm i @firetable/project-xiaochunAlso built in (all opt-in or sensible defaults, details in the package docs):
- Move / resize the iframe with the mouse:
draggable(left-drag the avatar) andresizable(drag a corner; default min 120×180, max = viewport), driven by the same gesture state machine as the desktop app (src/core/gesture); long-press / Cmd / Ctrl still do the 3D turn. RuntimesetDraggable/setResizable/setSize/getBox. - Rounded corners like the desktop window: opaque scenes get a 20 px radius by default (
borderRadiusoption,setBorderRadius(),border-radiusattribute,--xc-radius). - The iframe remembers the outfit and scene itself (
xiaochun_wearing_outfit/xiaochun_scene_themein its own localStorage); priority is URL / SDK options > saved > default;persistoptionally mirrors them in the host page.
👉 Package docs: packages/project-xiaochun/README.md · Design & protocol: docs/EMBED.md
| Layer | Technology | Description |
|---|---|---|
| 3D Core Engine | three.js 0.185 + @pixiv/three-vrm 3.5 | Decoupled modular core (Dual-Theme LineworkWorld, Studio 3-Channel Lighting, VRMMaterialManager, GazeController, BubbleTracker) |
| Post-Processing | Custom PostFx Pipeline (postFxPipeline.ts) |
UnrealBloomPass (background luminance bypass) + ToneMapping (ACESFilmic, etc.) + BC/HS Color Grading |
| Wardrobe & Delta Addons | Delta Addons + WASM bspatch + Pose Pre-restore | .vrmbase + 5 .vrmaddon files, Web Worker patch assembly, 2-tier IDB cache, zero-frame T-pose swapping |
| Motion Pipeline | Custom Layered Universal Pipeline (MotionPipeline) |
Quintic smootherstep inbetweening, bone masking, biomechanical limits, inverse quaternion decoupling & T-Pose protection |
| App Framework | React 19 + TanStack Start | Full-stack SSR with cookie-based i18n hydration |
| Router | TanStack Router | Type-safe file-based routing |
| LLM | WebLLM + custom OpenAI-compatible providers (Ollama / LM Studio / vLLM / cloud) | MiniCPM5 2B q4f16_1 on WebGPU (fallback Qwen2.5 0.5B), streaming; unified runChat(opts) factory pattern |
| Memory | IndexedDB + Custom 3-Tier Pipeline | 100% client-side multi-tier persistence, entity extraction & n-gram note retrieval |
| Motion | EMAGE + ONNX Runtime Web | Dedicated Worker, wasm EP + INT8 (no WebGPU; int64); streaming motion_chunk T=64 |
| TTS | Native WebSocket client (src/lib/edge-tts-core.ts) |
XiaoyiNeural zh-CN +10 Hz, emoji-stripped text; no third-party TTS SDK |
| Edge Runtime | Cloudflare Workers + @cloudflare/vite-plugin | SSR streaming + WebSocket Edge-TTS + Static Assets |
| Object Storage | Cloudflare R2 | EMAGE ONNX models served from https://cdn.firetable.tech/xiaochun/emage/ (the 71.59 MB Step 3 INT8 set; older FP32 504 MB / INT8 ~195 MB files stay at the parent path for older builds); browser downloads via the useInt8 toggle and APP_CONFIG.emage.base in src/config.ts |
| Styling | Tailwind CSS 4 + tailwindcss-animate |
liquid-glass aesthetic, mobile-first |
| i18n | i18next + react-i18next | 3 languages, SSR-hydrated |
| UI Primitives | Radix UI (Dropdown Menu, Slot) | shadcn-style components |
| Language | TypeScript 6 | Strict typing end-to-end |
| Build | Vite 8 + @cloudflare/vite-plugin |
Automated artifact slimming for lean edge bundles |
Requirements: Node.js 18+, package manager: pnpm (enforced via preinstall hook).
# 1. Install dependencies
pnpm install
# 2. (Optional) Configure environment — copy .env.example to .env.local and edit
cp .env.example .env.local
# - VITE_EMAGE_BASE / VITE_EMAGE_BASE_PROD: ONNX model base URLs
# - TTS_PROXY_URL: Forward Edge-TTS to your Cloudflare Worker (auto-appends /api/tts)
# 3. Start dev server (SSR + Miniflare local edge runtime)
pnpm dev # → http://localhost:5185
# 4. Production build & preview
pnpm build # outputs dist/client & dist/server (Worker)
pnpm preview # preview production Worker behavior locallyOpen the in-app Custom Model Service dialog (chat-bar menu → provider icon, or top-bar gear). Pick a template (Ollama / LM Studio / vLLM / LocalAI / Generic OpenAI), fill the base URL and (optionally) an API key, hit Test connection, then Save & enable. Active providers appear with a folded-corner checkmark in the list. Switching to a custom provider automatically unloads WebLLM to free 1-2 GB VRAM.
This project uses Cloudflare's modern Workers + Static Assets architecture (replacing legacy Pages) for lightning-fast SSR hydration and duplex WebSocket TTS streaming.
# Builds production bundle and deploys directly to Cloudflare edge
pnpm deploy
# Equivalent to: pnpm build && wrangler deployIf not logged in, Wrangler will automatically open your browser for OAuth authentication. Once deployed, attach your custom domain (e.g. xiaochun.firetable.tech) under Worker -> Domains & Routes.
To deploy automatically on every git push:
- Log in to Cloudflare Dashboard -> navigate to Workers & Pages -> Create Application -> Workers tab;
- Select Connect to Git and connect this repository;
- Build settings (
⚠️ Critical):- Root directory: Leave completely blank (do NOT enter
/) — entering/points the runner to the Linux root filesystem and causes builds to hang; - Build command:
pnpm build - Deploy command:
npx wrangler deploy
- Root directory: Leave completely blank (do NOT enter
- Pushing new commits to
mainwill now trigger automatic build and deployment.
Project-XiaoChun/
├── .github/ # Continuous Integration & Delivery
│ └── workflows/
│ └── release-tauri.yml # Multi-platform (macOS/Win/Linux) matrix build & GitHub release workflow
├── public/ # Static assets (hosted via Cloudflare Workers Assets)
│ ├── vrm/ # Modular VRM base and delta clothing addons
│ │ ├── xiaochun_base.vrmbase # Stripped base model (~5.9 MB, oxipng compressed textures)
│ │ ├── xiaochun_default.vrmaddon # Default Techwear outfit
│ │ ├── xiaochun_cheongsam.vrmaddon # Traditional Cheongsam
│ │ ├── xiaochun_bikini.vrmaddon # Beach Bikini
│ │ ├── xiaochun_maid.vrmaddon # Classic Maid dress
│ │ ├── xiaochun_swimsuit.vrmaddon # Summer Swimsuit
│ │ ├── xiaochun_dinner_dress.vrmaddon # Dinner Dress
│ │ ├── xiaochun_office_lady.vrmaddon # Office Lady
│ │ ├── xiaochun_techwear.vrmaddon # Techwear
│ │ ├── xiaochun_wedding.vrmaddon # Wedding Dress (13 MB)
│ │ └── xiaochun_shroud.vrmaddon # Bandage Wrap (with per-outfit bodyMorph baseline)
│ ├── thinking.vrma # Idle thinking animation loop
│ ├── materials/ # Loading screen key visual (xiaochun_keyvisual*.webp)
│ ├── onnx/ # EMAGE body motion model weights (vq_*, emage_step, postprocess)
│ ├── _headers # Cache-Control and security headers
│ ├── robots.txt / sitemap.* # Search engine crawler contracts
│ ├── llms.txt / llms-full.* # AI agent & LLM GEO documentation specs
│ ├── og.jpg # OpenGraph share card
│ └── logo.png / favicon.* # Brand and icon assets
├── src-tauri/ # Tauri 2.x native desktop hybrid application layer
│ ├── src/lib.rs # 60Hz alpha bitmask click-through, single-instance IPC, deep-link routing
│ ├── capabilities/ # Desktop permission capability declarations
│ └── tauri.conf.json # Desktop window, transparent compositing & URL scheme configs
├── docs/ # Architecture and module deep-dive docs
│ ├── README.md # Documentation index and agent routing guide
│ ├── INTERACTION_AND_CONTROLS.md # User interaction, shortcuts, mouse gestures & platform differences
│ ├── HYBRID_DESKTOP_APP.md # Desktop companion mode, click-through, and Tauri 2.x setup
│ ├── PROTOCOL.md # Inter-process protocol (xiaochun://) specification & actions
│ ├── AGENT_SKILL.md # Autonomous AI Agent skill specification for companion control
│ ├── INTERACTION_AND_3D_GUIDES.md # 3D holographic guides, wind dynamics & quaternion fixes
│ ├── ARCHITECTURE_AND_RULES.md # System architecture, 10-step lifecycle & developer rules
│ ├── OUTFIT_SWAP.md # 0-frame T-pose state snapshot, delta addons & IDB caching
│ ├── POSTFX.md # UnrealBloomPass halos, background bypass & tone grading
│ ├── VRM_BUILD_WORKFLOW.md # VRM build toolchain, bspatch delta splitting & byte idempotency
│ ├── VRM_ENGINE_AND_WORKER.md # VRMEngine facade coordinator & vrmWorker binary synthesis
│ ├── MOTION_PIPELINE.md # Universal motion blending & quintic smootherstep inbetweening
│ ├── BODY_TURN_AND_GAZE.md # Stepping locomotion state machine & saccadic gaze
│ ├── BONE_MORPH.md # 28-parameter orthogonal biomechanical morphing whitepaper
│ ├── FOOT_IK.md # Ground anchoring & foot inverse kinematics solver
│ ├── CHAT_DIRECTOR.md # LLM + TTS + EMAGE streaming orchestration
│ ├── TTS.md # Edge-TTS proxy & Tone.js vocal post-processing pipeline
│ ├── STT.md # SenseVoice ChatBar dictation (VAD + ORT Worker)
│ ├── ONBOARDING.md # Galgame-style immersive onboarding system specification & schema
│ ├── EMAGE_MODEL.md # EMAGE status & known limits (wasm/INT8, streaming, isolation)
│ └── PERFORMANCE.md # Performance & size notes: measured numbers, rejected options, unmeasured items (Chinese)
├── packages/ # pnpm workspace packages
│ ├── project-xiaochun/ # Embed SDK (@firetable/project-xiaochun, published to npm)
│ ├── three-vrm-materials-mtoon/ # MToon fork with NPR shading controls
│ └── emage-onnx/ # Private offline toolchain: slim, quantize and optimize the EMAGE ONNX set, download 191 → 72 MB (not published)
├── scripts/ # Build, offline asset processing, and protocol runners
│ ├── build-vrm/ # VRM delta packaging and texture optimization (workflow.mjs)
│ └── protocol-call.mjs # Fast CLI runner for protocol speech testing
├── wrangler.jsonc # Cloudflare Workers declarative configuration
├── src/
│ ├── routes/ # TanStack Start file-based routes
│ │ ├── __root.tsx # Root layout (i18n SSR hydration, GEO JSON-LD & meta tags)
│ │ └── index.tsx # Main index route
│ ├── components/ # React UI components
│ │ ├── onboarding/ # Immersive Galgame-style onboarding tour
│ │ │ ├── OnboardingTour.tsx # Tour orchestrator, camera checklist gating (4 tasks web / 5 on Tauri) & reply cards
│ │ │ └── onboardingSchema.ts # Finite-state machine schema, speech lines & branching options
│ │ ├── TauriWindowFrame.tsx # Native desktop frameless resize handles & corner glow
│ │ ├── TauriTopHeader.tsx # Desktop ⋯ menu (updates / reload / quit)
│ │ ├── TopHeader.tsx # Responsive top navigation (scene / wardrobe / lang / settings)
│ │ ├── ChatBar.tsx # Streaming chat interface & model switcher
│ │ ├── HeadBubble.tsx # 3D projected speech dialogue bubbles
│ │ ├── dev-drawer/ # Schema-driven 8-section debug drawer (localhost only)
│ │ ├── SliderWithAnchors.tsx # Optimized slider with onTick / onCommit split
│ │ └── ui/ # Radix UI primitives
│ ├── hooks/ # Reusable UI & platform hooks
│ │ ├── usePetUiVisibility.ts # Click-to-toggle desk-pet UI; hover/dialog holds auto-hide
│ │ └── useDeferredUnmount.ts # Optional delayed unmount (pet chrome uses opacity instead)
│ ├── core/ # 3D rendering & scene core (Decoupled Facade architecture)
│ │ ├── vrmEngine.ts # Central engine coordinator (slim Facade, render loop, VRM loading)
│ │ ├── protocol/ # Inter-process protocol layer (types.ts, handler.ts, index.ts)
│ │ ├── interaction/ # 3D holographic controls (interactionController, turnGuide, pitchGuide, cameraYGuide)
│ │ ├── wind/ # Aerodynamic localized cursor wind forces (windForce.ts)
│ │ ├── scene/ # Scene manager, lineworkWorld, beach3d, voidWorld, and passthroughManager
│ │ ├── lighting/ # 3-channel studio lighting (studioLighting.ts: dir / hemi / fill)
│ │ ├── postfx/ # Post-processing pipeline (postFxPipeline.ts: Bloom/ToneMapping/ColorGrading)
│ │ ├── material/ # MToon material management (vrmMaterialManager.ts: saturation injection)
│ │ ├── morph/ # 28-parameter bone-by-bone scale bindings (vrmBodyMorph.ts)
│ │ ├── outfitSwap.ts # Atomic outfit swapping (pose snapshot capture & in-memory pre-restore)
│ │ ├── vrmWorker.ts # Background worker (WASM bspatch assembly & 2-tier IDB cache)
│ │ └── ui/ # Spatial UI tracker (bubbleTracker.ts: raw crown → 2D bubble)
│ ├── motion/ # Motion system (plugin pipeline + bio-inspired posture)
│ │ ├── pipeline/ # tick / selectLiveMotionSource / PoseBuffer / transition
│ │ ├── sources/ # idle, vrma, emage (+ worker), clip, speakIdle
│ │ ├── constraints/ # footIK, bodyTurn, gaze (on draft, then compose)
│ │ └── vrmaRetarget.ts # VRMA bone retargeting & normalization
│ ├── memory/ # Client-side multi-tier memory (IndexedDB / entity extraction / n-gram retrieval)
│ ├── llm/ # LLM layer — WebLLM + custom OpenAI-compatible provider factory
│ ├── director/
│ │ ├── chatDirector.ts # LLM → TTS → EMAGE streaming coordinator pipeline
│ │ └── vocalProcessor.ts # Tone.js vocal post-processing pipeline & 3D spatial acoustics
│ ├── i18n/ # zh-CN / en / ja translation dictionaries + server cookie helper
│ ├── lib/ # Cross-cutting utilities
│ │ ├── platform.ts # Desktop/Tauri/Browser environment detection & window actions
│ │ ├── utils.ts # Shared class merge and speech chunk slicer
│ │ ├── constants.ts
│ │ └── edge-tts-core.ts # Native Edge-TTS WebSocket client (no third-party TTS SDK)
│ ├── styles/
│ │ └── main.css # Tailwind v4 @theme tokens + liquid-glass styles
│ ├── App.tsx # Main application component & event bindings
│ ├── client.tsx # Client hydration entry
│ ├── server.ts # Cloudflare Worker entry (SSR + /api/tts WS + COOP/COEP isolation headers)
│ ├── router.tsx # TanStack Router factory
│ ├── routeTree.gen.ts # Auto-generated type-safe route tree
│ └── config.ts # SSOT (R2 / camera / lights / postfx / LLM / bodyMorph / emage.models + emage.motion)
├── vite.config.ts # Vite 8 + TanStack Start + @cloudflare/vite-plugin
├── vite/ # Custom vite plugins (extracted from vite.config.ts)
│ ├── localApiPlugin.ts # /api/tts dev middleware (TTS proxy with EU fallback)
│ └── dropDockerfatAssets.ts # Strip 25 MiB+ WASM blobs for Cloudflare Pages limit
├── AGENTS.md # Coding agent engineering standards (motion pipeline + T-Pose / LookAt / state-machine pitfalls)
├── CLAUDE.md # Claude-specific runtime context
├── README.md / README-CN.md # This file (English / 中文)
├── .env.example # Documented template for all env vars (copy to .env.local)
└── tsconfig.json
Language is stored in a lang cookie and resolved during SSR via @tanstack/react-start/server's getCookie() — wrapped in createServerOnlyFn to keep it out of the client bundle. On the client, the same lang cookie is read via document.cookie, ensuring the SSR-rendered HTML matches the first client paint with zero hydration mismatch.
Switching language:
- Updates the
langcookie. - Calls
i18n.changeLanguage(). - Re-renders all
t(...)consumers without reloading.
This project is licensed under the MIT License.
Third-party assets used by the default scene:
public/vrm/xiaochun_base.vrmbaseand.vrmaddonbundles — VRM character model and wardrobe addons, generated with VRoid Studio (Pixiv Inc.). Subject to the VRoid Studio License — free for personal use, modification, and non-commercial redistribution with attribution. For commercial use, please review the VRoid Studio License terms or contact Pixiv Inc. for a separate agreement.public/thinking.vrma— VRM animation. License unknown — verify before redistribution.
/api/tts in src/server.ts (and the Vite middleware in vite/localApiPlugin.ts) uses a hand-rolled native WebSocket client (src/lib/edge-tts-core.ts) to talk to Microsoft's Edge-TTS service directly — no third-party TTS SDK. The TRUSTED_CLIENT_TOKEN constant is a publicly known shared token (the same value used by every open-source Edge-TTS implementation) and is not a personal secret.
- pixiv / three-vrm — VRM runtime
- Pixiv / VRoid Studio — default character authoring
- MLC AI / WebLLM — in-browser LLM
- TanStack Start — full-stack React framework
- Tailwind CSS — utility-first styling
- Qiuner / Qiuner.github.io (
src/worlds/linework/) — linework outdoor scene visual inspiration - Animation Inc. — First tried their Ani-2 product and got hooked on the idea of real-time on-device full-body motion synthesis; this project's EMAGE pipeline is our homegrown attempt at the same dream, running entirely in the browser
- PantoMatrix / EMAGE by Haiyang Liu*, Zihao Zhu*, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Naoya Iwamoto, Bo Zheng and Michael J. Black (*equal contribution), "EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling", CVPR 2024; weights: H-Liu1997/emage_audio (Apache-2.0 per the model card). It is the full-body co-speech motion model that powers our chat-time gesture generation. This project modifies the weights (ONNX export, cross-attention layers removed, INT8 quantization, graph optimization), see
packages/emage-onnx/NOTICE. - VolgaGerm / emage-onnx-export — the PyTorch → ONNX export script and the pre-converted
.onnxweights we run in-browser - snomiao / otoji (otoji.org) — open on-device voice graph; our ChatBar SenseVoice dictation (energy VAD + Worker) learned a great deal from their
?simplepath and sherpa-onnx wiring









