An open-source streaming MoE inference framework
Think big. Run here.
Sparse. By design.
macOS · iOS · Android · Linux · Windows
One program for every Edge0 model
Project
Latest release
Old release
English · 中文 · 日本語 · Español · Français
News · About · Models · Quality · Benchmark · Getting started
More updates
Full release notes
- [2026-11-13] We released
edge0-145bandedge0-120b, latent mixture-of-experts models derived from Qwen3.8-Flash-Next for a single NVIDIA GPU on Linux or Windows. They run on edge0-strata, whichedge0-infer serveandedge0-infer chatstart for them; see the model page. Weights on Hugging Face:Edge0-145B-q3k·Edge0-145B-NVFP4·Edge0-120B-q3k·Edge0-120B-NVFP4. - [2026-11-04] We released the unified inference engine
edge0-infer: built on GGML, one engine and one native C++ API run the same model on macOS, iOS, Android and Windows, with a CLI, an OpenAI-compatible local server, and chat app examples for desktop, Android and iOS. The first release supportsedge0-35b;edge0-8bfollows in a later release. It replaces the Python framework and the per-platform engines; the previous code is kept at tagpre-edge0-infer. - [2026-09-30] We released the edge0 inference engines for four platforms — iOS, macOS, Android and Windows (source at tag
pre-edge0-infer). - [2026-09-16] Our technical report is on arXiv: The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction.
- [2026-09-08] Initial open-source release of edge0, together with both model tiers —
Edge0-35B-A3B-previewandEdge0-8B-A1B-preview— on Hugging Face and ModelScope.
→ 35B / 8B release guide — model downloads, architecture, benchmarks, supported platforms and setup instructions.
Edge0 is an open-source streaming MoE inference framework for running large sparse models on consumer hardware.
Use it through the command line, a local API server, or the Edge0 Chat app.
The latest release brings 145B and 120B latent-MoE models to a single NVIDIA GPU.
Trained natively at 4 bits, both models are available in Q3_K and NVFP4 for local agent workloads.
edge0-infer provides shared serve and chat commands and selects the engine from the model folder. The latest models run on edge0-strata, a fork of Strata; the previous 35B release runs on the GGML-based native engine.
Support depends on the model and its engine.
| Platform | edge0-145b, edge0-120b (edge0-strata) |
edge0-35b (native engine) |
|---|---|---|
| Linux | NVIDIA GPU (CUDA 13) | CPU |
| Windows | NVIDIA GPU (CUDA 13) | CPU, Vulkan |
| macOS (Apple Silicon) | — | Metal, CPU |
| iOS | — | Metal |
| Android | — | CPU |
Use the 145B / 120B setup for the latest release. The previous-release guide covers native backends, platform-specific builds and chat examples. The 8B checkpoint is available; native-engine support is planned.
145B / 120B are the current latent-MoE series, derived from Qwen3.8-Flash-Next and trained natively at 4 bits.
They run on edge0-strata through edge0-infer serve and edge0-infer chat, on Linux or Windows with an NVIDIA GPU and CUDA 13.
145B Q3_K · RTX 5090 D 24 GB: 121.2 tok/s agent decode · 3554.9 tok/s 16K prefill.
| Tier | Released checkpoint | Inference profile | Engine |
|---|---|---|---|
edge0-145b |
Edge0/Edge0-145B-q3k · NVFP4 |
Q3_K or NVFP4 latent experts (1792), 48 layers, 512 experts, K=10 | edge0-strata ✅ (Linux or Windows, NVIDIA GPU) |
edge0-120b |
Edge0/Edge0-120B-q3k · NVFP4 |
Q3_K or NVFP4 latent experts (1280), 48 layers, 512 experts, K=10 | edge0-strata ✅ (Linux or Windows, NVIDIA GPU) |
Q3_K · RTX 5090 D 24 GB. Prefill uses a 16K-token input. Quality scores are model-level results.
Six benchmarks, six models. Higher is better. Scores are reported by model, not as separate Q3_K and NVFP4 evaluations.
Copyable scores and all model names
Copyable scores · all six models
Bold values indicate the best score in each column.
| Model | deepswe 1.1 | terminal-bench 2.1 | AA-LCR | IFBench | GPQA-Diamond | HLE |
|---|---|---|---|---|---|---|
| Edge0-145B | 64.6 | 86.5 | 79.3 | 80.6 | 90.4 | 35.4 |
| Edge0-120B | 51.3 | 75.3 | 81.0 | 79.3 | 87.4 | 31.0 |
| DeepSeek-V4-Flash-0731 | 54.4 | 82.7 | 79.7 | 79.2 | 90.8 | 33.8 |
| Qwen3.8-Flash-Next | 58.7 | 83.3 | 79.1 | 81.3 | 91.7 | 35.9 |
| Qwen3.8-27B | 42.2 | 73.0 | 82.0 | 79.5 | 89.2 | 30.8 |
| Claude-Opus-4.6 (Max) | / | 78.2 | 78.0 | 62.5 | 91.3 | 40.0 |
All throughput figures are in tok/s. The headline uses 145B Q3_K on RTX 5090 D 24 GB.
145B speed table · all machines and builds
Highlighted rows show Q3_K. Click the table to view it at full size.
Copyable 145B speed data
| Machine | Build | 4K prefill | 16K prefill | Chat decode | Agent decode |
|---|---|---|---|---|---|
| RTX 5090 D 24 GB | Q3_K | 2169.9 | 3554.9 | 99.7 | 121.2 |
| RTX 5090 D 24 GB | NVFP4 | 1839.6 | 3165.7 | 79.6 | 98.3 |
| RTX 4090 24 GB | Q3_K | 2042.0 | 3179.4 | 69.9 | 91.7 |
| RTX Spark N1X 60 GiB | Q3_K | 674.0 | 770.3 | 37.7 | 36.7 |
| RTX Spark N1X 60 GiB | NVFP4 | 821.2 | 922.4 | 40.1 | 46.0 |
| Apple M2 Ultra (Metal) | Q3_K | 755.6 | 800.3 | 60.9 | 57.0 |
| 16 GiB VRAM + 20 GiB RAM | Q3_K | 249 | 335 | 36.8 | 44.9 |
145B measurement conditions · constrained budget and memory residency
16 GiB VRAM + 20 GiB RAM rows are the RTX 5090 D 24 GB again, constrained by cgroup to a 16 GiB VRAM and
20 GiB system-RAM budget — not a different card. With residency on, the same card holds Q3_K (40.81 GiB of weights)
as 23.14 GiB VRAM + 31.28 GiB RAM, and NVFP4 (47.79 GiB) as 23.25 GiB VRAM + 38.27 GiB RAM — disjoint expert sets,
nothing counted twice.
The 16 GiB VRAM + 20 GiB RAM budget is a test limit, not a minimum system requirement. The Apple M2 Ultra row is a Metal measurement, not an announcement of macOS release availability.
120B speed table · all machines and builds
Copyable 120B speed data
| Machine | Build | 4K prefill | 16K prefill | Chat decode | Agent decode |
|---|---|---|---|---|---|
| RTX 5090 D 24 GB | Q3_K | 3236.7 | 4576.5 | 130.2 | 144.6 |
| RTX 5090 D 24 GB | NVFP4 | 2668.5 | 4357.1 | 124.9 | 135.5 |
| RTX 4090 24 GB | Q3_K | 2637.5 | 3551.1 | 95.0 | 104.9 |
| RTX Spark N1X 60 GiB | Q3_K | 987.0 | 1081.7 | 47.2 | 68.4 |
| RTX Spark N1X 60 GiB | NVFP4 | 996.4 | 1066.9 | 45.1 | 68.9 |
| Apple M2 Ultra (Metal) | Q3_K | 867.6 | 908.7 | 60.4 | 63.0 |
| 16 GiB VRAM + 20 GiB RAM | Q3_K | 430 | 778 | 62.4 | 73.3 |
| 16 GiB VRAM + 20 GiB RAM | NVFP4 | 361.1 | 619.7 | 51.3 | 59.9 |
16 GiB VRAM + 20 GiB RAM rows are the RTX 5090 D 24 GB again, constrained by cgroup to a 16 GiB VRAM and
20 GiB system-RAM budget — not a different card. With residency on, the same card holds Q3_K (27.98 GiB of weights)
as 22.93 GiB VRAM + 18.40 GiB RAM, and NVFP4 (32.96 GiB) as 23.02 GiB VRAM + 23.40 GiB RAM — disjoint expert sets,
nothing counted twice.
Recorded with Edge0-145B · Open video ↗
Five demonstrations · chapter guide
| Time | Demonstration |
|---|---|
| 0:00 | Local Chat & Coding |
| 0:36 | Video Production |
| 0:58 | Manhattan in 3D |
| 1:19 | Cathedral World Building |
| 1:38 | PCB Design |
Latent MoE compresses expert inputs while keeping attention at full width. The 145B model uses latent width 1792; the 120B model uses 1280. Both route each token to 10 of 512 experts, alongside one shared expert.
Architecture details · diagram and configuration
Each model folder (Q3_K or NVFP4, Edge0-145B or Edge0-120B) runs through edge0-infer, from the command line or from the Edge0 Chat desktop app. Keep the folder as it is:
edge0-infer identifies the model by model_name in its model.json.
For 145B and 120B, use serve or chat; run belongs to the native engine.
Platform & documentation notes
The build instructions target NVIDIA GPUs with CUDA 13 on Linux or Windows (x64 or ARM64). The Apple M2 Ultra measurements use Metal; they do not establish macOS release availability.
Build & command-line instructions
1. Build. Clone the repository, then build two parts. Run the launch commands below from edge0-infer/:
git clone https://github.com/Edge0-AI/Edge0.git edge0- edge0-strata, the engine: follow "Build (Linux, NVIDIA)" or "Build (Windows, NVIDIA)" in edge0-strata/EDGE0.md.
- edge0-infer: follow "Build" (Linux) or "Windows" in edge0-infer/docs/getting-started.md.
2. Run from the command line. Pass one model folder, such as Edge0-145B-q3k, not the folder that holds several of
them. On Windows the program is build\windows\bin\edge0-infer.exe.
build/cpu/bin/edge0-infer serve --model /path/to/Edge0-145B-q3kIt prints Listening: http://127.0.0.1:8000 model=<name> once the weights are loaded, where <name> is edge0_145b,
and then serves OpenAI- and Anthropic-compatible endpoints:
curl localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '{"model":"edge0_145b","messages":[{"role":"user","content":"hi"}],"max_tokens":64}'edge0-infer chat --model /path/to/Edge0-145B-q3k opens an interactive session instead. More options:
edge0-infer/docs/edge0-145b.md.
Desktop app · Windows
⬇️ Download: GitHub Releases
Open Edge0 Chat on Windows and choose a model folder, such as Edge0-145B-q3k, to run the model locally. To build the app from source, follow the Edge0 Chat guide.
Latest release · 145B / 120B
- Models, measurements and architecture
- edge0-strata · build and launch contract
- edge0-infer model guides: 145B / 120B
Framework
- edge0-infer · command line, local server and chat applications
- Local server / Testing
- Model asset tools and adding a model
Previous release · models and native-engine documentation
- 35B / 8B release guide
- MoE routing and experts / prerouter
- Model references: edge0-35b / edge0-8b
- Getting started: native build, model assets, run, chat and serve across supported platforms
- Configuration / Native API
- Architecture / Native-engine design / Known limitations
- Benchmarking
- Technical report: The Other Half of the Memory Wall (PDF)
Q4 2026 · architecture and inference
Models & algorithms
Q4 works two fronts: bringing a next-generation architecture into the framework, and turning latent reasoning into a real latency saving rather than just an arithmetic one.
- Next-gen architecture support (Qwen3.8-Flash class) — run hybrid linear attention (GDN + QSA), gated multi-branch residual, and N-gram embedding topologies on edge0. These designs suit SSD streaming offload naturally: O(1)-state attention keeps long thinking from becoming a KV-cache problem, and lookup-only N-gram tables stream on demand. Goal: the tier runs on a single device and benchmarks within an acceptable gap of the fp16 base.
- Latent thinking + batched expert pre-prediction — make latent reasoning a latency saving, not only an arithmetic one. The core engineering problem: move expert routing from per-position to once per block, so one prediction covers every position and round of a block and expert load volume decouples from the reasoning loop count — plus cross-block prefetch that loads the next block's experts inside the current block's compute window. Progress is measured as end-to-end thinking-phase time at matched accuracy (never tokens/s).
Previous-release milestones and planned native-engine support.
Development workflow and checks
Contributions are welcome — issues, PRs, benchmark reports and model ports all count.
Building and testing are described in Testing; the checks that need no model assets:
cd edge0-infer
cmake --preset cpu && cmake --build --preset cpu
ctest --preset cpu -L "L0|L1"For changes under edge0-infer/, open issues and PRs in this repository
as usual; the directory is developed in a separate repository, and an
accepted change is applied there first by the maintainers and arrives here
with the next sync. See CONTRIBUTING.
Workflow: fork → feature branch → PR against main. Please keep the
tests passing and add tests for new behavior.
The technical report covers 35B SSD streaming and trained routing prediction.
If you find edge0 useful, please cite our technical report:
@article{lin2026other,
title = {The Other Half of the Memory Wall: Serving 35B MoEs from SSD
with Trained Routing Prediction},
author = {Lin, Yu and Wang, Yiming and Cai, Runyuan and Liu, Hanze and
Zeng, Xiaodong},
journal = {arXiv preprint arXiv:2609.18063},
year = {2026},
url = {https://arxiv.org/abs/2609.18063}
}- Discord: edge0.ai/discord — community chat and support
- Email: samuel@edge0.ai
For bugs and feature requests, please use GitHub Issues.
The framework code remains under Apache-2.0. Bundled components retain their own licenses and notices, including the MIT license for Strata.
Edge0-specific model contributions are licensed under the Edge0 Community Model License 1.0, which applies only to contributions that Edge0 owns or is authorized to license. Prior valid licenses remain unchanged.
Subject to applicable upstream terms, the license permits personal use, research, education, modification, noncommercial sharing, internal use and commercial use of generated outputs. It does not require disclosure of private modifications or an Edge0 interface badge.
Separate written Edge0 permission is required for:
- Model sales and paid distribution, including covered adapters, deltas and commercial bundles whose principal value is model access.
- Commercial model services offering inference or fine-tuning, funded by fees, subscriptions, advertising or related sponsorship.
- Independent commercial coding or office assistants, whether inference runs on a server or on the user's device.
See the full license for definitions, exceptions and redistribution requirements. Commercial licensing: info@edge0.ai.
Qwen materials remain governed by their applicable upstream license. The retained Qwen Community License 1.0 is reproduced unchanged; Edge0 permission does not waive Qwen's attribution or separate commercial authorization requirements.
See the model NOTICE for attribution and licensing scope. Framework and third-party code licenses are unaffected.
Acknowledgements · model lineage and engine credits
We thank the teams behind Qwen3.8-Flash-Next and Strata. Edge0 adds latent MoE support to the Strata engine. edge0-strata is a fork of Strata by Niko1221 and the Strata contributors and retains its MIT license.
The Edge0 framework code is released under Apache-2.0; bundled third-party components keep their own licenses (see NOTICE). The earlier releases retain their Qwen3.6 and Ling model lineage.