diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index cc899a94..8154c839 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -21,7 +21,7 @@ platform/backend work.
| Path | What |
|---|---|
| [`edge0-infer/`](edge0-infer/) | The program you run: the native inference engine (C++, GGML), its CLI and local OpenAI- and Anthropic-compatible server, the model asset tools, and the chat app examples for desktop, Android and iOS |
-| [`edge0-strata/`](edge0-strata/EDGE0.md) | The engine for `edge0-145b` and `edge0-120b` (NVIDIA GPU, Linux or Windows), a fork of Strata that `edge0-infer` starts for these models |
+| [`edge0-strata/`](edge0-strata/EDGE0.md) | The engine for `edge0-145b` and `edge0-120b` (NVIDIA GPU on Linux or Windows, Metal on macOS), a fork of Strata that `edge0-infer` starts for these models |
| [`docs/`](docs/) | Model design docs: MoE routing, the prerouter, and the model tiers |
| [`paper/`](paper/) | The technical report |
diff --git a/README.md b/README.md
index 963832ce..c09868e7 100644
--- a/README.md
+++ b/README.md
@@ -14,7 +14,7 @@
Project
-
+
Latest release
@@ -74,7 +74,7 @@
Use it through the command line, a local API server, or the Edge0 Chat app.
-The latest release brings 145B and 120B latent-MoE models to a single NVIDIA GPU.
+The latest release brings 145B and 120B latent-MoE models to a single NVIDIA GPU or an Apple Silicon Mac.
Trained natively at 4 bits, both models are available in Q3_K and NVFP4 for local agent workloads.
@@ -88,7 +88,7 @@ Support depends on the model and its engine.
| --- | --- | --- |
| **Linux** | NVIDIA GPU (CUDA 13) | CPU |
| **Windows** | NVIDIA GPU (CUDA 13) | CPU, Vulkan |
-| **macOS** (Apple Silicon) | — | Metal, CPU |
+| **macOS** (Apple Silicon) | Metal | Metal, CPU |
| **iOS** | — | Metal |
| **Android** | — | CPU |
@@ -98,14 +98,14 @@ Use the [145B / 120B setup](#getting-started) for the latest release. The [previ
145B / 120B are the current latent-MoE series, derived from Qwen3.8-Flash-Next and trained natively at 4 bits.
-They run on edge0-strata through `edge0-infer serve` and `edge0-infer chat`, on Linux or Windows with an NVIDIA GPU and CUDA 13.
+They run on edge0-strata through `edge0-infer serve` and `edge0-infer chat`, on Linux or Windows with an NVIDIA GPU and CUDA 13, or on macOS with Metal on Apple Silicon.
**145B Q3_K · RTX 5090 D 24 GB:** 121.2 tok/s agent decode · 3554.9 tok/s 16K prefill.
| Tier | Released checkpoint | Inference profile | Engine |
|---|---|---|---|
-| [`edge0-145b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-145B-q3k`](https://huggingface.co/Edge0/Edge0-145B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-145B-NVFP4) | Q3_K or NVFP4 latent experts (1792), 48 layers, 512 experts, K=10 | edge0-strata ✅ (Linux or Windows, NVIDIA GPU) |
-| [`edge0-120b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-120B-q3k`](https://huggingface.co/Edge0/Edge0-120B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-120B-NVFP4) | Q3_K or NVFP4 latent experts (1280), 48 layers, 512 experts, K=10 | edge0-strata ✅ (Linux or Windows, NVIDIA GPU) |
+| [`edge0-145b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-145B-q3k`](https://huggingface.co/Edge0/Edge0-145B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-145B-NVFP4) | Q3_K or NVFP4 latent experts (1792), 48 layers, 512 experts, K=10 | edge0-strata ✅ (Linux or Windows, NVIDIA GPU; macOS Apple Silicon, Metal, Q3_K) |
+| [`edge0-120b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-120B-q3k`](https://huggingface.co/Edge0/Edge0-120B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-120B-NVFP4) | Q3_K or NVFP4 latent experts (1280), 48 layers, 512 experts, K=10 | edge0-strata ✅ (Linux or Windows, NVIDIA GPU; macOS Apple Silicon, Metal, Q3_K) |
Model overview · 145B Q3_K
@@ -183,7 +183,7 @@ nothing counted twice.
See [Usage & troubleshooting](#usage-help) for memory-saving options.
-The 16 GiB VRAM + 20 GiB RAM budget is a test limit, not a minimum system requirement. The Apple M2 Ultra row is a Metal measurement, not an announcement of macOS release availability.
+The 16 GiB VRAM + 20 GiB RAM budget is a test limit, not a minimum system requirement. The Apple M2 Ultra row is a Metal measurement.
@@ -268,7 +268,7 @@ For 145B and 120B, use `serve` or `chat`; `run` belongs to the native engine.
Platform & documentation notes
-The build instructions target NVIDIA GPUs with CUDA 13 on Linux or Windows (x64 or ARM64). The Apple M2 Ultra measurements use Metal; they do not establish macOS release availability.
+The build instructions cover NVIDIA GPUs with CUDA 13 on Linux or Windows (x64 or ARM64), and Metal on Apple Silicon macOS.
@@ -281,7 +281,7 @@ The build instructions target NVIDIA GPUs with CUDA 13 on Linux or Windows (x64
git clone https://github.com/Edge0-AI/Edge0.git edge0
```
-- **edge0-strata**, the engine: follow "Build (Linux, NVIDIA)" or "Build (Windows, NVIDIA)" in [edge0-strata/EDGE0.md](edge0-strata/EDGE0.md).
+- **edge0-strata**, the engine: follow "Build (Linux, NVIDIA)", "Build (Windows, NVIDIA)" or "Build (macOS, Apple Silicon)" in [edge0-strata/EDGE0.md](edge0-strata/EDGE0.md).
- **edge0-infer**: follow "Build" (Linux) or "Windows" in [edge0-infer/docs/getting-started.md](edge0-infer/docs/getting-started.md).
**2. Run from the command line.** Pass one model folder, such as `Edge0-145B-q3k`, not the folder that holds several of
@@ -304,11 +304,11 @@ curl localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '
-Desktop app · Windows
+Desktop app · Windows · macOS
⬇️ **Download:** **[GitHub Releases](https://github.com/Edge0-AI/Edge0/releases)**
-Open **Edge0 Chat** on Windows and choose a model folder, such as `Edge0-145B-q3k`, to run the model locally. To build the app from source, follow the [Edge0 Chat](edge0-infer/examples/desktop/README.md) guide.
+Open **Edge0 Chat** on Windows or macOS (Apple Silicon) and choose a model folder, such as `Edge0-145B-q3k`, to run the model locally. To build the app from source, follow the [Edge0 Chat](edge0-infer/examples/desktop/README.md) guide.
diff --git a/README_es.md b/README_es.md
index 0a6cb379..05c9eaec 100644
--- a/README_es.md
+++ b/README_es.md
@@ -14,7 +14,7 @@
Proyecto
-
+
Última versión
@@ -80,7 +80,7 @@
Úsalo desde la línea de comandos, un servidor API local o la aplicación Edge0 Chat.
-La última versión permite ejecutar modelos MoE latentes de 145B y 120B en una sola GPU NVIDIA.
+La última versión permite ejecutar modelos MoE latentes de 145B y 120B en una sola GPU NVIDIA o en un Mac con Apple Silicon.
Ambos se entrenan de forma nativa a 4 bits y se ofrecen en Q3_K y NVFP4 para tareas de agentes locales.
@@ -96,7 +96,7 @@ La compatibilidad depende del modelo y de su motor.
| --- | --- | --- |
| **Linux** | NVIDIA GPU (CUDA 13) | CPU |
| **Windows** | NVIDIA GPU (CUDA 13) | CPU, Vulkan |
-| **macOS** (Apple Silicon) | — | Metal, CPU |
+| **macOS** (Apple Silicon) | Metal | Metal, CPU |
| **iOS** | — | Metal |
| **Android** | — | CPU |
@@ -108,14 +108,14 @@ Para la última versión, sigue la [instalación de 145B / 120B](#getting-starte
145B / 120B es la serie MoE latente actual, derivada de Qwen3.8-Flash-Next y entrenada de forma nativa a 4 bits.
-Se ejecuta en edge0-strata mediante `edge0-infer serve` y `edge0-infer chat`, en Linux o Windows con una GPU NVIDIA y CUDA 13.
+Se ejecuta en edge0-strata mediante `edge0-infer serve` y `edge0-infer chat`, en Linux o Windows con una GPU NVIDIA y CUDA 13, o en macOS con Metal en Apple Silicon.
**145B Q3_K · RTX 5090 D 24 GB:** 121.2 tok/s de decodificación de agente · 3554.9 tok/s de prefill de 16K.
| Tamaño | Checkpoint publicado | Perfil de inferencia | Motor |
|---|---|---|---|
-| [`edge0-145b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-145B-q3k`](https://huggingface.co/Edge0/Edge0-145B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-145B-NVFP4) | Expertos latentes Q3_K o NVFP4 (1792), 48 capas, 512 expertos, K=10 | edge0-strata ✅ (Linux o Windows, GPU NVIDIA) |
-| [`edge0-120b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-120B-q3k`](https://huggingface.co/Edge0/Edge0-120B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-120B-NVFP4) | Expertos latentes Q3_K o NVFP4 (1280), 48 capas, 512 expertos, K=10 | edge0-strata ✅ (Linux o Windows, GPU NVIDIA) |
+| [`edge0-145b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-145B-q3k`](https://huggingface.co/Edge0/Edge0-145B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-145B-NVFP4) | Expertos latentes Q3_K o NVFP4 (1792), 48 capas, 512 expertos, K=10 | edge0-strata ✅ (Linux o Windows, GPU NVIDIA; macOS Apple Silicon, Metal, Q3_K) |
+| [`edge0-120b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-120B-q3k`](https://huggingface.co/Edge0/Edge0-120B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-120B-NVFP4) | Expertos latentes Q3_K o NVFP4 (1280), 48 capas, 512 expertos, K=10 | edge0-strata ✅ (Linux o Windows, GPU NVIDIA; macOS Apple Silicon, Metal, Q3_K) |
@@ -197,7 +197,7 @@ Las filas resaltadas corresponden a Q3_K. Haz clic en la tabla para verla a tama
Las filas `16 GiB VRAM + 20 GiB RAM` usan la misma RTX 5090 D 24 GB, no otra tarjeta. La VRAM se limita con `--vram-reserve-mib 8192`; la RAM del sistema, con un cgroup (`MemoryMax=20G`, sin swap). Con la residencia activada, **Q3_K** (40.81 GiB de pesos) ocupa 23.14 GiB VRAM + 31.28 GiB RAM, y **NVFP4** (47.79 GiB) ocupa 23.25 GiB VRAM + 38.27 GiB RAM en la misma tarjeta. Los conjuntos de expertos son disjuntos: nada se contabiliza dos veces.
-El presupuesto de 16 GiB VRAM + 20 GiB RAM es un límite de prueba, no un requisito mínimo del sistema. La fila Apple M2 Ultra es una medición con Metal, no un anuncio de disponibilidad para macOS.
+El presupuesto de 16 GiB VRAM + 20 GiB RAM es un límite de prueba, no un requisito mínimo del sistema. La fila Apple M2 Ultra es una medición con Metal.
@@ -282,7 +282,7 @@ Para 145B y 120B, usa `serve` o `chat`; `run` corresponde al motor nativo.
Notas sobre plataformas y documentación
-Las instrucciones de compilación están destinadas a GPU NVIDIA con CUDA 13 en Linux o Windows (x64 o ARM64). Las mediciones de Apple M2 Ultra usan Metal; no establecen la disponibilidad de una versión para macOS.
+Las instrucciones de compilación cubren las GPU NVIDIA con CUDA 13 en Linux o Windows (x64 o ARM64) y Metal en macOS con Apple Silicon.
@@ -295,7 +295,7 @@ Las instrucciones de compilación están destinadas a GPU NVIDIA con CUDA 13 en
git clone https://github.com/Edge0-AI/Edge0.git edge0
```
-- **edge0-strata**, el motor: sigue «Build (Linux, NVIDIA)» o «Build (Windows, NVIDIA)» en [edge0-strata/EDGE0.md](edge0-strata/EDGE0.md).
+- **edge0-strata**, el motor: sigue «Build (Linux, NVIDIA)», «Build (Windows, NVIDIA)» o «Build (macOS, Apple Silicon)» en [edge0-strata/EDGE0.md](edge0-strata/EDGE0.md).
- **edge0-infer**: sigue «Build» (Linux) o «Windows» en [edge0-infer/docs/getting-started.md](edge0-infer/docs/getting-started.md).
**2. Ejecutar desde la línea de comandos.** Indica una sola carpeta de modelo, como `Edge0-145B-q3k`, no la carpeta que contiene varios modelos. En Windows el programa es `build\windows\bin\edge0-infer.exe`.
@@ -315,11 +315,11 @@ curl localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '
-Aplicación de escritorio · Windows
+Aplicación de escritorio · Windows · macOS
⬇️ **Descargar:** **[GitHub Releases](https://github.com/Edge0-AI/Edge0/releases)**
-Abre **Edge0 Chat** en Windows y selecciona una carpeta de modelo, como `Edge0-145B-q3k`, para ejecutarlo localmente. Para compilar la aplicación desde el código fuente, sigue la guía de [Edge0 Chat](edge0-infer/examples/desktop/README.md).
+Abre **Edge0 Chat** en Windows o macOS (Apple Silicon) y selecciona una carpeta de modelo, como `Edge0-145B-q3k`, para ejecutarlo localmente. Para compilar la aplicación desde el código fuente, sigue la guía de [Edge0 Chat](edge0-infer/examples/desktop/README.md).
diff --git a/README_fr.md b/README_fr.md
index 7c6a937d..37ffe60b 100644
--- a/README_fr.md
+++ b/README_fr.md
@@ -14,7 +14,7 @@
Projet
-
+
Dernière version
@@ -80,7 +80,7 @@
Utilisez-le en ligne de commande, via un serveur API local ou avec l’application Edge0 Chat.
-La dernière version permet d’exécuter les modèles MoE latents de 145B et 120B sur un seul GPU NVIDIA.
+La dernière version permet d’exécuter les modèles MoE latents de 145B et 120B sur un seul GPU NVIDIA ou sur un Mac Apple Silicon.
Tous deux sont entraînés nativement en 4 bits et proposés aux formats Q3_K et NVFP4 pour les tâches d’agents locaux.
@@ -96,7 +96,7 @@ La prise en charge dépend du modèle et de son moteur.
| --- | --- | --- |
| **Linux** | NVIDIA GPU (CUDA 13) | CPU |
| **Windows** | NVIDIA GPU (CUDA 13) | CPU, Vulkan |
-| **macOS** (Apple Silicon) | — | Metal, CPU |
+| **macOS** (Apple Silicon) | Metal | Metal, CPU |
| **iOS** | — | Metal |
| **Android** | — | CPU |
@@ -108,14 +108,14 @@ Pour la dernière version, suivez l’[installation de 145B / 120B](#getting-sta
145B / 120B est la série MoE latente actuelle, dérivée de Qwen3.8-Flash-Next et entraînée nativement en 4 bits.
-Ces modèles fonctionnent sur edge0-strata via `edge0-infer serve` et `edge0-infer chat`, sous Linux ou Windows avec un GPU NVIDIA et CUDA 13.
+Ces modèles fonctionnent sur edge0-strata via `edge0-infer serve` et `edge0-infer chat`, sous Linux ou Windows avec un GPU NVIDIA et CUDA 13, ou sous macOS avec Metal sur Apple Silicon.
**145B Q3_K · RTX 5090 D 24 GB :** décodage agent à 121.2 tok/s · préremplissage 16K à 3554.9 tok/s.
| Taille | Checkpoint publié | Profil d’inférence | Moteur |
|---|---|---|---|
-| [`edge0-145b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-145B-q3k`](https://huggingface.co/Edge0/Edge0-145B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-145B-NVFP4) | Experts latents Q3_K ou NVFP4 (1792), 48 couches, 512 experts, K=10 | edge0-strata ✅ (Linux ou Windows, GPU NVIDIA) |
-| [`edge0-120b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-120B-q3k`](https://huggingface.co/Edge0/Edge0-120B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-120B-NVFP4) | Experts latents Q3_K ou NVFP4 (1280), 48 couches, 512 experts, K=10 | edge0-strata ✅ (Linux ou Windows, GPU NVIDIA) |
+| [`edge0-145b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-145B-q3k`](https://huggingface.co/Edge0/Edge0-145B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-145B-NVFP4) | Experts latents Q3_K ou NVFP4 (1792), 48 couches, 512 experts, K=10 | edge0-strata ✅ (Linux ou Windows, GPU NVIDIA ; macOS Apple Silicon, Metal, Q3_K) |
+| [`edge0-120b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-120B-q3k`](https://huggingface.co/Edge0/Edge0-120B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-120B-NVFP4) | Experts latents Q3_K ou NVFP4 (1280), 48 couches, 512 experts, K=10 | edge0-strata ✅ (Linux ou Windows, GPU NVIDIA ; macOS Apple Silicon, Metal, Q3_K) |
@@ -197,7 +197,7 @@ Les lignes mises en évidence correspondent à Q3_K. Cliquez sur le tableau pour
Les lignes `16 GiB VRAM + 20 GiB RAM` utilisent la même RTX 5090 D 24 GB, pas une autre carte. La VRAM est limitée par `--vram-reserve-mib 8192` ; la RAM système par un cgroup (`MemoryMax=20G`, sans swap). Avec la résidence activée, **Q3_K** (40.81 GiB de poids) occupe 23.14 GiB VRAM + 31.28 GiB RAM, et **NVFP4** (47.79 GiB) occupe 23.25 GiB VRAM + 38.27 GiB RAM sur cette même carte. Les ensembles d’experts sont disjoints, sans double comptage.
-Le budget de 16 GiB VRAM + 20 GiB RAM est une limite de test, pas une configuration minimale requise. La ligne Apple M2 Ultra est une mesure sous Metal, et non une annonce de disponibilité pour macOS.
+Le budget de 16 GiB VRAM + 20 GiB RAM est une limite de test, pas une configuration minimale requise. La ligne Apple M2 Ultra est une mesure sous Metal.
@@ -280,7 +280,7 @@ Pour 145B et 120B, utilisez `serve` ou `chat` ; `run` est réservé au moteur na
Notes sur les plateformes et la documentation
-Les instructions de compilation ciblent les GPU NVIDIA avec CUDA 13 sous Linux ou Windows (x64 ou ARM64). Les mesures sur Apple M2 Ultra utilisent Metal ; elles n’établissent pas la disponibilité d’une version pour macOS.
+Les instructions de compilation couvrent les GPU NVIDIA avec CUDA 13 sous Linux ou Windows (x64 ou ARM64), et Metal sur macOS Apple Silicon.
@@ -293,7 +293,7 @@ Les instructions de compilation ciblent les GPU NVIDIA avec CUDA 13 sous Linux o
git clone https://github.com/Edge0-AI/Edge0.git edge0
```
-- **edge0-strata**, le moteur : suivez « Build (Linux, NVIDIA) » ou « Build (Windows, NVIDIA) » dans [edge0-strata/EDGE0.md](edge0-strata/EDGE0.md).
+- **edge0-strata**, le moteur : suivez « Build (Linux, NVIDIA) », « Build (Windows, NVIDIA) » ou « Build (macOS, Apple Silicon) » dans [edge0-strata/EDGE0.md](edge0-strata/EDGE0.md).
- **edge0-infer** : suivez « Build » (Linux) ou « Windows » dans [edge0-infer/docs/getting-started.md](edge0-infer/docs/getting-started.md).
**2. Exécuter en ligne de commande.** Indiquez un seul dossier de modèle, tel que `Edge0-145B-q3k`, et non le dossier contenant plusieurs modèles. Sous Windows, le programme est `build\windows\bin\edge0-infer.exe`.
@@ -313,11 +313,11 @@ curl localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '
-Application de bureau · Windows
+Application de bureau · Windows · macOS
⬇️ **Télécharger :** **[GitHub Releases](https://github.com/Edge0-AI/Edge0/releases)**
-Ouvrez **Edge0 Chat** sous Windows et choisissez un dossier de modèle, tel que `Edge0-145B-q3k`, pour l’exécuter localement. Pour compiler l’application depuis les sources, suivez le guide [Edge0 Chat](edge0-infer/examples/desktop/README.md).
+Ouvrez **Edge0 Chat** sous Windows ou macOS (Apple Silicon) et choisissez un dossier de modèle, tel que `Edge0-145B-q3k`, pour l’exécuter localement. Pour compiler l’application depuis les sources, suivez le guide [Edge0 Chat](edge0-infer/examples/desktop/README.md).
diff --git a/README_ja.md b/README_ja.md
index c8d6dd2a..97baf4a9 100644
--- a/README_ja.md
+++ b/README_ja.md
@@ -14,7 +14,7 @@
-通过 `edge0-infer serve` 和 `edge0-infer chat` 调用 edge0-strata，在配备 NVIDIA GPU 和 CUDA 13 的 Linux 或 Windows 系统上运行。
+通过 `edge0-infer serve` 和 `edge0-infer chat` 调用 edge0-strata，在配备 NVIDIA GPU 和 CUDA 13 的 Linux 或 Windows 上运行，或在 macOS 的 Apple Silicon 上用 Metal 运行。
**145B Q3_K · RTX 5090 D 24 GB：** agent 解码 121.2 tok/s · 16K 预填充 3554.9 tok/s。
| 型号 | 已发布检查点 | 推理配置 | 引擎 |
|---|---|---|---|
-| [`edge0-145b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-145B-q3k`](https://huggingface.co/Edge0/Edge0-145B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-145B-NVFP4) | Q3_K 或 NVFP4 潜空间专家（1792），48 层，512 个专家，K=10 | edge0-strata ✅（Linux 或 Windows，NVIDIA GPU） |
-| [`edge0-120b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-120B-q3k`](https://huggingface.co/Edge0/Edge0-120B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-120B-NVFP4) | Q3_K 或 NVFP4 潜空间专家（1280），48 层，512 个专家，K=10 | edge0-strata ✅（Linux 或 Windows，NVIDIA GPU） |
+| [`edge0-145b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-145B-q3k`](https://huggingface.co/Edge0/Edge0-145B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-145B-NVFP4) | Q3_K 或 NVFP4 潜空间专家（1792），48 层，512 个专家，K=10 | edge0-strata ✅（Linux 或 Windows，NVIDIA GPU；macOS Apple Silicon，Metal，Q3_K） |
+| [`edge0-120b`](docs/models/edge0-145b.md) | [`Edge0/Edge0-120B-q3k`](https://huggingface.co/Edge0/Edge0-120B-q3k) · [NVFP4](https://huggingface.co/Edge0/Edge0-120B-NVFP4) | Q3_K 或 NVFP4 潜空间专家（1280），48 层，512 个专家，K=10 | edge0-strata ✅（Linux 或 Windows，NVIDIA GPU；macOS Apple Silicon，Metal，Q3_K） |
@@ -197,7 +197,7 @@ Q3_K · RTX 5090 D 24 GB。预填充使用 16K token 输入。能力分数为模
`16 GiB VRAM + 20 GiB RAM` 行仍使用同一张 RTX 5090 D 24 GB，并非另一张显卡。显存通过 `--vram-reserve-mib 8192` 限制，系统内存通过 cgroup（`MemoryMax=20G`，无 swap）限制。启用驻留后，同一张显卡上 **Q3_K**（权重 40.81 GiB）占用 23.14 GiB VRAM + 31.28 GiB RAM，**NVFP4**（47.79 GiB）占用 23.25 GiB VRAM + 38.27 GiB RAM；两处存放的专家集合互不重叠，没有重复计数。
-16 GiB VRAM + 20 GiB RAM 是测试时的预算限制，不是最低系统要求。Apple M2 Ultra 行是 Metal 后端的测量结果，不代表已发布 macOS 支持。
+16 GiB VRAM + 20 GiB RAM 是测试时的预算限制，不是最低系统要求。Apple M2 Ultra 行是 Metal 后端的测量结果。
@@ -282,7 +282,7 @@ Q3_K · RTX 5090 D 24 GB。预填充使用 16K token 输入。能力分数为模
平台与文档说明
-构建说明适用于 Linux 或 Windows（x64 或 ARM64）上的 NVIDIA GPU 与 CUDA 13。Apple M2 Ultra 的测量使用 Metal，不代表已发布 macOS 支持。
+构建说明涵盖 Linux 或 Windows（x64 或 ARM64）上带 CUDA 13 的 NVIDIA GPU，以及 Apple Silicon macOS 上的 Metal。
@@ -295,7 +295,7 @@ Q3_K · RTX 5090 D 24 GB。预填充使用 16K token 输入。能力分数为模
git clone https://github.com/Edge0-AI/Edge0.git edge0
```
-- **edge0-strata** 引擎：参照 [edge0-strata/EDGE0.md](edge0-strata/EDGE0.md) 中的“Build (Linux, NVIDIA)”或“Build (Windows, NVIDIA)”。
+- **edge0-strata** 引擎：参照 [edge0-strata/EDGE0.md](edge0-strata/EDGE0.md) 中的“Build (Linux, NVIDIA)”、“Build (Windows, NVIDIA)”或“Build (macOS, Apple Silicon)”。
- **edge0-infer**：参照 [edge0-infer/docs/getting-started.md](edge0-infer/docs/getting-started.md) 中的“Build”（Linux）或“Windows”。
**2. 从命令行运行。** 传入单个模型文件夹，例如 `Edge0-145B-q3k`，而非包含多个模型的上级目录。Windows 程序路径为 `build\windows\bin\edge0-infer.exe`。
@@ -315,11 +315,11 @@ curl localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '
-桌面应用 · Windows
+桌面应用 · Windows · macOS
⬇️ **下载：** **[GitHub Releases](https://github.com/Edge0-AI/Edge0/releases)**
-在 Windows 上打开 **Edge0 Chat**，选择模型文件夹（如 `Edge0-145B-q3k`）即可本地运行。如需从源码构建应用，请参阅 [Edge0 Chat](edge0-infer/examples/desktop/README.md) 指南。
+在 Windows 或 macOS（Apple Silicon）上打开 **Edge0 Chat**，选择模型文件夹（如 `Edge0-145B-q3k`）即可本地运行。如需从源码构建应用，请参阅 [Edge0 Chat](edge0-infer/examples/desktop/README.md) 指南。
diff --git a/assets/edge0-release/badges/releases-badge.svg b/assets/edge0-release/badges/releases-badge.svg
new file mode 100644
index 00000000..abe78cea
--- /dev/null
+++ b/assets/edge0-release/badges/releases-badge.svg
@@ -0,0 +1,16 @@
+
+
diff --git a/assets/edge0-release/badges/windows-releases-badge.svg b/assets/edge0-release/badges/windows-releases-badge.svg
deleted file mode 100644
index 641a9154..00000000
--- a/assets/edge0-release/badges/windows-releases-badge.svg
+++ /dev/null
@@ -1,21 +0,0 @@
-
-
diff --git a/docs/models/edge0-145b.md b/docs/models/edge0-145b.md
index 4f6fced9..efb6f37d 100644
--- a/docs/models/edge0-145b.md
+++ b/docs/models/edge0-145b.md
@@ -1,6 +1,6 @@
# Edge0 145B and Edge0 120B
-Two latent mixture-of-experts models for a single NVIDIA GPU, derived from Qwen3.8-Flash-Next: Edge0 145B and Edge0 120B. Their experts read compressed inputs, and both models run on [edge0-strata](../../edge0-strata/EDGE0.md), a fork of [Strata](https://github.com/Niko1221/Strata) whose scheduling of CPU, GPU and PCIe work they depend on. [edge0-infer](../../edge0-infer/README.md) starts edge0-strata for `serve` and `chat`, so the commands and endpoints are those of the other models.
+Two latent mixture-of-experts models for a single NVIDIA GPU on Linux or Windows, or for Apple Silicon on macOS, derived from Qwen3.8-Flash-Next: Edge0 145B and Edge0 120B. Their experts read compressed inputs, and both models run on [edge0-strata](../../edge0-strata/EDGE0.md), a fork of [Strata](https://github.com/Niko1221/Strata) whose scheduling of CPU, GPU and PCIe work they depend on. [edge0-infer](../../edge0-infer/README.md) starts edge0-strata for `serve` and `chat`, so the commands and endpoints are those of the other models.
Weights on Hugging Face: [`Edge0-145B-q3k`](https://huggingface.co/Edge0/Edge0-145B-q3k) · [`Edge0-145B-NVFP4`](https://huggingface.co/Edge0/Edge0-145B-NVFP4) · [`Edge0-120B-q3k`](https://huggingface.co/Edge0/Edge0-120B-q3k) · [`Edge0-120B-NVFP4`](https://huggingface.co/Edge0/Edge0-120B-NVFP4).
@@ -137,15 +137,13 @@ Each model folder holds one model in one expert format: `Edge0-145B-q3k`, `Edge0
## Requirements
-Linux or Windows with an NVIDIA GPU and CUDA 13. The constrained-memory results in the speed tables are measured test budgets, not general minimum requirements. A 16 GiB VRAM + 20 GiB host RAM test is available for 145B Q3_K and both 120B builds; no such result is available for 145B NVFP4.
-
-Apple M2 Ultra (Metal) measurements are retained in the speed tables. They do not establish macOS release availability.
+Linux or Windows with an NVIDIA GPU and CUDA 13, or macOS on Apple Silicon with Metal. On macOS only the Q3_K folders are supported, with `du=q4` and `ple-io=direct`. The Metal template uses a context length of 32768, and the first start writes `mtp-q2_0.gguf` into the model folder, which must be writable. The constrained-memory results in the speed tables are measured test budgets, not general minimum requirements. A 16 GiB VRAM + 20 GiB host RAM test is available for 145B Q3_K and both 120B builds; no such result is available for 145B NVFP4.
## Running
Use `serve` or `chat` for these models; `run` is for the native engine.
-Both models run on edge0-strata, the engine built for them, which edge0-infer starts for these models (see [Documentation](../../README.md#documentation)). Build edge0-strata as [Build (Linux, NVIDIA)](../../edge0-strata/EDGE0.md#build-linux-nvidia) or [Build (Windows, NVIDIA)](../../edge0-strata/EDGE0.md#build-windows-nvidia) in `edge0-strata/EDGE0.md` describes, build edge0-infer as in its [Getting started](../../edge0-infer/docs/getting-started.md#build), and start a model folder with `edge0-infer serve --model ` or `edge0-infer chat --model `. The pages [Edge0 145B](../../edge0-infer/docs/edge0-145b.md) and [Edge0 120B](../../edge0-infer/docs/edge0-120b.md) in the edge0-infer documentation cover `serve`, `chat` and `inspect` for these models, how `edge0-infer` finds edge0-strata, the options (`--device`, `du`, `ple-io`, `--context`, `--strata-arg`) and the known limitations.
+Both models run on edge0-strata, the engine built for them, which edge0-infer starts for these models (see [Documentation](../../README.md#documentation)). Build edge0-strata as [Build (Linux, NVIDIA)](../../edge0-strata/EDGE0.md#build-linux-nvidia), [Build (Windows, NVIDIA)](../../edge0-strata/EDGE0.md#build-windows-nvidia) or [Build (macOS, Apple Silicon)](../../edge0-strata/EDGE0.md#build-macos-apple-silicon) in `edge0-strata/EDGE0.md` describes, build edge0-infer as in its [Getting started](../../edge0-infer/docs/getting-started.md#build), and start a model folder with `edge0-infer serve --model ` or `edge0-infer chat --model `. The pages [Edge0 145B](../../edge0-infer/docs/edge0-145b.md) and [Edge0 120B](../../edge0-infer/docs/edge0-120b.md) in the edge0-infer documentation cover `serve`, `chat` and `inspect` for these models, how `edge0-infer` finds edge0-strata, the options (`--device`, `du`, `ple-io`, `--context`, `--strata-arg`) and the known limitations.
## Acknowledgements
diff --git a/edge0-infer/README.md b/edge0-infer/README.md
index fdbaa9e3..aeac59c6 100644
--- a/edge0-infer/README.md
+++ b/edge0-infer/README.md
@@ -5,13 +5,13 @@ English | [Chinese](README_zh.md) | [Japanese](README_ja.md) | [Español](README
`edge0-infer` runs mixture-of-experts (MoE) language models on your own hardware: a CLI, a local OpenAI- and Anthropic-compatible server and chat applications, with the same commands for every model. Each model runs on the engine built for it, and `edge0-infer` picks the engine from the model folder; there is nothing to choose:
- **Edge0 35B** (Qwen3.6-35B-A3B with a LoRA and a trained pre-router) runs on the **native engine** in this directory, built on [GGML](https://github.com/ggml-org/ggml) and with a native C++ API, on macOS, iOS, Android, Linux and Windows. Expert weights are loaded from storage on demand into a fixed pool of slots, so a model larger than the device memory budget runs without loading all of its weights. On an Apple M5 Mac it decodes at about 25 tokens per second with a peak physical footprint of about 3.4 GiB; figures for every platform are in [Benchmarking](docs/benchmarking.md#reference-figures).
-- **Edge0 145B** and **Edge0 120B** run on **edge0-strata**, a fork of [Strata](https://github.com/Niko1221/Strata) in `edge0-strata/` of the Edge0 repository, on an NVIDIA GPU on Linux or Windows; see [Edge0 145B](docs/edge0-145b.md) and [Edge0 120B](docs/edge0-120b.md).
+- **Edge0 145B** and **Edge0 120B** run on **edge0-strata**, a fork of [Strata](https://github.com/Niko1221/Strata) in `edge0-strata/` of the Edge0 repository, on an NVIDIA GPU on Linux or Windows, and on Apple Silicon with Metal on macOS; see [Edge0 145B](docs/edge0-145b.md) and [Edge0 120B](docs/edge0-120b.md).
## Platforms
| Platform | Edge0 35B (native engine) | Edge0 145B, Edge0 120B (edge0-strata) |
| --- | --- | --- |
-| macOS (Apple silicon) | Metal, CPU | — |
+| macOS (Apple silicon) | Metal, CPU | Metal |
| iOS | Metal | — |
| Android | CPU | — |
| Linux | CPU | NVIDIA GPU (CUDA 13) |
diff --git a/edge0-infer/README_es.md b/edge0-infer/README_es.md
index 8f7a1c59..d5eae734 100644
--- a/edge0-infer/README_es.md
+++ b/edge0-infer/README_es.md
@@ -5,7 +5,7 @@
`edge0-infer` ejecuta modelos de lenguaje de mezcla de expertos (MoE) en tu propio hardware: una CLI, un servidor local compatible con OpenAI y Anthropic y aplicaciones de chat, con los mismos comandos para todos los modelos. Cada modelo se ejecuta en el motor preparado para él, y `edge0-infer` elige el motor a partir de la carpeta del modelo; no hay nada que elegir:
- **Edge0 35B** (Qwen3.6-35B-A3B con un LoRA y un pre-router entrenado) se ejecuta en el **motor nativo** de este directorio, construido sobre [GGML](https://github.com/ggml-org/ggml) y con una API nativa en C++, en macOS, iOS, Android, Linux y Windows. Los pesos de los expertos se cargan desde el almacenamiento bajo demanda en un conjunto fijo de ranuras, de modo que un modelo más grande que el presupuesto de memoria del dispositivo se ejecuta sin cargar todos sus pesos. En un Mac con Apple M5 decodifica a unos 25 tokens por segundo, con un pico de huella física de unos 3,4 GiB; las cifras de cada plataforma están en [Benchmarking](docs/benchmarking.md#reference-figures).
-- **Edge0 145B** y **Edge0 120B** se ejecutan en **edge0-strata**, un fork de [Strata](https://github.com/Niko1221/Strata) en `edge0-strata/` del repositorio Edge0, en una GPU NVIDIA con Linux o Windows; consulta [Edge0 145B](docs/edge0-145b.md) y [Edge0 120B](docs/edge0-120b.md).
+- **Edge0 145B** y **Edge0 120B** se ejecutan en **edge0-strata**, un fork de [Strata](https://github.com/Niko1221/Strata) en `edge0-strata/` del repositorio Edge0, en una GPU NVIDIA con Linux o Windows, y en Apple Silicon con Metal en macOS; consulta [Edge0 145B](docs/edge0-145b.md) y [Edge0 120B](docs/edge0-120b.md).
> Esta página es una traducción del [README](README.md) en inglés; los documentos enlazados están en inglés.
@@ -13,7 +13,7 @@
| Plataforma | Edge0 35B (motor nativo) | Edge0 145B, Edge0 120B (edge0-strata) |
| --- | --- | --- |
-| macOS (Apple silicon) | Metal, CPU | — |
+| macOS (Apple silicon) | Metal, CPU | Metal |
| iOS | Metal | — |
| Android | CPU | — |
| Linux | CPU | GPU NVIDIA (CUDA 13) |
diff --git a/edge0-infer/README_fr.md b/edge0-infer/README_fr.md
index 098152f0..fcac81d3 100644
--- a/edge0-infer/README_fr.md
+++ b/edge0-infer/README_fr.md
@@ -5,7 +5,7 @@
`edge0-infer` exécute des modèles de langage à mélange d'experts (MoE) sur votre propre matériel : une CLI, un serveur local compatible OpenAI et Anthropic et des applications de discussion, avec les mêmes commandes pour tous les modèles. Chaque modèle s'exécute sur le moteur conçu pour lui, et `edge0-infer` choisit le moteur d'après le dossier du modèle ; il n'y a rien à choisir :
- **Edge0 35B** (Qwen3.6-35B-A3B avec un LoRA et un pre-router entraîné) s'exécute sur le **moteur natif** de ce répertoire, construit sur [GGML](https://github.com/ggml-org/ggml) et doté d'une API C++ native, sous macOS, iOS, Android, Linux et Windows. Les poids des experts sont chargés depuis le stockage à la demande dans un ensemble fixe d'emplacements, si bien qu'un modèle plus grand que le budget mémoire de l'appareil s'exécute sans charger tous ses poids. Sur un Mac Apple M5, il décode à environ 25 tokens par seconde, avec un pic d'empreinte physique d'environ 3,4 Gio ; les chiffres de chaque plateforme se trouvent dans [Benchmarking](docs/benchmarking.md#reference-figures).
-- **Edge0 145B** et **Edge0 120B** s'exécutent sur **edge0-strata**, un fork de [Strata](https://github.com/Niko1221/Strata) dans `edge0-strata/` du dépôt Edge0, sur un GPU NVIDIA sous Linux ou Windows ; voir [Edge0 145B](docs/edge0-145b.md) et [Edge0 120B](docs/edge0-120b.md).
+- **Edge0 145B** et **Edge0 120B** s'exécutent sur **edge0-strata**, un fork de [Strata](https://github.com/Niko1221/Strata) dans `edge0-strata/` du dépôt Edge0, sur un GPU NVIDIA sous Linux ou Windows, et sur Apple Silicon avec Metal sous macOS ; voir [Edge0 145B](docs/edge0-145b.md) et [Edge0 120B](docs/edge0-120b.md).
> Cette page est une traduction du [README](README.md) en anglais ; les documents liés sont en anglais.
@@ -13,7 +13,7 @@
| Plateforme | Edge0 35B (moteur natif) | Edge0 145B, Edge0 120B (edge0-strata) |
| --- | --- | --- |
-| macOS (Apple silicon) | Metal, CPU | — |
+| macOS (Apple silicon) | Metal, CPU | Metal |
| iOS | Metal | — |
| Android | CPU | — |
| Linux | CPU | GPU NVIDIA (CUDA 13) |
diff --git a/edge0-infer/README_ja.md b/edge0-infer/README_ja.md
index fa6b954d..74c6a137 100644
--- a/edge0-infer/README_ja.md
+++ b/edge0-infer/README_ja.md
@@ -5,7 +5,7 @@
`edge0-infer` は、混合エキスパート（MoE）言語モデルを手元のハードウェアで実行します。CLI、ローカルの OpenAI／Anthropic 互換サーバー、チャットアプリを備え、どのモデルも同じコマンドで使えます。各モデルはそれぞれ専用のエンジンで動作し、`edge0-infer` がモデルフォルダからエンジンを自動で選ぶため、選択する必要はありません。
- **Edge0 35B**（LoRA と学習済み pre-router を備えた Qwen3.6-35B-A3B）は、このディレクトリの**ネイティブエンジン**で動作します。ネイティブエンジンは [GGML](https://github.com/ggml-org/ggml) をベースにネイティブ C++ API を備え、macOS、iOS、Android、Linux、Windows で動作します。エキスパートの重みはストレージから必要に応じて固定数のスロットに読み込まれるため、デバイスのメモリ予算より大きなモデルでも、すべての重みを読み込まずに実行できます。Apple M5 Mac ではデコードが毎秒約 25 トークン、物理メモリフットプリントのピークが約 3.4 GiB です。各プラットフォームの数値は [Benchmarking](docs/benchmarking.md#reference-figures) を参照してください。
-- **Edge0 145B** と **Edge0 120B** は **edge0-strata** で動作します。[Strata](https://github.com/Niko1221/Strata) のフォークで、Edge0 リポジトリの `edge0-strata/` にあり、Linux または Windows の NVIDIA GPU で動作します。[Edge0 145B](docs/edge0-145b.md) と [Edge0 120B](docs/edge0-120b.md) を参照してください。
+- **Edge0 145B** と **Edge0 120B** は **edge0-strata** で動作します。[Strata](https://github.com/Niko1221/Strata) のフォークで、Edge0 リポジトリの `edge0-strata/` にあり、Linux または Windows の NVIDIA GPU、および macOS の Apple Silicon（Metal）で動作します。[Edge0 145B](docs/edge0-145b.md) と [Edge0 120B](docs/edge0-120b.md) を参照してください。
> このページは英語版 [README](README.md) の翻訳です。リンク先のドキュメントはすべて英語です。
@@ -13,7 +13,7 @@
| プラットフォーム | Edge0 35B（ネイティブエンジン） | Edge0 145B、Edge0 120B（edge0-strata） |
| --- | --- | --- |
-| macOS（Apple シリコン） | Metal、CPU | — |
+| macOS（Apple シリコン） | Metal、CPU | Metal |
| iOS | Metal | — |
| Android | CPU | — |
| Linux | CPU | NVIDIA GPU（CUDA 13） |
diff --git a/edge0-infer/README_zh.md b/edge0-infer/README_zh.md
index 638a2502..51801f29 100644
--- a/edge0-infer/README_zh.md
+++ b/edge0-infer/README_zh.md
@@ -5,7 +5,7 @@
`edge0-infer` 在你自己的硬件上运行混合专家（MoE）语言模型：命令行、本地 OpenAI／Anthropic 兼容服务与聊天应用，所有模型使用相同的命令。每个模型运行在为它准备的引擎上，`edge0-infer` 根据模型目录自动选择引擎，无需用户选择：
- **Edge0 35B**（Qwen3.6-35B-A3B，带 LoRA 与训练好的 pre-router）运行在本目录的**原生引擎**上。原生引擎基于 [GGML](https://github.com/ggml-org/ggml)，提供原生 C++ API，支持 macOS、iOS、Android、Linux 和 Windows。专家权重从存储按需读入固定数量的专家槽，因此比设备内存预算更大的模型无需载入全部权重即可运行。在 Apple M5 Mac 上解码约每秒 25 个 token，物理内存占用峰值约 3.4 GiB；各平台的数据见 [Benchmarking](docs/benchmarking.md#reference-figures)。
-- **Edge0 145B** 和 **Edge0 120B** 运行在 **edge0-strata** 上。它是 [Strata](https://github.com/Niko1221/Strata) 的分支，位于 Edge0 仓库的 `edge0-strata/`，在 Linux 或 Windows 的 NVIDIA GPU 上运行；见 [Edge0 145B](docs/edge0-145b.md) 与 [Edge0 120B](docs/edge0-120b.md)。
+- **Edge0 145B** 和 **Edge0 120B** 运行在 **edge0-strata** 上。它是 [Strata](https://github.com/Niko1221/Strata) 的分支，位于 Edge0 仓库的 `edge0-strata/`，在 Linux 或 Windows 的 NVIDIA GPU 上运行，也在 macOS 的 Apple Silicon 上用 Metal 运行；见 [Edge0 145B](docs/edge0-145b.md) 与 [Edge0 120B](docs/edge0-120b.md)。
> 本页为英文版 [README](README.md) 的翻译；下面链接的文档均为英文。
@@ -13,7 +13,7 @@
| 平台 | Edge0 35B（原生引擎） | Edge0 145B、Edge0 120B（edge0-strata） |
| --- | --- | --- |
-| macOS（Apple Silicon） | Metal、CPU | — |
+| macOS（Apple Silicon） | Metal、CPU | Metal |
| iOS | Metal | — |
| Android | CPU | — |
| Linux | CPU | NVIDIA GPU（CUDA 13） |
diff --git a/edge0-infer/docs/architecture.md b/edge0-infer/docs/architecture.md
index 6f487e27..3d39f602 100644
--- a/edge0-infer/docs/architecture.md
+++ b/edge0-infer/docs/architecture.md
@@ -9,7 +9,7 @@ An overview of how `edge0-infer` is organized. It names the parts and how they w
| Model | Engine | Hardware | How `edge0-infer` uses the engine |
| --- | --- | --- | --- |
| Edge0 35B | Native engine | GGML backends: Metal, CPU, Vulkan | A library in this directory, called in process; applications can also call it through the [native API](native-api.md) |
-| Edge0 145B, Edge0 120B | edge0-strata | NVIDIA GPU with CUDA | A separate engine in `edge0-strata/` of the Edge0 repository, a fork of [Strata](https://github.com/Niko1221/Strata); `edge0-infer` starts it as a child process and talks to it over a loopback port |
+| Edge0 145B, Edge0 120B | edge0-strata | NVIDIA GPU with CUDA, or Metal on macOS (Apple Silicon) | A separate engine in `edge0-strata/` of the Edge0 repository, a fork of [Strata](https://github.com/Niko1221/Strata); `edge0-infer` starts it as a child process and talks to it over a loopback port |
`edge0-infer` picks the engine from the model folder given with `--model`; there is nothing to choose. A `model.json` with an `architecture` field is an asset manifest of the native engine. A `model.json` whose `model_name` names an Edge0 145B or Edge0 120B folder selects edge0-strata: `edge0-infer` starts the engine with that model's configuration template from `edge0-strata/edge0/models/`, waits until it is ready, and then forwards generation requests to it. `serve` keeps answering `/health`, `/v1/models` and `/status` itself, and `chat` keeps its own commands, so clients see the same interface for every model. The pages [Edge0 145B](edge0-145b.md) and [Edge0 120B](edge0-120b.md) describe this path; edge0-strata's own build and its launch contract with `edge0-infer` are in `edge0-strata/EDGE0.md`.
diff --git a/edge0-infer/docs/edge0-120b.md b/edge0-infer/docs/edge0-120b.md
index d2db844c..dd9e7dc1 100644
--- a/edge0-infer/docs/edge0-120b.md
+++ b/edge0-infer/docs/edge0-120b.md
@@ -1,15 +1,15 @@
# Edge0 120B
-Edge0 120B is a mixture-of-experts model for a single NVIDIA GPU. It runs on **edge0-strata**, the engine built for it (see [Two layers](architecture.md#two-layers)): the `edge0-strata/` directory of the Edge0 repository, a fork of [Strata](https://github.com/Niko1221/Strata). `edge0-infer serve` and `edge0-infer chat` start edge0-strata for this model, wait until it is ready and talk to it on a loopback port, so clients use the same commands and endpoints as for every other model. [Edge0 145B](edge0-145b.md) runs the same way.
+Edge0 120B is a mixture-of-experts model for one NVIDIA GPU on Linux or Windows, or for Apple Silicon on macOS. It runs on **edge0-strata**, the engine built for it (see [Two layers](architecture.md#two-layers)): the `edge0-strata/` directory of the Edge0 repository, a fork of [Strata](https://github.com/Niko1221/Strata). `edge0-infer serve` and `edge0-infer chat` start edge0-strata for this model, wait until it is ready and talk to it on a loopback port, so clients use the same commands and endpoints as for every other model. [Edge0 145B](edge0-145b.md) runs the same way.
This page is the reference for running Edge0 120B with `edge0-infer`; [Getting started](getting-started.md#edge0-145b-and-edge0-120b-edge0-strata) walks through the steps. The model itself (architecture and measured speed) is described on the model page of Edge0 145B and Edge0 120B, `docs/models/edge0-145b.md` in the Edge0 repository.
## Requirements
-- Linux or Windows with an NVIDIA GPU and CUDA 13.
-- At least 16 GiB of GPU memory and 20 GiB of system memory.
-- `edge0-infer` built as in [Build](getting-started.md#build) (the `cpu` preset on Linux) or [Windows](getting-started.md#windows); it needs no GPU backend of its own for these models.
-- edge0-strata built as described in `edge0-strata/EDGE0.md` in the Edge0 repository. `edge0-infer` uses the engine `edge0-strata/build/strata` (`strata.exe` on Windows) and the Python environment `edge0-strata/.venv` it produces.
+- Linux or Windows with an NVIDIA GPU and CUDA 13, or macOS on Apple Silicon with Metal. The macOS measurements were taken on an Apple M2 Ultra.
+- On Linux and Windows, at least 16 GiB of GPU memory and 20 GiB of system memory.
+- `edge0-infer` built as in [Build](getting-started.md#build) (the `cpu` preset on Linux, or the `cpu` or `mac-metal` preset on macOS) or [Windows](getting-started.md#windows); it needs no GPU backend of its own for these models.
+- edge0-strata built as described in `edge0-strata/EDGE0.md` in the Edge0 repository. `edge0-infer` uses `edge0-strata/build/strata` on Linux, `edge0-strata/build/strata.exe` on Windows, or `edge0-strata/build-metal/strata` on macOS, and the Python environment `edge0-strata/.venv` that the build produces.
## Model directory
@@ -53,7 +53,7 @@ build/cpu/bin/edge0-infer inspect --model "$EDGE0_120B" --json
The report lists files with `exists` and `bytes`; `ok` is `true` (exit code 0) when the selected configuration is complete.
The expert format comes from `model_name` and is not a model option; `--model-option experts=` is rejected.
-D/U defaults to `q4`; add `--model-option du=q8` to check the other choice.
+On Linux and Windows, D/U defaults to `q4`; add `--model-option du=q8` to check the other choice. On macOS the only accepted value is `q4`.
Only Q3_K folders list or require `scales/` (in the report its `required` value is `experts=q3k`).
When files are missing, standard output remains valid JSON and standard error lists the missing files.
@@ -91,13 +91,13 @@ The commands are those of the native chat (`/file`, `"""`, `/think on|off`, `/ne
| Option | Values | Default |
| --- | --- | --- |
-| `--device` | `cuda` or `cuda:` | `cuda:0` |
-| `--model-option du=` | `q8` or `q4` | `q4` |
+| `--device` | `metal` on macOS; `cuda` or `cuda:` on Linux and Windows | `metal` on macOS, `cuda:0` otherwise |
+| `--model-option du=` | `q8` or `q4` | `q4` (`q4` on macOS) |
| `--model-option ple-io=` | `direct` (read the embedding table from disk as needed) or `ram` (load it into memory; to keep it locked there, raise the locked-memory limit, e.g. `ulimit -l unlimited`) | `direct` |
-| `--context` | tokens | 131072 |
+| `--context` | tokens | 131072 (`32768` on macOS) |
| `--strata-arg` | one word of edge0-strata's engine command line; repeat it for each word, e.g. `--strata-arg --pool-workers --strata-arg 24` | none |
-The model settings (automatic expert cache and prefill, speculative decoding, and int8 KV cache) are in `edge0-strata/edge0/models/edge0-120b.json`. CPU worker count, PCIe mode and fraction, VRAM reserve, and KV residency use the engine's defaults unless given with `--strata-arg`. This is meant for tuning to one machine, for example:
+On Linux and Windows the model settings (automatic expert cache and prefill, speculative decoding, and int8 KV cache) are in `edge0-strata/edge0/models/edge0-120b.json`. On macOS the same settings, with the limits below, are in `edge0-120b-metal.json`. CPU worker count, PCIe mode and fraction, VRAM reserve, and KV residency use the engine's defaults unless given with `--strata-arg`. This is meant for tuning to one machine, for example:
```sh
build/cpu/bin/edge0-infer serve --model "$EDGE0_120B" \
@@ -106,7 +106,9 @@ build/cpu/bin/edge0-infer serve --model "$EDGE0_120B" \
--strata-arg --kv-resident --strata-arg 32768
```
-The words are appended in the order given, after the template's arguments; write each option and its value as two words (not `--option=value`). An option the template already sets (such as `--kv` or `--pack`), `--max-context` (use `--context`) and an empty value are rejected before anything starts. Other words are not checked by `edge0-infer`: edge0-strata decides, and since the engine process's own output is not shown, a word it does not accept appears as a failed start (run edge0-strata directly to see why). The template requires launch contract version 4; an older engine template is rejected with both versions in the error.
+The words are appended in the order given, after the template's arguments; write each option and its value as two words (not `--option=value`). An option the template already sets (such as `--kv` or `--pack`), `--max-context` (use `--context`) and an empty value are rejected before anything starts. Other words are not checked by `edge0-infer`: edge0-strata decides, and since the engine process's own output is not shown, a word it does not accept appears as a failed start (run edge0-strata directly to see why). On macOS, `--device` accepts only `metal`. An `fp4` folder is rejected. `du` accepts only `q4`, and `ple-io` accepts only `direct`. The Metal template `edge0-120b-metal.json` sets the context length to 32768. If `mtp-q2_0.gguf` is absent, the first start runs `tools/mtp_gguf.py` with edge0-strata's Python and writes that file into the model folder, which must be writable; a later start leaves the file in place. The file is not part of the released folder listing above.
+
+The template requires launch contract version 5; an older engine template is rejected with both versions in the error.
## CUDA libraries on Windows
diff --git a/edge0-infer/docs/edge0-145b.md b/edge0-infer/docs/edge0-145b.md
index d373190b..380a0a6b 100644
--- a/edge0-infer/docs/edge0-145b.md
+++ b/edge0-infer/docs/edge0-145b.md
@@ -1,15 +1,15 @@
# Edge0 145B
-Edge0 145B is a mixture-of-experts model for a single NVIDIA GPU. It runs on **edge0-strata**, the engine built for it (see [Two layers](architecture.md#two-layers)): the `edge0-strata/` directory of the Edge0 repository, a fork of [Strata](https://github.com/Niko1221/Strata). `edge0-infer serve` and `edge0-infer chat` start edge0-strata for this model, wait until it is ready and talk to it on a loopback port, so clients use the same commands and endpoints as for every other model. [Edge0 120B](edge0-120b.md) runs the same way.
+Edge0 145B is a mixture-of-experts model for one NVIDIA GPU on Linux or Windows, or for Apple Silicon on macOS. It runs on **edge0-strata**, the engine built for it (see [Two layers](architecture.md#two-layers)): the `edge0-strata/` directory of the Edge0 repository, a fork of [Strata](https://github.com/Niko1221/Strata). `edge0-infer serve` and `edge0-infer chat` start edge0-strata for this model, wait until it is ready and talk to it on a loopback port, so clients use the same commands and endpoints as for every other model. [Edge0 120B](edge0-120b.md) runs the same way.
This page is the reference for running Edge0 145B with `edge0-infer`; [Getting started](getting-started.md#edge0-145b-and-edge0-120b-edge0-strata) walks through the steps. The model itself (architecture and measured speed) is described on the model page of Edge0 145B and Edge0 120B, `docs/models/edge0-145b.md` in the Edge0 repository.
## Requirements
-- Linux or Windows with an NVIDIA GPU and CUDA 13.
-- At least 16 GiB of GPU memory and 20 GiB of system memory.
-- `edge0-infer` built as in [Build](getting-started.md#build) (the `cpu` preset on Linux) or [Windows](getting-started.md#windows); it needs no GPU backend of its own for these models.
-- edge0-strata built as described in `edge0-strata/EDGE0.md` in the Edge0 repository. `edge0-infer` uses the engine `edge0-strata/build/strata` (`strata.exe` on Windows) and the Python environment `edge0-strata/.venv` it produces.
+- Linux or Windows with an NVIDIA GPU and CUDA 13, or macOS on Apple Silicon with Metal. The macOS measurements were taken on an Apple M2 Ultra.
+- On Linux and Windows, at least 16 GiB of GPU memory and 20 GiB of system memory.
+- `edge0-infer` built as in [Build](getting-started.md#build) (the `cpu` preset on Linux, or the `cpu` or `mac-metal` preset on macOS) or [Windows](getting-started.md#windows); it needs no GPU backend of its own for these models.
+- edge0-strata built as described in `edge0-strata/EDGE0.md` in the Edge0 repository. `edge0-infer` uses `edge0-strata/build/strata` on Linux, `edge0-strata/build/strata.exe` on Windows, or `edge0-strata/build-metal/strata` on macOS, and the Python environment `edge0-strata/.venv` that the build produces.
## Model directory
@@ -53,7 +53,7 @@ build/cpu/bin/edge0-infer inspect --model "$EDGE0_145B" --json
The report lists files with `exists` and `bytes`; `ok` is `true` (exit code 0) when the selected configuration is complete.
The expert format comes from `model_name` and is not a model option; `--model-option experts=` is rejected.
-D/U defaults to `q8`; add `--model-option du=q4` to check the other choice.
+On Linux and Windows, D/U defaults to `q8`; add `--model-option du=q4` to check the other choice. On macOS the only accepted value is `q4`.
Only Q3_K folders list or require `scales/` (in the report its `required` value is `experts=q3k`).
When files are missing, standard output remains valid JSON and standard error lists the missing files.
@@ -91,13 +91,13 @@ The commands are those of the native chat (`/file`, `"""`, `/think on|off`, `/ne
| Option | Values | Default |
| --- | --- | --- |
-| `--device` | `cuda` or `cuda:` | `cuda:0` |
-| `--model-option du=` | `q8` or `q4` | `q8` |
+| `--device` | `metal` on macOS; `cuda` or `cuda:` on Linux and Windows | `metal` on macOS, `cuda:0` otherwise |
+| `--model-option du=` | `q8` or `q4` | `q8` (`q4` on macOS) |
| `--model-option ple-io=` | `direct` (read the embedding table from disk as needed) or `ram` (load it into memory; to keep it locked there, raise the locked-memory limit, e.g. `ulimit -l unlimited`) | `direct` |
-| `--context` | tokens | 131072 |
+| `--context` | tokens | 131072 (`32768` on macOS) |
| `--strata-arg` | one word of edge0-strata's engine command line; repeat it for each word, e.g. `--strata-arg --pool-workers --strata-arg 24` | none |
-The model settings (automatic expert cache and prefill, speculative decoding, and int8 KV cache) are in `edge0-strata/edge0/models/edge0-145b.json`. CPU worker count, PCIe mode and fraction, VRAM reserve, and KV residency use the engine's defaults unless given with `--strata-arg`. This is meant for tuning to one machine, for example:
+On Linux and Windows the model settings (automatic expert cache and prefill, speculative decoding, and int8 KV cache) are in `edge0-strata/edge0/models/edge0-145b.json`. On macOS the same settings, with the limits below, are in `edge0-145b-metal.json`. CPU worker count, PCIe mode and fraction, VRAM reserve, and KV residency use the engine's defaults unless given with `--strata-arg`. This is meant for tuning to one machine, for example:
```sh
build/cpu/bin/edge0-infer serve --model "$EDGE0_145B" \
@@ -106,7 +106,9 @@ build/cpu/bin/edge0-infer serve --model "$EDGE0_145B" \
--strata-arg --kv-resident --strata-arg 32768
```
-The words are appended in the order given, after the template's arguments; write each option and its value as two words (not `--option=value`). An option the template already sets (such as `--kv` or `--pack`), `--max-context` (use `--context`) and an empty value are rejected before anything starts. Other words are not checked by `edge0-infer`: edge0-strata decides, and since the engine process's own output is not shown, a word it does not accept appears as a failed start (run edge0-strata directly to see why). The template requires launch contract version 4; an older engine template is rejected with both versions in the error.
+The words are appended in the order given, after the template's arguments; write each option and its value as two words (not `--option=value`). An option the template already sets (such as `--kv` or `--pack`), `--max-context` (use `--context`) and an empty value are rejected before anything starts. Other words are not checked by `edge0-infer`: edge0-strata decides, and since the engine process's own output is not shown, a word it does not accept appears as a failed start (run edge0-strata directly to see why). On macOS, `--device` accepts only `metal`. An `fp4` folder is rejected. `du` accepts only `q4`, and `ple-io` accepts only `direct`. The Metal template `edge0-145b-metal.json` sets the context length to 32768. If `mtp-q2_0.gguf` is absent, the first start runs `tools/mtp_gguf.py` with edge0-strata's Python and writes that file into the model folder, which must be writable; a later start leaves the file in place. The file is not part of the released folder listing above.
+
+The template requires launch contract version 5; an older engine template is rejected with both versions in the error.
## CUDA libraries on Windows
diff --git a/edge0-infer/docs/getting-started.md b/edge0-infer/docs/getting-started.md
index 37f19b22..e6519563 100644
--- a/edge0-infer/docs/getting-started.md
+++ b/edge0-infer/docs/getting-started.md
@@ -9,11 +9,11 @@ From a fresh clone to a chat and a local OpenAI- and Anthropic-compatible server
| Model | Engine | Hardware | Where the engine is |
| --- | --- | --- | --- |
| Edge0 35B | Native engine | Metal, CPU or Vulkan on macOS, iOS, Android, Linux and Windows | this directory, also usable in process through the [native API](native-api.md) |
-| Edge0 145B, Edge0 120B | edge0-strata | NVIDIA GPU (CUDA 13) on Linux and Windows | `edge0-strata/` in the Edge0 repository; `edge0-infer` starts it as a separate process |
+| Edge0 145B, Edge0 120B | edge0-strata | NVIDIA GPU (CUDA 13) on Linux and Windows, or Metal on macOS (Apple Silicon) | `edge0-strata/` in the Edge0 repository; `edge0-infer` starts it as a separate process |
[Architecture](architecture.md) describes how the two layers work together.
-Edge0 35B is verified on macOS on Apple silicon (Metal and CPU), on Android (CPU, PKR110), on Windows (CPU on an Intel Core i5-12400; Vulkan on an NVIDIA GeForce RTX 4070 SUPER) and on iOS (Metal, iPhone 17 Pro Max); measured speed and memory are under [Benchmarking](benchmarking.md#reference-figures). The measured speed of Edge0 145B and Edge0 120B is on their model page, `docs/models/edge0-145b.md` in the Edge0 repository.
+Edge0 35B is verified on macOS on Apple silicon (Metal and CPU), on Android (CPU, PKR110), on Windows (CPU on an Intel Core i5-12400; Vulkan on an NVIDIA GeForce RTX 4070 SUPER) and on iOS (Metal, iPhone 17 Pro Max); measured speed and memory are under [Benchmarking](benchmarking.md#reference-figures). The measured speed of Edge0 145B and Edge0 120B is on their model page, `docs/models/edge0-145b.md` in the Edge0 repository. The macOS measurements were taken on an Apple M2 Ultra.
## Requirements
@@ -86,9 +86,9 @@ The [asset tools guide](../tools/assets/README.md) describes the asset directory
### Edge0 145B and Edge0 120B (edge0-strata)
-These models come as released model folders, one model and one expert format per folder (for example `Edge0-145B-q3k`); the asset tools above do not make them. They need an NVIDIA GPU with CUDA 13 on Linux or Windows.
+These models come as released model folders, one model and one expert format per folder (for example `Edge0-145B-q3k`); the asset tools above do not make them. They need an NVIDIA GPU with CUDA 13 on Linux or Windows, or Apple Silicon with Metal on macOS. On macOS only the `q3k` folder is accepted, with `du=q4` and `ple-io=direct`. The first start writes `mtp-q2_0.gguf` into the model folder, so that folder must be writable.
-1. **Build edge0-strata**, the `edge0-strata/` directory beside `edge0-infer/` in the Edge0 repository, as its `EDGE0.md` describes (sections "Build (Linux, NVIDIA)" and "Build (Windows, NVIDIA)"). `edge0-infer` finds it there and uses the engine and the Python environment that the build produces.
+1. **Build edge0-strata**, the `edge0-strata/` directory beside `edge0-infer/` in the Edge0 repository, as its `EDGE0.md` describes (sections "Build (Linux, NVIDIA)", "Build (Windows, NVIDIA)" and "Build (macOS, Apple Silicon)"). `edge0-infer` finds it there and uses the engine and the Python environment that the build produces. On macOS the engine is `build-metal/strata`.
2. **Check the model folder**, the one that contains `model.json`:
diff --git a/edge0-infer/docs/limitations.md b/edge0-infer/docs/limitations.md
index d90a9aa8..a7d3bcf4 100644
--- a/edge0-infer/docs/limitations.md
+++ b/edge0-infer/docs/limitations.md
@@ -10,4 +10,4 @@ Behavior of the current release that users and integrators should be aware of.
## Edge0 145B and Edge0 120B run only through serve and chat
-Edge0 145B and Edge0 120B run on the edge0-strata engine, not on the native engine (see [Two layers](architecture.md#two-layers)): only through `edge0-infer serve` and `edge0-infer chat` (and `inspect`), and not through the native API. Their server queues concurrent requests instead of answering 429, and part of `/status` is null for them (no memory breakdown or expert statistics; progress of a request in flight is counted from the forwarded text fragments). See Known limitations for [Edge0 145B](edge0-145b.md#known-limitations) and [Edge0 120B](edge0-120b.md#known-limitations).
+Edge0 145B and Edge0 120B run on the edge0-strata engine, not on the native engine (see [Two layers](architecture.md#two-layers)): only through `edge0-infer serve` and `edge0-infer chat` (and `inspect`), and not through the native API. On macOS they use Metal on Apple Silicon: only a `q3k` folder, `du=q4` and `ple-io=direct`, with a context length of 32768, and the first start writes `mtp-q2_0.gguf` into the model folder. Their server queues concurrent requests instead of answering 429, and part of `/status` is null for them (no memory breakdown or expert statistics; progress of a request in flight is counted from the forwarded text fragments). See Known limitations for [Edge0 145B](edge0-145b.md#known-limitations) and [Edge0 120B](edge0-120b.md#known-limitations).
diff --git a/edge0-infer/examples/desktop/README.md b/edge0-infer/examples/desktop/README.md
index a300a3d8..717dbdf9 100644
--- a/edge0-infer/examples/desktop/README.md
+++ b/edge0-infer/examples/desktop/README.md
@@ -146,6 +146,9 @@ Build `edge0-infer` using the [build guide](../../docs/getting-started.md).
On Choose a model, expand Other location… to enter the full executable path
and one asset directory per line. This section starts collapsed, including on
first launch before an executable has been configured.
+With no executable configured, the app uses the `edge0-infer` that a release
+package ships with it: beside its own executable, or in `Contents/Resources` of
+the macOS app bundle.
If no executable is configured, or the configured file has been moved or deleted,
loading or saving reports which one it is and points to Settings > Engine > Executable
and Other location… to set it again.
@@ -165,7 +168,7 @@ The default device is `metal` on macOS and `cpu` on Windows. Settings also allow
chosen backend. Model loading can take several minutes. Cancel stops a pending
load. Changing model or saving settings ends the old service and clears history.
A model that `inspect --json` reports as `delegated` is started without the device
-settings: it runs on the GPU its external engine uses. Its `edge0-infer` finds edge0-strata through
+settings. On macOS it runs on Metal through edge0-strata; on Linux and Windows it runs on the NVIDIA GPU that engine uses. Its `edge0-infer` finds edge0-strata through
`EDGE0_STRATA_DIR` in the app's environment or an `edge0-strata/` directory beside an ancestor of the
selected executable.
diff --git a/edge0-infer/examples/desktop/src-tauri/src/service.rs b/edge0-infer/examples/desktop/src-tauri/src/service.rs
index 4708882b..411af2fa 100644
--- a/edge0-infer/examples/desktop/src-tauri/src/service.rs
+++ b/edge0-infer/examples/desktop/src-tauri/src/service.rs
@@ -296,7 +296,7 @@ impl Manager {
} else {
Settings::default()
};
- // No executable chosen yet: use the edge0-infer that the release package puts beside this app.
+ // No executable chosen yet: use the edge0-infer that the release package ships with this app.
if settings.executable.trim().is_empty() {
if let Some(found) = std::env::current_exe()
.ok()
@@ -829,11 +829,21 @@ impl Drop for Manager {
let _ = self.shutdown();
}
}
-/// The edge0-infer in `app_dir` (the directory of this app's own executable), as the release package lays
-/// them out side by side; None when it is not there (a development build, an app bundle).
+/// The edge0-infer that a release package ships with this app, given `app_dir`, the directory of this app's
+/// own executable: beside it (the Windows package), or in the bundle's `Contents/Resources` when the app runs
+/// from `Contents/MacOS` of a macOS app bundle. None when neither is there (a development build).
pub(crate) fn bundled_executable(app_dir: &Path) -> Option {
- let path = app_dir.join(format!("edge0-infer{}", std::env::consts::EXE_SUFFIX));
- path.is_file().then(|| path.to_string_lossy().into_owned())
+ let name = format!("edge0-infer{}", std::env::consts::EXE_SUFFIX);
+ let mut candidates = vec![app_dir.join(&name)];
+ if app_dir.file_name().is_some_and(|n| n == "MacOS") {
+ if let Some(contents) = app_dir.parent().filter(|c| c.file_name().is_some_and(|n| n == "Contents")) {
+ candidates.push(contents.join("Resources").join(&name));
+ }
+ }
+ candidates
+ .into_iter()
+ .find(|path| path.is_file())
+ .map(|path| path.to_string_lossy().into_owned())
}
#[cfg(test)]
@@ -864,6 +874,30 @@ mod tests {
fs::remove_dir_all(dir).unwrap();
}
#[test]
+ fn finds_the_bundled_executable_in_the_app_bundle_resources() {
+ let root = directory();
+ let name = format!("edge0-infer{}", std::env::consts::EXE_SUFFIX);
+ let macos = root.join("Edge0 Chat.app").join("Contents").join("MacOS");
+ let resources = macos.parent().unwrap().join("Resources");
+ fs::create_dir_all(&macos).unwrap();
+ fs::create_dir_all(&resources).unwrap();
+ assert_eq!(bundled_executable(&macos), None);
+ let inside = resources.join(&name);
+ fs::write(&inside, b"").unwrap();
+ assert_eq!(bundled_executable(&macos), Some(inside.to_string_lossy().into_owned()));
+ // The one beside the executable comes first.
+ let beside = macos.join(&name);
+ fs::write(&beside, b"").unwrap();
+ assert_eq!(bundled_executable(&macos), Some(beside.to_string_lossy().into_owned()));
+ // Resources is only looked at from Contents/MacOS.
+ let other = root.join("bin");
+ fs::create_dir_all(&other).unwrap();
+ fs::create_dir_all(root.join("Resources")).unwrap();
+ fs::write(root.join("Resources").join(&name), b"").unwrap();
+ assert_eq!(bundled_executable(&other), None);
+ fs::remove_dir_all(root).unwrap();
+ }
+ #[test]
fn memory_is_steady_only_after_a_full_window_within_tolerance() {
let gb = |v: f64| (v * GIB as f64) as u64;
let mut committed: VecDeque = (0..STEADY_SAMPLES - 1).map(|_| gb(90.0)).collect();
diff --git a/edge0-infer/tests/integration/chat_strata_test.py b/edge0-infer/tests/integration/chat_strata_test.py
index ea01a769..5c9cffd8 100644
--- a/edge0-infer/tests/integration/chat_strata_test.py
+++ b/edge0-infer/tests/integration/chat_strata_test.py
@@ -180,8 +180,12 @@ def failures(cli, root, engine, model):
kept_log(record, result.stderr)
assert_no_leftovers(record)
record = root / 'rejected'
- result, _ = chat(cli, model, engine, record, 'hi\n', ['--device', 'metal'])
- assert result.returncode == 2 and 'CUDA GPU' in result.stderr and not record.exists()
+ if sys.platform == 'darwin':
+ result, _ = chat(cli, model, engine, record, 'hi\n', ['--device', 'cuda'])
+ assert result.returncode == 2 and 'must be metal' in result.stderr and not record.exists()
+ else:
+ result, _ = chat(cli, model, engine, record, 'hi\n', ['--device', 'metal'])
+ assert result.returncode == 2 and 'CUDA GPU' in result.stderr and not record.exists()
result, _ = chat(cli, model, engine, record, 'hi\n', ['--temperature', '0'])
assert result.returncode == 2 and '--temperature' in result.stderr and not record.exists()
print('PASS failures: start failure and engine exit exit 1 with the log tail and a kept log, rejected options start nothing')
@@ -220,14 +224,35 @@ def main():
cancel(args.cli, root, engine, model)
failures(args.cli, root, engine, model)
passthrough(args.cli, root, engine, model)
+ if sys.platform == 'darwin':
+ assert (model / 'mtp-q2_0.gguf').read_bytes() == b'mtp-q2_0'
+ failed_model = make_model(root / 'prepare-fail-model')
+ failed_record = root / 'prepare-fail'
+ failed_env = environment(failed_record)
+ failed_env['FAKE_MTP_MODE'] = 'fail'
+ failed = subprocess.run([str(args.cli), 'chat', '--model', str(failed_model), *strata_args(engine)],
+ input='hi\n', capture_output=True, encoding='utf-8', env=failed_env, timeout=120)
+ assert failed.returncode == 1 and 'preparation failed' in failed.stderr, failed.stderr
+ assert 'log kept at' not in failed.stderr
+ assert not (failed_model / 'mtp-q2_0.gguf').exists()
+ assert not list(temp_dir(failed_record).iterdir()), list(temp_dir(failed_record).iterdir())
+ assert len((model / 'prepare-calls.jsonl').read_text().splitlines()) == 1
+ again, _ = chat(args.cli, model, engine, root / 'chat-again', 'hi\n')
+ assert again.returncode == 0 and 'preparing' not in again.stderr, again.stderr
+ assert len((model / 'prepare-calls.jsonl').read_text().splitlines()) == 1
+ assert_no_leftovers(root / 'chat-again')
fp4 = make_model(root / 'fp4', model_name='edge0-120b-fp4')
record = root / 'fp4-chat'
result, bodies = chat(args.cli, fp4, engine, record, 'hi\n')
- assert result.returncode == 0 and bodies[0]['model'] == 'edge0_120b', result.stderr
- config = json.loads((record / 'config.json').read_text())
- assert config['env']['STRATA_LAT_FUSED_Q3K'] == '0' and not (fp4 / 'scales').exists()
- assert config['args'][1] == str(fp4)
- assert_no_leftovers(record)
+ if sys.platform == 'darwin':
+ assert result.returncode == 2 and 'uses fp4 experts' in result.stderr and not record.exists(), result.stderr
+ assert bodies == [] and not (fp4 / 'mtp-q2_0.gguf').exists()
+ else:
+ assert result.returncode == 0 and bodies[0]['model'] == 'edge0_120b', result.stderr
+ config = json.loads((record / 'config.json').read_text())
+ assert config['env']['STRATA_LAT_FUSED_Q3K'] == '0' and not (fp4 / 'scales').exists()
+ assert config['args'][1] == str(fp4)
+ assert_no_leftovers(record)
print('PASS chat strata')
diff --git a/edge0-infer/tests/integration/fake_strata.py b/edge0-infer/tests/integration/fake_strata.py
index e4979867..dcd1cd73 100644
--- a/edge0-infer/tests/integration/fake_strata.py
+++ b/edge0-infer/tests/integration/fake_strata.py
@@ -41,6 +41,54 @@
ANSWER = 'Hello from fake Strata, answering in several pieces.'
+# Stand-in for edge0-strata's tools/mtp_gguf.py. The delegate launcher runs it with the engine Python
+# when a Metal template's prepare output is missing. FAKE_MTP_MODE selects a failure:
+# fail exit 1 with an error on stderr and no output file
+# exit0 exit 0 with no output file
+# sleep record the call, then sleep (Ctrl-C must kill this process)
+MTP_SCRIPT = '''"""Fake MTP conversion used by delegate-mode tests."""
+import json
+import os
+import sys
+import time
+from pathlib import Path
+
+
+def main():
+ args = sys.argv[1:]
+ out = None
+ if "--out" in args:
+ out = Path(args[args.index("--out") + 1])
+ if out is not None:
+ log = out.parent / "prepare-calls.jsonl"
+ log.parent.mkdir(parents=True, exist_ok=True)
+ with log.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps({
+ "argv": args,
+ "cwd": os.getcwd(),
+ "api_key": os.environ.get("STRATA_API_KEY"),
+ "pid": os.getpid(),
+ }) + "\\n")
+ mode = os.environ.get("FAKE_MTP_MODE", "")
+ if mode == "fail":
+ sys.stderr.write("mtp conversion failed\\n")
+ return 1
+ if mode == "exit0":
+ return 0
+ if mode == "sleep":
+ time.sleep(30)
+ return 0
+ if out is None:
+ sys.stderr.write("missing --out\\n")
+ return 2
+ out.write_bytes(b"mtp-q2_0")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
+'''
+
class Fake:
def __init__(self, config, record):
@@ -520,6 +568,31 @@ def call(method, path, body=None):
if process.poll() is None:
process.kill()
process.wait(10)
+ with tempfile.TemporaryDirectory() as folder:
+ root = Path(folder)
+ script = root / 'mtp_gguf.py'
+ script.write_text(MTP_SCRIPT)
+ out = root / 'model' / 'mtp-q2_0.gguf'
+
+ def run(mode):
+ env = dict(os.environ)
+ env.pop('STRATA_API_KEY', None)
+ if mode:
+ env['FAKE_MTP_MODE'] = mode
+ else:
+ env.pop('FAKE_MTP_MODE', None)
+ return subprocess.run([sys.executable, str(script), '--rt', str(root / 'mtp'), '--out', str(out)],
+ cwd=root, env=env, capture_output=True, text=True)
+
+ ok = run('')
+ assert ok.returncode == 0 and out.read_bytes() == b'mtp-q2_0', (ok.returncode, ok.stderr)
+ recorded = json.loads((out.parent / 'prepare-calls.jsonl').read_text().splitlines()[0])
+ assert Path(recorded['cwd']).resolve() == root.resolve() and recorded['pid'] > 0 and recorded['api_key'] is None
+ out.unlink()
+ failed = run('fail')
+ assert failed.returncode == 1 and 'mtp conversion failed' in failed.stderr and not out.exists()
+ silent = run('exit0')
+ assert silent.returncode == 0 and not out.exists() and silent.stderr == ''
print('PASS fake strata: command line, ready file, status, streaming with timings, echo, records, failure, child cleanup')
diff --git a/edge0-infer/tests/integration/serve_strata_test.py b/edge0-infer/tests/integration/serve_strata_test.py
index 9fcaca5e..ff8fe525 100644
--- a/edge0-infer/tests/integration/serve_strata_test.py
+++ b/edge0-infer/tests/integration/serve_strata_test.py
@@ -194,7 +194,7 @@ def run(name, extra=(), engine_dir=engine, **fake):
assert 'Python for edge0-strata not found' in text and not record.exists()
other = make_engine(root / 'f-engine-2', contract='3')
record, text = run('contract', engine_dir=other)
- assert 'version 3 is not supported' in text and 'supports version 4' in text and not record.exists()
+ assert 'version 3 is not supported' in text and 'supports version 5' in text and not record.exists()
record, text = run('start-failure', fail_start=5)
assert 'exited before it was ready (exit code 5)' in text, text
assert_no_leftovers(record)
@@ -247,7 +247,11 @@ def proxy(cli, root, snapshots):
assert initial['current'] is None and initial['last'] is None and initial['totals']['requests'] == 0
assert initial['instance']['pid'] == server.process.pid and initial['listen'] == {'host': '127.0.0.1', 'port': port, 'auth': False}
assert initial['engine']['version'] == 'fake-strata 0.0' and initial['model']['context_length'] == 131072
- assert initial['engine']['devices'][0]['name'] == 'cuda:0' and initial['engine']['devices'][0]['description'] == 'Fake GPU'
+ device = initial['engine']['devices'][0]
+ expected_device = 'metal' if sys.platform == 'darwin' else 'cuda:0'
+ expected_backend = 'metal' if sys.platform == 'darwin' else 'cuda'
+ assert device['name'] == expected_device and device['description'] == 'Fake GPU'
+ assert device['backend'] == expected_backend and initial['model']['device'] == expected_device
assert initial['memory'] == {'ledger': None, 'resident': None, 'physical': None}
# Non-streaming chat: the request and the response bodies arrive byte for byte.
@@ -680,20 +684,35 @@ def config_of(extra):
return json.loads((record / 'config.json').read_text())
finally:
assert server.stop() == 0
+ mac = sys.platform == 'darwin'
config = config_of([])
- assert config['gpu'] == 0 and config['model_name'] == ALIAS
- assert config['env']['STRATA_LATENT_DENSE'] == str(model / 'latent_dense_q8.bin') and config['env']['STRATA_LATENT_DENSE_FMT'] == '8'
+ assert config['gpu'] == 0 and config['model_name'] == ALIAS and 'prepare' not in config
+ default_du = 'q4' if mac else 'q8'
+ assert config['env']['STRATA_LATENT_DENSE'] == str(model / f'latent_dense_{default_du}.bin')
+ assert config['env']['STRATA_LATENT_DENSE_FMT'] == default_du[1:]
assert config['args'] == ['--pack', str(model), '--ple-io', 'direct', '--max-context', '131072']
assert Path(config['exe']) == engine / 'build' / ('strata.exe' if os.name == 'nt' else 'strata') and config['cwd'] == str(engine)
- config = config_of(['--device', 'cuda:1', '--model-option', 'du=q4', '--model-option', 'ple-io=ram', '--context', '8192',
- '--alias', 'big', '--prefill-device', 'cuda:1'])
- assert config['gpu'] == 1 and config['model_name'] == 'big'
- assert config['env']['STRATA_LATENT_DENSE'] == str(model / 'latent_dense_q4.bin') and config['env']['STRATA_LATENT_DENSE_FMT'] == '4'
- assert config['args'] == ['--pack', str(model), '--ple-io', 'ram', '--max-context', '8192']
- assert config_of(['--device', 'cuda'])['gpu'] == 0
- for extra, message in ((['--device', 'cpu'], 'CUDA GPU'), (['--device', 'metal'], 'CUDA GPU'), (['--device', 'vulkan'], 'CUDA GPU'),
- (['--device', 'cuda:x'], 'CUDA GPU'), (['--prefill-device', 'cpu'], 'must equal --device'),
- (['--model-option', 'du=q5'], 'q8|q4'), (['--model-option', 'ple-io=mmap'], 'direct|ram'),
+ if mac:
+ config = config_of(['--device', 'metal', '--context', '8192', '--alias', 'big'])
+ assert config['gpu'] == 0 and config['model_name'] == 'big' and 'prepare' not in config
+ assert config['args'] == ['--pack', str(model), '--ple-io', 'direct', '--max-context', '8192']
+ rejected = ((['--device', 'cpu'], 'must be metal'), (['--device', 'cuda'], 'must be metal'),
+ (['--device', 'cuda:0'], 'must be metal'), (['--device', 'cuda:1'], 'must be metal'),
+ (['--device', 'vulkan'], 'must be metal'), (['--prefill-device', 'cuda:0'], 'must be metal'),
+ (['--model-option', 'du=q8'], 'only supports du=q4'), (['--model-option', 'du=q5'], 'must be q4'),
+ (['--model-option', 'ple-io=ram'], 'only supports ple-io=direct'),
+ (['--model-option', 'ple-io=mmap'], 'must be direct'))
+ else:
+ config = config_of(['--device', 'cuda:1', '--model-option', 'du=q4', '--model-option', 'ple-io=ram', '--context', '8192',
+ '--alias', 'big', '--prefill-device', 'cuda:1'])
+ assert config['gpu'] == 1 and config['model_name'] == 'big'
+ assert config['env']['STRATA_LATENT_DENSE'] == str(model / 'latent_dense_q4.bin') and config['env']['STRATA_LATENT_DENSE_FMT'] == '4'
+ assert config['args'] == ['--pack', str(model), '--ple-io', 'ram', '--max-context', '8192']
+ assert config_of(['--device', 'cuda'])['gpu'] == 0
+ rejected = ((['--device', 'cpu'], 'CUDA GPU'), (['--device', 'metal'], 'CUDA GPU'), (['--device', 'vulkan'], 'CUDA GPU'),
+ (['--device', 'cuda:x'], 'CUDA GPU'), (['--prefill-device', 'cpu'], 'must equal --device'),
+ (['--model-option', 'du=q5'], 'q8|q4'), (['--model-option', 'ple-io=mmap'], 'direct|ram'))
+ for extra, message in (*rejected,
(['--model-option', 'experts=resident'], 'unknown model option experts'),
(['--prefill-chunk', '64'], '--prefill-chunk'), (['--temperature', '0'], '--temperature'),
(['--max-tokens', '4'], '--max-tokens'), (['--no-warmup'], '--no-warmup'),
@@ -752,19 +771,28 @@ def passthrough(cli, root):
def model_formats(cli, root):
- from strata_support import TEMPLATE
+ from strata_support import TEMPLATE, write_templates
engine = make_engine(root / 'formats-engine')
release = dict(TEMPLATE)
release['args'] = [*TEMPLATE['args'], '--native', '${MODEL_DIR}/model-skeleton.gguf',
'--native-dense-gguf', '${MODEL_DIR}/model-skeleton.gguf',
'--expert-profile', '${MODEL_DIR}/profile.bin']
- for template in ('edge0-145b.json', 'edge0-120b.json'):
- (engine / 'edge0' / 'models' / template).write_text(json.dumps(release))
- for identifier, du in (('edge0_145b', 'q8'), ('edge0_120b', 'q4')):
+ write_templates(engine, release)
+ mac = sys.platform == 'darwin'
+ for identifier, du in (('edge0_145b', 'q4' if mac else 'q8'), ('edge0_120b', 'q4')):
for experts in ('q3k', 'fp4'):
name = identifier.replace('_', '-') + '-' + experts
model = make_model(root / name, model_name=name)
record = root / (name + '-record')
+ if mac and experts == 'fp4':
+ result = subprocess.run([str(cli), 'serve', '--model', str(model), '--port', '0', *strata_args(engine)],
+ env=environment(record), capture_output=True, text=True, timeout=60)
+ assert result.returncode == 2 and 'uses fp4 experts' in result.stderr and 'Metal supports q3k' in result.stderr, result.stderr
+ assert not record.exists() and not (model / 'mtp-q2_0.gguf').exists()
+ inspected = subprocess.run([str(cli), 'inspect', '--model', str(model), '--json'], capture_output=True, text=True)
+ report = json.loads(inspected.stdout)
+ assert inspected.returncode == 1 and report['ok'] is False and 'uses fp4 experts' in inspected.stderr
+ continue
server = Running(cli, model, engine, record)
try:
config = json.loads((record / 'config.json').read_text())
@@ -779,7 +807,7 @@ def model_formats(cli, root):
assert config['env']['STRATA_LAT_FUSED_Q3K'] == ('1' if experts == 'q3k' else '0')
assert Path(config['env']['STRATA_LATENT_DENSE']) == model / f'latent_dense_{du}.bin'
assert config['env']['STRATA_LATENT_DENSE_FMT'] == du[1:]
- assert 'log' not in config and config['exe'] != 'ignored'
+ assert 'log' not in config and 'prepare' not in config and config['exe'] != 'ignored'
finally:
assert server.stop() == 0
assert_no_leftovers(record)
@@ -809,9 +837,12 @@ def model_formats(cli, root):
def inspect_defaults(cli, root):
import shutil
+ mac = sys.platform == 'darwin'
keys = {'adapter', 'delegated', 'engine', 'directory', 'ok', 'files'}
file_keys = {'filename', 'required', 'exists', 'bytes'}
for identifier, default_du, other_du in (('edge0_145b', 'q8', 'q4'), ('edge0_120b', 'q4', 'q8')):
+ if mac:
+ default_du, other_du = 'q4', 'q8'
model = make_model(root / ('inspect-' + identifier), model_name=identifier.replace('_', '-') + '-q3k')
base = [str(cli), 'inspect', '--model', str(model), '--json']
@@ -827,12 +858,20 @@ def inspect(extra, ok):
inspect([], True)
(model / ('latent_dense_' + default_du + '.bin')).unlink()
result, _ = inspect([], False)
- assert result.stderr.rstrip().endswith('use --model-option du=' + other_du + ' instead.'), result.stderr
- inspect(['--model-option', 'du=' + other_du], True)
- result, _ = inspect(['--model-option', 'du=' + default_du], False)
- suggested = result.stderr.rsplit(' use ', 1)[1].removesuffix(' instead.\n').split()
- assert suggested == ['--model-option', 'du=' + other_du]
- inspect(suggested, True)
+ if mac:
+ assert 'latent_dense_q4.bin' in result.stderr and '--model-option' not in result.stderr, result.stderr
+ rejected = subprocess.run([*base, '--model-option', 'du=' + other_du], capture_output=True, text=True)
+ report = json.loads(rejected.stdout)
+ assert rejected.returncode == 1 and report['ok'] is False and 'only supports du=q4' in rejected.stderr, rejected.stderr
+ (model / 'latent_dense_q4.bin').write_bytes(b'q4')
+ inspect([], True)
+ else:
+ assert result.stderr.rstrip().endswith('use --model-option du=' + other_du + ' instead.'), result.stderr
+ inspect(['--model-option', 'du=' + other_du], True)
+ result, _ = inspect(['--model-option', 'du=' + default_du], False)
+ suggested = result.stderr.rsplit(' use ', 1)[1].removesuffix(' instead.\n').split()
+ assert suggested == ['--model-option', 'du=' + other_du]
+ inspect(suggested, True)
shutil.rmtree(model / 'scales')
result, _ = inspect([], False)
assert 'scales/' in result.stderr and '--model-option' not in result.stderr
@@ -1005,12 +1044,126 @@ def launch_handshake(cli, root):
print('PASS launch handshake: environment cleared, banner skipped, missing PID reported, no leftovers')
+def mac_prepare(cli, root):
+ """Metal prepare and the Mac refusals. Other hosts skip these cases inside this process.
+
+ Exiting 77 would skip the CUDA cases that share this ctest, and a separate test is outside this change's scope.
+ """
+ if sys.platform != 'darwin':
+ print('SKIP macOS prepare: this host is not macOS')
+ return
+ engine = make_engine(root / 'mac-engine')
+ model = make_model(root / 'mac-model')
+ record = root / 'mac-first'
+ env = environment(record)
+ env['STRATA_API_KEY'] = 'secret-key'
+ server = Running(cli, model, engine, record, env=env)
+ try:
+ assert 'preparing' in server.text() and 'mtp-q2_0.gguf' in server.text() and '(one time)' in server.text()
+ assert (model / 'mtp-q2_0.gguf').read_bytes() == b'mtp-q2_0'
+ calls = [json.loads(line) for line in (model / 'prepare-calls.jsonl').read_text().splitlines()]
+ assert len(calls) == 1 and calls[0]['api_key'] is None, calls
+ assert Path(calls[0]['cwd']).resolve() == engine.resolve()
+ assert calls[0]['argv'][:2] == ['--rt', str(model / 'mtp')]
+ config = json.loads((record / 'config.json').read_text())
+ assert 'prepare' not in config
+ status = json.loads(request(server.port, 'GET', '/status')[2])
+ assert status['engine']['devices'][0]['name'] == 'metal'
+ assert status['engine']['devices'][0]['backend'] == 'metal'
+ assert status['model']['device'] == 'metal'
+ finally:
+ assert server.stop() == 0
+ assert_no_leftovers(record)
+ rel_model = make_model(root / 'mac-rel-model')
+ rel_record = root / 'mac-rel'
+ rel_server = Running(cli, rel_model, engine, rel_record, extra=['--strata-python', os.path.relpath(sys.executable)])
+ try:
+ assert (rel_model / 'mtp-q2_0.gguf').read_bytes() == b'mtp-q2_0'
+ rel_calls = [json.loads(line) for line in (rel_model / 'prepare-calls.jsonl').read_text().splitlines()]
+ assert Path(rel_calls[0]['cwd']).resolve() == engine.resolve()
+ finally:
+ assert rel_server.stop() == 0
+ assert_no_leftovers(rel_record)
+ record = root / 'mac-second'
+ server = Running(cli, model, engine, record)
+ try:
+ assert 'preparing' not in server.text(), server.text()
+ assert len((model / 'prepare-calls.jsonl').read_text().splitlines()) == 1
+ finally:
+ assert server.stop() == 0
+ assert_no_leftovers(record)
+
+ def rejected(name, extra, message, mode=None):
+ target = make_model(root / name, model_name='edge0-145b-q3k')
+ log = root / (name + '-record')
+ run_env = environment(log)
+ if mode:
+ run_env['FAKE_MTP_MODE'] = mode
+ result = subprocess.run([str(cli), 'serve', '--model', str(target), '--port', '0', *strata_args(engine), *extra],
+ capture_output=True, text=True, env=run_env, timeout=60)
+ assert result.returncode != 0 and 'Listening:' not in result.stderr, (name, result.returncode, result.stderr)
+ assert message in result.stderr, (name, result.stderr)
+ assert not log.exists(), ('a service process started', name)
+ return result
+
+ failed = rejected('mac-fail', [], 'mtp conversion failed', mode='fail')
+ assert failed.returncode == 1 and 'mtp-q2_0.gguf' in failed.stderr and 'mtp_gguf.py' in failed.stderr, failed.stderr
+ assert 'output was not created' in failed.stderr
+ assert not (root / 'mac-fail' / 'edge0-145b' / 'mtp-q2_0.gguf').exists()
+ silent = rejected('mac-exit0', [], 'output was not created', mode='exit0')
+ assert silent.returncode == 1 and 'exit code 0' in silent.stderr and 'mtp_gguf.py' in silent.stderr, silent.stderr
+ assert not (root / 'mac-exit0' / 'edge0-145b' / 'mtp-q2_0.gguf').exists()
+ for name, extra, message in (
+ ('mac-fp4', [], 'uses fp4 experts'),
+ ('mac-q8', ['--model-option', 'du=q8'], 'only supports du=q4'),
+ ('mac-ram', ['--model-option', 'ple-io=ram'], 'only supports ple-io=direct'),
+ ('mac-cuda', ['--device', 'cuda'], 'must be metal')):
+ if name == 'mac-fp4':
+ directory = make_model(root / name, model_name='edge0-120b-fp4')
+ log = root / (name + '-record')
+ result = subprocess.run([str(cli), 'serve', '--model', str(directory), '--port', '0', *strata_args(engine)],
+ capture_output=True, text=True, env=environment(log), timeout=60)
+ assert result.returncode == 2 and message in result.stderr and not log.exists(), result.stderr
+ assert not (directory / 'mtp-q2_0.gguf').exists()
+ else:
+ result = rejected(name, extra, message)
+ assert result.returncode == 2, (name, result.returncode, result.stderr)
+ assert not (root / name / 'edge0-145b' / 'mtp-q2_0.gguf').exists()
+ # Ctrl-C while the script is still running kills it and does not start the service.
+ sleeping = make_model(root / 'mac-sleep')
+ log = root / 'mac-sleep-record'
+ run_env = environment(log)
+ run_env['FAKE_MTP_MODE'] = 'sleep'
+ process = subprocess.Popen([str(cli), 'serve', '--model', str(sleeping), '--port', '0', *strata_args(engine)],
+ stderr=subprocess.PIPE, stdout=subprocess.PIPE, text=True, env=run_env)
+ calls = sleeping / 'prepare-calls.jsonl'
+ deadline = time.time() + 15
+ while time.time() < deadline and not calls.exists():
+ if process.poll() is not None:
+ raise AssertionError(process.stderr.read())
+ time.sleep(0.05)
+ assert calls.exists(), 'prepare script did not start'
+ pid = json.loads(calls.read_text().splitlines()[0])['pid']
+ process.send_signal(signal.SIGINT)
+ try:
+ _, stderr = process.communicate(timeout=15)
+ except subprocess.TimeoutExpired:
+ process.kill()
+ process.communicate(timeout=10)
+ raise AssertionError('Ctrl-C during prepare did not stop edge0-infer')
+ assert process.returncode == 1 and 'interrupted while preparing' in stderr, (process.returncode, stderr)
+ assert not log.exists() and not (sleeping / 'mtp-q2_0.gguf').exists()
+ assert gone(pid), 'prepare script outlived Ctrl-C'
+ print('PASS macOS prepare: once, then skipped; failure and exit 0 leave no file or service; fp4, du=q8, ple-io=ram and cuda rejected; Ctrl-C kills the script')
+
+
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--cli', required=True)
args = parser.parse_args()
with tempfile.TemporaryDirectory() as folder:
root = Path(folder)
+ mac_prepare(args.cli, root)
launch_handshake(args.cli, root)
interrupted_responses(args.cli, root)
inspect_defaults(args.cli, root)
diff --git a/edge0-infer/tests/integration/strata_support.py b/edge0-infer/tests/integration/strata_support.py
index 48659919..ea7b1ae3 100644
--- a/edge0-infer/tests/integration/strata_support.py
+++ b/edge0-infer/tests/integration/strata_support.py
@@ -27,16 +27,37 @@
}
# The desktop App's readiness rule (examples/desktop/src-tauri/src/service.rs, ready_url).
APP_READY = re.compile(r'^Listening: http://(?:127\.0\.0\.1|0\.0\.0\.0):(\d+) model=(\S.*)$')
-
-
-def make_engine(root, contract='4'):
+# Only the Metal template declares this. The fake tools/mtp_gguf.py writes the output unless FAKE_MTP_MODE says otherwise.
+PREPARE = [{
+ 'output': '${MODEL_DIR}/mtp-q2_0.gguf',
+ 'script': 'tools/mtp_gguf.py',
+ 'args': ['--rt', '${MODEL_DIR}/mtp', '--out', '${MODEL_DIR}/mtp-q2_0.gguf'],
+}]
+
+
+def write_templates(engine, body):
+ """CUDA templates as given; Metal templates are the same body plus the prepare step."""
+ models = Path(engine) / 'edge0' / 'models'
+ models.mkdir(parents=True, exist_ok=True)
+ metal = dict(body)
+ metal['prepare'] = PREPARE
+ text = json.dumps(body, indent=1)
+ metal_text = json.dumps(metal, indent=1)
+ for name in ('edge0-145b.json', 'edge0-120b.json'):
+ (models / name).write_text(text)
+ (models / name.replace('.json', '-metal.json')).write_text(metal_text)
+
+
+def make_engine(root, contract='5'):
+ from fake_strata import MTP_SCRIPT
engine = Path(root) / 'edge0-strata'
(engine / 'serve').mkdir(parents=True)
- (engine / 'edge0' / 'models').mkdir(parents=True)
+ (engine / 'tools').mkdir(parents=True)
shutil.copy(HERE / 'fake_strata.py', engine / 'serve' / 'server.py')
+ (engine / 'tools' / 'mtp_gguf.py').write_text(MTP_SCRIPT)
+ (engine / 'edge0').mkdir(parents=True)
(engine / 'edge0' / 'CONTRACT').write_text(contract + '\n')
- (engine / 'edge0' / 'models' / 'edge0-145b.json').write_text(json.dumps(TEMPLATE, indent=1))
- (engine / 'edge0' / 'models' / 'edge0-120b.json').write_text(json.dumps(TEMPLATE, indent=1))
+ write_templates(engine, TEMPLATE)
return engine
diff --git a/edge0-infer/tests/platform/strata/check_mtp_gguf.py b/edge0-infer/tests/platform/strata/check_mtp_gguf.py
new file mode 100644
index 00000000..1cb54931
--- /dev/null
+++ b/edge0-infer/tests/platform/strata/check_mtp_gguf.py
@@ -0,0 +1,114 @@
+"""Compare a rebuilt MTP GGUF with the original mtp-q2_0.gguf.
+
+Same-type tensors must match byte for byte. A Q8_0 tensor rebuilt from the runtime files is compared with the
+original BF16 tensor quantized the way the Metal engine does at load (ref::quantize_q8_0: fp32 d = amax / 127,
+fp16 d by round-to-nearest-even, roundf of x * (1 / d)).
+"""
+import struct
+import sys
+
+import numpy as np
+
+
+def read(path):
+ f = open(path, "rb")
+ magic, ver, nt, nkv = struct.unpack("<4sIQQ", f.read(24))
+ assert magic == b"GGUF" and ver == 3
+
+ def rs():
+ n, = struct.unpack(" Q8_0 vs engine quantization: {bad_blocks}/{ref.shape[0]} blocks differ "
+ f"(scales {bad_scales}, values {int((qd > 0).sum())}, max step {int(qd.max())})")
+ if bad_blocks > 0:
+ ok = False
+ else:
+ print(f"{name}: type {ot} vs {nt}")
+ ok = False
+ print("RESULT:", "same-type tensors identical" if ok else "MISMATCH")
+ return 0 if ok else 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/edge0-infer/tests/platform/strata/compare_outputs.py b/edge0-infer/tests/platform/strata/compare_outputs.py
index a2468489..30d539d3 100644
--- a/edge0-infer/tests/platform/strata/compare_outputs.py
+++ b/edge0-infer/tests/platform/strata/compare_outputs.py
@@ -54,8 +54,11 @@ def main():
parser.add_argument('--device', default='cuda:0')
parser.add_argument('--strata-dir')
parser.add_argument('--max-tokens', type=int, default=200)
+ parser.add_argument('--question', action='append', default=[],
+ help='ask these instead of the built-in three; repeat the option for each question')
parser.add_argument('--out', help='write all answers as JSON')
args = parser.parse_args()
+ questions = args.question or QUESTIONS
command = [args.cli, 'serve', '--model', args.model, '--device', args.device, '--port', '0']
if args.strata_dir:
command += ['--strata-dir', args.strata_dir]
@@ -81,7 +84,7 @@ def read():
if 'proxy' not in ports or 'direct' not in ports:
raise SystemExit('edge0-infer serve did not become ready')
results = []
- for question in QUESTIONS:
+ for question in questions:
first = ask(ports['direct'], ports['alias'], question, args.max_tokens)
second = ask(ports['direct'], ports['alias'], question, args.max_tokens)
proxied = ask(ports['proxy'], ports['alias'], question, args.max_tokens)
diff --git a/edge0-infer/tests/unit/cli/strata_registry_test.cpp b/edge0-infer/tests/unit/cli/strata_registry_test.cpp
index 93e7e324..96e7e233 100644
--- a/edge0-infer/tests/unit/cli/strata_registry_test.cpp
+++ b/edge0-infer/tests/unit/cli/strata_registry_test.cpp
@@ -114,7 +114,8 @@ int main() {
}
// Devices and model options.
- CHECK(parse_device("cuda") == 0 && parse_device("cuda:0") == 0 && parse_device("cuda:1") == 1 && parse_device("cuda:12") == 12);
+ CHECK(parse_device("cuda").gpu == 0 && parse_device("cuda").name == "cuda:0" && parse_device("cuda:0").gpu == 0 &&
+ parse_device("cuda:1").gpu == 1 && parse_device("cuda:1").name == "cuda:1" && parse_device("cuda:12").gpu == 12);
for (const auto* bad : {"cpu", "metal", "vulkan", "cuda:", "cuda:x", "cuda:-1", "CUDA", "cuda:1 "})
CHECK(throws_config([&] { parse_device(bad); }, "CUDA GPU"));
const auto defaults = resolve_options(registration, {});
@@ -206,6 +207,45 @@ int main() {
CHECK(given.strata_args.size() == 2);
}
+ // prepare: placeholders expand like every other field, a relative script is under the engine directory,
+ // and the field is not part of the config handed to server.py.
+ {
+ const auto with_prepare = Json::parse(R"({"exe":"${STRATA_DIR}/build/strata","args":["--mtp","${MODEL_DIR}/mtp"],
+ "prepare":[{"output":"${MODEL_DIR}/mtp-q2_0.gguf","script":"tools/mtp_gguf.py",
+ "args":["--rt","${MODEL_DIR}/mtp","--out","${MODEL_DIR}/mtp-q2_0.gguf"]}]})");
+ std::vector steps;
+ const auto expanded = expand_template(with_prepare, registration, options, Platform::macos, &steps);
+ CHECK(!expanded.contains("prepare"));
+ // The template keeps the slash after ${MODEL_DIR}; the expanded text is not re-normalized.
+ const auto mtp = options.model.string() + "/mtp";
+ const auto mtp_file = options.model.string() + "/mtp-q2_0.gguf";
+ CHECK(expanded["args"][1] == mtp);
+ CHECK(steps.size() == 1);
+ CHECK(steps[0].output == fs::path(mtp_file));
+ CHECK(steps[0].script == options.strata_dir / "tools" / "mtp_gguf.py");
+ CHECK((steps[0].args == std::vector{"--rt", mtp, "--out", mtp_file}));
+ auto absolute = with_prepare;
+ absolute["prepare"][0]["script"] = "${STRATA_DIR}/tools/mtp_gguf.py";
+ expand_template(absolute, registration, options, Platform::macos, &steps);
+ CHECK(steps.size() == 1 && steps[0].script == options.strata_dir / "tools" / "mtp_gguf.py");
+ expand_template(Json::parse(R"({"exe":"x","args":[]})"), registration, options, Platform::posix, &steps);
+ CHECK(steps.empty());
+ const auto stripped = expand_template(with_prepare, registration, options, Platform::posix);
+ CHECK(!stripped.contains("prepare"));
+ CHECK(throws_config([&] {
+ expand_template(Json::parse(R"({"exe":"x","args":[],"prepare":[{"output":"${NOPE}","script":"t.py","args":[]}]})"),
+ registration, options, Platform::posix, &steps);
+ }, "${NOPE}"));
+ CHECK(throws_config([&] {
+ expand_template(Json::parse(R"({"exe":"x","args":[],"prepare":{"output":"a"}})"),
+ registration, options, Platform::posix, &steps);
+ }, "prepare must be an array"));
+ CHECK(throws_config([&] {
+ expand_template(Json::parse(R"({"exe":"x","args":[],"prepare":[{"output":"","script":"t.py","args":[]}]})"),
+ registration, options, Platform::posix, &steps);
+ }, "needs output, script and args"));
+ }
+
// The release templates: every model/format resolves all model files and D/U choices.
CHECK(registry()[1].id == "edge0_120b");
CHECK(registry()[1].config_template == "edge0/models/edge0-120b.json");
@@ -288,11 +328,13 @@ int main() {
CHECK(throws_config([&] { locate_engine(engine.string(), ""); }, "not a directory"));
fs::create_directories(engine / "edge0");
CHECK(throws_config([&] { locate_engine(engine.string(), ""); }, "edge0/CONTRACT"));
- write(engine / "edge0" / "CONTRACT", "4\n");
+ write(engine / "edge0" / "CONTRACT", "5\n");
CHECK(locate_engine(engine.string(), "") == fs::absolute(engine));
check_contract(engine);
write(engine / "edge0" / "CONTRACT", "3\n");
- CHECK(throws_config([&] { check_contract(engine); }, "version 3 is not supported; this edge0-infer supports version 4"));
+ CHECK(throws_config([&] { check_contract(engine); }, "version 3 is not supported; this edge0-infer supports version 5"));
+ write(engine / "edge0" / "CONTRACT", "4\n");
+ CHECK(throws_config([&] { check_contract(engine); }, "version 4 is not supported; this edge0-infer supports version 5"));
write(engine / "edge0" / "CONTRACT", "one\n");
CHECK(throws_config([&] { check_contract(engine); }, "unreadable"));
@@ -321,7 +363,7 @@ int main() {
platform_args.strata_dir = engine.string();
const auto cuda = t.path / "CUDA toolkit";
const auto platform_template = Json::parse(R"({"exe":"${STRATA_DIR}/build/strata${EXE_SUFFIX}","args":[],"lib_dirs":["${CUDA_LIB_DIR}"]})");
- for (const auto platform : {Platform::posix, Platform::windows}) {
+ for (const auto platform : {Platform::posix, Platform::windows, Platform::macos}) {
fs::remove_all(cuda);
for (int layout = 0; layout < 4; ++layout) {
if (layout == 2) fs::create_directories(cuda / "bin");
@@ -329,6 +371,10 @@ int main() {
const auto launch = resolve(platform_args, *detect(model), installed_exe, platform,
layout == 0 ? "" : cuda.string());
const bool windows = platform == Platform::windows;
+ CHECK(launch.device == default_device(platform));
+ CHECK(launch.gpu == 0);
+ CHECK(launch.model_options.at("du") == (platform == Platform::macos ? "q4" : "q8"));
+ CHECK(launch.model_options.at("ple-io") == "direct");
CHECK(launch.python == engine / ".venv" / (windows ? fs::path("Scripts") / "python.exe" : fs::path("bin") / "python"));
const auto expanded = expand_template(platform_template, registration, launch, platform);
CHECK(fs::path(expanded["exe"].get()) == engine / "build" / (windows ? "strata.exe" : "strata"));
@@ -354,20 +400,110 @@ int main() {
CHECK(throws_config([&] { expand_template(Json::parse(bad), registration, launch, platform); }, "${CUDA_LIB_DIR}"));
}
platform_args.python = (t.path / "custom python").string();
- CHECK(resolve(platform_args, *detect(model), installed_exe, platform, "").python == *platform_args.python);
+ CHECK(resolve(platform_args, *detect(model), installed_exe, platform, "").python == fs::absolute(*platform_args.python));
+ platform_args.python = "rel-python";
+ CHECK(resolve(platform_args, *detect(model), installed_exe, platform, "").python == fs::absolute("rel-python"));
platform_args.python.reset();
CHECK(resolve(platform_args, *detect(model), installed_exe, platform, "").strata_args.empty());
platform_args.strata_args = {"--pool-workers", "24"};
CHECK((resolve(platform_args, *detect(model), installed_exe, platform, "").strata_args == platform_args.strata_args));
platform_args.strata_args.clear();
- for (const auto* version : {"0", "1", "2", "3"}) {
+ for (const auto* version : {"0", "1", "2", "3", "4"}) {
write(engine / "edge0" / "CONTRACT", version);
- CHECK(throws_config([&] { check_contract(engine); }, "supports version 4"));
+ CHECK(throws_config([&] { check_contract(engine); }, "supports version 5"));
}
- write(engine / "edge0" / "CONTRACT", "4\n");
+ write(engine / "edge0" / "CONTRACT", "5\n");
check_contract(engine);
}
+ // Device rules for every platform parameter: defaults, rejection, the Metal template, and inspect.
+ {
+ auto run_inspect = [&](Platform platform, const std::map& given) {
+ return inspect_report(*detect(model), model, given, default_device(platform));
+ };
+ const auto metal_template = fs::path("edge0") / "models" / "edge0-145b-metal.json";
+ write(engine / registration.config_template, R"({"exe":"cuda-template","args":[]})");
+ write(engine / metal_template, R"({"exe":"metal-template","args":["--max-context","32768"]})");
+ for (const auto platform : {Platform::posix, Platform::windows, Platform::macos}) {
+ CHECK(default_device(platform) == (platform == Platform::macos ? "metal" : "cuda:0"));
+ if (platform == Platform::macos) {
+ CHECK(parse_device("metal", platform).name == "metal" && parse_device("metal", platform).gpu == 0);
+ for (const auto* bad : {"cuda", "cuda:0", "cuda:1"})
+ CHECK(throws_config([&] { parse_device(bad, platform); }, "must be metal"));
+ } else {
+ CHECK(throws_config([&] { parse_device("metal", platform); }, "CUDA GPU"));
+ }
+ complete(model);
+ write(engine / "edge0" / "CONTRACT", "5\n");
+ CommonArgs bare;
+ bare.model = model.string();
+ bare.strata_dir = engine.string();
+ const auto launch = resolve(bare, *detect(model), installed_exe, platform, "");
+ CHECK(launch.device == default_device(platform));
+ CHECK(load_template(engine, registration, launch.device)["exe"] == (platform == Platform::macos ? "metal-template" : "cuda-template"));
+ if (platform == Platform::macos) {
+ CHECK(launch.model_options.at("du") == "q4" && launch.model_options.at("ple-io") == "direct");
+ write(model / "model.json", R"({"model_name":"edge0-120b-q3k"})");
+ CHECK(resolve(bare, *detect(model), installed_exe, platform, "").model_options.at("du") == "q4");
+ write(model / "model.json", R"({"model_name":"edge0-145b-q3k"})");
+ fs::remove(model / "experts.bin");
+ fs::remove(model / "latent_dense_q4.bin");
+ write(model / "model.json", R"({"model_name":"edge0-145b-fp4"})");
+ fs::remove_all(model / "scales");
+ CHECK(throws_config([&] { resolve(bare, *detect(model), installed_exe, platform, ""); },
+ "unsupported on Metal: edge0-145b-fp4 uses fp4 experts; Metal supports q3k"));
+ CommonArgs illegal = bare;
+ illegal.model_options = {{"du", "q5"}};
+ CHECK(throws_config([&] { resolve(illegal, *detect(model), installed_exe, platform, ""); }, "must be"));
+ CHECK(throws_config([&] { run_inspect(platform, {{"du", "q5"}}); }, "must be"));
+ auto result = run_inspect(platform, {});
+ CHECK(result.code == 1 && result.message.find("uses fp4 experts") != std::string::npos &&
+ result.message.find("Metal supports q3k") != std::string::npos);
+ CHECK(result.report["ok"] == false && result.report["files"].is_array() && !result.report["files"].empty());
+ complete(model);
+ CommonArgs rejected = bare;
+ rejected.model_options = {{"du", "q8"}};
+ fs::remove(model / "latent_dense_q4.bin");
+ CHECK(throws_config([&] { resolve(rejected, *detect(model), installed_exe, platform, ""); },
+ "unsupported on Metal: du=q8; Metal only supports du=q4"));
+ result = run_inspect(platform, {{"du", "q8"}});
+ CHECK(result.code == 1 && result.report["ok"] == false && result.message.find("only supports du=q4") != std::string::npos);
+ rejected.model_options = {{"ple-io", "ram"}};
+ CHECK(throws_config([&] { resolve(rejected, *detect(model), installed_exe, platform, ""); },
+ "Metal only supports ple-io=direct"));
+ result = run_inspect(platform, {{"ple-io", "ram"}});
+ CHECK(result.code == 1 && result.report["ok"] == false && result.message.find("ple-io=direct") != std::string::npos);
+ CHECK(throws_config([&] { run_inspect(platform, {{"experts", "q3k"}}); }, "unknown model option experts"));
+ write(model / "latent_dense_q4.bin", "latent_dense_q4.bin");
+ result = run_inspect(platform, {});
+ CHECK(result.code == 0 && result.report["ok"] == true && result.message.empty());
+ fs::remove(engine / metal_template);
+ CHECK(throws_config([&] { load_template(engine, registration, "metal"); }, "edge0-145b-metal.json"));
+ write(engine / metal_template, R"({"exe":"metal-template","args":[]})");
+ } else {
+ CommonArgs prefill = bare;
+ prefill.prefill_device = "cpu";
+ CHECK(throws_config([&] { resolve(prefill, *detect(model), installed_exe, platform, ""); }, "must equal"));
+ prefill.device = "cuda";
+ prefill.prefill_device = "cuda:0";
+ CHECK(throws_config([&] { resolve(prefill, *detect(model), installed_exe, platform, ""); }, "must equal"));
+ prefill.prefill_device = "cuda:01";
+ CHECK(throws_config([&] { resolve(prefill, *detect(model), installed_exe, platform, ""); }, "must equal"));
+ CommonArgs cuda = bare;
+ cuda.device = "cuda:1";
+ CHECK(resolve(cuda, *detect(model), installed_exe, platform, "").device == "cuda:1");
+ cuda.model_options = {{"du", "q8"}};
+ CHECK(resolve(cuda, *detect(model), installed_exe, platform, "").model_options.at("du") == "q8");
+ write(model / "model.json", R"({"model_name":"edge0-120b-fp4"})");
+ fs::remove_all(model / "scales");
+ CHECK(resolve(bare, *detect(model), installed_exe, platform, "").experts == "fp4");
+ const auto result = run_inspect(platform, {});
+ CHECK(result.code == 0 && result.report["ok"] == true);
+ }
+ }
+ complete(model);
+ }
+
// Response accounting: plain JSON, SSE split at arbitrary points, other dialects.
{
bool active = true;int finished = 0;
@@ -622,7 +758,14 @@ int main() {
input.active = ActiveRequest{"/v1/messages", "127.0.0.1", input.now - std::chrono::seconds(2), Accounting{}};
auto status = status_json(input);
CHECK(status["engine"]["version"] == "0.1.55" && status["engine"]["ggml_version"].is_null() && status["engine"]["backends"].is_null());
- CHECK(status["engine"]["devices"][0]["name"] == "cuda:1" && status["engine"]["devices"][0]["memory_total_bytes"] == 4096LL * 1048576);
+ CHECK(status["engine"]["devices"][0]["name"] == "cuda:1" && status["engine"]["devices"][0]["backend"] == "cuda" &&
+ status["engine"]["devices"][0]["memory_total_bytes"] == 4096LL * 1048576);
+ input.device = "metal";
+ status = status_json(input);
+ CHECK(status["engine"]["devices"][0]["name"] == "metal" && status["engine"]["devices"][0]["backend"] == "metal" &&
+ status["model"]["device"] == "metal");
+ input.device = "cuda:1";
+ status = status_json(input);
CHECK(status["engine"]["devices"][0]["memory_free_bytes"] == 3072LL * 1048576 && status["engine"]["devices"][0]["description"] == "RTX");
CHECK(status["model"]["context_length"] == 131072 && status["model"]["device"] == "cuda:1" && status["model"]["prefill_device"] == "cuda:1");
CHECK(status["memory"]["ledger"].is_null() && status["memory"]["resident"].is_null() && status["memory"]["physical"].is_null());
diff --git a/edge0-infer/tools/checks/check_layers.py b/edge0-infer/tools/checks/check_layers.py
index 41d1a18b..dfcbe130 100644
--- a/edge0-infer/tools/checks/check_layers.py
+++ b/edge0-infer/tools/checks/check_layers.py
@@ -10,10 +10,12 @@
_spec.loader.exec_module(_common)
cmake_commands, files = _common.cmake_commands, _common.files
-# Windows console/stdio setup and delegated process management outside src/platform/.
-# Only these files, and only these headers.
+# Platform headers used by delegated process management, outside src/platform/.
+# tools/cli/serve_delegate_process.cpp: Windows process APIs, and on macOS the
+# header that locates this executable so a sibling edge0-strata/ can be found
+# when the program is started through PATH. Only these files, and only these headers.
CONSOLE_SETUP = {
- "tools/cli/serve_delegate_process.cpp": {"windows.h"},
+ "tools/cli/serve_delegate_process.cpp": {"windows.h", "mach-o/dyld.h", "spawn.h", "unistd.h", "fcntl.h", "sys/wait.h"},
"tools/cli/main.cpp": {"windows.h", "io.h", "fcntl.h"},
"tests/unit/app/template_tools_probe.cpp": {"io.h", "fcntl.h"},
}
diff --git a/edge0-infer/tools/cli/serve_delegate.h b/edge0-infer/tools/cli/serve_delegate.h
index 805e752a..2deefe10 100644
--- a/edge0-infer/tools/cli/serve_delegate.h
+++ b/edge0-infer/tools/cli/serve_delegate.h
@@ -22,7 +22,7 @@ namespace delegate {
using Json = nlohmann::ordered_json;
// The launch contract version this edge0-infer understands (edge0-strata/edge0/CONTRACT).
-constexpr int supported_contract = 4;
+constexpr int supported_contract = 5;
struct RegisteredFile {
std::string path; // relative to the model directory
@@ -37,6 +37,8 @@ struct Registration {
std::vector>> options; // name, allowed values (first = default)
bool thinking = false, thinking_default = false;
std::map names; // model_name -> expert format
+ // Allowed values for one device name. A device that is absent uses `options`. The first value is the default.
+ std::map>>> device_options;
};
const std::vector& registry();
@@ -53,29 +55,47 @@ struct FileState {
struct Match {
const Registration* registration = nullptr;
std::vector files;
+ std::string model_name; // the model.json value that selected this registration
std::string experts; // selected by model_name, never a user option
// Required files (for the given options) that are absent.
std::vector missing(const std::map& options) const;
// Inspect requires the selected configuration, using the same defaults as serve for omitted options.
- bool complete(const std::map& selected = {}) const;
+ // `device` is "metal" or "cuda:"; the default is the CUDA device, so a caller that has not chosen a platform
+ // keeps the CUDA rules. inspect passes the host's default device.
+ bool complete(const std::map& selected = {}, const std::string& device = "cuda:0") const;
// Option choices ("
+