BROKE Runs On Keen Efficiency

BROKE Cluster: Multi-Node LLM Routing for Apple Silicon

What MLX Knife is to one Mac, BROKE is to a handful of them.

You have a few Apple Silicon Macs and a few local models spread across them. You send a request to one address, and BROKE works out which machine and which model should answer it:

Prompt → {node, model}

One OpenAI-compatible entry point, no node addresses in your clients, and the data never leaves the building.

Status — July 2026

This repository publishes documentation only. There is no code here yet.

The mission above is intact and has not moved since 2025. The design underneath it is being reworked, and until that settles there is nothing worth publishing. The effort so far has gone into MLX Knife maturing into a solid building block for BROKE — gaining workspace management and multimodality it wasn't originally built for. That groundwork is why BROKE can stay a thin layer on top.

• A prototype exists, but it is not the direction we are keeping. It does route across a small Mac cluster — by having the router estimate how hard each incoming prompt is (the "Complexity Analysis" box in the diagram below). Putting that guess inside the router is the decision we are walking back, so it gets redesigned rather than released. The cluster-commissioning tooling is being rebuilt alongside it, against the current MLX Knife.

• No release date, no timeline. BROKE is a side project. It ships when it is honest to ship.

The division of labour

Don't rebuild what the node already does. Model management and inference are solved, per machine, by MLX Knife. Deciding which node, which model is not — and that is the only thing BROKE has to be good at. Every feature that does not serve that decision belongs somewhere else.

That split is settled; the rework only touches how models get chosen. The line it draws:

• MLX Knife is the node executive, and it is stateless. Everything that happens on a machine is its job: which models are present, their health and identity, loading them, serving them over an OpenAI-compatible API. It keeps no state and no configuration beyond environment variables, and it knows nothing about any other machine — whatever you ask it, the answer is about the box it runs on. It is reached across the network; it never reaches across the network itself.
• BROKE owns exactly what spans machines, and nothing else: which nodes exist, which model lives where, getting the models onto the nodes in the first place, and choosing the node for a given request. That a model belongs on a node is recorded nowhere in MLX Knife — that binding is BROKE's, established when a cluster is commissioned and spent when a request is routed. BROKE re-implements no inference: it delegates to the node's MLX Knife and reads its published, versioned JSON API (mlxk list --json, mlxk health --json) instead of parsing human-readable CLI output.
• One endpoint, no node addresses in your clients. A client is configured once with a single base_url. Moving or retiring a model across nodes stays a central concern, not a client edit.
• Local-first. Apple Silicon and MLX; models and prompts stay on your own hardware. Escalation to hosted APIs is a decision you make in your own agent framework, alongside BROKE — not something BROKE does on your behalf.
• Primitive first. SSH workers, explicit configuration, no container orchestration. Right-sized for a handful of Macs and a small team, not for a data center.

Where this is heading

The next milestone is a usable cluster (v0.2): a pinned-model client — Zed, an editor config, any OpenAI tool with a model field — works against BROKE with nothing changed but its base_url. Two pieces get there: commissioning (turn a few SSH-enabled Macs into a running cluster — install the runtime, place the models, verify them) and the location router (one endpoint that knows which model lives where and routes to a node that has it).

After that: a benchmark that measures the real routed path, a thin web UI the router serves itself, and an opt-in model-selection layer (named aliases like auto:code — never prompt-sniffing). One 0.x version line; the old dual versioning is gone.

No dates. It ships when the promise above is demonstrable: an unchanged OpenAI client, only its base_url swapped, working against the cluster.

What you can use today: MLX Knife

MLX Knife is the per-node component, and unlike BROKE it is released, working software:

pip install mlx-knife          # Python 3.10+ (3.9 is not supported)
mlxk list                      # what's on this machine
mlxk serve                     # OpenAI-compatible server

Version baseline for BROKE: the rework builds on the current MLX Knife 2.x line. An exact minimum version gets pinned once there is BROKE code to pin it in — until then, treat any version reference in the older BROKE material as stale.

Architecture

(The diagram reflects the 2025 design. The execution layer — a per-node MLX Knife behind an OpenAI-compatible API — holds. The routing box and the commissioning path are what is being reworked.)

Hardware

Nothing exotic — a couple of Apple Silicon Macs on a local network:

• Minimal: one Mac with 32 GB+ (M1 Pro/Max or better), 4-bit quantized MLX models.
• Small team: a Mac Studio for the large models, one or two Mac Minis for the smaller ones, gigabit LAN between them.

Getting involved

There is no code to contribute to yet, so the useful thing right now is conversation, not pull requests:

• 💬 Discussions — how you route between models today, what your agent framework actually needs, where this would break for you.
• 🧪 MLX Knife — run the single-node piece; that is the layer BROKE will orchestrate.
• 🐛 Issues — for the published material, including anything in here that is still wrong.

---

BROKE = BROKE Runs On Keen Efficiency 🦫
The beaver builds a dam so your data stops flowing to the cloud.