A gateway that sorts each LLM request onto the right model.
One small Rust binary in front of your providers: it speaks OpenAI Chat, OpenAI Responses and Anthropic Messages, routes by what each request is, fails over between providers, and tells you what it decided and why.
Ask a question, share your setup, tell us what to build next.
If humpyard saves you time or money, a star helps other people find it.
Status: pre-alpha. It runs real traffic (Claude Code, Codex, OpenClaw) but expect rough edges. Early users shape what comes next: tell us what you run it with on Discord.
No API keys needed: run the demo yourself. It uses a mock provider; the routing, the ledger and the cost math are real.
Why the name: a hump yard is the railroad yard where cars are classified and sent down the right track, one by one. That is the job here, with requests and models.
Agent tools such as Claude Code and Codex send a stream of very different requests: a one-line question, a 60,000-token tool loop, a sub-agent doing narrow work. Sending all of it to one model is slow or expensive. humpyard is the routing and policy layer in front of your providers.
- Content-aware routing. Rules pick the pool from the key, headers, agent metadata, prompt size, tools and images. Clients can narrow, never widen, what a key may reach.
- Explainable and measured. Ask what a request would do before sending it, then see what it did: every decision, fallback and refusal is recorded, with latency and cost per route.
- Resilient. Ordered provider pools, per-endpoint circuit breakers, and control over which failures may fall through to the next pool.
- Easy to run. A static binary or a small container, a SQLite ledger with virtual keys and budgets, hot config reload, and money handled as integers that fail closed.
The fastest way: let humpyard init write the config. It checks your provider's models live and sends a test request.
export GROQ_API_KEY=... # your key; init reads it from the environment and never stores it
humpyard init --provider groq && humpyard serve --config config.tomlOr by hand with the container:
curl -fsSLO https://raw.githubusercontent.com/gschaetz/humpyard/main/examples/quickstart.toml
export GROQ_API_KEY=... # your key; the config reads it from the environment
docker run -d -p 127.0.0.1:8080:8080 -e GROQ_API_KEY \
-v "$PWD/quickstart.toml:/etc/humpyard/config.toml:ro" ghcr.io/gschaetz/humpyard:0.21.0
curl localhost:8080/v1/chat/completions -H 'content-type: application/json' \
-d '{"model":"fast","messages":[{"role":"user","content":"hi"}]}'quickstart.toml is one provider and one model (any OpenAI-compatible
endpoint works), published on 127.0.0.1 only. It has no virtual keys, so keep it local until you add
some. No keys and no network? Try the demo first.
No Docker? Download a binary from the releases
(check it against SHA256SUMS; on macOS a browser-downloaded binary may need
xattr -d com.apple.quarantine humpyard), or cargo build --release, then
humpyard serve --config quickstart.toml. That config listens on all interfaces because it is meant
for the container, so set listen = "127.0.0.1:8080" if the machine is on a shared network. A fuller
starting config, with failover across two providers, is in examples/config.toml.
| I want to... | Go to |
|---|---|
| install and run it (binary, container, macOS service) | Deployment |
| write a config: providers, targets, routes, keys | Configuration · Examples |
| point Claude Code, Codex or OpenClaw at it | Clients |
route one auto model to the right pool |
Auto routing recipe |
| understand failover, selectors and health | Routing |
| set budgets, read the usage and decision log, see stats | Keys, budgets and the ledger |
| call the API and its endpoints | API and endpoints |
| look at a running gateway from the command line (status, stats, decisions, explain) | humpyard ctl |
| move from LiteLLM | Migrating from LiteLLM |
| move from ModelRelay | Moving from ModelRelay |
| understand the design and why | Architecture · Decision records |
| see what has shipped and what is next | Roadmap |
Questions, feedback and show-and-tell are welcome on Discord. Bugs and feature requests belong in GitHub issues, so they stay searchable. Please never paste API keys or full configs in chat. The roadmap says where routing is going and lists where help is wanted.
Specs live in openspec/ and decisions in docs/adr/; the rules that must
keep holding, and what enforces them, are in Invariants.
Contributions need a DCO sign-off: see CONTRIBUTING.md.
Locally: cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test.
Apache-2.0, see LICENSE and NOTICE.
humpyard is an independent project. It is built on NVIDIA NeMo Switchyard (Apache-2.0) and is not affiliated with or endorsed by NVIDIA. NVIDIA and NeMo are trademarks of NVIDIA Corporation.