Gateway fallback routing: a slow engine can hand one request to an engine that needs no card

CHANGELOG.md

@@ -8,6 +8,16 @@ has a matching GitHub release whose body comes from its section below.
 
 ### Added
 
+- Gateway fallback routing: an `[[engine]]` may name a `fallback`, another engine in the file that
+  needs no card (a CPU llama-server, an Ollama on another box), and `fallback_after` (10 s by
+  default). A request that has waited that long for its own engine is proxied to the fallback when
+  the fallback answers its probe at that moment, with the body's model renamed when `fallback_model`
+  is set. A non-streaming answer carries `X-Berth-Fallback: <fallback>`; a streaming one, whose SSE
+  preamble was already sent, gets the fallback's stream after one `: berth: fallback <fallback>`
+  comment. The gateway never starts, leases or drains the fallback, and the primary's start is never
+  cancelled, so later requests land on it. The request reaches the fallback without its
+  `Authorization` header, so the node's API key never leaves the gateway. One hop: `berth config check`
+  refuses chains.
 - Cgroup attribution closes the one known stowaway window (chaos scenario 3: a child
   whose parent exits within one 2 s tick). On Linux with a systemd user manager,
   `berth run` execs the engine in its own transient cgroup scope (`systemd-run --user

README.md

@@ -630,6 +630,39 @@ than `--ready` (10 minutes); if the holder has not left by then, the request fai
 dropped, so it cannot take the card hours later. Asking llama-swap for one of its
 models later drains the gateway's engine the same way, so either side can take the card back.
 
+**A slow engine can name a fallback.** An interactive client should not sit through a 90 s cold start
+when an engine that needs no card can answer now. Give an engine a `fallback` (another engine in the
+same file, such as a CPU llama-server) and a request that has waited `fallback_after` (10 s by
+default) goes there instead, when the fallback answers its probe at that moment. The fallback is
+never started, leased or drained by the gateway: one that is not answering means no fallback, and the
+request keeps waiting for its own engine. The primary's start is never cancelled either way, so
+requests that come later land on it. `fallback_model` names the model the fallback lists the request
+under when the two disagree, a non-streaming answer carries `X-Berth-Fallback: <fallback>`, and a
+streaming answer, whose headers went out with the keep-alive preamble, carries one
+`: berth: fallback <fallback>` comment before its data. The fallback gets the request without its
+`Authorization` header: the node's API key stays with the gateway. One hop, per engine, opt-in: `berth config check`
+refuses chains and a fallback that is not an engine in the file.
+
+```toml
+[[engine]]
+name     = "strata"
+type     = "llamacpp"
+url      = "http://127.0.0.1:8080"
+start    = ["/path/to/strata/.venv/bin/python", "serve/server.py", "--engine", "strata", "--port", "8080"]
+serves   = ["qwen3.8-flash-next-iq2_xs"]
+priority = "interactive"
+fallback       = "strata-cpu"
+fallback_after = "10s"
+fallback_model = "qwen3.8-cpu"   # what the CPU server lists the model as
+
+[[engine]]
+name  = "strata-cpu"
+type  = "llamacpp"
+url   = "http://127.0.0.1:8010"
+start = ["llama-server", "-m", "/models/qwen3.8-cpu.gguf", "--port", "8010"]   # no GPU
+serves = ["qwen3.8-cpu"]
+```
+
 **A start reads the engine's files into the page cache while the holder drains.** Give an engine a
 `prefetch` list of absolute files and directories, and the gateway asks the kernel to read them the
 moment it decides to start that engine, before its `berth run` waits for the lease. The kernel reads

internal/config/config.go

@@ -101,6 +101,19 @@ type Engine struct {
 	// gateway's wake page (/gui/<name>) starts the engine on a visit and
 	// follows it there once it answers.
 	GUI string `toml:"gui,omitempty" json:"gui,omitempty" yaml:"gui,omitempty"`
+	// Fallback is the name of another engine in this file: while a request
+	// waits for this engine to come up, past FallbackAfter, the gateway
+	// sends it to the fallback instead, if the fallback answers its probe
+	// at that moment. The gateway never starts the fallback, and the start
+	// of this engine is never cancelled: later requests land on it. No
+	// chains: a fallback has no fallback of its own.
+	Fallback string `toml:"fallback,omitempty" json:"fallback,omitempty" yaml:"fallback,omitempty"`
+	// FallbackAfter is how long a request waits for this engine before it
+	// may go to the fallback. 0 = 10 s when Fallback is set.
+	FallbackAfter Duration `toml:"fallback_after,omitzero" json:"fallback_after,omitempty" yaml:"fallback_after,omitempty"`
+	// FallbackModel is the model name sent to the fallback; unset keeps
+	// the request's model name.
+	FallbackModel string `toml:"fallback_model,omitempty" json:"fallback_model,omitempty" yaml:"fallback_model,omitempty"`
 }
 
 // Types are the engine readers that exist.
@@ -613,6 +626,32 @@ func (c *Config) Validate() error {
 			}
 		}
 	}
+	// A fallback is resolved by name after every engine is known, so it is
+	// checked in a second pass. An error names both engines.
+	byName := map[string]int{}
+	for i, e := range c.Engine {
+		if e.Name != "" {
+			byName[e.Name] = i
+		}
+	}
+	for i, e := range c.Engine {
+		if e.Fallback == "" {
+			continue
+		}
+		where := fmt.Sprintf("engine[%d] %q", i, e.Name)
+		if e.FallbackAfter.Duration < 0 {
+			errs = append(errs, fmt.Errorf("%s: fallback_after %s must be >= 0", where, e.FallbackAfter))
+		}
+		j, ok := byName[e.Fallback]
+		switch {
+		case !ok:
+			errs = append(errs, fmt.Errorf("%s: fallback %q is not an engine in this file", where, e.Fallback))
+		case j == i:
+			errs = append(errs, fmt.Errorf("%s: fallback names the engine itself", where))
+		case c.Engine[j].Fallback != "":
+			errs = append(errs, fmt.Errorf("%s: fallback %q has a fallback of its own; no chains", where, e.Fallback))
+		}
+	}
 	for _, p := range c.Engines.Ports {
 		if p <= 0 || p > 65535 {
 			errs = append(errs, fmt.Errorf("engines.discover_ports: %d is not a port", p))

internal/config/config_test.go

@@ -4,6 +4,7 @@
 package config
 
 import (
+	"fmt"
 	"os"
 	"path/filepath"
 	"strconv"
@@ -718,3 +719,63 @@ func TestLibraryFloorDefaults(t *testing.T) {
 		t.Errorf("explicit 0 = %g, want the guard off", c.LibraryFloor())
 	}
 }
+
+func TestEngineFallbackParsedAndValidated(t *testing.T) {
+	const engines = `
+[[engine]]
+name   = "strata"
+type   = "llamacpp"
+url    = "http://127.0.0.1:8080"
+serves = ["qwen3.8"]
+%s
+[[engine]]
+name   = "strata-cpu"
+type   = "llamacpp"
+url    = "http://127.0.0.1:8081"
+serves = ["qwen3.8-cpu"]
+`
+	c, err := Load(write(t, fmt.Sprintf(engines, `
+fallback = "strata-cpu"
+fallback_after = "30s"
+fallback_model = "qwen3.8-cpu"
+`)))
+	if err != nil {
+		t.Fatalf("a valid fallback was refused: %v", err)
+	}
+	if c.Engine[0].Fallback != "strata-cpu" || c.Engine[0].FallbackAfter.Seconds() != 30 || c.Engine[0].FallbackModel != "qwen3.8-cpu" {
+		t.Fatalf("fallback keys not loaded: %+v", c.Engine[0])
+	}
+	// Without fallback_after the gateway applies its own default.
+	c, err = Load(write(t, fmt.Sprintf(engines, `fallback = "strata-cpu"`)))
+	if err != nil || c.Engine[0].FallbackAfter.Duration != 0 {
+		t.Fatalf("no fallback_after: %+v err=%v", c.Engine[0], err)
+	}
+
+	for _, tc := range []struct {
+		name    string
+		keys    string
+		wantErr string
+	}{
+		{"unknown engine", `fallback = "nope"`, `engine[0] "strata": fallback "nope" is not an engine in this file`},
+		{"itself", `fallback = "strata"`, `engine[0] "strata": fallback names the engine itself`},
+		{"negative after", `fallback = "strata-cpu"
+fallback_after = "-5s"`, `engine[0] "strata": fallback_after -5s must be >= 0`},
+	} {
+		_, err := Load(write(t, fmt.Sprintf(engines, tc.keys)))
+		if err == nil || !strings.Contains(err.Error(), tc.wantErr) {
+			t.Errorf("%s: error lacks %q: %v", tc.name, tc.wantErr, err)
+		}
+	}
+	// A fallback that has a fallback of its own is refused, and the error
+	// names both engines.
+	_, err = Load(write(t, fmt.Sprintf(engines, `fallback = "strata-cpu"`)+`
+[[engine]]
+name   = "third"
+type   = "llamacpp"
+url    = "http://127.0.0.1:8082"
+fallback = "strata"
+`))
+	if err == nil || !strings.Contains(err.Error(), `engine[2] "third": fallback "strata" has a fallback of its own; no chains`) {
+		t.Errorf("a chain was accepted or names neither engine: %v", err)
+	}
+}

internal/gateway/fallback.go

@@ -0,0 +1,121 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+// Fallback routing: an engine may name another engine to take one request
+// while it is still coming up, so an interactive client does not sit
+// through a 90 s cold start when something that needs no card can answer
+// now. Opt-in per engine, one hop, and the primary's start is never
+// cancelled by it.
+package gateway
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"time"
+
+	"github.com/T-Crypt/berth/internal/config"
+)
+
+// defaultFallbackAfter is how long a request waits for its engine before
+// it may go to the fallback, when the engine names one but sets no time.
+const defaultFallbackAfter = 10 * time.Second
+
+func (g *Gateway) fallbackAfter(e *config.Engine) time.Duration {
+	if e.FallbackAfter.Duration > 0 {
+		return e.FallbackAfter.Duration
+	}
+	return defaultFallbackAfter
+}
+
+// waitPrimary runs ensure for the engine a request resolved to, and waits
+// it out. When the engine names a fallback, the wait watches the clock:
+// the engine still not answering after fallback_after, and the fallback
+// answering its probe right then, this request goes to the fallback
+// instead. The probe is all the gateway ever sends a fallback it did not
+// see up: one that is not answering means no fallback, and the request
+// keeps waiting for the primary as today. ensure is never cancelled
+// either way, so the primary's start goes on and later requests land on
+// it.
+//
+// The return says where the request goes: fb nil means the primary (err
+// is ensure's outcome); fb set means proxy it there, waited long. A
+// streaming response (w non-nil) gets its loading comments every
+// keepAliveEvery while it waits, as before.
+func (g *Gateway) waitPrimary(ctx context.Context, e *config.Engine, w http.ResponseWriter) (fb *config.Engine, waited time.Duration, err error) {
+	start := time.Now()
+	// A request that switches to the fallback returns without its engine's
+	// wait being over, and the handler's context goes with it: the wait for
+	// the primary rides on one that survives, so the start is never
+	// cancelled and answering late still counts for the next request.
+	ensureCtx := ctx
+	if e.Fallback != "" {
+		ensureCtx = context.WithoutCancel(ctx)
+	}
+	ch := make(chan error, 1)
+	go func() { ch <- g.ensure(ensureCtx, e) }()
+	var fbAt <-chan time.Time
+	if e.Fallback != "" {
+		at := time.NewTimer(g.fallbackAfter(e))
+		defer at.Stop()
+		fbAt = at.C // a Timer fires once: the fallback gets one look
+	}
+	var tick <-chan time.Time
+	if w != nil {
+		every := time.NewTicker(keepAliveEvery)
+		defer every.Stop()
+		tick = every.C
+	}
+	for {
+		select {
+		case err := <-ch:
+			return nil, time.Since(start), err
+		case <-ctx.Done():
+			return nil, time.Since(start), ctx.Err()
+		case <-tick:
+			fmt.Fprintf(w, ": berth: loading %s, %d s\n\n", e.Name, int64(time.Since(start).Seconds()))
+			_ = http.NewResponseController(w).Flush()
+		case <-fbAt:
+			f := g.engineByName(e.Fallback)
+			if f != nil && g.answers(ctx, f.URL) {
+				return f, time.Since(start), nil
+			}
+		}
+	}
+}
+
+// tookFallback hands one request to the fallback: it journals the switch,
+// rewrites the body's model when the fallback lists it under another
+// name, and marks the response, a header while the headers are still open
+// and an SSE comment once the preamble is already on the wire. It returns
+// the base URL to proxy to and the body to send.
+func (g *Gateway) tookFallback(w http.ResponseWriter, preamble bool, e, fb *config.Engine, waited time.Duration, body []byte) (string, []byte) {
+	if e.FallbackModel != "" {
+		body = rewriteModel(body, e.FallbackModel)
+	}
+	g.log().Info("gateway request fell back", "engine", e.Name, "fallback", fb.Name, "waited", waited.Round(time.Millisecond))
+	if preamble {
+		fmt.Fprintf(w, ": berth: fallback %s\n\n", fb.Name)
+		_ = http.NewResponseController(w).Flush()
+	} else {
+		w.Header().Set("X-Berth-Fallback", fb.Name)
+	}
+	return fb.URL, body
+}
+
+// rewriteModel replaces the request body's model name. Everything else in
+// the body goes through byte for byte.
+func rewriteModel(body []byte, model string) []byte {
+	var req map[string]json.RawMessage
+	if json.Unmarshal(body, &req) != nil {
+		return body
+	}
+	name, _ := json.Marshal(model)
+	req["model"] = name
+	out, err := json.Marshal(req)
+	if err != nil {
+		return body
+	}
+	return out
+}