Repository navigation
Gateway fallback routing: a slow engine can hand one request to an engine that needs no card - #114
Merged
Merged
Conversation
An [[engine]] may name a fallback (another engine in the file), fallback_after (default 10 s at the gateway) and fallback_model (what the fallback lists the model as). berth config check refuses a fallback that is not an engine, names the engine itself, or has a fallback of its own; the error names both engines.
A request for an engine that is coming up runs ensure as before, but when fallback_after passes, the engine still is not answering and the fallback answers its probe right then, this one request is proxied there: the body's model rewritten when fallback_model is set, the response marked with X-Berth-Fallback, and a streaming request whose SSE preamble is already on the wire served through the same body-only path as the primary, with one comment marking the switch. The gateway never starts, leases or drains the fallback, and the primary's start is never cancelled: the wait rides on a context that survives the switch, so later requests land on the primary. One INFO line per switch names the engine, the fallback and the wait. The old keepAlive loop is waitPrimary now; an engine without a fallback behaves exactly as before.
Fake engines and the fake runner: a slow primary falls back after the wait (non-streaming and with the SSE preamble already sent); the primary answering first, or a fallback that is down, leaves the request on the primary; the model is rewritten only when fallback_model is set; X-Berth-Fallback marks a non-streaming answer; one INFO line per switch; the gateway takes no lease for the fallback; the primary's start answers after a switch just the same.
One paragraph next to the model-routing section: a slow engine names a fallback in the same berth.toml, the request goes there after fallback_after when the fallback answers now, the fallback is never started or leased by the gateway, and the primary's start goes on. A toml example pairs Strata with a CPU llama-server. A CHANGELOG Added line under Unreleased.
The client's bearer token authenticates it to the gateway. The proxy passed it on, and a fallback may run on another box. TestFallbackDropsTheNodeKey fails without the change.
…thorization header
# Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OME APEPGNW. README "Next: v0.3.x": gateway fallback routing.
Opt-in per engine. An
[[engine]]may name afallback(another engine in the file),fallback_after(10 s by default) andfallback_model. A request that has waitedfallback_afterfor its own engine is proxied to the fallback if the fallback answers its probe at that moment. Otherwise it keeps waiting as before. An engine withoutfallbackbehaves exactly as before.X-Berth-Fallback: <name>. A streaming one, whose SSE headers already went out, gets one: berth: fallback <name>comment.Authorizationheader, so the node's API key never leaves the gateway. A fallback can run on another box.berth config checkrefuses chains, self-references and unknown names.Proof:
TestFallbackDropsTheNodeKeyfails without the header drop and passes with it.go test ./..., Windows build and vet, golangci-lint (0 issues).