perf: improve cache hits and request latency

CONTEXT.md

@@ -0,0 +1,61 @@
+# LLM Proxy
+
+This context describes the language used to discuss interactive model requests
+and their performance through routatic-proxy.
+
+## Language
+
+**Time to First Token (TTFT)**:
+For a streaming request, the elapsed time from when the proxy starts reading
+the request until it writes the first non-empty text, thinking, or tool content
+to the client.
+_Avoid_: First-byte time, total response time
+
+**Repeated Conversation**:
+A model request that includes a stable prefix from earlier turns plus a new
+turn.
+_Avoid_: Warm request, duplicate request
+
+**Cache Directive**:
+A client-supplied instruction that marks a prompt boundary as eligible for
+reuse by the Provider Prompt Cache.
+_Avoid_: Automatic cache rule, proxy cache marker
+
+**Cache Usage**:
+Provider-reported token counts describing prompt data read from or written to
+the Provider Prompt Cache.
+_Avoid_: Token Count Cache hit, inferred cache use
+
+**Provider Prompt Cache**:
+An upstream provider feature that reuses an unchanged prompt prefix across
+model requests.
+_Avoid_: Token Count Cache, response cache
+
+**Token Count Cache**:
+A process-local cache that reuses tokenizer results without storing raw prompt
+text.
+_Avoid_: Provider Prompt Cache, response cache
+
+**Completion Record**:
+Best-effort analytics data describing one finished model request. Losing this
+record must not change the model response seen by the client.
+_Avoid_: Response, durable event
+
+**Catalog Snapshot**:
+The last successfully loaded set of providers, models, and routing scenarios
+used for model selection.
+_Avoid_: Live catalog, catalog request
+
+**Known Model**:
+A model present in the active configuration or catalog snapshot.
+_Avoid_: Requested model, arbitrary model name
+
+**Model Identity**:
+The canonical `provider/model` name used to distinguish a model across
+providers.
+_Avoid_: Short model name, display name
+
+**Unknown Content Block**:
+A client content block whose type is not yet modeled by the proxy but whose raw
+data and position remain intact.
+_Avoid_: Invalid block, ignored block

docs/adr/0001-user-responses-before-analytics.md

@@ -0,0 +1,7 @@
+# User responses take priority over analytics completeness
+
+Completion records are best-effort analytics. The proxy never delays a model
+response when the storage queue is full, so it drops the newest analytics
+record and reports the drop. During normal shutdown it drains accepted records
+until the existing shutdown deadline, then reports any remaining loss. This
+trades complete analytics for predictable user-facing latency.

docs/adr/0002-client-owned-cache-directives.md

@@ -0,0 +1,6 @@
+# Provider Prompt Cache directives are owned by the client
+
+The proxy preserves cache directives supplied at the request, tool, system, and
+message-block levels, but it does not add, move, or infer cache boundaries.
+Provider adapters may omit a directive only when their wire format cannot use
+it. This favors predictable request behavior over automatic cache tuning.

docs/adr/0003-ordered-normalized-content.md

@@ -0,0 +1,8 @@
+# Normalized content keeps client block order
+
+The canonical request model uses one ordered list of content blocks for system
+and message content. Text, images, thinking, tool calls, tool results, and cache
+directives stay at their original positions. This requires a larger core
+refactor, but it prevents separate convenience fields from changing request
+order or losing cache boundaries. Unknown block types remain as ordered raw
+JSON for compatible provider formats and are never silently dropped.

docs/adr/0004-last-valid-catalog-remains-usable.md

@@ -0,0 +1,7 @@
+# The last valid catalog remains usable
+
+After the proxy loads a valid catalog, refresh failures never make that catalog
+unusable solely because of age. The proxy reports snapshot age and refresh
+errors, but keeps routing with the last valid data. Legacy routing is used only
+when no valid catalog has ever loaded, favoring request availability over
+catalog freshness.