Derives from: ADR-0015, ADR-0004, ADR-0011 (security standard).
LearnStack uses Apache APISIX in standalone mode as the API gateway in front of both the LearnStack core API and the LearnStack Hub API.
Status (2026-08-08). APISIX is demand-gated to Phase 11 per ADR-0035; its trigger is a non-development deployment that needs edge rate limiting, host routing, or JWT pre-validation. Until then the same responsibilities are carried by ASP.NET middleware inside the API process: correlation-id assignment, CORS, JWT bearer validation, rate limiting, and header normalisation. ADR-0015 stands as the decision that APISIX is the gateway LearnStack uses; ADR-0035 decides when.
Backend boundary — P02b-5 Step 4, 2026-10-19. The gateway remains gated and route 100's
openid-connectblock remains commented out. The backend now validates the closed tenant bearer profile under ADR-0064, before tenant reconciliation. Its production surface is the existing anonymous reads plus exact AccountSite GET; application authorization still passes through until Phase 03. No authoring command is routed. Present invalid credentials return 401 even on public routes; absent AccountSite credentials return masked 404. The “validated twice” property in § 5 remains the Phase 11 target. The Step 4 delivery record owns validation and review status, rather than this topology claiming completion.
Accepted P02d-5 local topology — 2026-10-08. ADR-0053 uses native loopback HTTPS Next ingress and loopback HTTP API with authenticated host/visitor metadata. Step 2 replaces socket-only anonymous partitioning with canonical-IP and peer budgets. APISIX remains Phase 11's gated adapter; the topology below is that target, not a prerequisite or delivery claim for the walking skeleton.
The API validates one canonical tenant-realm issuer and retrieves discovery/JWKS only from descendant paths at that exact origin. Private or split-horizon DNS must preserve that identity. Startup checks configuration syntax, not DNS, TLS or issuer availability; a separate metadata origin is not authorized.
A cold cache or unavailable required signing key refuses authentication. A warm
current configuration can still validate a known key during refresh failure;
UseLastKnownGoodConfiguration=false does not discard that current cache.
TenantAuthenticationProfileTests.Transport proves a request-driven new-key
refresh, manually requested withdrawal and manually requested outage refresh.
It does not prove an elapsed five-minute automatic refresh or immediate key
revocation. Acceptance remains bounded by the token lifetime and observed cache.
Current Information outcomes distinguish fixed credential/profile/validation stages and accepted/rejected results. They are not an operational failure taxonomy. Provider event-3 adapter injection tests the stock exception/log path, not a natural metadata-outage occurrence. Authentication also precedes tenant resolution, so early correlation/tenant attribution can be absent. Row 10's membership refusal has no audit row and only Debug visibility today.
Phase 11's activation gate owns reachable issuer/DNS/TLS drills, operational failure signals, rotation, readiness, alerts and exercised runbooks before production exposure. P02b-6 owns real realm issuance; P02b-7 owns session custody and recovery. These are separate from P02b-5's merged API boundary.
flowchart LR
subgraph Internet
Browser["Tenant browser<br/>(app.learnstack.dev or<br/>tenant custom domain)"]
OpBrowser["Operator browser<br/>(hub.learnstack.dev)"]
end
subgraph Edge["Edge (per region)"]
APISIX["APISIX gateway<br/>:9080 (TLS:443 prod)<br/>standalone mode"]
end
subgraph LS_Plane["LearnStack data plane"]
LSApi["LearnStack.Host<br/>:5000"]
LSDapr["Dapr sidecar"]
end
subgraph Hub_Plane["LearnStack Hub data plane"]
HubApi["learnstack-hub.Api<br/>:5000"]
HubDapr["Dapr sidecar"]
HubInternal["Hub internal listener<br/>(mTLS only, never via APISIX)"]
end
Browser -- "HTTPS + JWT<br/>{tenant-domain}/api/v*/..." --> APISIX
OpBrowser -- "HTTPS + JWT<br/>hub.learnstack.dev/api/v*/..." --> APISIX
APISIX -- "host: app.learnstack.dev OR<br/>{tenant-custom-domain}" --> LSApi
APISIX -- "host: hub.learnstack.dev" --> HubApi
HubApi -- "mTLS + signed JWT + HMAC<br/>NEVER via APISIX" --> LSApi
LSApi -- "metering / verify (mTLS + JWT + HMAC)" --> HubInternal
LSApi <-.-> LSDapr
HubApi <-.-> HubDapr
Two hosts on the same APISIX instance:
app.learnstack.devand tenant custom domains → LearnStack core API.hub.learnstack.dev→ Hub API.
Tenant ↔ LearnStack internal API (/api/internal/*) is bound to a different listener
inside the LearnStack pod, accessible only via mTLS from Hub; not proxied by APISIX.
Standalone mode (no etcd). Routes loaded from YAML files mounted into the APISIX container; APISIX hot-reloads on file change.
infra/apisix/
├── config.yaml # APISIX core config: deployment role, plugin list, observability
├── apisix.yaml # Static route definitions (priority bands)
└── partials/
├── routes-custom-domains.yaml # Generated by Hub; updated on CustomDomainActivated event
└── ssl.yaml # SNI certs; cert/key referenced via Vault Agent sidecar
apisix.yaml is reloadable; the gateway watches the file for changes. Hub publishes
updates to routes-custom-domains.yaml via Kubernetes ConfigMap rolling update (or
direct file mount in dev).
When dynamic operator-driven config becomes a hard requirement (Phase 11+), we revisit etcd-backed mode + APISIX Admin API — out of scope for now.
Every request flows through this chain (in order):
Client request
│
▼
1. real-ip ─── trust X-Forwarded-For from L4 load balancer; reject if origin not allow-listed
│
▼
2. cors ─── handle preflight (OPTIONS) for allowed origins; inject Access-Control-* headers
│
▼
3. request-id ─── inject X-Correlation-Id (or echo if present)
│
▼
4. openid-connect ─── JWT validation against Keycloak realm (signature, expiry, issuer, audience)
│ only on authenticated routes
│
▼
5. limit-req ─── leaky-bucket rate limit per route + per remote-addr (or per consumer when authenticated)
│
▼
6. limit-count ─── per-window count limit for write endpoints (60/min/token)
│
▼
6b. basic-auth ─── consumer credentials from the secret store; used only by the
│ `/admin/hangfire*` infrastructure route, never on `/api/v*`
│
▼
7. proxy-rewrite ─── header normalisation; strip Authorization-Internal if leaked
│
▼
Upstream backend (LearnStack.Host or learnstack-hub)
│
▼
8. response-rewrite ─── strip internal headers from response (e.g. X-Internal-Trace)
│
▼
9. prometheus ─── /apisix/prometheus/metrics endpoint scraped by OTel collector
│
▼
10. gzip ─── compress response if Accept-Encoding allows
│
▼
Client response
config.yaml declares the global plugin list:
deployment:
role: data_plane
role_data_plane:
config_provider: yaml
plugins:
- real-ip
- cors
- request-id
- openid-connect
- limit-req
- limit-count
- basic-auth # /admin/hangfire* only; a route may not use an undeclared plugin
- proxy-rewrite
- response-rewrite
- prometheus
- gzipThree rules govern this table, and each exists because its violation has already been found in the corpus.
lua-resty-radixtree, the matcher APISIX uses, supports one wildcard and only at the
end of the path. There is no ** form, and a * in the middle of a path is not a
wildcard — it is a literal asterisk. A route declared as /api/v*/** therefore does not
mean "any version, any sub-path"; it means something no client will ever send.
Because ADR-0024 puts the version in the URL and the version set is small and enumerable, the correct spelling is one route per live version:
| Wrong | Right |
|---|---|
/api/v*/** |
/api/v1/* (and /api/v2/* when v2 ships) |
/api/v*/localization/* |
/api/v1/localization/* |
Adding a version adds routes. That is the intended cost of URL versioning, and it keeps the deprecation window in ADR-0024 visible in the gateway config rather than hidden behind a wildcard.
APISIX resolves overlapping routes by priority, higher wins, and an omitted
priority defaults to 0. The id field is an identifier, not an ordering — a fact the
earlier version of this table obscured by using id values that looked like priority
bands.
The consequence of that confusion was concrete: the public GET /api/v*/localization/*
route carried no priority (so 0) while the authenticated catch-all carried
priority: 100. Both matched a localization request, the catch-all won, and a route
documented as public anonymous would have demanded a bearer token — an anonymous
learner loading a public page would have received 401 from the gateway.
The banding is therefore specific-beats-general, written down:
| Band | Priority | Contents |
|---|---|---|
| Infrastructure | 400 | /healthz, /openapi/* and /docs* (environment-gated), /admin/hangfire* |
| Public anonymous | 300 | Named public paths: localization, every anonymous read enumerated in API Standards § Public surface, auth endpoints with their own strict limits. The route table below does not yet carry a route per enumerated read; they arrive with the reviewed route allow-list in Phase 11 |
| CORS preflight | 200 | OPTIONS on the versioned prefix — must beat the authenticated band so preflight never reaches openid-connect |
| Authenticated catch-all | 100 | Everything else under the versioned prefix |
A public route always outranks the catch-all. The invariant is not asserted in CI
today — there is no route-table lint, and calling a rule enforced when the check does not
exist is precisely the failure
Standards 21 is written against. The
lint ships with APISIX itself in
Phase 11. It will fail when any route omits
priority, and when any route whose plugin
set lacks openid-connect shares a path prefix with a higher-or-equal-priority route
that has it.
P02d-4's accepted bootstrap and Education route set is defined by ADR-0052, with mandatory marked pipeline dispatch and a host ceiling. Gateway banding does not replace backend admission or turn caller-supplied tenant ids into authority.
Route 100 previously carried
client_secret_ref: vault://learnstack/hub/internal-api-hmac-key. That path is the
Hub internal API HMAC body-signing key, from a different trust domain.
Possession permits forged body signatures for arbitrary tenant pushes; successful
internal admission still independently requires accepted mTLS and a valid service
JWT under Security Standards.
Giving that secret to the gateway compromises the body-signature boundary even
when the other two controls continue to hold.
The line is deleted, and no substitute is needed: the route is bearer_only: true,
which means APISIX validates a presented bearer token against the realm's public JWKS and
never performs a code exchange. A bearer_only OIDC route has no use for a client
secret at all.
routes:
# ---- Infrastructure band — priority 400 -------------------------------
- id: 1
uri: /healthz
methods: [GET]
priority: 400
plugins:
limit-req: { rate: 10, burst: 5, key: remote_addr, rejected_code: 429 }
upstream: { type: roundrobin, nodes: { "learnstack-api:5000": 1 } }
- id: 2
uris: ["/openapi/*", "/docs", "/docs/*"]
priority: 400
upstream: { type: roundrobin, nodes: { "learnstack-api:5000": 1 } }
# The document and the reference console are separate paths on purpose:
# Scalar mounts a `{documentName?}` catch-all inside whatever prefix it is
# given, and under /openapi that shadowed the document namespace. Gating is
# a per-environment decision here; the application serves both everywhere,
# because the document is the contract the SDK generates from.
# Hangfire dashboard — gateway BasicAuth + backend Platform-scope permission
- id: 3
uri: /admin/hangfire*
priority: 400
plugins:
basic-auth: {} # consumer credentials from the secret store
upstream: { type: roundrobin, nodes: { "learnstack-api:5000": 1 } }
# ---- Public anonymous band — priority 300 -----------------------------
# MUST outrank the authenticated catch-all, or these become 401s.
- id: 10
uri: /api/v1/localization/*
methods: [GET]
priority: 300
plugins:
cors: { allow_origins: "https://app.learnstack.dev,https://hub.learnstack.dev" }
limit-req: { rate: 50, burst: 20, key: remote_addr, rejected_code: 429 }
upstream: { type: roundrobin, nodes: { "learnstack-api:5000": 1 } }
- id: 11
uri: /api/v1/auth/*
methods: [POST]
priority: 300
plugins:
# allow_origins takes EXACT origins or `**`. It has no `*.domain` wildcard —
# `*.learnstack.app` matches nothing and silently blocks every tenant origin.
# Subdomain patterns go through allow_origins_by_regex, anchored at both ends.
cors:
allow_origins: "https://app.learnstack.dev"
allow_origins_by_regex: ["^https://[a-z0-9-]+\\.learnstack\\.app$"]
limit-count: { count: 5, time_window: 60, key: remote_addr, rejected_code: 429 }
upstream: { type: roundrobin, nodes: { "learnstack-api:5000": 1 } }
# ---- CORS preflight band — priority 200 -------------------------------
# OPTIONS carries no Authorization header; openid-connect would reject it.
- id: 20
uri: /api/v1/*
methods: [OPTIONS]
priority: 200
plugins:
cors:
allow_origins: "https://app.learnstack.dev"
allow_origins_by_regex: ["^https://[a-z0-9-]+\\.learnstack\\.app$"]
allow_methods: "GET,POST,PUT,PATCH,DELETE,OPTIONS"
allow_headers: "Authorization,Content-Type,X-Tenant-Id,X-Organization-Id,X-Correlation-Id"
max_age: 3600
# ---- Authenticated catch-all — priority 100 ---------------------------
- id: 100
uri: /api/v1/*
methods: [GET, POST, PUT, PATCH, DELETE]
priority: 100
plugins:
openid-connect:
client_id: learnstack-gateway
discovery: http://keycloak:8080/realms/learnstack/.well-known/openid-configuration
bearer_only: true # validates a presented token; no code exchange,
realm: learnstack # therefore no client secret — see Rule 3
scope: openid
set_access_token_header: true
access_token_in_authorization_header: true
cors:
allow_origins: "https://app.learnstack.dev"
allow_origins_by_regex: ["^https://[a-z0-9-]+\\.learnstack\\.app$"]
limit-req: { rate: 100, burst: 50, key: remote_addr, rejected_code: 429 }
request-id: { include_in_response: true }
upstream: { type: roundrobin, nodes: { "learnstack-api:5000": 1 } }
# ---- Hub host — same banding, scoped by host --------------------------
- id: 200
host: hub.learnstack.dev
uri: /api/v1/*
methods: [OPTIONS]
priority: 200
plugins:
cors:
allow_origins: "https://hub.learnstack.dev"
allow_methods: "GET,POST,PUT,PATCH,DELETE,OPTIONS"
allow_headers: "Authorization,Content-Type,X-Correlation-Id"
- id: 201
host: hub.learnstack.dev
uri: /api/v1/*
methods: [GET, POST, PUT, PATCH, DELETE]
priority: 100
plugins:
openid-connect:
client_id: learnstack-hub-gateway
discovery: http://keycloak:8080/realms/learnstack-hub/.well-known/openid-configuration
bearer_only: true
realm: learnstack-hub
cors: { allow_origins: "https://hub.learnstack.dev" }
limit-req: { rate: 60, burst: 30, key: remote_addr, rejected_code: 429 } # stricter on Hub
request-id: { include_in_response: true }
upstream: { type: roundrobin, nodes: { "learnstack-hub:5000": 1 } }The corrections in Rules 1–3 land in the committed infra/apisix/apisix.yaml
scaffold under Phase 03 G18.
That file already has priorities and no Hub-secret reference; its remaining
version-wildcard correction must preserve disabled openid-connect. Phase 03
does not copy this complete target example into live routes. The reviewed
per-public-read allowlist, route lint, OIDC/TLS/provenance qualification and
gateway activation remain
Phase 11.
Why validate JWT both at APISIX and at the backend?
- At APISIX: signature, expiry, audience, issuer. Failure → 401 returned by gateway before any backend roundtrip. This is the cheap, fast first line.
- At LearnStack backend (
AddJwtBearermiddleware): same checks again + tenant context resolution + organization context resolution + permission policy evaluation. The backend never trusts the gateway alone.
If APISIX is misconfigured, bypassed (someone hits the backend directly on the internal network), or compromised, the backend still enforces its own credential and scope boundary. Eligible public GET/HEAD reads remain available without credentials; AccountSite returns 404 for absent credentials. Present invalid credentials return 401 after the earlier admission gates, including on public routes. Phase 03 supplies the permission checks for its protected routes.
The backend currently carries the bearer-validation boundary on its own. Phase 11 activates the gateway's independent check when ADR-0035's trigger fires; Phase 03 supplies application permissions. Neither changes the backend's duty to validate and reconcile its own trusted inputs.
The running registration is
TenantAuthenticationRegistration,
installed in the explicit order in
Program.cs.
ADR-0064 owns its strict
issuer, purpose, actor, JOSE, diagnostic and refresh contract. Configure
Authentication:Tenant:Issuer as the canonical tenant-realm issuer. Only
Development permits loopback HTTP; other deployment modes require explicit HTTPS.
Metadata is fetched from that configured issuer, never a URL selected by claims.
No API client secret, live membership or BFF session is introduced here.
The API must resolve and reach the configured issuer itself, with the required TLS
trust. Private HTTPS addresses and internal DNS names are supported; there is no
separate Authentication:Tenant:MetadataAddress setting for an internal discovery
origin paired with a different public issuer. Startup validates the URL without
network I/O, so successful startup does not establish issuer reachability. Deployment
DNS, proxy and certificate configuration must preserve the issuer's exact identity
and the discovery/JWKS containment boundary.
The current API maps /healthz liveness only. It has neither mapped authentication
readiness nor a dedicated authentication outcome counter. Its stock manager caches
current metadata; disabling last-known-good fallback does not promise either an
outage grace period or immediate refusal of every warm-cache request. The
Phase 11 authentication activation gate
owns topology, cold/warm-cache and key-rotation drills, diagnostic/readiness decisions
and operational runbooks before production use. P02b-6 delivers actual realm
issuance and safe development convergence; its
Step 4 record
owns qualification. P02b-7 supplies native browser-session custody under
ADR-0070; production gateway and
proxy activation remain Phase 11's separate qualification.
This is the edge policy: it limits a noisy client to protect the platform, and it is keyed on the remote address. It is deliberately looser per second than the per-principal fair-use policy in Standards 11 § Rate Limiting, which is keyed on the token and is the only one that can express a plan limit. The two are not competing statements of one number — a request passes both or neither.
| Surface | Rate | Burst | Key |
|---|---|---|---|
/healthz |
10/s | 5 | remote_addr |
/api/v1/localization/* |
50/s | 20 | remote_addr |
/api/v1/auth/* (login, password reset) |
5/min | 0 | remote_addr |
| Authenticated GET/POST/PUT/PATCH/DELETE (main API) | 100/s | 50 | remote_addr |
| Hub authenticated API | 60/s | 30 | remote_addr |
| Write endpoints (POST/PUT/PATCH/DELETE) on authenticated routes | 60/min/token | — | consumer (when consumer-key authn is wired, Phase 11) |
remote_addr keying limits a noisy client, not a noisy tenant: one tenant behind
many addresses is unaffected, and many tenants behind one NAT are throttled together.
Per-tenant fairness — at the gateway and inside the database — is
Phase 11 work, described in
25-deployment-models.md § 2. Consumer-key-keyed limits require
etcd-backed mode for dynamic consumer issuance and land in the same phase.
The openid-connect plugin rejects OPTIONS requests because they don't carry an
Authorization header. This breaks browser CORS preflight unless OPTIONS is handled
separately.
Solution: an OPTIONS-only route in the preflight band (priority 200) carries the
cors plugin without openid-connect. Because 200 outranks the authenticated
catch-all's 100, the browser's preflight succeeds; the subsequent actual request, with
its Authorization header and a non-OPTIONS method, falls through to the authenticated
route.
Nexora pattern verbatim (see Nexora/docs/decisions/0003-deployment-strategy.md and
Nexora/docs/operations/HELM_INSTALLATION.md).
APISIX request-id plugin injects X-Correlation-Id on every request (or echoes the
client's value if present and validated as UUID-shaped). The same header propagates
through:
- The backend's
Activity.Current?.TraceId(viaASP.NET Core RequestId middleware). - The audit log entry's
correlation_idcolumn (ADR-0016). - Every emitted log line (via Serilog
LogContext). - Every outbound HTTP call (via
HttpClientinterceptor). - Every Dapr pub/sub event (
metadata.correlationId). - Every Hangfire job parameter.
End-to-end observability: a correlation_id traces a request from browser through
gateway, backend, Dapr, Kafka topic, consumer module, Hangfire job, audit log row.
Production: APISIX terminates TLS. Certs:
app.learnstack.devand*.learnstack.app— wildcard cert from Let's Encrypt or a commercial CA.- Tenant custom domains — per-domain certs via cert-manager + Let's Encrypt DNS-01 (ADR-0022).
hub.learnstack.dev— wildcard or dedicated cert.
Certs are stored in Vault (secret/learnstack/certs/<domain>); APISIX SSL resources are
hot-loaded via Vault Agent sidecar.
Backend (LearnStack.Host): HTTP only on the internal network; Kestrel TLS off. The
internal mTLS listener for /api/internal/* uses a separate cert (mutual TLS with Hub
client cert).
Phase 11 deliverables:
- TLS termination at APISIX with Let's Encrypt or commercial CA.
- HSTS header injection (
max-age=31536000; includeSubDomains; preload). - Strict CSP header via
response-rewriteplugin (production only). - Per-tenant rate-limit override (consumer keys; requires etcd mode).
-
KC_HOSTNAME_BACKCHANNEL_DYNAMIC=truefor Keycloak issuer split (Docker network vs browser); document in 13-identity-and-auth.md. - Hangfire dashboard route gated by BasicAuth at the gateway and the application's Platform-scope permission; Phase 03 owns the dated ADR-0015 disclosure before mapping, and Phase 11 owns gateway qualification. (backend).
- Production CORS allow-list (no
localhost). - OpenAPI dev route gated by environment (404 in prod).
- Real client IP forwarding via
real-ipconfigured with trusted L4 LB ranges only. - APISIX runtime metrics scraped by OTel collector.
- APISIX access log shipped to Loki.
- Hot-reload smoke test in CI: malformed YAML push must fail before the file lands.
Apisix_RouteYaml_IsValid— CI step runningapisix testagainstapisix.yaml.Apisix_Routes_Declare_Explicit_Priority— route-table lint: every route setspriority; a default-0route is a latent ordering bug (Rule 2).Apisix_Public_Routes_Outrank_Authenticated_Catchall— route-table lint: no route lackingopenid-connectshares a path prefix with a higher-or-equal-priority route that has it (Rule 2).Apisix_Uri_Patterns_Are_RadixtreeValid— route-table lint: at most one*, and only as the final path segment (Rule 1).Apisix_NeverFronts_InternalApi— integration test asserts requests toapp.learnstack.dev/api/internal/*return 404 from APISIX (route not defined).Backend_RequiresJwt_OnAllAuthenticatedRoutes—WebApplicationFactoryintegration test under ADR-0064: present invalid credentials on known hosts return 401, including public routes. Exact AccountSite absent credentials return maskednot_found404 on tenant, organization and platform hosts. Unknown hosts refuse before JWT work; the generic unresolved MediatR gate is unchanged. P02b-5 Step 4 activates this boundary; its delivery record above owns the actual proof and review receipts. This is the test that holds the line while the gateway's OIDC block is commented out.
- Service mesh. APISIX is an edge gateway, not Istio / Linkerd. Modular monolith has one process per pod; no east-west service mesh needed.
- API key management for tenants. Tenant-facing API keys (Phase 11) live in LearnStack's own auth surface — not in APISIX as a consumer system.
- L7 WAF rules. Phase 11+ may add ModSecurity via APISIX
wafplugin; out of scope for initial release.
- ADR-0015 — API Gateway with APISIX (what).
- ADR-0035 — Demand-gated infrastructure (when), and what carries the gateway's responsibilities until then.
- ADR-0024 — URL versioning; why the route table enumerates versions instead of wildcarding them.
- ADR-0004 Amendment 1 —
learnstack-hubrealm. - 11-security.md — defense-in-depth.
- 13-identity-and-auth.md — Keycloak realm topology.
- 29-dapr-integration.md — pub/sub and state transports.
- Nexora reference:
Nexora/docs/decisions/0003-deployment-strategy.md,Nexora/docs/operations/HELM_INSTALLATION.md,Nexora/docs/auth/IDENTITY.md.