fix(b2b_learner_records): verify the bearer JWT instead of trusting X-Userinfo

README.md

@@ -144,6 +144,20 @@ refuses identically whether the organization is ungranted or doesn't exist.
 There's no round-trip and no grant store; removing the client revokes the
 access. See `docs/b2b-learner-records-provider-authorization.md`.
 
+That tenant does **not** read `X-Userinfo`. It verifies the bearer token
+itself (`tenants/b2b_learner_records/token.py`): RS256 against the realm's
+JWKS, checking issuer, audience, token class and lifetime, and it takes the
+claims it authorizes on from the verified payload. The gateway rebuilds
+`X-Userinfo` only for traffic that goes through the gateway, and the pod is
+reachable without doing so — the pod security group admits the whole pod
+subnet and the CNI runs with network policy disabled on both data clusters.
+Aggregate k-anonymized figures can live with that; records naming individual
+learners can't. `OL_ANALYTICS_API_B2B_LEARNER_RECORDS_ISSUER` and
+`..._AUDIENCE` come from Vault via the Pulumi stack, out of the same entry
+the gateway route reads, so the two can't check different realms. Both
+default to production's, and the app refuses to start if the issuer is left
+at that default in any other deployed environment.
+
 The org-manager round-trip authenticates with this service's **own** OAuth2
 client-credentials token and names the subject user explicitly
 (`?user_global_id=<keycloak sub>`), rather than forwarding the caller's

docs/b2b-learner-records-provider-authorization.md

@@ -107,15 +107,41 @@ definition is the only place access is recorded.
 - Whether APISIX's `openid-connect` plugin passes the hardcoded claim and
   scopes through in `X-Userinfo` on a bearer-only route. The learner-records
   mount needs a bearer-only route, but today's routes use the redirect flow.
-  Check on QA.
-- That APISIX *overwrites* a caller-supplied `X-Userinfo` rather than passing
-  it through. `core/auth/userinfo.py` decodes whatever header arrives without
-  validating a token, and the organization check reads from it. Also confirm
-  the pod can't be reached except through the gateway route: `k8s/` defines no
-  NetworkPolicy. Check both on QA.
+  Check on QA. This no longer gates the tenant (see below), but the claim
+  still has to arrive in the token.
 - The access-token lifespan these clients will get, since it is the
   revocation window.
 
+## The app verifies the token; the gateway is not the trust boundary
+
+Settled 2026-09-18, after Copilot raised it on
+[ol-infrastructure#5939](https://github.com/mitodl/ol-infrastructure/pull/5939).
+
+The question above was whether APISIX overwrites a caller-supplied
+`X-Userinfo`. It does, at the start of its `rewrite` phase. That is beside the
+point, because a caller does not have to go through APISIX. The pod security
+group admits the whole pod subnet, and `aws-eks-nodeagent` runs with
+`--enable-network-policy=false` on both data clusters, so the NetworkPolicies
+that exist are no-ops. Any compromised in-cluster workload can post a forged
+`X-Userinfo` naming `learner-records:read` and any organization UUID, straight
+to the pod.
+
+For k-anonymized aggregates that is a risk worth arguing about. For records
+that name individual learners it is not, so this tenant verifies the bearer
+token itself (`tenants/b2b_learner_records/token.py`): RS256 against the
+realm's JWKS, checking issuer, audience and lifetime, and taking `scope`,
+`learner_records_organizations` and `azp` from the verified payload.
+`X-Userinfo` is not read at all. The gateway route stays as it is; it is now
+defence in depth rather than the only check.
+
+Rejected: locking down the pod security group, and enabling CNI network
+policy cluster-wide. Both are larger changes that protect one tenant by
+changing how every workload on the cluster is reached.
+
+`b2b_dashboard` still authorizes from `X-Userinfo`. Its exposure is aggregate
+and k-anonymized and it has a browser session flow, so it is a separate
+decision, not a follow-up to this one.
+
 ## Follow-ups
 
 Not needed to write the tenant, but each needs an owner before partners

docs/openapi/b2b-learner-records-v1.yaml

@@ -362,11 +362,12 @@ components:
     oauth2ClientCredentials:
       type: oauth2
       description: >-
-        Keycloak client credentials, validated at the gateway. One client is
-        issued per contracted integration and lists the organizations it may
-        read, every contract under each included. Scopes bound to a client are
-        a contractual limit on that credential; they are not the consent
-        mechanism.
+        Keycloak client credentials. The token's signature, issuer, audience
+        and lifetime are verified by the service itself, not only at the
+        gateway. One client is issued per contracted integration and lists the
+        organizations it may read, every contract under each included. Scopes
+        bound to a client are a contractual limit on that credential; they are
+        not the consent mechanism.
       flows:
         clientCredentials:
           tokenUrl: https://sso.ol.mit.edu/realms/olapps/protocol/openid-connect/token
@@ -933,11 +934,14 @@ components:
           example: { code: invalid_parameter, detail: 'limit: Input should be less than or equal to 1000' }
 
     Unauthorized:
-      description: Missing, malformed or expired token.
+      description: >-
+        No token, or one that failed verification: bad signature, unknown
+        signing key, wrong issuer or audience, or outside its validity window.
+        The body does not say which, so it cannot be used to probe.
       content:
         application/json:
           schema: { $ref: '#/components/schemas/Error' }
-          example: { code: unauthorized, detail: 'Invalid or expired access token' }
+          example: { code: unauthorized, detail: 'Invalid or missing bearer token' }
 
     Forbidden:
       description: >-
@@ -949,7 +953,7 @@ components:
       content:
         application/json:
           schema: { $ref: '#/components/schemas/Error' }
-          example: { code: no_organization_access, detail: 'No access to the requested organization' }
+          example: { code: no_organization_access, detail: 'No grant for the requested organization' }
 
     TooManyRequests:
       description: Per-client rate limit exceeded.

pyproject.toml

@@ -19,6 +19,7 @@ dependencies = [
     "opentelemetry-instrumentation-fastapi>=0.52b0",
     "opentelemetry-instrumentation-httpx>=0.52b0",
     "granian[reload,uvloop]>=2.7",
+    "pyjwt[crypto]>=2.10",
 ]
 
 [tool.uv]
@@ -86,6 +87,7 @@ dev = [
     "cyclopts>=4.22.5",
     "pyyaml>=6.0.3",
     "types-pyyaml>=6.0.12.20260724",
+    "cryptography>=44",
 ]
 
 [build-system]
@@ -127,7 +129,7 @@ ignore = [
 [tool.ruff.lint.per-file-ignores]
 # S105-S107: fixture values named `token` are fake OAuth2 tokens for the
 # MITx Online client tests, not real credentials.
-"tests/*" = ["S101", "ANN", "PLR2004", "S105", "S106", "S107"]
+"tests/*" = ["S101", "ANN", "PLR2004", "PLR0913", "S105", "S106", "S107"]
 # structlog's processor signature is conventionally (logger: Any, method_name: str,
 # event_dict: dict) — matches mitol-django-observability's own typing, ported verbatim.
 "src/ol_analytics_api/core/observability/processors.py" = ["ANN401"]

src/ol_analytics_api/tenants/b2b_learner_records/app.py

@@ -15,19 +15,24 @@
 
 from ol_analytics_api.core.errors import add_shared_error_handlers
 from ol_analytics_api.core.health import register_readiness_check
+from ol_analytics_api.tenants.b2b_learner_records.errors import (
+    ErrorCode,
+    add_error_handlers,
+    error_body,
+)
 from ol_analytics_api.tenants.b2b_learner_records.routers import organizations
 
 TENANT_NAME = "b2b_learner_records"
 
 
 async def _bad_request(_request: Request, exc: RequestValidationError) -> JSONResponse:
-    # The contract's error body is {"detail": "<message>"} with a 400, not
-    # FastAPI's 422 carrying a list of error objects.
+    # The contract's error body is {"code": ..., "detail": ...} with a 400,
+    # not FastAPI's 422 carrying a list of error objects.
     error = exc.errors()[0]
     location = ".".join(str(part) for part in error["loc"][1:])
     return JSONResponse(
         status_code=status.HTTP_400_BAD_REQUEST,
-        content={"detail": f"{location}: {error['msg']}"},
+        content=error_body(ErrorCode.INVALID_PARAMETER, f"{location}: {error['msg']}"),
     )
 
 
@@ -41,6 +46,9 @@ def create_app() -> FastAPI:
     )
     app.include_router(organizations.router)
     add_shared_error_handlers(app)
+    # After the shared handlers: this tenant overrides the 503 so it carries
+    # the code its contract documents.
+    add_error_handlers(app)
     app.add_exception_handler(RequestValidationError, _bad_request)  # type: ignore[arg-type]
     register_readiness_check(TENANT_NAME)
     return app

src/ol_analytics_api/tenants/b2b_learner_records/auth.py

@@ -3,9 +3,14 @@
 There is no user in this flow. MIT issues one Keycloak client-credentials
 client per contracted integration, and that client carries the organization
 UUIDs its contract covers as a hardcoded claim
-(docs/b2b-learner-records-provider-authorization.md). APISIX validates the
-token and forwards its claims in X-Userinfo, so authorization here is a set
-membership test: no call to MITx Online, no grant store.
+(docs/b2b-learner-records-provider-authorization.md). Authorization is
+therefore a set membership test over the token's own claims: no call to MITx
+Online, no grant store.
+
+Unlike b2b_dashboard, this tenant does not read X-Userinfo. It verifies the
+bearer token's signature itself (token.py), because a header rebuilt by
+APISIX only proves anything about traffic that went through APISIX, and
+these records name individual learners.
 """
 
 from __future__ import annotations
@@ -15,19 +20,21 @@
 from typing import Annotated, Any
 
 import structlog
-from fastapi import Depends, HTTPException, Security, status
+from fastapi import Depends, Security, status
 from fastapi.openapi.models import OAuthFlowClientCredentials, OAuthFlows
 from fastapi.security import OAuth2
 
-from ol_analytics_api.core.auth.userinfo import get_userinfo
 from ol_analytics_api.tenants.b2b_learner_records.config import settings
+from ol_analytics_api.tenants.b2b_learner_records.errors import ApiError, ErrorCode
+from ol_analytics_api.tenants.b2b_learner_records.token import verified_claims
 
 ORGANIZATIONS_CLAIM = "learner_records_organizations"
 READ_SCOPE = "learner-records:read"
 
 # Declares the contract's security scheme in this tenant's OpenAPI, so generated
-# clients obtain and send a token. It enforces nothing: auto_error=False, and
-# the checks below read the claims APISIX forwards after validating the token.
+# clients obtain and send a token. It enforces nothing on its own
+# (auto_error=False); the token is verified by the TokenClaims dependency and
+# the checks below run against the verified payload.
 oauth2_client_credentials = OAuth2(
     flows=OAuthFlows(
         clientCredentials=OAuthFlowClientCredentials(
@@ -45,13 +52,13 @@
 
 log = structlog.get_logger(__name__)
 
-UserInfo = Annotated[dict[str, Any], Depends(get_userinfo)]
+TokenClaims = Annotated[dict[str, Any], Depends(verified_claims)]
 
 
-def _granted_organizations(userinfo: dict[str, Any]) -> set[uuid.UUID]:
+def _granted_organizations(claims: dict[str, Any]) -> set[uuid.UUID]:
     # Keycloak emits the claim as a JSON array only when the mapper's claim
     # type is JSON. Any other shape grants nothing rather than being guessed at.
-    claim = userinfo.get(ORGANIZATIONS_CLAIM)
+    claim = claims.get(ORGANIZATIONS_CLAIM)
     if not isinstance(claim, list):
         return set()
     granted = set()
@@ -63,21 +70,26 @@ def _granted_organizations(userinfo: dict[str, Any]) -> set[uuid.UUID]:
 
 def require_organization_grant(
     organization_id: uuid.UUID,
-    userinfo: UserInfo,
+    claims: TokenClaims,
     _token: Annotated[str | None, Security(oauth2_client_credentials, scopes=[READ_SCOPE])],
 ) -> None:
-    scopes = userinfo.get("scope")
+    scopes = claims.get("scope")
     if not isinstance(scopes, str) or READ_SCOPE not in scopes.split():
-        raise HTTPException(
+        raise ApiError(
             status_code=status.HTTP_403_FORBIDDEN,
+            code=ErrorCode.MISSING_SCOPE,
             detail=f"Token lacks the {READ_SCOPE} scope",
         )
-    if organization_id not in _granted_organizations(userinfo):
-        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail=NO_GRANT_DETAIL)
+    if organization_id not in _granted_organizations(claims):
+        raise ApiError(
+            status_code=status.HTTP_403_FORBIDDEN,
+            code=ErrorCode.NO_ORGANIZATION_ACCESS,
+            detail=NO_GRANT_DETAIL,
+        )
     # Every granted read discloses identifiable learner records, so record which
     # client read which organization. The access log has the path but not the client.
     log.info(
         "learner_records_access",
-        client_id=userinfo.get("azp") or userinfo.get("client_id"),
+        client_id=claims.get("azp") or claims.get("client_id"),
         organization_id=str(organization_id),
     )

src/ol_analytics_api/tenants/b2b_learner_records/config.py

@@ -2,11 +2,19 @@
 
 from __future__ import annotations
 
-from pydantic import field_validator
+from pydantic import field_validator, model_validator
 from pydantic_settings import BaseSettings, SettingsConfigDict
 
+from ol_analytics_api.core.config import settings as core_settings
 from ol_analytics_api.core.db.identifiers import validate_sql_identifier
 
+PRODUCTION_ISSUER = "https://sso.ol.mit.edu/realms/olapps"
+
+# Where leaving the issuer at its production default is not evidence of a
+# misconfiguration: production itself, and a developer's machine, which has
+# no partner tokens to verify either way.
+_ISSUER_DEFAULT_IS_FINE = ("development", "production")
+
 
 class B2BLearnerRecordsSettings(BaseSettings):
     model_config = SettingsConfigDict(env_prefix="OL_ANALYTICS_API_B2B_LEARNER_RECORDS_")
@@ -21,9 +29,38 @@ class B2BLearnerRecordsSettings(BaseSettings):
     def _validate_starrocks_schema(cls, value: str) -> str:
         return validate_sql_identifier(value)
 
+    # The Keycloak realm that issues partner client-credentials tokens. Every
+    # other SSO URL below is derived from it, so pointing a deployment at a
+    # different realm is one setting, not four that can drift apart.
+    # ol-infrastructure templates it out of the same Vault entry the gateway
+    # route reads. If that ever fails to render, the default below would have
+    # a QA pod verifying against the production realm while APISIX in front of
+    # it used QA's -- every partner token refused, for a reason nothing in the
+    # refusal names. _reject_the_wrong_realm turns that into a failed start.
+    issuer: str = PRODUCTION_ISSUER
+
+    # Tokens must name this service in `aud`. The learner-records client scope
+    # adds it through an audience mapper (ol-infrastructure
+    # substructure/keycloak/learner_records.py); a token minted for another
+    # olapps client is signed by the same realm key and is refused on this
+    # check alone.
+    audience: str = "ol-analytics-api-client"
+
     # Advertised in this tenant's OpenAPI security scheme so generated clients
-    # know where to get a token. APISIX, not this service, validates tokens.
-    token_url: str = "https://sso.ol.mit.edu/realms/olapps/protocol/openid-connect/token"  # noqa: S105
+    # know where to get a token.
+    token_url: str = ""
+
+    # Verification keys. Cached for the TTL and refetched early on a kid this
+    # service has not seen, so a realm key rotation costs one fetch rather
+    # than a TTL of 401s.
+    jwks_url: str = ""
+    jwks_cache_ttl_seconds: float = 3600.0
+    jwks_timeout_seconds: float = 5.0
+
+    # Tolerance for clock skew between Keycloak and this pod when checking
+    # exp/nbf. The partner access token lifespan is 300s, so this stays well
+    # under it.
+    token_leeway_seconds: float = 30.0
 
     default_page_size: int = 100
     max_page_size: int = 1000
@@ -35,5 +72,34 @@ def _validate_starrocks_schema(cls, value: str) -> str:
     # discloses nothing; deployments opt in through ol-infrastructure.
     consent_fail_open: bool = False
 
+    @model_validator(mode="after")
+    def _reject_the_wrong_realm(self) -> B2BLearnerRecordsSettings:
+        """Refuse to start rather than verify against another environment.
+
+        Every failure mode of this setting is silent: a pod that verifies
+        partner tokens against a realm that never issued them refuses every
+        one of them, and the 401 it returns looks like a bad credential.
+        """
+        if self.issuer == PRODUCTION_ISSUER and core_settings.environment not in (
+            _ISSUER_DEFAULT_IS_FINE
+        ):
+            msg = (
+                f"{self.model_config['env_prefix']}ISSUER is unset in the "
+                f"{core_settings.environment!r} environment, so partner tokens would be "
+                f"verified against the production realm ({PRODUCTION_ISSUER}). Set it to "
+                "this environment's Keycloak realm URL."
+            )
+            raise ValueError(msg)
+        return self
+
+    @model_validator(mode="after")
+    def _derive_sso_urls(self) -> B2BLearnerRecordsSettings:
+        base = self.issuer.rstrip("/")
+        if not self.token_url:
+            self.token_url = f"{base}/protocol/openid-connect/token"
+        if not self.jwks_url:
+            self.jwks_url = f"{base}/protocol/openid-connect/certs"
+        return self
+
 
 settings = B2BLearnerRecordsSettings()