Skip to content

feat(b2b_learner_records): refuse tokens past the contract end date - #73

Open
blarghmatey wants to merge 9 commits into
learner-records-jwt-verifyfrom
feat/learner-records-contract-end
Open

blarghmatey wants to merge 9 commits into
learner-records-jwt-verifyfrom
feat/learner-records-contract-end

Conversation

@blarghmatey

Copy link
Copy Markdown
Member

What are the relevant tickets?

N/A. Stacked on #69, which adds token.py.

Description (What does it do?)

Learner-records clients carry learner_records_contract_end_date (hardcoded from the contract in mitodl/ol-infrastructure#5939), but nothing checked it. A partner whose contract ended kept reading identifiable records until someone deleted its Keycloak client. This refuses the partner's tokens with a 401 once that date has passed.

Implementation details
  • The check runs in verified_claims, after the signature and token-class checks. It returns 401 unauthorized, not 403, because the credential itself is no longer valid.
  • The date is inclusive and read as Anywhere on Earth (UTC-12), so access stops at 12:00 UTC the day after the end date. The contract names no timezone, and this is the reading that never cuts a partner off before its end date by its own clock. Partners east of UTC-12 get up to a day of extra access.
  • An absent claim means no end date. Existing clients were provisioned without one (the mapper is only created when contract_end_date is set), and treating absence as expired would revoke all of them on deploy.
  • A present claim that isn't exactly YYYY-MM-DD (what the mapper writes with date.isoformat()) refuses. date.fromisoformat also accepts 20270630 and ISO week dates, so the parse round-trips to catch those. 9999-12-31 is a valid end date, not an overflow.
  • The contract-ended refusal gets its own detail naming the date, unlike the uniform verification refusals. Only a validly signed token reaches that check, so the message tells a forger nothing, and it tells a partner why its sync broke.
  • It logs learner_records_contract_ended at warning level when it fires. That means a client outlived its contract and nobody removed it. learner_records_access now carries contract_end_date, so a Loki alert can warn MIT ahead of a lapse. The alert rule itself isn't in this PR.
  • docs/b2b-learner-records-provider-authorization.md replaces the open "does access expire with the contract?" question with this behavior. @pdpinch, that question was addressed to you, so please confirm the end-of-day reading before this merges. Partners should be told about it too.

How can this be tested?

ruff check, ruff format --check, mypy src, bin/generate-openapi-spec --check, and the full pytest suite (340 tests) pass locally. New tests cover an expired claim (refused), a future one (accepted), an absent one (accepted), nine malformed values (refused), 9999-12-31, the access-log field, and the UTC-12 boundary through the endpoint at 2027-07-01 11:59:59Z vs. 12:00:00Z. Nothing was run against a live Keycloak token.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GoWu48siDM16978y9JKmhr

blarghmatey and others added 8 commits September 22, 2026 14:22
…-Userinfo

The tenant authorized from X-Userinfo, which APISIX rebuilds from a token it
has already validated. That only says anything about traffic that went
through APISIX, and nothing forces a caller to: the pod security group admits
the whole pod subnet, and aws-eks-nodeagent runs with
--enable-network-policy=false on both data clusters, so the NetworkPolicies
that exist are no-ops. A compromised in-cluster workload could post a forged
X-Userinfo carrying learner-records:read and any organization UUID and read
identifiable learner records.

This verifies the token in the app instead. token.py checks RS256 against the
olapps realm JWKS, with issuer, audience and lifetime, and require_organization_grant
reads scope, learner_records_organizations and azp from the verified payload.
X-Userinfo is not read by this tenant at all. Anything that fails verification
gets one 401 that does not say which check failed.

The key set is cached for a TTL and refetched once on an unseen kid, so a
realm key rotation costs a fetch rather than a TTL of refusals; a cooldown
keeps forged kids from turning the endpoint into an amplifier pointed at
Keycloak. An unreachable JWKS refuses rather than falling back.

Config derives the token, JWKS and issuer URLs from one issuer setting, so
moving a deployment to another realm is one environment variable.

b2b_dashboard still reads X-Userinfo. Its exposure is aggregate and
k-anonymized and it has a browser session flow, so that is a separate call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qoz9pfQxUU8VLsRxr1tTnm
The security scheme said the credential is 'validated at the gateway' and the
401 example named a detail string the tenant no longer returns. Both describe
the behaviour before the tenant started verifying the bearer JWT itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qoz9pfQxUU8VLsRxr1tTnm
…lass

Review findings on the verification added in the previous commit.

A JWKS body that parses as JSON but holds no usable key (an empty set, an
error document served as 200, a bare list) raised out of the dependency as a
500 rather than the 401 it exists to return, and since the fetch never
populated the cache it recurred on every request. _fetch now turns every way
the endpoint can go wrong into one JWKSUnavailableError.

A fetch failure was neither held off nor backed by the key set already in
hand, so once the TTL lapsed with Keycloak unreachable every request took the
lock and paid the timeout in turn while a still-valid key set sat unused.
Failures now hold off further attempts briefly, and a stale key set is served
through an outage: realm keys turn over on the order of months, so refusing
every request because a refresh failed is an outage this service inflicts on
itself.

An ID token minted for ol-analytics-api-client carries the realm's signature,
this issuer and this audience, so it cleared verification and was stopped
only by carrying no organization claim. The token class is now required and
checked.

Also: a forced load no longer refetches what another request just fetched;
the unknown-kid cooldown starts only once a freshly fetched key set really
lacked the kid, so a failed refetch doesn't lock every other kid out for a
minute; and the tenant refuses to start when the issuer is left at the
production realm in another deployed environment, since every failure mode of
that setting is a silent 401.

The concurrency test pinned the cache rather than the lock -- a mocked
transport never suspends, so the gathered requests serialised themselves and
it passed with the lock removed. It now fetches through a callback that
awaits. Added cases for the token class, for HS256 signed with the public key
out of the JWKS, for the multi-audience shape Keycloak actually emits, and
for the unusable-key-set bodies above.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qoz9pfQxUU8VLsRxr1tTnm
…fetch

A forced refetch that fails hands back the stale key set rather than
raising, so the unknown kid still isn't found and the cooldown started on
evidence nobody gathered. The comment there already claimed this didn't
happen.

The case it breaks is a Keycloak blip during a key rotation: the refetch
fails, the cooldown starts, and tokens carrying the new kid keep getting 401s
for a minute after Keycloak comes back, with no refetch attempted. Now the
cooldown starts only when _fetched_at advanced, which is what says a freshly
fetched key set really lacked the kid. A failed fetch is already held off by
_retry_after.

The regression test fails against the previous code.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qoz9pfQxUU8VLsRxr1tTnm
…ys required

The Error schema in docs/openapi/b2b-learner-records-v1.yaml declares every
error response as {code, detail}, with code required and drawn from a fixed
enum, and tells clients to branch on it because the wording of detail may
change. Nothing produced it. FastAPI's HTTPException renders {"detail": ...},
so a client generated from the spec would reject every error this tenant
returned, on all five documented statuses.

errors.py adds ApiError, carrying the code, plus the handler that renders it.
The code can't be derived from the status, which is why it rides on the
exception: 403 is missing_scope when the token's scopes fall short and
no_organization_access when the organization isn't granted, and those stay
distinct to a client while remaining indistinguishable to an attacker, since
both keep the same detail.

400 (invalid_parameter), 401 (unauthorized), both 403s and 503 (unavailable)
now carry their code. Errors the contract doesn't document, a 404 for an
unrouted path or a 405, keep FastAPI's own body rather than being dressed in
a code that claims a contract covering them.

Scoped to this tenant. b2b_dashboard has no published contract to honour, so
adding a field to its error bodies would be a change with no reader. 429 is
enforced at APISIX, not here, so nothing in this service produces that body.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qoz9pfQxUU8VLsRxr1tTnm
… main

Two things #70 changed under this branch.

_partner_header became _partner_token here while #70 added a test that calls
it. Both sides touched different parts of the file, so the rebase merged them
cleanly and the result referenced a helper that no longer exists.

#70 also added pyyaml, which was the only reason the error-code test asserted
against the ErrorCode enum rather than the contract the enum exists to mirror.
It now reads the enum out of docs/openapi/b2b-learner-records-v1.yaml, so a
code added on one side and not the other fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qoz9pfQxUU8VLsRxr1tTnm
Learner-records clients carry learner_records_contract_end_date from their
contract, but nothing checked it, so a partner whose contract ended kept
reading identifiable records until someone deleted the Keycloak client.

The tenant now refuses a verified token once its contract end date has
passed, reading the date as inclusive and Anywhere on Earth so no partner
is cut off early by its own clock. An absent claim stays open-ended, since
existing clients were provisioned without one. A malformed claim refuses,
because the alternative fails open. The access log now carries the end
date so MIT can alert before a lapse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GoWu48siDM16978y9JKmhr
Adding a day to date.max raised OverflowError, which the ValueError
handler didn't catch, so a client provisioned with the obvious "never"
sentinel got a 500 on every request. The check now compares against the
end of the end date instead of the start of the next.

Also drives the Anywhere-on-Earth boundary through the endpoint rather
than the helper, covers the access log's contract_end_date field, and
reuses _unauthorized for the contract-ended refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GoWu48siDM16978y9JKmhr
blarghmatey added a commit to mitodl/ol-infrastructure that referenced this pull request Sep 25, 2026
The comment said nothing checks learner_records_contract_end_date. The
learner-records tenant now refuses tokens past it
(mitodl/ol-analytics-api#73).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GoWu48siDM16978y9JKmhr
@blarghmatey
blarghmatey marked this pull request as ready for review September 25, 2026 20:21
@blarghmatey
blarghmatey added this pull request to stack #74 September 25, 2026 20:22
@blarghmatey
blarghmatey requested a balanced review from Copilot September 25, 2026 20:22

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The operational contract-ended warning event lacks regression coverage.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
What changed in this PR

Adds contract-end enforcement for learner-record clients using an inclusive Anywhere-on-Earth cutoff.

Changes:

  • Rejects expired or malformed contract-date claims.
  • Logs contract dates and expiration events.
  • Documents behavior and adds boundary tests.
File Description
token.py Enforces contract expiration.
auth.py Adds contract dates to access logs.
test_learner_records_token.py Tests expiration and date validation.
b2b-learner-records-v1.yaml Documents expired-token responses.
b2b-learner-records-provider-authorization.md Documents the authorization policy.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/ol_analytics_api/tenants/b2b_learner_records/token.py
The expired-token test checked only the response, so demoting or
dropping the learner_records_contract_ended warning would pass while
silently disabling the alert that keys on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GoWu48siDM16978y9JKmhr
@blarghmatey
blarghmatey requested a review from pdpinch September 30, 2026 13:22
blarghmatey added a commit that referenced this pull request Sep 30, 2026
The partner's lookup and unwrap commands passed the wrapping token as a curl
argument, where another local process could read it and unwrap first. They
now read it without echo and pipe it on stdin. Planned removal said "on the
end date", which cuts a partner off before the inclusive, Anywhere-on-Earth
cutoff #73 enforces; it now deploys after 12:00 UTC the following day.

Claude-Session: https://claude.ai/code/session_01S7nfyd7Pky17JRQS3fUy1C
blarghmatey added a commit to mitodl/ol-infrastructure that referenced this pull request Oct 1, 2026
* feat(keycloak): issue per-contract learner-records clients

The learner-records tenant in ol-analytics-api authorizes a caller by the
organization UUIDs in its token, and access is settled when a contract is
signed (docs/b2b-learner-records-provider-authorization.md). Nothing issued
those tokens yet.

This adds the learner-records:read client scope to the olapps realm, with an
audience mapper that addresses its tokens to ol-analytics-api-client, and one
client-credentials client per entry in the keycloak stack's
olapps-learner-records-clients config. Each client carries its organizations
as a JSON-typed hardcoded claim, since a String-typed claim grants nothing in
the API. Credentials land in Vault under sso/learner-records/<name> for
handoff.

ol-analytics-api gets a bearer-only APISIX rule for /api/v1/learner-records
on every host. It verifies tokens against the JWKS (introspection reports
client-credentials tokens inactive) and requires the scope and the audience
at the gateway, so a token minted for another olapps client is refused before
the app sees it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rhFxFFvpxfnHNAYDyfttX

* refactor(keycloak): fail when learner-records clients lack the API client

Learner-records entries were skipped with no error when the
olapps-ol-analytics-api-client-secret config was absent, since the clients
sit inside that guard. They now raise at deploy time. Also drops a JSON
round-trip for the Vault document and notes that removing the
service_account scope (and its client_id claim) is intended.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rhFxFFvpxfnHNAYDyfttX

* feat(ol_analytics_api): give the learner-records tenant its realm and audience

The learner-records tenant now verifies the partner's bearer JWT itself
instead of trusting the X-Userinfo this route forwards, because the pod is
reachable without going through APISIX: the pod security group admits the
whole pod subnet, and aws-eks-nodeagent runs with
--enable-network-policy=false on both data clusters, so the NetworkPolicies
that exist are no-ops. Without an issuer it falls back to production SSO,
which is wrong everywhere but production.

This templates OL_ANALYTICS_API_B2B_LEARNER_RECORDS_ISSUER and _AUDIENCE out
of secret-operations/sso/ol-analytics-api, the same entry the route's
openid-connect plugin reads for its discovery URL and client id, so the
gateway and the app cannot end up checking different realms or a different
audience. The app derives its JWKS and token URLs from the issuer.

The app side is mitodl/ol-analytics-api's
fix(b2b_learner_records): verify the bearer JWT instead of trusting X-Userinfo.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qoz9pfQxUU8VLsRxr1tTnm

* doc(keycloak): point the contract-end claim at the API's enforcement

The comment said nothing checks learner_records_contract_end_date. The
learner-records tenant now refuses tokens past it
(mitodl/ol-analytics-api#73).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GoWu48siDM16978y9JKmhr

* style(keycloak): reflow the contract-end claim comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GoWu48siDM16978y9JKmhr

* feat(keycloak): rotate a learner-records client secret with secret_version

Without a rotation knob the only way to reissue a partner's secret was
replacing the client, which deletes and recreates it and every mapper on
it. Bumping secret_version feeds client_secret_regenerate_when_changed, so
Keycloak issues a new secret in place and the Vault document picks it up
on the same apply.

Claude-Session: https://claude.ai/code/session_01S7nfyd7Pky17JRQS3fUy1C

* doc(keycloak): say when a rotated learner-records secret can be handed over

Claude-Session: https://claude.ai/code/session_01S7nfyd7Pky17JRQS3fUy1C

* fix(apisix): run the integration tests on the APISIX the chart ships

APISIX_CHART_VERSION is 2.17.0, whose appVersion is 3.18.0, but the
integration suite still defaulted to 3.17.0, so it tested a gateway
production no longer runs.

Claude-Session: https://claude.ai/code/session_01S7nfyd7Pky17JRQS3fUy1C

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants