diff --git a/docs/b2b-learner-records-credential-runbook.md b/docs/b2b-learner-records-credential-runbook.md new file mode 100644 index 0000000..9b135a7 --- /dev/null +++ b/docs/b2b-learner-records-credential-runbook.md @@ -0,0 +1,202 @@ +# B2B Learner Records: partner credential runbook + +How a partner gets a learner-records API credential, how it is rotated, and how +it is taken away. Companion to +[`b2b-learner-records-provider-authorization.md`](b2b-learner-records-provider-authorization.md), +which explains why access is encoded this way. + +Each credential is a Keycloak client-credentials client in the `olapps` realm, +defined by one entry in the `ol-substructure-keycloak` stack config +(`ol-infrastructure`, `substructure/keycloak/learner_records.py`). That entry is +the only record of what the partner can read. Adding it grants access, removing +it revokes access, and the PR that changes it is the audit trail. + +## Before issuing: the opt-in and the contract + +Not every contracted organization gets a credential. The organization opts in +to this integration, and that opt-in is its own recorded decision. Don't infer +it from a contract existing, because the credential reads identifiable learner +records (email and full name are always returned). + +The opt-in is recorded on the organization's contract in MITx Online. That +field doesn't exist yet. Until it does, the PR that adds the client must link +the organization's written request (email or ticket). + +The contract has to state: + +- Which organizations the partner reads. Access is organization-wide: a client + reads every learner under every contract of each organization it lists, not + only one contract's cohort. +- The end date. +- The partner's obligations for restricting learner records inside its own LMS. + Nothing on MIT's side enforces these. + +A provider working for two organizations under two contracts gets two clients. + +## Issuing a credential + +1. Collect the terms for each environment. Organization UUIDs are the + organization's `sso_organization_id` in MITx Online (the Keycloak + organization id), and they differ between QA and Production. The end date is + the contract's `contract_end` in MITx Online. +2. Open a PR against `ol-infrastructure` adding an entry to + `keycloak_realm:olapps-learner-records-clients` in + `substructure/keycloak/Pulumi..yaml`: + + ```yaml + keycloak_realm:olapps-learner-records-clients: + - name: contoso-lms + description: Contoso LMS integration for Contoso Manufacturing + organizations: + - 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 + contract_end_date: "2027-06-30" + ``` + + `name` becomes the client id `learner-records-` and the Vault path. + Always set `contract_end_date` for a partner. It is optional in the schema + only because internal test clients have no contract. + + Anyone can file the PR. Platform engineering reviews it and checks that the + opt-in is linked, that the UUIDs belong to the right organizations in that + environment, and that the end date matches the contract. +3. Merge, then deploy. In the `docker-packer-pulumi-keycloak` Concourse + pipeline, only CI deploys on merge. QA and Production each wait on a + `[bot] Pulumi ol-substructure-keycloak ready to deploy.` GitHub + issue, and closing it deploys that stack (Production only after QA). Close + the QA gate, then the Production gate, and confirm the + `deploy-ol-substructure-keycloak-` job succeeded. A gate deploys the + newest previewed commit, so check its preview for unrelated Keycloak changes + before closing it. +4. The apply writes the credential to Vault at + `secret-operations/sso/learner-records/`: `client_id`, + `client_secret`, `issuer`, `token_url`, `scope`, `organizations` and + `contract_end_date`. That document is what the partner receives. The API + itself has no read access to this path. +5. Smoke-test before handing it over. This keeps the secret out of files and + out of process arguments: + + ```bash + CREDS=$(vault read -format=json secret-operations/sso/learner-records/) + TOKEN=$(jq -r '.data | "grant_type=client_credentials&client_id=\(.client_id)&client_secret=\(.client_secret)"' <<<"$CREDS" \ + | curl -s --data @- "$(jq -r .data.token_url <<<"$CREDS")" | jq -r .access_token) + curl -s -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $TOKEN" \ + "https://analytics-qa.ol.mit.edu/api/v1/learner-records/organizations//courses" + unset CREDS TOKEN + ``` + + Expect `200`. Decoding the token should show `learner_records_organizations` + as a JSON array and `learner-records:read` in `scope`. The Production host + is `analytics.ol.mit.edu`. + +## Handing the credential to the partner + +Send the Vault document to the partner's named technical contact, in this +order of preference. Staff with read access to +`secret-operations/sso/learner-records/` perform the handoff. + +1. Vault response wrapping. Wrap the document in a single-use token: + + ```bash + vault read -format=json -wrap-ttl=72h \ + secret-operations/sso/learner-records/ | jq .wrap_info + ``` + + Send the partner `token` and tell them the Vault address, the TTL and the + expected creation path. Keep `accessor` in the handoff record. If the + partner reports a failed unwrap before the TTL is up, + `vault token lookup -accessor ` tells you which case it is: the + lookup succeeds if the wrapping token is still unused (the partner has the + wrong token), and fails if it was already unwrapped. + + The partner checks the token before using it, then unwraps it once. The + token is read without echo and piped to `curl` on stdin, so it never + appears in process arguments or shell history, where another local process + could take it and unwrap first: + + ```bash + read -rs WRAP + printf '{"token": "%s"}' "$WRAP" | curl -s -X POST --data @- \ + https://vault-production.odl.mit.edu/v1/sys/wrapping/lookup + # creation_path must be secret-operations/sso/learner-records/ + printf 'X-Vault-Token: %s\n' "$WRAP" | curl -s -X POST -H @- \ + https://vault-production.odl.mit.edu/v1/sys/wrapping/unwrap + unset WRAP + ``` + + A wrong `creation_path` means the token was swapped in transit. A failed + unwrap on a token that hasn't expired means someone else unwrapped it + first. In both cases, rotate the secret (below) and send it again. The + Vault hostname resolves publicly, but unwrapping from outside MIT's network + hasn't been tried with a partner yet. If the partner can't reach it, use the + next option. +2. A LastPass share to the partner contact. Remove the share once they confirm + receipt. +3. GPG-encrypted email to the partner contact's public key. Confirm the key + fingerprint with them over a separate channel first. + +Tell the partner: + +- Access tokens last 300 seconds. Request a new one with the client-credentials + grant as needed. +- Access ends at 12:00 UTC on the day after the contract's end date. The end + date is inclusive and read as Anywhere on Earth, the same cutoff + [ol-analytics-api#73](https://github.com/mitodl/ol-analytics-api/pull/73) + enforces once it merges. +- Who to contact to rotate the secret or report it exposed. + +## How fast a change takes effect + +Every change below lands when its stack deploys, not when the PR merges, and +QA and Production deploy only once their gates are closed (see step 3 above). +After that, tokens already issued keep working until they expire: 300 seconds +(`ACCESS_TOKEN_LIFESPAN_SECONDS`, pinned on each client so a realm-wide change +can't widen it), plus the API's 30-second clock leeway once +[ol-analytics-api#69](https://github.com/mitodl/ol-analytics-api/pull/69) +merges. The API verifies tokens against the realm signing keys and doesn't ask +Keycloak whether the client still exists, so there is no way to cut off a +token already issued. For an exposure or an early termination, close the gates +as soon as the PR merges. + +## Rotating a secret + +Rotate on suspected exposure, when the partner's staff with access change, or +when the partner asks. No routine rotation interval has been set. + +1. Increment `secret_version` on the client's entry (it defaults to `1`) in a + PR. On deploy, Keycloak issues a new secret, the old one stops working, and + the Vault document is updated in the same apply. +2. For a planned rotation, agree the deploy window with the partner, since + their integration fails from the deploy until they have the new secret. For + an exposure, don't wait. +3. After the deploy, hand the new secret over as above. + +## Revoking access + +Removing the client's entry from the stack config deletes the Keycloak client +and its Vault document. + +- Planned end: open the removal PR before the end date, and deploy it after + 12:00 UTC on the day after the end date. Deploying earlier cuts the partner + off before the end of the end date in its own timezone. Once #73 merges, every `learner_records_access` log line carries + the client's `contract_end_date`, so a Loki alert can warn ahead of a lapse. + That alert rule doesn't exist yet. +- Early termination: the same removal PR, deployed immediately. +- An organization dropping out of a multi-organization client: remove its UUID + from `organizations`. Tokens issued before the deploy still list it until + they expire. The other organizations keep working. +- Backstop: once #73 merges (it is stacked on #69), the API refuses a token + whose `learner_records_contract_end_date` has passed, from 12:00 UTC on the + day after the end date (the end date is inclusive and read as Anywhere on + Earth). It logs `learner_records_contract_ended` at warning level when it + does. That means a client outlived its contract, so remove it now. Until #73 + merges there is no backstop, and a forgotten client keeps reading. + +When the bulk export channel exists, a partner's cross-account IAM role and +SFTP account are separate credentials and have to be revoked alongside the +client. + +## QA credentials for partner integration testing + +A partner building against QA gets its own client in `Pulumi.QA.yaml`, pointed +at a QA organization of synthetic learners, and receives it the same way from +`vault-qa.odl.mit.edu`. QA credentials never list a real organization's UUID. diff --git a/docs/b2b-learner-records-provider-authorization.md b/docs/b2b-learner-records-provider-authorization.md index fdcc3ef..0233062 100644 --- a/docs/b2b-learner-records-provider-authorization.md +++ b/docs/b2b-learner-records-provider-authorization.md @@ -121,9 +121,8 @@ definition is the only place access is recorded. Not needed to write the tenant, but each needs an owner before partners multiply: -- **Client lifecycle.** One client per contract covers creation only. Who - delivers the secret to the partner, who rotates it, and who removes it when - the contract ends are unassigned. +- **Client lifecycle.** Issuance, handoff, rotation and removal are in + [`b2b-learner-records-credential-runbook.md`](b2b-learner-records-credential-runbook.md). - **Audit logging, before the first partner.** `auth.py` logs the client ID and organization on every authorized request. It doesn't log how many records were returned, and nothing yet says where those logs are kept or who reviews