From 070990829ca5bd7b6096190ebc5fd2ef82ec741e Mon Sep 17 00:00:00 2001 From: Bo Wu Date: Tue, 15 Sep 2026 15:28:25 -0700 Subject: [PATCH 1/5] docs: record historical production integration evidence --- .../2026-08-production-integration.md | 124 ++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 docs/evidence/2026-08-production-integration.md diff --git a/docs/evidence/2026-08-production-integration.md b/docs/evidence/2026-08-production-integration.md new file mode 100644 index 0000000..4bdfe5d --- /dev/null +++ b/docs/evidence/2026-08-production-integration.md @@ -0,0 +1,124 @@ +# Initial production integration: recorded deployment evidence + +Reviewed on September 15, 2026. The operational observations below are from +August 22–27, 2026; this documentation review did not query the live deployment +or rerun the integration test. + +## Summary + +The historical deployment records support three completed milestones: + +- prod-mcp was configured to trust the supervisor through an immutable AWS KMS + key ARN, an explicit key ID, and a matching region. +- A deployed multiagent session read actual testnet Loki logs through prod-mcp. +- The session retained its request, successful operation receipt, and trace + artifacts in its scoped S3 prefix. + +These observations correct the previously unchecked setup items in the +[canonical TODO](../TODO.md#production-mcp-integration-status). They establish +an internally deployed integration with real operational data, not a simulated +Grafana response. They do not establish current availability, organic adoption, +or completion of every later architecture requirement. + +## Key pinning + +The deployment catalog read on August 22 at 10:29:59 UTC contained a non-empty +kmsKeys entry with: + +- keyArn: an immutable AWS KMS key resource ARN, not an alias; +- kid: the configured supervisor permit-verification key ID; +- region: the same region as the key ARN. + +The recorded verifier implementation resolves the configured key through +GetPublicKey at startup, checks the immutable ARN and region, and imports the +public key for permit verification. Pinning here means selecting an immutable +KMS key resource; it does not mean checking a PEM public key into this repository. + +An August 27 deployment record also contains the same key ARN, key ID, and +region in the recorded prod-mcp Helm values. The real Grafana receipt below +comes from the intervening deployed integration. The cloud account, full key +ARN, key material, and authentication values are intentionally omitted here. + +## Real Grafana/Loki execution + +Session: grafana-e2e-20260823-v28. + +The session report records discovery of 13 Loki label names and 35 namespace +values, followed by retrieval of real logs selected from the testnet environment. +The deployment report identifies multiagent Helm revision 45. + +Selected fields from the retained operation receipt: + + operationId: op-41f65b2358d1353e01a384f335a8e9af + actionId: ops-1787477155582-1630 + requestedOperation.id: grafana.read + state: succeeded + executor: fixed-command + acceptedAt: 2026-08-23T09:25:56.022Z + updatedAt: 2026-08-23T09:25:56.654Z + +The receipt includes a permit digest and returned Loki content. Its recorded +result contains real cluster-autoscaler log entries. This note omits service +identifiers, private endpoints, raw logs, and the complete permit/request. + +At that point the integration worked on the internal path. The contemporaneous +report still listed public DNS/Ingress work as pending; this run should not be +described as evidence that the public reverse-proxy path was ready. + +## S3 persistence + +The August 23 object listing confirms artifacts beneath the session prefix, +including these relative keys: + + production/sessions/grafana-e2e-20260823-v28/logs/transcript-index.json + production/sessions/grafana-e2e-20260823-v28/operations/ops-1787477155582-1630/request.json + production/sessions/grafana-e2e-20260823-v28/operations/ops-1787477155582-1630/receipt.json + +The bucket name is omitted. The listing establishes observed persistence of the +session artifacts; it does not prove retention policy, rollback detection, or +complete trace coverage for every execution. + +## Source provenance + +This is an operator-readable summary derived from retained private deployment +records, not a publicly downloadable or independently reproduced test bundle. +The source is the Codex deployment session +01a02145-afcc-7601-8e7e-d078dc1b0296, started on August 20, 2026. + +The following one-based JSONL record locations identify the observations: + +| Record | Observation time (UTC) | Evidence | +| --- | --- | --- | +| 6338 | August 22, 10:29:59 | Deployment catalog with the configured KMS key pin | +| 17685 | August 23, 09:30:40 | Session report with discovery results, operation identifiers, Helm revision and artifact locations | +| 18090 | August 23, 09:55:40 | Tool output containing the retained successful Grafana operation receipt | +| 18100 | August 23, 09:56:03 | Tool output listing persisted S3 request, receipt and trace artifacts | +| 41354 | August 27, 08:04:18 | Deployment record retaining the configured KMS pin in Helm values | + +The two raw tool-output records used for receipt and S3 verification have these +SHA-256 digests, computed over each original JSONL line including its newline: + + record 18090: 9e1dc7e77b7e29fa961cb02a99cf673589a139b896743bb5b43fa58d840fe4db + record 18100: 408d3b2a27d9ca19e9cb10d6936882eb5ada038dc57d8d3b9408043b22217f4a + +These digests help an authorized reviewer match the retained records; they are +not signatures or independent attestations of the deployment outcome. No local +user filesystem paths or private cloud object URLs are required by this note. + +Later public evidence in [PR #87](https://github.com/areshand/multiagent/pull/87), +merged September 5, records a completed production Wiki thread, a deployed +111-repository catalog, and S3 trace export. It supports closing the old +repository-provisioning setup item; it is not a rerun of the Grafana scenario. + +## What remains open + +The [current architecture acceptance path](../architecture/system-architecture.md#required-end-to-end-acceptance-path) +also requires the independent Logger to receive trace commitments, expose chain +and signed checkpoint evidence, and retain continuity after restart. Those +requirements postdate this Grafana run. A complete current-deployment acceptance +run remains unchecked in the TODO, alongside the separate Logger rollback +detection and durable-delivery backlog items. + +This update does not change the historical benchmark interpretation: 36/50 +remains a tuned cumulative aggregate, not a reproducible single run. + From 05178fa8341cd87b0ce2950333c209b70e0e038f Mon Sep 17 00:00:00 2001 From: Bo Wu Date: Tue, 15 Sep 2026 15:28:31 -0700 Subject: [PATCH 2/5] docs: reconcile production integration milestones --- docs/TODO.md | 40 ++++++++++++++++++++++++++++++++++------ 1 file changed, 34 insertions(+), 6 deletions(-) diff --git a/docs/TODO.md b/docs/TODO.md index c427ccc..7a41cfa 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -84,17 +84,44 @@ must also be represented or linked from this file. the query service nor the maintenance agent receives Wiki-write or trace-read credentials. -## Production MCP Joint Test +## Production MCP Integration Status + +The initial deployed integration completed in August 2026. The milestones below +record that historical result; they do not certify the current deployment +against later architecture requirements. See the +[deployment evidence note](evidence/2026-08-production-integration.md) for dates, +receipt identifiers, source provenance, and verification limits. + +### Completed Deployment Milestones - [x] Activate the existing Linux UID and Landlock boundary in the production container. - [x] Prevent role processes from inheriting AWS, KMS, prod-mcp, and GitHub credentials. - [x] Add supervisor-owned KMS permit signing for generic runbook operations with a sealed independent reviewer gate. -- [x] Configure the multiagent StatefulSet with the correct runtime variables and credential group. -- [ ] Pin the immutable multiagent KMS public key in prod-mcp. -- [ ] Persist multiagent traces to a scoped S3 prefix. -- [ ] Provision an approved repository for task execution. +- [x] Configure the initial multiagent StatefulSet with the correct runtime variables and credential group. +- [x] Pin the supervisor verification key in prod-mcp using an immutable KMS key + ARN, explicit key ID, and region. The August 22 deployment configuration and + later deployment records contain the pin. +- [x] Persist session traces, requests, and operation receipts to a scoped S3 + prefix. The August 23 Grafana session has recorded S3 object verification. +- [x] Provision a catalog-backed repository for task execution. The later + [deployed Wiki MVP](https://github.com/areshand/multiagent/pull/87) also records + a completed production thread initially targeting multiagent and a catalog + covering 111 repositories. - [x] Pass the local malicious-agent boundary and joint contract tests. -- [ ] Pass the deployed testnet Grafana log-read joint test. +- [x] Pass the initial deployed testnet Grafana log-read integration: the + August 23 session retrieved real Loki logs through prod-mcp and retained a + successful operation receipt and S3 evidence. + +### Remaining Deployment Verification + +- [ ] Run the full + [current end-to-end acceptance path](architecture/system-architecture.md#required-end-to-end-acceptance-path) + on the intended deployed revisions. Record component image/commit identifiers, + authenticated caller and confined-role evidence, reviewer approval, KMS permit + verification, a real Grafana/Loki receipt, S3 trace persistence, independent + Logger commitment ingestion, checkpoint readback and restart continuity, and + accurate integration health/readiness. The August Grafana result predates the + independent Logger and does not close this broader acceptance item. ## OSS Positioning And Adoption @@ -222,3 +249,4 @@ Evidence rules for checking these boxes (see - [x] Add a reproducible benchmark section with exact commands and a compact, relocatable provenance validator built on reusable framework primitives. - [x] Complete these credibility basics before significant external promotion. + From cd091495cdb599f8179e135b5295c975bb48ae2f Mon Sep 17 00:00:00 2001 From: Bo Wu Date: Tue, 15 Sep 2026 15:30:35 -0700 Subject: [PATCH 3/5] docs: keep deployment status directly in TODO --- docs/TODO.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/TODO.md b/docs/TODO.md index 7a41cfa..3c2677c 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -88,9 +88,8 @@ must also be represented or linked from this file. The initial deployed integration completed in August 2026. The milestones below record that historical result; they do not certify the current deployment -against later architecture requirements. See the -[deployment evidence note](evidence/2026-08-production-integration.md) for dates, -receipt identifiers, source provenance, and verification limits. +against later architecture requirements. Status was reconciled with retained +deployment records on September 15, 2026; no live acceptance test was rerun. ### Completed Deployment Milestones @@ -250,3 +249,4 @@ Evidence rules for checking these boxes (see relocatable provenance validator built on reusable framework primitives. - [x] Complete these credibility basics before significant external promotion. + From 3196f6447ef7863a23963881c73b9562dc06b6d6 Mon Sep 17 00:00:00 2001 From: Bo Wu Date: Tue, 15 Sep 2026 15:30:41 -0700 Subject: [PATCH 4/5] docs: remove separate deployment evidence note --- .../2026-08-production-integration.md | 124 ------------------ 1 file changed, 124 deletions(-) delete mode 100644 docs/evidence/2026-08-production-integration.md diff --git a/docs/evidence/2026-08-production-integration.md b/docs/evidence/2026-08-production-integration.md deleted file mode 100644 index 4bdfe5d..0000000 --- a/docs/evidence/2026-08-production-integration.md +++ /dev/null @@ -1,124 +0,0 @@ -# Initial production integration: recorded deployment evidence - -Reviewed on September 15, 2026. The operational observations below are from -August 22–27, 2026; this documentation review did not query the live deployment -or rerun the integration test. - -## Summary - -The historical deployment records support three completed milestones: - -- prod-mcp was configured to trust the supervisor through an immutable AWS KMS - key ARN, an explicit key ID, and a matching region. -- A deployed multiagent session read actual testnet Loki logs through prod-mcp. -- The session retained its request, successful operation receipt, and trace - artifacts in its scoped S3 prefix. - -These observations correct the previously unchecked setup items in the -[canonical TODO](../TODO.md#production-mcp-integration-status). They establish -an internally deployed integration with real operational data, not a simulated -Grafana response. They do not establish current availability, organic adoption, -or completion of every later architecture requirement. - -## Key pinning - -The deployment catalog read on August 22 at 10:29:59 UTC contained a non-empty -kmsKeys entry with: - -- keyArn: an immutable AWS KMS key resource ARN, not an alias; -- kid: the configured supervisor permit-verification key ID; -- region: the same region as the key ARN. - -The recorded verifier implementation resolves the configured key through -GetPublicKey at startup, checks the immutable ARN and region, and imports the -public key for permit verification. Pinning here means selecting an immutable -KMS key resource; it does not mean checking a PEM public key into this repository. - -An August 27 deployment record also contains the same key ARN, key ID, and -region in the recorded prod-mcp Helm values. The real Grafana receipt below -comes from the intervening deployed integration. The cloud account, full key -ARN, key material, and authentication values are intentionally omitted here. - -## Real Grafana/Loki execution - -Session: grafana-e2e-20260823-v28. - -The session report records discovery of 13 Loki label names and 35 namespace -values, followed by retrieval of real logs selected from the testnet environment. -The deployment report identifies multiagent Helm revision 45. - -Selected fields from the retained operation receipt: - - operationId: op-41f65b2358d1353e01a384f335a8e9af - actionId: ops-1787477155582-1630 - requestedOperation.id: grafana.read - state: succeeded - executor: fixed-command - acceptedAt: 2026-08-23T09:25:56.022Z - updatedAt: 2026-08-23T09:25:56.654Z - -The receipt includes a permit digest and returned Loki content. Its recorded -result contains real cluster-autoscaler log entries. This note omits service -identifiers, private endpoints, raw logs, and the complete permit/request. - -At that point the integration worked on the internal path. The contemporaneous -report still listed public DNS/Ingress work as pending; this run should not be -described as evidence that the public reverse-proxy path was ready. - -## S3 persistence - -The August 23 object listing confirms artifacts beneath the session prefix, -including these relative keys: - - production/sessions/grafana-e2e-20260823-v28/logs/transcript-index.json - production/sessions/grafana-e2e-20260823-v28/operations/ops-1787477155582-1630/request.json - production/sessions/grafana-e2e-20260823-v28/operations/ops-1787477155582-1630/receipt.json - -The bucket name is omitted. The listing establishes observed persistence of the -session artifacts; it does not prove retention policy, rollback detection, or -complete trace coverage for every execution. - -## Source provenance - -This is an operator-readable summary derived from retained private deployment -records, not a publicly downloadable or independently reproduced test bundle. -The source is the Codex deployment session -01a02145-afcc-7601-8e7e-d078dc1b0296, started on August 20, 2026. - -The following one-based JSONL record locations identify the observations: - -| Record | Observation time (UTC) | Evidence | -| --- | --- | --- | -| 6338 | August 22, 10:29:59 | Deployment catalog with the configured KMS key pin | -| 17685 | August 23, 09:30:40 | Session report with discovery results, operation identifiers, Helm revision and artifact locations | -| 18090 | August 23, 09:55:40 | Tool output containing the retained successful Grafana operation receipt | -| 18100 | August 23, 09:56:03 | Tool output listing persisted S3 request, receipt and trace artifacts | -| 41354 | August 27, 08:04:18 | Deployment record retaining the configured KMS pin in Helm values | - -The two raw tool-output records used for receipt and S3 verification have these -SHA-256 digests, computed over each original JSONL line including its newline: - - record 18090: 9e1dc7e77b7e29fa961cb02a99cf673589a139b896743bb5b43fa58d840fe4db - record 18100: 408d3b2a27d9ca19e9cb10d6936882eb5ada038dc57d8d3b9408043b22217f4a - -These digests help an authorized reviewer match the retained records; they are -not signatures or independent attestations of the deployment outcome. No local -user filesystem paths or private cloud object URLs are required by this note. - -Later public evidence in [PR #87](https://github.com/areshand/multiagent/pull/87), -merged September 5, records a completed production Wiki thread, a deployed -111-repository catalog, and S3 trace export. It supports closing the old -repository-provisioning setup item; it is not a rerun of the Grafana scenario. - -## What remains open - -The [current architecture acceptance path](../architecture/system-architecture.md#required-end-to-end-acceptance-path) -also requires the independent Logger to receive trace commitments, expose chain -and signed checkpoint evidence, and retain continuity after restart. Those -requirements postdate this Grafana run. A complete current-deployment acceptance -run remains unchecked in the TODO, alongside the separate Logger rollback -detection and durable-delivery backlog items. - -This update does not change the historical benchmark interpretation: 36/50 -remains a tuned cumulative aggregate, not a reproducible single run. - From 162101c06497ba3c5d0718c86daf09f5f99bc1ef Mon Sep 17 00:00:00 2001 From: Bo Wu Date: Tue, 15 Sep 2026 15:31:00 -0700 Subject: [PATCH 5/5] docs: remove trailing blank lines --- docs/TODO.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/TODO.md b/docs/TODO.md index 3c2677c..ebf34c4 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -248,5 +248,3 @@ Evidence rules for checking these boxes (see - [x] Add a reproducible benchmark section with exact commands and a compact, relocatable provenance validator built on reusable framework primitives. - [x] Complete these credibility basics before significant external promotion. - -