diff --git a/docs/AWS_HYBRID_STORAGE.md b/docs/AWS_HYBRID_STORAGE.md index 987b08e..6a7b07e 100644 --- a/docs/AWS_HYBRID_STORAGE.md +++ b/docs/AWS_HYBRID_STORAGE.md @@ -6,6 +6,9 @@ certification. No AWS account, credentials, live deployment, or cloud validation was used to implement this sample. Review the draft before changing settings. Nothing here creates a GitHub deployment workflow or stores secrets in GitHub. +See the [AWS topology and write/restore diagrams](STORAGE_TOPOLOGY_DIAGRAMS.md) +for a visual guide separating the foundation from operator-supplied peers. + ## What is implemented `infra/aws/storage-foundation.json` is a manually deployable CloudFormation diff --git a/docs/DISTRIBUTED_STORAGE.md b/docs/DISTRIBUTED_STORAGE.md index ab39bea..f13094d 100644 --- a/docs/DISTRIBUTED_STORAGE.md +++ b/docs/DISTRIBUTED_STORAGE.md @@ -9,6 +9,9 @@ verify bytes against the pointer CID before decoding. The storage adapter does not encrypt: callers must supply authenticated ciphertext, never plaintext FHIR, biometrics, tokens, or derived keys. +See the [AWS and local topology diagrams](STORAGE_TOPOLOGY_DIAGRAMS.md) for +deployment boundaries and the ordered write and explicit-restore flows. + ## Atomicity and failure model The injected `BlobStore` index supplies atomic create and numeric-version CAS. diff --git a/docs/STORAGE_DESIGN.md b/docs/STORAGE_DESIGN.md index f54fbe5..36f24a9 100644 --- a/docs/STORAGE_DESIGN.md +++ b/docs/STORAGE_DESIGN.md @@ -7,6 +7,10 @@ and private-IPFS experiments without changing dual unlock, FHIR validation, signatures, MACs, or provenance verification. It does not claim clinical, regulatory, or production readiness. +The [storage topology diagrams](STORAGE_TOPOLOGY_DIAGRAMS.md) distinguish +implemented composition from proposed AWS infrastructure and local test setups. +Azure deployment topology is **To be decided**. + ```text PatientRecordService -> BlobStore |-- InMemoryBlobStore (test) diff --git a/docs/STORAGE_QUICKSTART.md b/docs/STORAGE_QUICKSTART.md index 15a9431..ecaab1e 100644 --- a/docs/STORAGE_QUICKSTART.md +++ b/docs/STORAGE_QUICKSTART.md @@ -10,6 +10,10 @@ Provider implementations and setup are segregated under for the folder layout and registering a custom adapter without editing the record service or built-in factory. +See the [storage topology diagrams](STORAGE_TOPOLOGY_DIAGRAMS.md) for the +proposed AWS deployment, default local app and optional private-IPFS fixture. +Azure deployment topology is **To be decided**. + ## Without any cloud accounts ```sh diff --git a/docs/STORAGE_TOPOLOGY_DIAGRAMS.md b/docs/STORAGE_TOPOLOGY_DIAGRAMS.md new file mode 100644 index 0000000..c98b877 --- /dev/null +++ b/docs/STORAGE_TOPOLOGY_DIAGRAMS.md @@ -0,0 +1,255 @@ +# Storage topology diagrams + +**Global Patient Record Project — synthetic-only research POC.** These diagrams +describe the implemented storage composition and a **proposed** AWS deployment, +not a deployed system or a clinical, regulatory, or production-readiness claim. +Azure deployment: **To be decided**; an adapter exists, but no Azure topology is +selected here. + +**Legend:** solid arrows show calls/data movement; dotted arrows show control, +credentials, or infrastructure encryption. Subgraph labels distinguish +CloudFormation resources from operator-supplied deployment prerequisites. +CID means content identifier; CAS means compare-and-swap. All persisted record +content is application-encrypted; storage adapters do not perform dual unlock. + +## AWS: proposed private three-peer deployment + +```mermaid +flowchart TB + subgraph APP["Application host - operator supplied"] + UI["Local browser UI"] + SERVICE["Loopback API and record service
Patient factor AND clinician private key
Encryption and verification in application"] + STORE["DistributedBlobStore
BackedUpContentStore and IpfsClusterClient"] + UI --> SERVICE + SERVICE -->|"Already-encrypted values"| STORE + end + + subgraph OPS["Operator supplied - private networking and runtime"] + AUTH["Secrets Manager or restricted external mounts
Infrastructure and API authentication only"] + PROXY["Authenticated private HTTPS endpoints
Separate Kubo and Cluster authorization"] + subgraph A["Proposed AZ A - optional foundation EC2 or operator host"] + C1["Cluster 1"] + K1["Kubo 1"] + E1["Separately encrypted EBS data volume
Kubo blocks and Cluster state"] + C1 -.->|"Pin management"| K1 + K1 --> E1 + C1 --> E1 + end + subgraph B["Proposed AZ B - additional operator-supplied EC2"] + C2["Cluster 2"] + K2["Kubo 2"] + E2["Separately encrypted EBS data volume"] + C2 -.->|"Pin management"| K2 + K2 --> E2 + C2 --> E2 + end + subgraph C["Proposed AZ C - additional operator-supplied EC2"] + C3["Cluster 3"] + K3["Kubo 3"] + E3["Separately encrypted EBS data volume"] + C3 -.->|"Pin management"| K3 + K3 --> E3 + C3 --> E3 + end + C1 -.->|"Cluster control membership - proposed Raft"| C2 + C2 -.->|"Not application metadata CAS"| C3 + C3 -.-> C1 + K1 <-->|"Private swarm - encrypted blocks"| K2 + K2 <-->|"Private swarm - encrypted blocks"| K3 + K3 <--> K1 + end + + subgraph FOUNDATION["CloudFormation foundation - if manually deployed"] + INDEX["DynamoDB single-Region key-to-CID index
Strong reads and atomic conditional writes
PITR and infrastructure encryption"] + BACKUP["Private versioned S3
Immutable CID-addressed encrypted backups
SSE-KMS"] + KMS["Separate infrastructure KMS keys
Not patient or clinician keys"] + OPTIONAL["CreateNode=false by default
At most ONE optional EC2 and data EBS
Approved prebuilt AMI required"] + end + + STORE -->|"Private route and authenticated HTTPS"| PROXY + PROXY -->|"Block put/get"| K1 + PROXY -->|"Pin request and replica observation"| C1 + STORE -->|"Synchronous backup before pointer publication"| BACKUP + STORE -->|"Read pointer; publish only after pins and backup"| INDEX + AUTH -.->|"API headers"| STORE + AUTH -.->|"Operator-injected swarm and Cluster secrets"| C1 + AUTH -.->|"Operator-injected private swarm key"| K1 + KMS -.-> INDEX + KMS -.-> BACKUP + KMS -.-> E1 + KMS -.->|"Operator-configured EBS encryption"| E2 + KMS -.->|"Operator-configured EBS encryption"| E3 + OPTIONAL -.->|"Compute only - no cluster bootstrap"| K1 +``` + +- The three nodes/AZs are a **proposal**, not stack-created replicas. The + [foundation template](../infra/aws/storage-foundation.json) provisions storage + and at most one optional node. The operator supplies additional peers, private + routes/SG rules, service endpoints, DNS/TLS/authenticated proxies, approved AMI, + secret injection, swarm configuration and monitoring. Repeat node-secret + provisioning for every peer; peer identities must be distinct. +- The app may run on the approved EC2 host or on a local host with reviewed + private connectivity, API credentials and scoped AWS identity. The current web + server remains loopback-only; this is **not** an internet-facing app deployment. + The foundation role trusts EC2; a local/other host needs its own reviewed + identity. No GitHub credentials belong in runtime secret mounts or configuration. +- Cluster places pins among distinct Kubo peers. Success requires observing at + least **two** live, distinct Kubo identities reporting pinned content (sample + maximum: three). This observation is not a continuing durability guarantee. + Raft, if selected by the operator, concerns Cluster control state/membership, + **not** the application's CAS; DynamoDB supplies that authority. +- A single EC2/EBS node remains a single point of failure and cannot satisfy the + remote two-copy policy alone. Even the proposed topology retains ingress, + metadata, AWS account/IAM and KMS dependencies; automatic API failover is not + implemented. A private swarm alone does not establish independently governed + decentralized durability. Operators still see CIDs, sizes, placement and timing. +- KMS adds infrastructure encryption; it never replaces patient/clinician + factors or receives the reconstructed record key. External secrets contain + infrastructure/API material, not patient factors or plaintext records. + +Implementation: [IPFS provider](../src/storage/adapters/ipfs/provider.ts), +[client](../src/storage/adapters/ipfs/ipfs-cluster-client.ts), +[DynamoDB adapter](../src/storage/adapters/dynamodb/dynamodb-blob-store.ts), +[S3 adapter](../src/storage/adapters/s3/s3-blob-store.ts). +Deployment details: [AWS runbook](AWS_HYBRID_STORAGE.md), +[runtime samples](../infra/aws/runtime), and +[nonsecret configuration](../examples/config/aws-hybrid.env.example). + +## Write ordering: replicated content, backup, then index + +```mermaid +sequenceDiagram + participant App as Application crypto and record service + participant Store as DistributedBlobStore + participant Wrapper as BackedUpContentStore + participant Primary as IpfsClusterClient via private HTTPS + participant S3 as S3 backup + participant Index as DynamoDB index + App->>Store: Create or CAS with already-encrypted value + Note over Store,Index: CAS first checks the current metadata version + Store->>Wrapper: Put serialized envelope bytes + Wrapper->>Primary: Put bytes + Note over Primary: Compute and verify raw CID, upload to Kubo, request Cluster pins + Note over Primary: Poll pin state and live peers, require at least two distinct Kubo copies + Primary-->>Wrapper: Verified CID after replication threshold + Wrapper->>S3: Conditional immutable create with SSE-KMS + Note over Wrapper,S3: Existing backup is accepted only after byte and CID verification + S3-->>Wrapper: Backup complete + Wrapper-->>Store: CID + Store->>Index: Atomic create or expected-version CAS of key-to-CID pointer + Index-->>Store: Published version + Store-->>App: Success +``` + +No pointer publication occurs if replication or backup fails. A lost CAS can +leave unreachable blocks/backups; no automatic unpin/delete occurs. A timeout +can have an ambiguous outcome, so reconcile before retrying. This is ordered +publication, not a transaction spanning IPFS, S3 and DynamoDB. +Sources: [distributed store](../src/storage/adapters/ipfs/distributed-blob-store.ts) +and [backup wrapper](../src/storage/adapters/ipfs/backed-up-content-store.ts). + +## Explicit restore: never an invisible read fallback + +```mermaid +sequenceDiagram + participant Operator as Operator + participant Store as DistributedBlobStore + participant Index as DynamoDB index + participant Wrapper as BackedUpContentStore + participant S3 as S3 backup + participant Primary as Private Kubo and Cluster via client + Operator->>Store: Restore opaque object key + Store->>Index: Strong read of current pointer + Index-->>Store: CID and version + Store->>Wrapper: Restore CID + Wrapper->>S3: Read immutable backup + S3-->>Wrapper: Encrypted bytes and CID envelope + Note over Wrapper: Validate format, version, bytes and matching CID + Wrapper->>Primary: Republish bytes and repin + Primary-->>Wrapper: Same CID after replica threshold + Wrapper-->>Store: Verified restored CID + Store-->>Operator: Complete, metadata unchanged +``` + +Ordinary reads use the index and primary Kubo bytes with CID verification; +primary failure is surfaced, not silently replaced with S3. Restore the metadata +index separately first if it is lost. Concurrent pointer changes can require +another recovery pass; content restore cannot recover lost factors or establish +snapshot freshness. See [recovery guidance](AWS_HYBRID_STORAGE.md). + +## Local default: no IPFS, Docker or cloud required + +With dependencies installed and no backend override, `npm start` runs this path: + +```mermaid +flowchart LR + UI["Browser UI
http://127.0.0.1:3000"] + API["Localhost-only API
Patient factor AND clinician private key"] + SERVICE["Record service
Application encryption and verification"] + FILE["FileBlobStore
.data/web
Encrypted records; single process"] + UI --> API + API --> SERVICE + SERVICE --> FILE +``` + +This is a single-host demonstration, not a distributed durability test. Existing +`.data/web` data is not automatically migrated when another backend is selected. +Sources: [web server](../src/web/server.ts) and +[file provider](../src/storage/adapters/file/provider.ts). + +## Optional local private-IPFS integration fixture + +`npm run test:integration:ipfs` is a **separate native Linux Docker** test path, +not part of `npm start`. Docker Desktop and remote Docker daemons are unsupported. + +```mermaid +flowchart TB + subgraph HOST["One native Linux host - synthetic integration only; no AWS"] + RUN["Integration runner and record-service tests
DistributedBlobStore and backup wrapper"] + FILES["Temporary local FileBlobStore
Metadata pointers and encrypted backups"] + RELAYS["Runner-owned TCP API relays
127.0.0.1 dynamic ports only"] + subgraph NET["Docker internal bridge - six containers"] + C1["Cluster 1"] -.->|"Pin control"| K1["Kubo 1"] + C2["Cluster 2"] -.->|"Pin control"| K2["Kubo 2"] + C3["Cluster 3"] -.->|"Pin control"| K3["Kubo 3"] + C1 -.->|"Raft control membership"| C2 + C2 -.-> C3 + C3 -.-> C1 + K1 <-->|"Private swarm"| K2 + K2 <--> K3 + K3 <--> K1 + VOLUMES["Per-node Docker data volumes
Separate restricted swarm and Cluster secret volume"] + K1 --> VOLUMES + K2 --> VOLUMES + K3 --> VOLUMES + end + RUN --> FILES + RUN -->|"Explicit loopback HTTP emulator mode"| RELAYS + RELAYS -->|"Suite ingress"| K1 + RELAYS -->|"Suite pin API"| C1 + RELAYS -->|"Probe and readiness APIs on other peers"| K2 + RELAYS --> K3 + RELAYS --> C2 + RELAYS --> C3 + end +``` + +The runner owns six API relays; no Docker host ports, public gateways or swarm +ports are published. APIs are unauthenticated for an isolated development host, +not safe for a shared untrusted machine. Only image pulls need internet access; +the running fixture stays on its internal bridge. Metadata/backups use temporary +local files, never AWS. Cluster state also persists in per-node Docker volumes. + +The suite exercises synthetic records and CID-verified restore. A separate +node-loss probe first confirms a synthetic block on **all three** Kubo peers, +then stops Kubo 1/Cluster 1 and checks the same bytes through Kubo 2. This is not +automatic ingress failover or proof of host/region failure tolerance: all peers, +metadata and backups still share one host. See the +[fixture contract](../test/fixtures/ipfs/README.md) and +[runner](../scripts/run-ipfs-integration.ts). + +## Azure: To be decided + +The Azure adapter does not select an Azure deployment architecture. Hosting, +networking, identity, metadata placement and recovery topology are intentionally +undefined here; no cross-cloud mirroring is implied.