Harden crypto service baseline

.github/workflows/ci.yml

@@ -0,0 +1,67 @@
+name: ci
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+permissions:
+  contents: read
+
+jobs:
+  verify:
+    runs-on: ubuntu-latest
+    env:
+      GOTOOLCHAIN: go1.27.0
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: 1.27.0
+          cache-dependency-path: services/crypto_svc/go.sum
+      - name: Install pinned tools
+        run: |
+          go install golang.org/x/vuln/cmd/govulncheck@v1.6.0
+          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.36.12
+          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.6.2
+      - name: Install protoc
+        uses: arduino/setup-protoc@v3
+        with:
+          version: "35.1"
+      - name: Verify common
+        working-directory: services/common
+        run: |
+          test -z "$(gofmt -l .)"
+          go test -race -count=1 ./...
+          go vet ./...
+          govulncheck ./...
+      - name: Verify crypto service
+        working-directory: services/crypto_svc
+        run: make proto-check test-race lint standalone
+      - name: Build container
+        run: docker build -f services/crypto_svc/Dockerfile -t fmcrypto:${{ github.sha }} services
+
+  container-scan:
+    needs: verify
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      security-events: write
+    steps:
+      - uses: actions/checkout@v4
+      - name: Build container
+        run: docker build -f services/crypto_svc/Dockerfile -t fmcrypto:scan services
+      - name: Scan container
+        uses: aquasecurity/trivy-action@v0.36.0
+        with:
+          image-ref: fmcrypto:scan
+          format: sarif
+          output: trivy-results.sarif
+          severity: CRITICAL,HIGH
+          ignore-unfixed: true
+          exit-code: "1"
+      - name: Upload scan result
+        if: always()
+        uses: github/codeql-action/upload-sarif@v4
+        with:
+          sarif_file: trivy-results.sarif

README.md

@@ -39,11 +39,13 @@ services/
   third-party runtime deps are `grpc` + `protobuf` (gRPC transport). No database,
   no web framework, no crypto libraries.
 - **HSM abstraction**: all symmetric-key and PKI operations flow through the
-  `hsm.HSM` interface. GP (software, LMK-wrapped TR-31 key blocks) is the default;
+  `hsm.HSM` interface. GP (software, LMK-wrapped TR-31 key blocks) is the local
+  development default;
   a real customer HSM (PS/AT) plugs in at the composition root (`CRYPTO_HSM`)
   with no usecase changes.
-- **Key hygiene**: keys never leave the engine clear — they are created and
-  returned wrapped under the LMK, mirroring a real HSM.
+- **Key hygiene**: ordinary operations return keys wrapped under the LMK. The
+  clear-key `unwrap` command is an opt-in internal operation with a dedicated
+  role and audit boundary.
 
 ## quick start
 
@@ -52,8 +54,8 @@ cd services/crypto_svc && make run    # REST :8001, gRPC :50051
 docker compose up --build             # containerized, from repo root
 ```
 
-The service is headless (service-to-service): auth is API-key based. Set
-`API_KEYS` to require keys on REST; gRPC always requires a key. See
+The service is headless (service-to-service): auth is API-key based and fails
+closed. An explicit insecure override exists for local development only. See
 [`docs/setup.md`](docs/setup.md).
 
 ## testing

docker-compose.yml

@@ -6,9 +6,14 @@ services:
     environment:
       PORT: ${PORT:-8001}
       GRPC_PORT: ${GRPC_PORT:-50051}
+      APP_ENV: ${APP_ENV:-development}
       # HSM provider: GP (software) / PS / AT (real HSMs)
       CRYPTO_HSM: ${CRYPTO_HSM:-GP}
-      # headless service-to-service auth via API keys (blank API_KEYS = open REST)
+      # Explicit local-only overrides. Disable them in deployments.
+      ALLOW_INSECURE_AUTH: ${ALLOW_INSECURE_AUTH:-true}
+      GRPC_REFLECTION: ${GRPC_REFLECTION:-true}
+      ENABLE_INTERNAL_UNWRAP: ${ENABLE_INTERNAL_UNWRAP:-false}
+      # Headless service-to-service auth via API keys.
       API_KEY_PREFIX: ${API_KEY_PREFIX:-crypto_}
       API_KEYS: ${API_KEYS:-}
       API_KEY_ROLES: ${API_KEY_ROLES:-admin}
@@ -19,4 +24,4 @@ services:
       test: ["CMD", "wget", "-q", "-O", "-", "http://localhost:8001/healthz"]
       interval: 5s
       timeout: 5s
-      retries: 5
+      retries: 5

docs/auth.md

@@ -5,8 +5,12 @@ The crypto service is a headless service (no browser clients, no IdP). Auth is
 
 | Transport | Auth | When enforced |
 |---|---|---|
-| REST | API key (`X-API-Key: crypto_<hex>` or `Authorization: Bearer crypto_<hex>`) | when `API_KEYS` is configured; otherwise open (`PermitAll`) |
-| gRPC | API key (`authorization: Bearer crypto_<hex>` metadata) | always (strict, via `APIKeyInterceptor`) |
+| REST | API key (`X-API-Key: crypto_<hex>` or `Authorization: Bearer crypto_<hex>`) | required unless explicit local insecure mode is enabled |
+| gRPC | API key (`authorization: Bearer crypto_<hex>` metadata) | same policy as REST |
+
+Startup fails when `API_KEYS` is empty unless `ALLOW_INSECURE_AUTH=true` is
+explicitly set. Production rejects that override. Local insecure mode supplies
+an admin development identity consistently to both transports.
 
 ## API keys (stateless)
 
@@ -32,4 +36,11 @@ role names. Wildcard `resource:*` matches all actions on a resource.
 |---|---|
 | `admin` | `crypto:*`, `serv:*` |
 | `operator` | `crypto:gen_sign`, `crypto:key_gen`, `crypto:kcv_gen`, `crypto:data_encr`, `crypto:data_decr`, `crypto:mac`, `crypto:rand_gen` |
-| `reader` | `crypto:rand_gen` |
+| `reader` | `crypto:rand_gen` |
+| `key_custodian` | `internal:unwrap` |
+
+`unwrap` is not covered by the `admin` role's `crypto:*` wildcard. Its REST
+route is absent and its gRPC method is disabled unless
+`ENABLE_INTERNAL_UNWRAP=true`. Enabling it also requires configured API keys.
+Calls are audited by operation and outcome only; request and response material
+is never included in those audit records.

docs/python-migration-parity.md

@@ -0,0 +1,138 @@
+# Python-to-Go migration parity
+
+This document is the migration ledger for the earlier Python/FastAPI crypto
+service in `dms-be/crypto_service`. It records what must remain compatible,
+what has deliberately changed, and which hardware-provider capabilities are
+still deferred. The old repository should not be required to establish the
+public endpoint inventory during future work.
+
+The status terms used here are:
+
+- **Ported**: the Go service exposes the operation over REST and gRPC and the GP
+  software provider implements it.
+- **Hardened difference**: the operation exists, but the Go service rejects an
+  unsafe or previously ignored input instead of silently accepting it.
+- **Deferred**: the API or adapter capability is known, but production-provider
+  implementation or parity evidence is not complete.
+- **Internal**: intentionally unavailable as an ordinary public crypto command.
+
+## Public API inventory
+
+The Python router exposed 16 crypto operations and three certificate/CRL
+operations. All 19 have a corresponding Go contract. This is API-surface
+parity, not yet production-provider parity: the current Go implementation has
+only the GP software provider. PayShield and AT integrations remain deferred.
+
+| Python operation | Go REST operation | Go API status | GP status | Production provider status | Notes |
+| --- | --- | --- | --- | --- | --- |
+| `kp_gen` | `POST /v1/crypto/kp_gen` | Ported | Ported | Deferred | RSA/EC key-pair generation. Golden-vector comparison remains open. |
+| `gen_sign` | `POST /v1/crypto/gen_sign` | Ported | Ported | Deferred | Malformed hexadecimal and base64 data is rejected as client input. |
+| `ecdh` | `POST /v1/crypto/ecdh` | Ported | Ported | Deferred | Shared-info and key-use semantics require an explicit compatibility decision. |
+| `exp_key` | `POST /v1/crypto/exp_key` | Ported | Ported | Deferred | Key type and use-mode behavior still requires parity vectors. |
+| `exp_tr31` | `POST /v1/crypto/exp_tr31` | Ported | Ported | Deferred | Short or malformed key blocks are rejected before header slicing. |
+| `rand_gen` | `POST /v1/crypto/rand_gen` | Ported | Ported | Deferred | Go caps one request at 4096 bytes to prevent unbounded allocation. |
+| `exp_tr34` | `POST /v1/crypto/exp_tr34` | Ported | Ported | Deferred | Requires shared positive and negative TR-34 vectors. |
+| `key_gen` | `POST /v1/crypto/key_gen` | Ported | Ported | Deferred | Key type/use-mode validation remains incomplete. |
+| `kcv_gen` | `POST /v1/crypto/kcv_gen` | Ported | Ported | Deferred | Output encoding must be compared with hardware providers. |
+| `ipek_derive` | `POST /v1/crypto/ipek_derive` | Hardened difference | Ported | Deferred | IKSN must be eight bytes. GP rejects transport-key export because it does not implement that parameter. |
+| `data_decr` | `POST /v1/crypto/data_decr` | Hardened difference | Ported | Deferred | Go honors the requested cipher mode instead of always selecting CBC with padding. |
+| `data_encr` | `POST /v1/crypto/data_encr` | Hardened difference | Ported | Deferred | Go honors the requested cipher mode instead of always selecting CBC with padding. |
+| `mac` | `POST /v1/crypto/mac` | Hardened difference | Partial | Deferred | GP supports generation only. Verification and KSN-based MAC requests are rejected rather than ignored. |
+| `trans_pin` | `POST /v1/crypto/trans_pin` | Hardened difference | Partial | Deferred | KSN must be 12 bytes and the PIN block 16 bytes. Destination KSN is rejected because GP does not implement it. |
+| `wrap` | `POST /v1/crypto/wrap` | Ported | Ported | Deferred | GP wrapping is for local development and test only. |
+| `unwrap` | Conditional `POST /v1/crypto/unwrap` | Internal | Ported | Deferred | Disabled by default. Requires `ENABLE_INTERNAL_UNWRAP=true`, authentication and the `key_custodian` role. Clear key material is never written to audit logs. |
+| certificate create | `POST /v1/serv/cert` | Ported | Ported | Deferred | Certificate behavior has Go service tests; cross-runtime vectors remain open. |
+| certificate renew | `PUT /v1/serv/cert` | Ported | Ported | Deferred | Certificate behavior has Go service tests; cross-runtime vectors remain open. |
+| CRL management | `POST /v1/serv/crl` | Ported | Ported | Deferred | Certificate behavior has Go service tests; cross-runtime vectors remain open. |
+
+## Provider boundary
+
+GP is a deterministic software HSM emulator for development and tests. Its
+fixed LMK derivation is intentional because reproducible wrapped values are
+useful in local tests. The Go service refuses to start with GP when
+`APP_ENV=production`. A production PayShield or AT adapter must use the
+device's native LMK, session and key-custody model; it must not inherit GP's
+deterministic LMK behavior.
+
+The earlier Python service had GP, PayShield (`PS`) and AT adapters. The Go
+service currently has only GP. Therefore no operation should be described as
+production-migration complete until at least one hardware adapter is ported,
+tested against a device or vendor simulator, and observed in shadow traffic.
+
+## PayShield capabilities outside the Python public API
+
+The Python PayShield adapter contains the following implementations, but its
+FastAPI router did not expose them. They are product-capability candidates, not
+missing public-route parity. Each requires an explicit API, authorization,
+audit and provider-support decision before being added to Go.
+
+| Capability | Python PayShield module | Status in Go | Required decision |
+| --- | --- | --- | --- |
+| ARQC verification and ARPC generation | `adapter/ps/verif_arqc.py` | Deferred | Define EMV input model, key references, methods 1/2 and option A/B support. |
+| CVV/CVC generation | `adapter/ps/cv_gen.py` | Deferred | Define permitted variants, PAN/expiry/service-code handling and PCI logging boundary. |
+| CVV/CVC verification | `adapter/ps/cv_verif.py` | Deferred | Define stable result codes without leaking verification detail. |
+| PVV generation | `adapter/ps/gen_pvv.py` | Deferred | Define PVKI/PVK reference model and issuer authorization policy. |
+| PVV verification | `adapter/ps/verif_pvv.py` | Deferred | Define PIN-failure response and rate-limit ownership. |
+| PIN generation | `adapter/ps/pin_gen.py` | Deferred | Decide whether this is an issuance-only internal API. |
+| IBM PIN offset | `adapter/ps/pin_offset.py` | Deferred | Define decimalization table and validation-data ownership. |
+| Visa PIN offset | `adapter/ps/pin_off_visa.py` | Deferred | Define scheme-specific contract and test vectors. |
+| TR-31 import | `adapter/ps/imp_tr31.py` | Deferred | Keep as a key-custodian operation with dual-control deployment procedures. |
+| RSA PIN translation | `adapter/ps/trans_pin_rsa.py` | Deferred | Define certificate/key reference, source format and PCI PIN controls. |
+
+## Intentional compatibility changes
+
+The Python GP adapter silently ignored several request fields or hardcoded a
+different value. Repeating that behavior would make the API misleading and
+would hide migration errors. The Go service therefore follows these rules:
+
+- An advertised option is implemented or rejected with a typed client error.
+- Malformed hexadecimal, base64, PAN, KSN, IKSN, PIN block and TR-31 inputs are
+  rejected before any provider call.
+- Requested data-encryption modes are honored. They are not silently replaced
+  with CBC plus padding.
+- Unsupported MAC verification, KSN-based MAC, destination-KSN PIN translation
+  and IPEK transport-key export are rejected.
+- Unwrap is retained as an internal command, disabled by default and separated
+  from the broad crypto administrator permission.
+- REST and gRPC authentication fail closed unless an explicit non-production
+  development override is enabled.
+
+These differences are migration safeguards. Consumers that depended on an
+ignored Python parameter must correct the request or wait for that capability
+to be implemented explicitly.
+
+## Parity proof still required
+
+Before switching an issuance or acquiring consumer, run both services with a
+shared vector corpus and compare normalized results. The corpus must contain:
+
+1. Known-answer vectors for every deterministic operation and algorithm/mode.
+2. Structural assertions for randomized operations, including key type,
+   length, encoding and successful round trips.
+3. Boundary and malformed values for every field accepted by REST and gRPC.
+4. Provider failures, timeouts and unavailable-session behavior.
+5. REST status/error envelopes and gRPC status-code equivalence.
+6. Output normalization checks for hex case, base64, absent fields and
+   provider-specific metadata.
+7. Shadow comparison using representative issuance and acquiring traffic with
+   sensitive values redacted from logs and reports.
+
+Random bytes, generated keys, signatures and certificates cannot be compared
+by raw equality unless the algorithm is deterministic. Compare their
+cryptographic properties or verify them using the corresponding public
+operation.
+
+## Migration completion criteria
+
+The Python service can be retired for a consumer only when:
+
+- every operation used by that consumer is marked supported for its selected
+  production provider;
+- shared positive, negative and boundary vectors pass;
+- authentication, role mapping, TLS ownership and audit retention are agreed;
+- provider readiness, timeout and recovery behavior has been exercised;
+- shadow comparison shows no unexplained semantic differences; and
+- rollback to the Python path has been tested for the migration window.
+
+Until then, this repository is a complete Go API and GP development reference,
+but not a declaration of PayShield or AT production parity.

docs/setup.md

@@ -21,17 +21,25 @@ interpolation. Copy `.env.example` to `.env` and adjust:
 cp services/crypto_svc/.env.example .env
 ```
 
-Key vars: `PORT`, `GRPC_PORT`, `CRYPTO_HSM` (provider: `GP` software, future
-`PS`/`AT` real HSMs), `SOFTWARE_LMK` (software-HSM master key), and the API-key
-auth block (`API_KEY_PREFIX`, `API_KEYS`, `API_KEY_ROLES`).
+Key vars: `APP_ENV`, `PORT`, `GRPC_PORT`, `CRYPTO_HSM` (`GP` software, future
+`PS`/`AT` real HSMs), `SOFTWARE_LMK`, and the API-key auth block
+(`API_KEY_PREFIX`, `API_KEYS`, `API_KEY_ROLES`).
 
-**Auth** — set `API_KEYS` (sha256 hex hashes) to require API keys on REST;
-gRPC always requires a key. Generate a hash with:
+**Auth** — startup requires `API_KEYS` (sha256 hex hashes) unless the explicit
+`ALLOW_INSECURE_AUTH=true` local-development override is set. The override is
+rejected in production and applies consistently to REST and gRPC. Generate a
+hash with:
 
 ```bash
 printf '%s' "<plaintext-key>" | shasum -a 256 | cut -d' ' -f1
 ```
 
+GP is a deterministic development/test emulator and is rejected when
+`APP_ENV=production`. A production deployment must select a hardware provider
+after its adapter is implemented. gRPC reflection is disabled unless
+`GRPC_REFLECTION=true`. The internal clear-key `unwrap` command is separately
+controlled by `ENABLE_INTERNAL_UNWRAP` and the `key_custodian` role.
+
 ## Build
 
 ```bash
@@ -66,4 +74,4 @@ make proto          # protoc → gen/crypto/v1 (after proto/crypto/v1 changes)
 
 ## Directory
 
-See the layout tree in `services/crypto_svc/README.md`.
+See the layout tree in `services/crypto_svc/README.md`.

docs/testing.md

@@ -7,7 +7,10 @@ From `services/crypto_svc/`:
 ```bash
 make test             # go test ./...
 make vet              # go vet ./...
-go test -race ./...   # race detector
+make test-race        # race detector, uncached
+make lint             # format, vet, govulncheck
+make proto-check      # regenerate protobuf output and require a clean diff
+make standalone       # prove both modules with GOWORK=off
 ```
 
 Workspace-wide (both modules), from `services/`:
@@ -18,11 +21,16 @@ go test ./common/... ./crypto_svc/...
 
 ## What's covered
 
-- **`internal/handlers`** — route registration for all 19 REST endpoints
-  (scaffold 501), RBAC 401/403 via injected `UserContext` (no mocks).
+- **`internal/handlers`** — route registration and implemented REST operations,
+  RBAC 401/403, internal-unwrap isolation, happy paths, and invalid-input 422
+  behavior via an injected `UserContext`.
+- **`internal/grpcserver`** — crypto validation error mapping, disabled internal
+  unwrap, and the explicit insecure-development identity.
 - **`internal/authz`** — role→permission matrix incl. wildcards.
-- **`internal/config`** — defaults, env overrides, flag precedence, invalid
-  values.
+- **`internal/config`** — fail-closed auth, production GP rejection, internal
+  unwrap prerequisites, malformed env values, overrides and flag precedence.
+- **`internal/health`** — liveness remains process-only while readiness follows
+  the configured HSM provider.
 - **`services/common`** — the shared framework's own suite: errs, middleware.
 
 Crypto engine known-answer vector tests cover the key-block, symmetric,
@@ -34,12 +42,13 @@ DUKPT, and asymmetric primitives.
 # health (public)
 curl -s localhost:8001/healthz    # {"status":"ok"}
 
-# REST (scaffold → 501 until engine lands)
-curl -s -X POST localhost:8001/v1/crypto/rand_gen -d '{"len":"12"}'
+# REST (local insecure mode from .env.example)
+curl -s -X POST localhost:8001/v1/crypto/rand_gen \
+  -H 'Content-Type: application/json' -d '{"len":"12"}'
 
-# gRPC (API key required)
+# gRPC (API key required outside explicit local insecure mode)
 grpcurl -plaintext -H "authorization: Bearer crypto_<key>" \
   localhost:50051 crypto.v1.CryptoService/RandGen
 ```
 
-Full auth + API-key setup is in [`auth.md`](auth.md).
+Full auth + API-key setup is in [`auth.md`](auth.md).

services/common/auth/auth.go

@@ -27,4 +27,4 @@ func WithUserContext(ctx context.Context, uc UserContext) context.Context {
 func UserFromContext(ctx context.Context) (UserContext, bool) {
 	uc, ok := ctx.Value(userCtxKey{}).(UserContext)
 	return uc, ok
-}
+}