fix(dp): retune clipping_norm for post_training mechanisms and add real-weight tests (BL-23)

Developer/BACK_LOG.md

@@ -29,7 +29,7 @@
 | BL-20 | Daemon | Container healthcheck can't detect `degraded` status | `HealthHandler.do_GET()` (`dincli/dind/health.py` — lands with [PR #32](https://github.com/InfiniteZeroFoundation/DevNet/pull/32), not yet on `develop`) always returns HTTP 200 unless `_build_payload()` itself raises — the computed `status: "degraded"` (stale last-tick) appears only in the JSON body, never in the status code. `docker-compose.yml`'s `curl -f http://127.0.0.1:8787/health` only fails on non-2xx, so a daemon whose event loop has hung while the health thread still answers passes the healthcheck forever. As wired, the check is crash-only, not hang-detecting — undercutting `ecf7aa3`'s "Docker auto-restart an unhealthy daemon" claim. Fix: map `degraded` to a non-2xx code, or document the healthcheck as crash-only. | P4-1.1 (harden before `dind` runs real jobs) | PR #32 code review finding No. 2, Jul 30, 2026 | 🟢 Fixed on the PR branch, unmerged — [PR #32](https://github.com/InfiniteZeroFoundation/DevNet/pull/32)'s `dincli/dind/health.py` now serves a `degraded` body with HTTP 503, so `curl -f` fails on a hung event loop. Lands with PR #32. Checked 2026-09-30 |
 | BL-21 | Daemon | `dind` persists no logs to disk | `configure_logging()` (`dincli/dind/logging.py` — lands with [PR #32](https://github.com/InfiniteZeroFoundation/DevNet/pull/32), not yet on `develop`) attaches a plain `StreamHandler` (stderr) with the `JsonFormatter` — no `RotatingFileHandler`, nothing written into `StateDirs` despite it already holding `jobs.db`/`dind.pid`/`preferences.json`. Under systemd (journald) or a container log driver the supervisor persists logs; run directly in a terminal — the documented and tested path — stderr is gone when the session closes, leaving nothing to `grep` for "why did my node skip a job three hours ago". For an always-on daemon aimed at non-expert operators (issue #21), `dind` should own a rotating log file under its own state dir. | P4-1.1 (operability, before `dind` is load-bearing) | PR #32 code review finding No. 6, Jul 30, 2026 | 🟢 Fixed on the PR branch, unmerged — [PR #32](https://github.com/InfiniteZeroFoundation/DevNet/pull/32)'s `dincli/dind/logging.py` adds a `RotatingFileHandler` in the state directory when `state_dir` is given, alongside stderr. Lands with PR #32. Checked 2026-09-30 |
 | BL-22 | Daemon / SDK | `NonceManager._instances` never evicted | `NonceManager` (`dincli/sdk/tx.py` — lands with [PR #31](https://github.com/InfiniteZeroFoundation/DevNet/pull/31), not yet on `develop`) keys its per-`(chain_id, address)` cache in a class-level `_instances` dict with no eviction, so it grows for the process lifetime. Correct and irrelevant for the CLI (one invocation per process), but it's process-wide mutable state — worth a look once `dind` runs long-lived. Already caused real test pollution: reservations leaked between test classes until each cleared `NonceManager._instances` in `setup_method`. | P4-1.1 (before `dind` runs long-lived) | `task_300726_8` implementation note, Jul 31, 2026 | 🟢 Fixed on the PR branch, unmerged — [PR #31](https://github.com/InfiniteZeroFoundation/DevNet/pull/31)'s `dincli/sdk/tx.py` bounds `_instances` at `_MAX_INSTANCES` (strong cache, with live instances tracked separately so an in-flight reservation isn't dropped). Lands with PR #31. Checked 2026-09-30 |
-| BL-23 | DP / Privacy (cache_model_0) | `clip_scope: global` default paired with an unreviewed `clipping_norm: 1.0` may over-clip real weights | [PR #109](https://github.com/InfiniteZeroFoundation/DevNet/pull/109) (`fix/dp-sensitivity`) correctly ties noise to the clip (`noise_scale = noise_multiplier * clipping_norm`) and defaults `clip_scope` to `global` so the clip actually bounds the combined L2 norm the noise is calibrated against — the sensitivity-calibration fix itself is sound (`per_layer` only bounds the combined norm at `clipping_norm * sqrt(n)`, not `clipping_norm`, so pairing it with the new noise formula would under-noise as tensor count grows). But `cache_model_0/manifest.json`'s `clipping_norm: 1.0` was never revisited: under the old `per_layer` scope it bounded each tensor independently to norm 1.0; under `global` it bounds the *entire model's combined* L2 norm to 1.0, a much tighter constraint for any model with more than one floating tensor (`sqrt(Σ‖tensor_i‖²)` grows with tensor count even if no individual tensor's norm changes). No test in PR #109 exercises this against `cache_model_0`'s real trained `state_dict` (the new tests use synthetic tensors), so it's unverified whether the new default clips — and thus degrades — the reference model's weights far more aggressively than before. Follow-up: run `apply_dp_mechanism`/`clip_state_dict` against `cache_model_0`'s actual trained weights under the new default and retune `clipping_norm` if the combined-norm bound turns out to be destructive. | P3 (verify before/at merge) | PR #109 review, Sep 8, 2026 | 🆕 New — unaddressed; `cache_model_0/manifest.json`'s `clipping_norm` still `1.0`, and `tests/test_cache_client_dp.py` still exercises synthetic tensors (`build_small_state_dict`), not real trained weights |
+| BL-23 | DP / Privacy (cache_model_0) | `clip_scope: global` default paired with an unreviewed `clipping_norm: 1.0` may over-clip real weights | [PR #109](https://github.com/InfiniteZeroFoundation/DevNet/pull/109) (`fix/dp-sensitivity`) correctly ties noise to the clip (`noise_scale = noise_multiplier * clipping_norm`) and defaults `clip_scope` to `global` so the clip actually bounds the combined L2 norm the noise is calibrated against — the sensitivity-calibration fix itself is sound (`per_layer` only bounds the combined norm at `clipping_norm * sqrt(n)`, not `clipping_norm`, so pairing it with the new noise formula would under-noise as tensor count grows). But `cache_model_0/manifest.json`'s `clipping_norm: 1.0` was never revisited: under the old `per_layer` scope it bounded each tensor independently to norm 1.0; under `global` it bounds the *entire model's combined* L2 norm to 1.0, a much tighter constraint for any model with more than one floating tensor (`sqrt(Σ‖tensor_i‖²)` grows with tensor count even if no individual tensor's norm changes). No test in PR #109 exercises this against `cache_model_0`'s real trained `state_dict` (the new tests use synthetic tensors), so it's unverified whether the new default clips — and thus degrades — the reference model's weights far more aggressively than before. Follow-up: run `apply_dp_mechanism`/`clip_state_dict` against `cache_model_0`'s actual trained weights under the new default and retune `clipping_norm` if the combined-norm bound turns out to be destructive. | P3 (verify before/at merge) | PR #109 review, Sep 8, 2026 | ✅ Fixed — PR No. 263 |
 | BL-24 | DP / Privacy (cache_model_0) | F4 — Laplace noise calibrated against an L2 clip, not the L1 sensitivity it's normally calibrated against | `post_training_laplace`'s `laplace_scale` is scaled by the same `clipping_norm` as the Gaussian mechanisms (`clip_state_dict`'s L2-norm clip), but the Laplace mechanism's textbook privacy guarantee is calibrated against L1 sensitivity, not L2. [PR #109](https://github.com/InfiniteZeroFoundation/DevNet/pull/109) flagged this explicitly rather than silently assuming it's fine (`Documentation/technical/services/clients.md:210-212`: "Laplace noise calibrated against an L2 clip is an open question tracked separately"), and [#99](https://github.com/InfiniteZeroFoundation/DevNet/issues/99)'s F4 (`Documentation/technical/audits/dp-mechanism-review.md` §F4) called the structural mismatch confirmed but left the exact magnitude/severity for a DP specialist's sign-off — explicitly excluded from #99's scope rather than fixed. [#119](https://github.com/InfiniteZeroFoundation/DevNet/pull/119) (which closed #99's remaining F8 items) left F4 untouched, so nothing currently tracks it now that #99 is closed. Needs either a DP specialist's opinion that L2-calibrated Laplace is acceptable as-is (and the open-question note downgraded to an explanation), or a fix (e.g. recalibrating against an actual L1 clip, or reclassifying `post_training_laplace` as experimental alongside F5's `per_layer` treatment) before any ε is claimed for it. | P3/P4 (needs DP specialist input first) | [#99](https://github.com/InfiniteZeroFoundation/DevNet/issues/99) F4 (excluded from scope), `Documentation/technical/audits/dp-mechanism-review.md` §F4, Sep 10, 2026 | 🆕 New — unaddressed; `Documentation/technical/services/clients.md`'s open-question note on L2-calibrated Laplace noise is unchanged since PR #109/#119 |
 | BL-25 | Tooling / reference model (cache_model_0) | `cache_model_0/abis/` task-contract artifacts are stale vs. `foundry/src` | `cache_model_0/abis/DINTaskCoordinator.json` and `DINTaskAuditor.json` are full hardhat artifacts (abi + bytecode) last regenerated 2026-06-09, missing 67 ABI entries present in `foundry/src` for the coordinator (104 vs 170) and 96 for the auditor (68 vs 158), with pre-foundry constructors. [PR #171](https://github.com/InfiniteZeroFoundation/DevNet/pull/171) regenerated the six bundled `dincli/abis/` files from `foundry/out` but deliberately left these alone: [cache_model_0/manifest.json](../cache_model_0/manifest.json)'s `task_contracts` block pins each artifact to an IPFS CID, records `"type": "hardhat-artifact"` and a hardhat/cancun/optimizer-off `compilation` block, and carries deployed addresses, and dincli resolves task artifacts by CID. Regenerating the files locally would leave the manifest describing artifacts it no longer points to. Needs one pass: regenerate both from `foundry/out` (with bytecode, e.g. `dincli system dump-abi --bytecode --output cache_model_0/abis`), update the manifest's artifact type and compilation metadata to foundry, re-pin to IPFS and update the CIDs. Addresses only change on a redeploy. | P3/P4 | PR #171 merge (2026-09-26): out of scope for the PR's ABI refresh | 🆕 New — unaddressed |
 | BL-26 | Solidity/foundry (task contracts) | Batch-assignment seed lock (H-2) shares BL-11's re-roll residual, plus a new pool-reshaping gap | `DINTaskCoordinator.autoCreateTier1AndTier2`/`createAuditorsBatches` (task_240926_18 Part B, issue #156 H-2 aggregation side) reuse BL-11's future-block-seed + permissionless-lock pattern (`lockAggSeed`/`lockAuditSeed`), so they inherit its residual (2): the model owner can decline to lock a seed they dislike and wait out the ~256-block `blockhash` window for a fresh draw — repeatable, since nothing locks automatically today. A second, distinct gap not present in BL-11's `_assignFreshSubgroup` (which shuffles a fixed pool and only lets an exiting validator drop itself, not re-roll the rest): `_activeAggregatorPool`/`_activeAuditorPool` are evaluated at `autoCreateTier1AndTier2`/`createAuditorsBatches` call time, *after* the seed is already public, and Fisher-Yates re-rolls the whole assignment on any pool-size change — so a validator can unstake between lock and create to reshape everyone else's batch. Cost to the attacker is real but modest (their own Sybils sit out the GI and enter unbonding). Ready-made fix for the pool-reshaping gap: adopt `_assignFreshSubgroup`'s pattern — shuffle the fixed per-GI registration list (`dinAggregators[_GI]` / the auditor equivalent) with the locked seed, then skip inactive validators while filling seats, so an unstake only forfeits the leaver's own seat instead of re-rolling everyone else's. | P3/P4 | [PR #191](https://github.com/InfiniteZeroFoundation/DevNet/pull/191) review, Sep 29, 2026 | 🟡 Partially mitigated — the decline-to-lock re-roll is narrowed by a validator-side lock in dincli, added at the PR #191 merge (`dincli aggregator lock-seed` / `dincli auditor lock-seed`, and automatically in `show-t1-batches`/`show-t2-batches`/`lms-evaluation show-batch` before batches exist): any validator online after the seed block locks it, so the owner only gets a re-roll if nobody else is. A contract-side cap on re-anchors was considered and deferred: with no GI-abort path, a GI capped out would be stuck forever (`endGI` needs `AggregatorsSlashed`, `startGI` needs `GIended`), locking validators' registration slots and the reward pool — it needs a GI-abort design (who may abort, where the reward pool goes) first. Pool-reshaping gap unaddressed. Documented as residuals in the PR (`DINTaskCoordinator.md` §6.3). Sequencer-trust residual (shared with BL-11) tracked in [issue #178](https://github.com/InfiniteZeroFoundation/DevNet/issues/178); the re-roll-by-declining-to-lock and pool-reshaping gaps above have no dedicated issue yet, same as BL-11's own un-issued residual (2) |

Documentation/public/manifest.md

@@ -42,7 +42,7 @@ For example:
 | `Genesis_Model_CID` | Yes | IPFS CID of the genesis model. Set after running `dincli model-owner model create-genesis` |
 | `DINTaskCoordinator_Contract` | Yes | Address of the deployed TaskCoordinator contract |
 | `DINTaskAuditor_Contract` | Yes | Address of the deployed TaskAuditor contract |
-| `dp` | No | Nested Differential Privacy configuration block used by services at runtime |
+| `dp` | No | Nested Differential Privacy configuration block used by services at runtime. See [Differential Privacy Parameters](#differential-privacy-parameters) below. |
 
 ### Service Entries
 
@@ -109,6 +109,40 @@ and then install the requirements.txt file using the following command:
 pip3 install -r requirements.txt
 ```
 
+---
+
+## Differential Privacy Parameters
+
+The optional `dp` block controls whether trained weights are clipped and noised before upload. Privacy is **off by default** — a manifest with no `dp` block or `"enabled": false` uploads raw weights.
+
+```json
+"dp": {
+  "enabled": false,
+  "mode": "afterTraining",
+  "mechanism": "post_training_gaussian",
+  "parameters": {
+    "clipping_norm": 10.0,
+    "noise_multiplier": 0.5,
+    "clip_scope": "global"
+  }
+}
+```
+
+### Choosing `clipping_norm`
+
+`clipping_norm` controls the L2-norm bound that is clipped and that the noise is calibrated against (`noise_scale = noise_multiplier × clipping_norm`). The right value depends on **what** the mechanism clips:
+
+| Mechanism | What is clipped | Guidance |
+|-----------|----------------|----------|
+| `post_training_gaussian` / `post_training_laplace` | Combined L2 norm of all final weights | Set to at least the model's combined weight norm. For the reference cache_model_0 architecture (measured ≈ 9.44 with seed=42, 2000 MNIST samples, 20 epochs, lr=0.01), use **≥ 10.0**. Too small a value discards most of the model. |
+| `update_gaussian` | Combined L2 norm of the per-round weight delta | The delta is the per-round weight change, not the absolute weights. For cache_model_0 (seed=42, 2000 MNIST samples, 20 epochs, lr=0.01), the measured delta norm is ≈ 3.62. **≥ 4.0** is appropriate for those settings; adjust based on your own training configuration. |
+
+> **Note**: raising `clipping_norm` to stop over-clipping also raises the noise by the same factor. If you increase `clipping_norm`, lower `noise_multiplier` proportionally to keep the absolute noise standard deviation unchanged (note: this does not preserve equivalent DP privacy protection — a larger `clipping_norm` provides weaker privacy at the same σ; privacy assessment is needed separately).
+
+For full technical details see [technical/manifest.md](../technical/manifest.md).
+
+---
+
 ## Example Manifest
 
 Please find the example/template manifest file at [cache_model_0/manifest.json](../../cache_model_0/manifest.json)

Documentation/technical/manifest.md

@@ -134,7 +134,7 @@ The DevNet manifest now supports a preferred nested `dp` block:
   "mode": "afterTraining",
   "mechanism": "post_training_gaussian",
   "parameters": {
-    "clipping_norm": 1.0,
+    "clipping_norm": 10.0,
     "noise_multiplier": 0.5,
     "laplace_scale": 0.35,
     "clip_scope": "global"
@@ -153,6 +153,19 @@ Resolution rules in the current service:
 - `clipping_norm` must be greater than zero whenever DP is enabled. A non-positive value is rejected rather than treated as "no clipping", because noise applied without a sensitivity bound behind it is not a privacy guarantee;
 - `post_training_laplace` requires its own `laplace_scale`. It does not fall back to `noise_multiplier`, which is a differently calibrated quantity measured in different units.
 
+### Per-mechanism `clipping_norm` guidance
+
+The two mechanism families clip fundamentally different quantities, so the right `clipping_norm` differs by roughly an order of magnitude:
+
+| Mechanism | What is clipped | Typical norm (cache_model_0) | Recommended `clipping_norm` |
+|-----------|----------------|------------------------------|-----------------------------|
+| `post_training_gaussian` / `post_training_laplace` | Combined L2 norm of all final weights | ≈ 9.44 (seed=42, 2000 MNIST samples, 20 epochs, lr=0.01) | ≥ 10.0 |
+| `update_gaussian` | Combined L2 norm of the per-round weight delta | ≈ 3.62 (seed=42, 2000 MNIST samples, 20 epochs, lr=0.01) | ≥ 4.0 |
+
+Using `clipping_norm: 1.0` with `post_training_*` and `clip_scope: global` clips the combined weight norm from ≈ 9.44 down to 1.0 — a scale factor of ≈ 0.11 — discarding 89% of each weight before noise is applied. Raising `clipping_norm` to cover the actual combined norm stops the destructive clipping, but raises the noise by the same factor (`noise_scale = noise_multiplier × clipping_norm`), so `noise_multiplier` should be reduced proportionally to keep the same absolute noise standard deviation σ. Note: reducing `noise_multiplier` proportionally preserves σ but does not preserve equivalent DP privacy — a larger `clipping_norm` provides weaker privacy at the same σ because the sensitivity bound is larger. The right tradeoff requires separate privacy assessment.
+
+`update_gaussian` clips only the round-to-round weight delta, not the absolute weights. The right `clipping_norm` depends on training settings; for the reference configuration (seed=42, 2000 MNIST samples, 20 epochs, lr=0.01) the measured delta is ≈ 3.62, so `clipping_norm=4.0` leaves it unclipped. Adjust for your own training duration and dataset.
+
 **Privacy is off unless you turn it on.** A manifest with no `dp` block, or
 with `dp.enabled: false`, uploads raw trained weights with no clipping and no
 noise. This is the default. Nothing warns about it at runtime, so a model owner

cache_model_0/manifest.json

@@ -71,7 +71,7 @@
       "mode": "afterTraining",
       "mechanism": "post_training_gaussian",
       "parameters": {
-         "clipping_norm": 1.0,
+         "clipping_norm": 10.0,
          "noise_multiplier": 0.5,
          "laplace_scale": 0.35,
          "clip_scope": "global"