From 082fd4c173410fa4b839f0969b80d674f32be56b Mon Sep 17 00:00:00 2001 From: Abidoyesimze Date: Wed, 28 Oct 2026 09:53:27 +0100 Subject: [PATCH] docs(dp): verify noise_multiplier is independently destructive (BL-23 follow-on) BL-23's clipping_norm question is already resolved on develop by PR #263, which raised the manifest default to 10.0 and added reproducible tests pinning the reference model's real post-training L2 norm (~9.44) and one-round update_gaussian delta norm (~3.62). This records a related, previously unscoped finding surfaced while cross-checking that work: at the manifest's default noise_multiplier (0.5), Gaussian DP noise collapses model utility to the random-guessing floor regardless of clipping_norm, because the noise's aggregate L2 norm scales with noise_multiplier * clipping_norm * sqrt(num_parameters), not just noise_multiplier * clipping_norm. scripts/dp_noise_multiplier_sweep.py reproduces this using the exact same fixture as PR #263's _build_and_train_reference() (the committed cache_model_0/dataset/test/test_dataset.pt, seed=42, 2000 samples, 20 epochs, lr=0.01) -- its printed post-training/delta norms match PR #263's pinned values exactly, confirming it's the same fixture, and it needs no network access. Finding 1 (the clipping_norm question) is left as PR #263's resolution; this doc's own exploratory clipping_norm sweep (a different, uncommitted config -- 400 samples/10 epochs/Adam lr=0.001, train_client_model's actual hyperparameters) is kept only as corroborating evidence, explicitly labelled as using a different config from PR #263's pinned numbers. BL-23's status cell on develop (set by PR #263) is left untouched. Renamed from dp-clipping-norm-verification.md to dp-noise-multiplier-verification.md to match the finding this doc now centers on. Co-Authored-By: Claude Sonnet 5 --- .gitignore | 1 + .../dp-noise-multiplier-verification.md | 63 ++++++ scripts/dp_noise_multiplier_sweep.py | 195 ++++++++++++++++++ 3 files changed, 259 insertions(+) create mode 100644 Documentation/technical/audits/dp-noise-multiplier-verification.md create mode 100644 scripts/dp_noise_multiplier_sweep.py diff --git a/.gitignore b/.gitignore index ef7fa490..586b09a0 100644 --- a/.gitignore +++ b/.gitignore @@ -23,6 +23,7 @@ tasks/local/0x4657105FC932625CD289107aAE7B2174a822b709/services/__pycache__/* /tasks/* /scripts/* !scripts/preflight.py +!scripts/dp_noise_multiplier_sweep.py /hardhat/ignition/* # /hardhat/scripts/* # /hardhat/test/* diff --git a/Documentation/technical/audits/dp-noise-multiplier-verification.md b/Documentation/technical/audits/dp-noise-multiplier-verification.md new file mode 100644 index 00000000..efe04643 --- /dev/null +++ b/Documentation/technical/audits/dp-noise-multiplier-verification.md @@ -0,0 +1,63 @@ +# DP Noise Multiplier Verification Against Real Trained Weights + +**Tracking:** [BL-23](../../../Developer/BACK_LOG.md); manifest `clipping_norm` fix in [PR No. 263](https://github.com/InfiniteZeroFoundation/DevNet/pull/263) +**Scope:** BL-23's `clipping_norm` question is already resolved by PR #263, which raised the manifest default to `10.0` and added reproducible tests (`tests/test_cache_client_dp.py`'s `_build_and_train_reference`) pinning the reference model's real post-training L2 norm at ≈ 9.44 and its one-round `update_gaussian` delta norm at ≈ 3.62. This document instead records a related, previously unscoped finding surfaced while verifying that work: the manifest's default `noise_multiplier: 0.5` is independently destructive, regardless of `clipping_norm`. +**Method (Finding 2, reproducible):** [`scripts/dp_noise_multiplier_sweep.py`](../../../scripts/dp_noise_multiplier_sweep.py) uses the exact same fixture as PR #263's `_build_and_train_reference()` — the committed `cache_model_0/dataset/test/test_dataset.pt`, seed=42, 2,000 samples, 20 epochs, lr=0.01 — so its numbers are directly comparable to that test's pinned values, and the script is runnable with no network access or extra dependencies. + +--- + +## Finding 1 (already fixed by PR #263): `clipping_norm: 1.0` was destructive + +This was BL-23's original question, resolved by PR #263 before this document was committed. `_build_and_train_reference`'s pinned numbers (reproducible via `pytest tests/test_cache_client_dp.py -k test_real_weights_combined_norm_fits_manifest_clipping_norm`): the reference model's real post-training combined L2 norm is **≈ 9.44**, well above the old default of `1.0` and comfortably under PR #263's new default of `10.0`. + +An earlier, separate exploratory sweep (config: 400 real MNIST samples via `sklearn.datasets.fetch_openml`, 10 local epochs, Adam lr=0.001 — `train_client_model`'s exact hyperparameters, not committed as a script since it depends on network access) found the same order of magnitude across more configs/seeds — global L2 norm consistently in **9.3–12.6** across 5 seeds × 3 `(sample count, epoch count)` combinations — and confirmed utility stops degrading from clipping once `clipping_norm` reaches roughly 4: + +| `clipping_norm` | accuracy after clip only (that sweep's config) | +|---|---| +| 1.0 | 36.25% | +| 2.0 | 69.80% | +| 4.0 | 85.15% | +| 6.0–20.0 | 85.25–85.60% (flat, matches that config's unclipped baseline of 85.25%) | + +The two configs differ (400 samples/10 epochs/Adam lr=0.001 vs. PR #263's 2,000 samples/20 epochs/plain-SGD lr=0.01), which is why the norms differ slightly (9.3–12.6 vs. 9.44) — both are well inside the same order of magnitude, and both are well below PR #263's `clipping_norm: 10.0`, which sits in the flat, non-destructive region either way. No further `clipping_norm` change is proposed here. + +--- + +## Finding 2 (new, not in BL-23's original scope): the default `noise_multiplier: 0.5` is independently destructive, regardless of `clipping_norm` + +Run `python3 scripts/dp_noise_multiplier_sweep.py` to reproduce the numbers below exactly (no network access needed — it loads the committed test dataset the same way `tests/test_cache_client_dp.py` does). + +``` +Baseline (no DP) holdout accuracy: 0.8600 +Post-training combined L2 norm: 9.44 +One-round delta L2 norm: 3.62 +``` + +The post-training norm (9.44) and delta norm (3.62) match PR #263's pinned values exactly, confirming the script reproduces the same fixture. + +**Raising `clipping_norm` alone does not fix the accuracy collapse**, because the Gaussian noise's scale (`noise_multiplier * clipping_norm`) moves in lockstep with the clip. Holding `clipping_norm` fixed at the measured delta norm (3.62 — no clipping distortion at all) and sweeping `noise_multiplier`: + +| `noise_multiplier` | accuracy | +|---|---| +| 0.5 (current manifest default) | 10.50% | +| 0.1 | 10.00% | +| 0.01 | 73.00% | +| 0.001 | 85.50% | +| 0.0005 | 85.75% | +| 0.0 (no noise) | 86.00% | + +At the current default (`noise_multiplier: 0.5`), accuracy collapses to 10.50% — at the 10-class random-guessing floor — even with zero clipping distortion. Utility only recovers once `noise_multiplier` drops to roughly `0.001`, about **500x** below the current default. + +**Why:** noise is added independently to every floating-point coordinate. For a `D`-parameter model, the *aggregate* L2 norm of the injected noise scales as `noise_multiplier * clipping_norm * √D`, not `noise_multiplier * clipping_norm`. `cache_model_0`'s MLP has ~109K parameters (`√D ≈ 330`), so the aggregate noise norm at the default `noise_multiplier: 0.5` is roughly 330x the per-coordinate scale — far exceeding the signal it's added to, regardless of how well-tuned `clipping_norm` is. This is the same `√d` dimension penalty `dp-mechanism-review.md` F4 describes for the Laplace/L1 case, showing up again here on the Gaussian/noise-multiplier side. + +**This is left unfixed here.** Retuning `noise_multiplier` is a privacy-level decision that properly belongs with an epsilon/accounting discussion (`dp-mechanism-review.md`'s priority list, item 3 — "Accountant integration" — and item 4, DP-SGD), not a standalone default-value change made without that context. Recorded here as a concrete, measured, reproducible finding for whoever picks up that work next — see [issue #279](https://github.com/InfiniteZeroFoundation/DevNet/issues/279). + +--- + +## Reproducing this + +```bash +python3 scripts/dp_noise_multiplier_sweep.py +``` + +The script trains the real `ModelArchitecture` (`cache_model_0/services/model.py`) on the committed `cache_model_0/dataset/test/test_dataset.pt`, then runs `apply_dp_mechanism` from `cache_model_0/services/client.py` with `update_gaussian` across the `noise_multiplier` sweep above. It depends only on `torch` (already required by `tests/test_cache_client_dp.py`), with no network access and no `torchvision` dependency (it stubs the torchvision classes referenced by the dataset pickle, the same technique `tests/test_cache_client_dp.py`'s `_load_test_dataset()` uses). diff --git a/scripts/dp_noise_multiplier_sweep.py b/scripts/dp_noise_multiplier_sweep.py new file mode 100644 index 00000000..5a0e44a1 --- /dev/null +++ b/scripts/dp_noise_multiplier_sweep.py @@ -0,0 +1,195 @@ +#!/usr/bin/env python3 +""" +scripts/dp_noise_multiplier_sweep.py — reproduces Finding 2 of +Documentation/technical/audits/dp-clipping-norm-verification.md: at the +manifest default noise_multiplier (0.5), Gaussian DP noise collapses model +utility regardless of clipping_norm, because the noise's *aggregate* L2 norm +scales with sqrt(num_parameters), not just noise_multiplier * clipping_norm. + +Uses the exact same training fixture as tests/test_cache_client_dp.py's +_build_and_train_reference() (the committed cache_model_0/dataset/test/ +test_dataset.pt, seed=42, n_samples=2000, 20 epochs, lr=0.01), so the +post-training/delta norms here are directly comparable to that test's pinned +values (~9.44 / ~3.62) -- no network access or extra dependencies beyond +what the repo already requires for tests/test_cache_client_dp.py (torch). + +Usage: + python3 scripts/dp_noise_multiplier_sweep.py +""" +from __future__ import annotations + +import sys +import types +from importlib.util import module_from_spec, spec_from_file_location +from pathlib import Path + +import torch +import torch.nn as nn + +_REPO_ROOT = Path(__file__).resolve().parent.parent +_CACHE_MODEL_0 = _REPO_ROOT / "cache_model_0" + + +def _load_module(name, path): + spec = spec_from_file_location(name, path) + module = module_from_spec(spec) + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +def load_model_module(): + return _load_module("cache_model_0_model", _CACHE_MODEL_0 / "services" / "model.py") + + +def load_client_module(): + return _load_module("cache_model_0_client", _CACHE_MODEL_0 / "services" / "client.py") + + +def load_test_dataset(): + """Load cache_model_0/dataset/test/test_dataset.pt without torchvision. + + Mirrors tests/test_cache_client_dp.py's _load_test_dataset(): stubs the + torchvision classes referenced by the pickle so the raw tensors can be + recovered with only torch installed. + """ + + class _VD: + def __setstate__(self, d): + self.__dict__.update(d) + + class _MNIST(_VD): + pass + + class _ST(_VD): + pass + + class _Compose(_VD): + pass + + class _ToTensor(_Compose): + pass + + class _Normalize(_Compose): + pass + + stub_specs = [ + ("torchvision", {}), + ("torchvision.datasets", {}), + ("torchvision.datasets.mnist", {"MNIST": _MNIST}), + ("torchvision.datasets.vision", {"StandardTransform": _ST}), + ("torchvision.transforms", {}), + ("torchvision.transforms.transforms", {"Compose": _Compose, "ToTensor": _ToTensor, "Normalize": _Normalize}), + ] + added = {} + for mod_name, attrs in stub_specs: + if mod_name not in sys.modules: + m = types.ModuleType(mod_name) + m.__path__ = [] + m.__package__ = mod_name + for attr, cls in attrs.items(): + setattr(m, attr, cls) + sys.modules[mod_name] = m + added[mod_name] = m + + try: + dataset_path = _CACHE_MODEL_0 / "dataset" / "test" / "test_dataset.pt" + obj = torch.load(dataset_path, weights_only=False) + x = obj.data.float().unsqueeze(1) / 255.0 + y = obj.targets.clone() + return x, y + finally: + for mod_name in added: + sys.modules.pop(mod_name, None) + + +def build_and_train_reference(n_samples=2000, num_epochs=20, lr=0.01, seed=42): + """Same fixture as tests/test_cache_client_dp.py's _build_and_train_reference().""" + model_module = load_model_module() + criterion = nn.CrossEntropyLoss() + + x_all, y_all = load_test_dataset() + rng_sample = torch.Generator().manual_seed(seed) + idx = torch.randperm(len(x_all), generator=rng_sample)[:n_samples] + x_sub, y_sub = x_all[idx], y_all[idx] + + n_train = int(0.8 * n_samples) + x_train, y_train = x_sub[:n_train], y_sub[:n_train] + x_test, y_test = x_sub[n_train:], y_sub[n_train:] + + torch.manual_seed(seed) + model = model_module.ModelArchitecture() + ref_sd = {k: v.detach().clone() for k, v in model.state_dict().items()} + + batch_size = 32 + rng_train = torch.Generator().manual_seed(seed) + for _ in range(num_epochs): + perm = torch.randperm(len(x_train), generator=rng_train) + for start in range(0, len(x_train) - batch_size + 1, batch_size): + batch_idx = perm[start:start + batch_size] + loss = criterion(model(x_train[batch_idx]), y_train[batch_idx]) + loss.backward() + with torch.no_grad(): + for p in model.parameters(): + if p.grad is not None: + p -= lr * p.grad + p.grad.zero_() + + trained_sd = {k: v.detach().clone() for k, v in model.state_dict().items()} + return ref_sd, trained_sd, x_test, y_test + + +def evaluate_accuracy(model_module, state_dict, x, y): + m = model_module.ModelArchitecture() + m.load_state_dict(state_dict) + m.eval() + with torch.no_grad(): + preds = m(x).argmax(dim=1) + return (preds == y).float().mean().item() + + +def combined_l2_norm(state_dict): + total_sq = 0.0 + for value in state_dict.values(): + if torch.is_tensor(value) and torch.is_floating_point(value): + total_sq += torch.norm(value).item() ** 2 + return total_sq ** 0.5 + + +def main(): + client = load_client_module() + model_module = load_model_module() + + ref_sd, trained_sd, x_test, y_test = build_and_train_reference() + + baseline_acc = evaluate_accuracy(model_module, trained_sd, x_test, y_test) + post_training_norm = combined_l2_norm(trained_sd) + local_delta = client.compute_state_dict_delta(trained_sd, ref_sd) + delta_norm = combined_l2_norm(local_delta) + + print(f"Baseline (no DP) holdout accuracy: {baseline_acc:.4f}") + print(f"Post-training combined L2 norm: {post_training_norm:.2f}") + print(f"One-round delta L2 norm: {delta_norm:.2f}") + print() + print("noise_multiplier sweep (update_gaussian, clipping_norm fixed at the") + print("measured delta norm so no clipping distortion is introduced):") + print(f"{'noise_multiplier':>18} | accuracy") + print("-" * 32) + + for noise_multiplier in [0.5, 0.1, 0.01, 0.001, 0.0005, 0.0]: + dp_config = { + "enabled": True, + "mechanism": "update_gaussian", + "parameters": { + "clipping_norm": delta_norm, + "noise_multiplier": noise_multiplier, + "clip_scope": "global", + }, + } + out = client.apply_dp_mechanism(trained_sd, dp_config, reference_state_dict=ref_sd) + acc = evaluate_accuracy(model_module, out, x_test, y_test) + print(f"{noise_multiplier:>18} | {acc:.4f}") + + +if __name__ == "__main__": + main()