Repository navigation
DP: default noise_multiplier (σ = 5.0 at clipping_norm 10) reduces the reference model to chance accuracy once DP is enabled #299
Description
Summary
With the reference manifest's DP block switched on, the default noise wipes out the model. cache_model_0/manifest.json ships noise_multiplier: 0.5 with clipping_norm: 10.0 (raised from 1.0 by PR No. 263 / issue No. 247). The noise scale is noise_multiplier × clipping_norm, so σ = 5.0 per coordinate, and accuracy falls to chance:
| Source | Baseline | Clip only | Clip + default noise |
|---|---|---|---|
PR No. 263 tests (seed 42, 2,000 samples from dataset/test/test_dataset.pt, reproducible) |
0.860 | 0.860 | 0.120 |
| PR No. 293 write-up (separate experiment, script not committed) | 85.25% | 85–86% | ~13%, at every clipping_norm from 1 to 20, and for update_gaussian too |
DP is "enabled": false by default, so nothing breaks today. But a model owner who turns DP on with the shipped parameters gets a useless model and no warning.
Why
Noise is added independently to every floating coordinate. For a model with D parameters, the total noise norm grows as σ·√D. cache_model_0's MLP has ~109K parameters (√D ≈ 330), so the noise swamps the signal regardless of how well clipping_norm is tuned. Raising clipping_norm to stop over-clipping (PR No. 263) also raised σ by the same factor. PR No. 293's sweep (not reproduced here) only recovers utility at noise_multiplier ≈ 0.001.
The current test pins this behaviour rather than flagging it: test_manifest_dp_full_noise_impact (tests/test_cache_client_dp.py) asserts dp_acc < baseline_acc - 0.3. Its docstring already suggests ~0.05 "would restore σ≈0.5" but leaves the choice open.
What needs deciding
Picking noise_multiplier is a privacy-budget decision, not a tuning knob. Documentation/technical/audits/dp-mechanism-review.md puts accountant integration (item 3) and DP-SGD (item 4) ahead of any ε claim. So the options are:
- Interim (docs/manifest only): keep DP off by default, but warn clearly in
Documentation/public/manifest.mdand the manifest example that the shippednoise_multipliergives no utility forcache_model_0, and give a measured utility-vs-noise table. No ε claim. - With the accountant (issue No. 279 / review item 3): choose a default
noise_multipliertogether with a reported (ε, δ) and a measured accuracy, and updatetest_manifest_dp_full_noise_impactto assert acceptable utility at that setting.
Option 1 can land now. Option 2 needs the DP specialist's input on issue No. 279.
References
- Issue No. 247 (closed), PR No. 263:
clipping_normretune and the real-weight accuracy tests - PR No. 293:
noise_multipliersweep (Finding 2), changes requested - Issue No. 279: DP mechanism fixes, DP specialist tagged
cache_model_0/manifest.jsondp.parameters;Documentation/public/manifest.md"Choosingclipping_norm"
Activity
abrahamnash commentedon Oct 29, 2026
@robertocarlous would you be up for Option 1 here? You wrote PR #263 and the "Choosing clipping_norm" section in manifest.md, so you're closest to it.
Scope for Option 1 (docs, manifest and test only, no ε claim):
@Abidoyesimze your #293 sweep is useful input for the table.
Option 2 (choosing a default noise_multiplier with a reported ε, δ) stays with #279 and the DP specialist. @walkonwayvs your view would be welcome there, given your DP review in #89.
walkonwayvs commentedon Oct 30, 2026
Thanks @abrahamnash. I'm a bit of a privacy nut personally, so I'd love to see DP on by default at some point.
Right now every client adds the full noise to its own update before uploading, so the averaged model only gets cleaner with the square root of the number of clients. The usual way (DP-FedAvg, McMahan et al. 2018) adds the noise once to the combined update, which hurts accuracy way less. Catch is the aggregator would see updates before the noise, so DIN would probably need secure aggregation with it.
Also, the right noise_multiplier and ε depend on how many clients are in a round. At devnet size I doubt any setting gets you both, which kinda matches #293. Accountant choice I'd leave to @tudorcebere.