Skip to content

docs(dp): verify noise_multiplier is independently destructive (BL-23 follow-on) - #293

Open
Abidoyesimze wants to merge 1 commit into
InfiniteZeroFoundation:developfrom
Abidoyesimze:feat/issue-279-clipping-norm-verification
Open

Abidoyesimze wants to merge 1 commit into
InfiniteZeroFoundation:developfrom
Abidoyesimze:feat/issue-279-clipping-norm-verification

Conversation

@Abidoyesimze

@Abidoyesimze Abidoyesimze commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

BL-23's clipping_norm question is already resolved on develop by PR #263, which raised the manifest default to 10.0 and added reproducible tests (_build_and_train_reference) pinning the reference model's real post-training L2 norm (~9.44) and one-round update_gaussian delta norm (~3.62).

This PR records a related, previously unscoped finding surfaced while cross-checking that work: at the manifest's default noise_multiplier: 0.5, Gaussian DP noise collapses model utility to the random-guessing floor regardless of clipping_norm, because the noise's aggregate L2 norm scales with noise_multiplier * clipping_norm * √(num_parameters), not just noise_multiplier * clipping_norm.

Changes

Test plan

  • python3 scripts/dp_noise_multiplier_sweep.py — prints post-training norm 9.44, delta norm 3.62 (matches PR fix(dp): retune clipping_norm for post_training mechanisms and add real-weight tests (BL-23) #263's pinned values), and the noise_multiplier sweep: 10.50% accuracy at the default 0.5, recovering to ~85.5% only once noise_multiplier drops to 0.001.
  • pytest tests/test_cache_client_dp.py -q — 31 passed, unaffected.
  • python3 .github/scripts/check_doc_links.py Documentation Developer — 573 links, all resolve.

Addresses review feedback on prior revisions of this PR — see the PR comments for the full discussion with @Santiagocetran, @robertocarlous, and @umeradl.

🤖 Generated with Claude Code

Copy link
Copy Markdown
Collaborator

Thanks, Abidoye. This overlaps with #247/#263, which already covers the clipping-default verification. Abraham also removed Track C from #279 for that reason. #263 now addresses the requested reference-data checks and sets the default to 10; this PR independently changes it to 15.

Could we reconcile the two before merging? I suggest keeping #263 as the existing scoped change. If the additional experiment supports a further adjustment, please include a reproducible script and exact settings so we can assess it as a separate follow-up. Otherwise, we should narrow the empirical claims. This isn’t a finding that 15 is inherently incorrect.

@robertocarlous

Copy link
Copy Markdown
Collaborator

Reviewed against develop in an isolated worktree (feat/issue-279-clipping-norm-verification on top of upstream develop tip 954fbf8d, 1 commit, no conflicts — confirmed by both git merge-tree --write-tree locally and GitHub's mergeable: MERGEABLE). All 3 actually-changed files verified against upstream develop.

Claimed: cache_model_0/manifest.json's clipping_norm: 1.0 was destructive against the reference model's real trained weights, and the new default clipping_norm: 15.0 sits comfortably above the observed combined L2 norm range.

Verified — manifest change: cache_model_0/manifest.json dp.parameters.clipping_norm is now 15.0 (was 1.0). The clip_scope: global and noise_multiplier: 0.5 are unchanged.

Verified — parameter count and √D math: counted cache_model_0/services/model.py's ModelArchitecture (784-128-64-10 MLP) directly:

Total params: 109,386   →   √D ≈ 330.7
Aggregate noise norm at noise_mult=0.5, clip_norm=15.0: ≈ 2480 (vs. trained weight L2 ≈ 12)

The document's "~109K parameters, √D ≈ 330, aggregate noise swamps signal" claim holds exactly. The √D penalty is the same structural issue dp-mechanism-review.md F4 names for the Laplace/L1 case; seeing it reappear here on the Gaussian side is consistent.

Verified — trained weight norm cross-reference: tests/test_cache_client_dp.py's _build_and_train_reference() (seed=42, 2000 MNIST samples, 20 epochs, lr=0.01) yields a trained (post-training) L2 norm of ~9.44 — within the document's 9.3-12.6 observed range (which uses Adam lr=0.001, 100-400 samples, 10 epochs). Different hyperparameters, consistent magnitude, no contradiction.

Verified — clipping-norm sweep inflection: the sweep table shows accuracy flat from clipping_norm ≥ 4.0 (85.15-85.60%, matching the unclipped 85.25% baseline). 15.0 is in this flat region with ~2.4x headroom over the top of the observed range (12.6). Sound choice.

Verified — Finding 2 is documented but not acted on: noise_multiplier: 0.5 collapses accuracy to ~13% regardless of clipping_norm (confirmed by the noise_multiplier sweep: utility only recovers below ~0.001). The PR explicitly leaves this unfixed pending an epsilon/accounting discussion, and the document cross-references dp-mechanism-review.md's priority list. The unfixed finding is carried in the BL-23 resolved note, not silently discarded.

Verified — BL-23 update: status field changes from 🆕 New — unaddressed to ✅ Resolved, with the link to the new audit document. The BL-23 resolved note accurately records both the fix (clipping_norm raised) and the open follow-on (noise_multiplier).

Verified — audit document placement: Documentation/technical/audits/dp-clipping-norm-verification.md exists, alongside the existing dp-mechanism-review.md.

Verified — issue No. 279 is open: title "DP: fix Laplace calibration (F4) and label post-training mechanisms experimental (F5)", state OPEN. The PR closes Track C of that issue; Tracks A (BL-24, Laplace reclassification) and B (Laplace recalibration) remain open — the document is correctly scoped to Track C only.

Verified — CI on 4166d9cb: CI OK ✅, Python ✅, Solidity ✅, Docs ✅, Integration paths ✅ all pass. Integration (full GI) skipping as usual. The 2 test_integration_marker.py failures are pre-existing on develop (confirmed by running on the current develop HEAD — same 2 failures). The 3 torch-dependent collection errors (test_cache_client_dp.py, test_resampling_policy.py, test_scoring.py) are also pre-existing (torch not installed in the CI-local pyDIN312 venv).

Not independently re-verified: the exact holdout accuracy figures (85.25%, 36.25%, 13.40% etc.) and the sweep tables rely on an uncommitted experiment script. The methodology is transparent (the document describes it fully, including why the script isn't committed — it hits sklearn.datasets.fetch_openml which is outside the test suite's permitted external-access perimeter). The figures are internally consistent and the math checks out. The key structural claim — that clipping_norm: 1.0 is destructive when the actual weight norm is 9-12× larger — is also supported by the existing Documentation/public/manifest.md and Documentation/technical/manifest.md tables (already updated in PR No. 263 to show clipping from 9.44 down to 1.0 as "discarding 89% of each weight").


All checkable claims held up. The scope is tight (one manifest value, one BL entry, one audit document), the reasoning is sound, and Finding 2 is correctly handled as a separate concern rather than an untested change. Looks good to merge.

@robertocarlous

Copy link
Copy Markdown
Collaborator

Merge proposal

Head at review: 4166d9cbd196ee43a401fd4182b97a0a672fa7ea
Merge base (upstream develop): 954fbf8d74058f4000f22760550eedd5d8a7f466

cache_model_0/manifest.json

Key Value
Change +1/-1 line: dp.parameters.clipping_norm raised from 1.0 to 15.0
Pending proposal None
Recommended merge proposal Merge as-is
Actual merge proposal Soon

Documentation/technical/audits/dp-clipping-norm-verification.md

Key Value
Change +79 lines (new file): full verification methodology — trained-weight norm sweep, clipping-norm sweep, noise_multiplier sweep, Finding 2 (noise_multiplier independently destructive, unfixed), reproduce instructions
Pending proposal None
Recommended merge proposal Merge as-is
Actual merge proposal Soon

Developer/BACK_LOG.md

Key Value
Change +1/-1 line: BL-23 status updated from 🆕 New to ✅ Resolved; resolved note cross-links the new audit document and records Finding 2 as an open follow-on
Pending proposal None
Recommended merge proposal Merge as-is
Actual merge proposal Soon

robertocarlous
robertocarlous previously approved these changes Oct 28, 2026

@robertocarlous robertocarlous left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verification and merge proposal in the comments above. All checks pass — approving.

Copy link
Copy Markdown
Collaborator

Thanks, Robbert. Before merging, could we resolve the coordination point in my earlier comment? #263 sets the clipping default to 10, while this PR sets it to 15. That overlap is still pending, so “Pending proposal: None” doesn’t reflect the current discussion.

My suggestion remains to keep #263 as the existing scoped change, then assess any further adjustment with reproducible evidence. This isn’t a claim that 15 is incorrect. Since the accuracy figures and sweeps haven’t been independently reproduced, please include the experiment script and exact settings, or narrow the empirical claims.

Also, Abraham removed Track C from #279 because the clipping-default work was already covered by #247/#263. Let’s update that scope reference and agree how the two PRs fit together before either lands.

@robertocarlous

Copy link
Copy Markdown
Collaborator

Hi @Abidoyesimze — three small changes needed before this can merge, to resolve the coordination point Santiagocetran raised. All in the existing commit; no new files.


1. cache_model_0/manifest.json — revert clipping_norm to 1.0

PR No. 263 already has the scoped manifest change (raising it to 10.0) and should land first. This PR should not touch the manifest — revert line 75 back to:

"clipping_norm": 1.0,

2. Documentation/technical/audits/dp-clipping-norm-verification.md — two edits

a) Line 3 — remove the "Track C" scope reference. Abraham removed Track C from issue No. 279 (the clipping-default work was already covered by PR No. 247 / PR No. 263). Replace:

**Tracking:** [issue #279](https://github.com/InfiniteZeroFoundation/DevNet/issues/279) Track C, [BL-23](../../../Developer/BACK_LOG.md)

with:

**Tracking:** [BL-23](../../../Developer/BACK_LOG.md); manifest default fix in [PR No. 263](https://github.com/InfiniteZeroFoundation/DevNet/pull/263)

b) Recommendation section — don't claim this PR raised the manifest value. Replace the current Recommendation paragraph and table outro:

Raised `cache_model_0/manifest.json`'s `clipping_norm` from `1.0` to **`15.0`** — comfortably above the full observed range, leaving headroom for training configurations this sweep didn't cover (more local epochs, more client data, different random seeds). A sweep across clipping norms (at the original 400-sample/10-epoch config) confirms utility stops degrading from clipping once the norm reaches roughly 4 and stays flat afterward:

...table...

`15.0` sits in this flat region with margin for the observed per-run variance above.

with:

The sweep confirms utility stops degrading from clipping once `clipping_norm` reaches roughly 4 and stays flat afterward:

...table (unchanged)...

Any value ≥ 4.0 avoids clipping-induced accuracy loss. A value at or above the observed norm range (9.3–12.6) provides headroom for training configurations this sweep did not cover (more local epochs, more client data, different random seeds). The manifest default change is tracked in [PR No. 263](https://github.com/InfiniteZeroFoundation/DevNet/pull/263), which raises `clipping_norm` to `10.0` — in the flat region and above the observed maximum. A further adjustment to `15.0` should wait for independent reproduction of the sweep figures in this document (see **Reproducing this** below).

3. Developer/BACK_LOG.md — BL-23 status line

In the BL-23 row, replace:

✅ Resolved — [issue #279](https://github.com/InfiniteZeroFoundation/DevNet/issues/279) Track C verified against a real trained `state_dict` ... `clipping_norm` raised to `15.0`. Full methodology ...

with:

✅ Resolved — verified against a real trained `state_dict` ... `clipping_norm` raised to `10.0` in [PR No. 263](https://github.com/InfiniteZeroFoundation/DevNet/pull/263). Full methodology ...

(Same text, just: remove "issue No. 279 Track C verified against", and change "raised to 15.0" → "raised to 10.0 in PR No. 263".)


Sequencing after these changes: PR No. 263 merges first (sets clipping_norm: 10.0), then this PR merges as the investigation/audit document that explains why. Both close BL-23 together.

Abidoyesimze added a commit to Abidoyesimze/DevNet that referenced this pull request Oct 28, 2026
…ights (BL-23)

Addresses PR InfiniteZeroFoundation#294/InfiniteZeroFoundation#293 review feedback: PR InfiniteZeroFoundation#263 already lands the scoped
manifest fix (clipping_norm 1.0 -> 10.0) for BL-23, so this PR no longer
touches cache_model_0/manifest.json — it's now just the investigation
writeup. Adds Documentation/technical/audits/dp-clipping-norm-verification.md,
quantifying how destructive the old clipping_norm: 1.0 default was against
the reference model's real trained weights (not the synthetic tensors
tests/test_cache_client_dp.py uses) and recommending any value >= 4.0,
noting InfiniteZeroFoundation#263's 10.0 already satisfies that with margin. Also documents a
related but separate, still-unfixed finding: the default noise_multiplier:
0.5 independently collapses accuracy regardless of clipping_norm, left for
a future epsilon/accounting decision rather than fixed here without that
context. Updates BL-23's status in Developer/BACK_LOG.md to resolved,
crediting PR InfiniteZeroFoundation#263 for the actual default change.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@Abidoyesimze
Abidoyesimze force-pushed the feat/issue-279-clipping-norm-verification branch from 4166d9c to c7c5777 Compare October 28, 2026 16:38
@Abidoyesimze

Copy link
Copy Markdown
Collaborator Author

Pushed c7c5777 (rebased onto develop, which now has #263 merged) with the 3 requested changes:

  1. cache_model_0/manifest.json no longer touched by this PR — rebasing onto develop made this concrete: the file now has zero diff against develop (clipping_norm stays at fix(dp): retune clipping_norm for post_training mechanisms and add real-weight tests (BL-23) #263's 10.0).
  2. dp-clipping-norm-verification.md: tracking line drops the "Track C" reference, and the Recommendation section no longer claims this PR raised the manifest value — it states any clipping_norm >= 4.0 avoids the clipping-induced accuracy loss and credits fix(dp): retune clipping_norm for post_training mechanisms and add real-weight tests (BL-23) #263's 10.0 as already satisfying that with margin, noting 15.0 would need independent reproduction of the sweep figures first.
  3. BACK_LOG.md's BL-23 resolved note now credits PR fix(dp): retune clipping_norm for post_training mechanisms and add real-weight tests (BL-23) #263 for the actual default change and drops the "Track C" phrasing, keeping the rest of the measured findings (including Finding 2 on noise_multiplier) as-is.

This PR is now just the investigation/audit writeup explaining why 10.0 (or higher) is needed, sequenced to merge after #263 as you both suggested. check_doc_links.py still passes (566 links, all resolve).

@umeradl

umeradl commented Oct 29, 2026

Copy link
Copy Markdown
Member

Reviewed against develop in an isolated worktree. The branch is one commit (c7c5777) on merge-base f2cff58; develop is now at f5b2284, which touches neither of this PR's files. It merges cleanly: git merge-tree --write-tree exits 0, and GitHub agrees (mergeable: MERGEABLE, mergeStateStatus: BEHIND). CI on c7c5777: CI OK green; the Docs job covers the new file's links. The PR is docs-only (a new audit write-up plus the BL-23 status cell), so there's no code to run. I checked its claims against what PR No. 263 merged.

Scope first. Abraham removed BL-23 from issue No. 279 on 26 Oct (comment) because issue No. 247 / PR No. 263 already covered it. PR No. 263 merged on 28 Oct. It raised clipping_norm to 10.0 and added reproducible real-weight tests in tests/test_cache_client_dp.py (fixed seed, the repo's own cache_model_0/dataset/test/test_dataset.pt), which pin the post-training norm (≈ 9.44), the update_gaussian delta norm (≈ 3.62) and baseline/clip-only/post-DP accuracy. So "Track C" is already done on develop. Despite the title ("fix(dp): retune clipping_norm"), this PR now retunes nothing; it adds a write-up of a parallel experiment.

Claimed: clipping_norm: 1.0 was destructive (85.25% → 36.25% from clipping alone); the real global L2 norm is 9.3–12.6.

Verified — consistent with develop, but not reproducible from this PR: PR No. 263's tests measure the same effect (norm ≈ 9.44). But the doc's own numbers come from an uncommitted script that fetches MNIST from fetch_openml ("Reproducing this" section), so nobody can re-run them. Its update_gaussian delta norm (8.69) also differs from develop's pinned 3.62. That's a different training config (400 samples × 10 epochs vs. the test fixture), but the doc doesn't say so.

Claimed (Finding 2): the default noise_multiplier: 0.5 collapses accuracy to ~13% at any clipping_norm, and utility only recovers around noise_multiplier ≈ 0.001.

Verified — partly new: PR No. 263 already records the collapse (post-DP accuracy 0.120) and the Documentation/public/manifest.md note about noise scaling with clipping_norm. The noise_multiplier sweep and the √D explanation are genuinely new detail, and useful for whoever picks up accountant integration.

Claimed (BACK_LOG.md): BL-23 → "✅ Resolved — verified against a real trained state_dict …".

Verified — accurate, but replaces a short correct cell with a long one: develop already says "✅ Fixed — PR No. 263".

Not independently re-verified: the sweep tables themselves, since the script isn't committed.


Recommendation: the part worth keeping is Finding 2. Two options:

  1. Make it reproducible and keep it as a docs PR: commit the sweep script (for example under scripts/, or as an integration-marked test, since it fetches external data). Cite develop's pinned numbers from PR No. 263 alongside your own and say how the training configs differ. Retitle it docs(dp): …, and leave BL-23's status cell as it is on develop.
  2. Or close this PR and post Finding 2 (the noise_multiplier sweep and the √D argument) as a comment on issue No. 279, where the DP specialist will see it.

Either way, it shouldn't be merged as "Track C of issue No. 279"; that track was removed from the issue before this PR was opened.

@umeradl

umeradl commented Oct 29, 2026

Copy link
Copy Markdown
Member

Files changed (2) — as of c7c5777 (PR head)

Diffed against merge-base f2cff58 (develop). develop has moved to f5b2284 since then, without touching either file. GitHub agrees: mergeable: MERGEABLE, mergeStateStatus: BEHIND. Confirmed with a local git merge-tree --write-tree dry run: clean.

Documentation/technical/audits/dp-clipping-norm-verification.md

Field Value
Change New
Lines +79/-0
Diff (what exactly is in this PR) A write-up of a parallel BL-23 experiment: the clipping_norm sweep (Finding 1) and the noise_multiplier sweep plus the √D noise-norm argument (Finding 2). The experiment script isn't committed.
Functionality — how & why How: docs only. Why: to record the measurements behind BL-23. PR No. 263 already landed the fix and reproducible tests for Finding 1, so the new part is Finding 2.
Diff vs current develop HEAD New file; there's no develop counterpart. develop's tests pin different numbers (norm ≈ 9.44, delta ≈ 3.62) from a different training config.
Recommended merge proposal Needs a fix before merge: commit the sweep script (or an integration-marked test), cite PR No. 263's pinned numbers next to these and explain the config difference, and retitle the PR as docs(dp). Alternative: close it and move Finding 2 to issue No. 279.
Actual merge proposal Soon
Pending proposal Author's choice between the two options.
Local merge conflict No
GitHub merge conflict No

Developer/BACK_LOG.md

Field Value
Change Modified
Lines +1/-1
Diff (what exactly is in this PR) BL-23 status: "✅ Fixed — PR No. 263" → a long "✅ Resolved — verified against a real trained state_dict …" paragraph linking the new doc.
Functionality — how & why How: status cell only. Why: to point BL-23 at the write-up.
Diff vs current develop HEAD Untouched by develop since merge-base.
Recommended merge proposal Drop this change and keep develop's cell. At most, append a link to the doc if it merges.
Actual merge proposal Soon
Pending proposal Revert to develop's cell.
Local merge conflict No
GitHub merge conflict No

Verification

Docs-only. CI Docs is green on c7c5777. The claims were checked against PR No. 263's merged tests and docs; details are in the verification comment above.

Local vs. GitHub agree: clean on both.

@umeradl umeradl left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested at c7c5777. Details are in the verification comment.

BL-23 was already fixed by PR No. 263 (merged, with reproducible real-weight tests) and was removed from issue No. 279 before this PR was opened. The new value here is Finding 2, the noise_multiplier sweep. Either:

  1. commit the sweep script (under scripts/, or as an integration-marked test), cite PR No. 263's pinned numbers and the training-config difference, retitle the PR as docs(dp), and leave BL-23's status cell as it is on develop; or
  2. close this PR and post Finding 2 as a comment on issue No. 279.

@robertocarlous

Copy link
Copy Markdown
Collaborator

Reviewed against develop in an isolated worktree (pr-293-review on top of c7c5777, the PR head, rebased onto develop after PR No. 263 merged). Peer check: OK (Abidoyesimze is the author; operator is not).

CI on c7c5777: All checks green ✅ — CI OK, Python, Solidity, Docs (566 links), Integration paths. Integration (full GI) skipping — non-blocking per MAINTAINING No. 4.9.

Two files changed vs. develop: Developer/BACK_LOG.md (+1/-1) and Documentation/technical/audits/dp-clipping-norm-verification.md (+79 new). cache_model_0/manifest.json has zero diff against develop — the rebase onto the post-PR No. 263 develop absorbed the clipping_norm: 1.0 → 10.0 change, leaving no manifest delta here.


Claimed — cache_model_0/manifest.json no longer touched by this PR. Verified: git diff upstream/develop...HEAD -- cache_model_0/manifest.json produces no output. The current develop already carries clipping_norm: 10.0 from PR No. 263.

Claimed — dp-clipping-norm-verification.md does not claim this PR raised the manifest value. Verified: the Recommendation section reads "The manifest default change is tracked in PR No. 263, which raises clipping_norm to 10.0" and notes 15.0 would need independent reproduction. No claim of raising the manifest value here ✓.

Claimed — BL-23 resolution credits PR No. 263 for the default change and drops "Track C". Verified: the new BL-23 status reads "✅ Resolved — verified against a real trained state_dict… clipping_norm raised to 10.0 in PR No. 263"; no "Track C" wording in the row ✓.

Claimed — Finding 1 (clipping_norm: 1.0 destructive) is quantitatively correct. The document reports the real global L2 norm at 9.3–12.6 across 5 seeds × 3 configs (consistent range, not a single-seed figure). The sweep table shows utility flat from ~4.0 onward, consistent with 10.0 being safely in the flat region. The √D scaling underpinning Finding 2: cache_model_0's MLP (784-128-64-10) has 784×128+128 + 128×64+64 + 64×10+10 = 109,386 parameters; √109,386 ≈ 330.7 — matches the doc's "~109K parameters (√D ≈ 330)". At noise_multiplier: 0.5, the aggregate noise L2 norm is ~165 at clipping_norm: 1.0, rising proportionally — consistent with the observed ~13% accuracy floor at every clipping_norm tested. The √D scaling also explains why lowering noise_multiplier by ~500x (to ~0.001) is needed for utility recovery (0.001 × 12.36 × 330 ≈ 4.1, comparable to the signal norm).

Claimed — Finding 2 (noise_multiplier: 0.5 independently destructive) is left unfixed. Verified: the doc explicitly defers to the epsilon/accounting context in dp-mechanism-review.md priority items 3–4. BACK_LOG.md records it as a concrete measured finding alongside the resolved Finding 1.

Claimed — doc links resolve. CI Docs check passes (566 links) ✓. Internal link [BL-23](../../../Developer/BACK_LOG.md) from Documentation/technical/audits/ path is correct (3 levels up to repo root, then into Developer/) ✓.

Merge status. Local git merge-tree --write-tree --name-only upstream/develop c7c5777 exits 0 (clean). GitHub shows CONFLICTING/DIRTY — stale, not yet recalculated after the rebase; update-branch will clear it (no actual conflict).


No findings. Merge-ready pending update-branch to clear GitHub's stale conflict flag. No foundry/src/** touched; no second sign-off required.

@robertocarlous

Copy link
Copy Markdown
Collaborator

Head at review: c7c5777
Merge base (upstream develop): behind but clean — merge-tree exit 0, GitHub's CONFLICTING flag is stale post-rebase; update-branch will clear it.


Documentation/technical/audits/dp-clipping-norm-verification.md

Key Value
Change +79 lines (new file): audit of clipping_norm against real trained state_dict. Finding 1: clipping_norm: 1.0 reduced accuracy 85.25% → 36.25% from clipping alone; real global L2 norm 9.3–12.6 across seeds/configs; flat recovery at ≥4.0; PR No. 263's 10.0 verified in flat region. Finding 2 (new): noise_multiplier: 0.5 collapses accuracy to ~13% at every clipping norm due to √D ≈ 330 scaling; left unfixed pending epsilon/accounting work.
Pending proposal None
Recommended merge proposal Merge as-is
Actual merge proposal Soon

Developer/BACK_LOG.md

Key Value
Change +1/-1 line: BL-23 status updated to "✅ Resolved — verified against a real trained state_dict"; credits PR No. 263 for the clipping_norm: 10.0 default; records Finding 2 (noise_multiplier) as a concrete measured finding; drops "Track C" phrasing; links to the new audit doc
Pending proposal None
Recommended merge proposal Merge as-is
Actual merge proposal Soon

robertocarlous
robertocarlous previously approved these changes Oct 31, 2026

@robertocarlous robertocarlous left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs and BACK_LOG only — clipping_norm verification doc internally consistent, √D ≈ 330 maths checks out, no manifest delta vs develop, BL-23 correctly credits PR No. 263, no Track C language. Verified at c7c5777

… follow-on)

BL-23's clipping_norm question is already resolved on develop by PR InfiniteZeroFoundation#263,
which raised the manifest default to 10.0 and added reproducible tests
pinning the reference model's real post-training L2 norm (~9.44) and
one-round update_gaussian delta norm (~3.62). This records a related,
previously unscoped finding surfaced while cross-checking that work: at
the manifest's default noise_multiplier (0.5), Gaussian DP noise collapses
model utility to the random-guessing floor regardless of clipping_norm,
because the noise's aggregate L2 norm scales with noise_multiplier *
clipping_norm * sqrt(num_parameters), not just noise_multiplier *
clipping_norm.

scripts/dp_noise_multiplier_sweep.py reproduces this using the exact same
fixture as PR InfiniteZeroFoundation#263's _build_and_train_reference() (the committed
cache_model_0/dataset/test/test_dataset.pt, seed=42, 2000 samples, 20
epochs, lr=0.01) -- its printed post-training/delta norms match PR InfiniteZeroFoundation#263's
pinned values exactly, confirming it's the same fixture, and it needs no
network access.

Finding 1 (the clipping_norm question) is left as PR InfiniteZeroFoundation#263's resolution;
this doc's own exploratory clipping_norm sweep (a different, uncommitted
config -- 400 samples/10 epochs/Adam lr=0.001, train_client_model's actual
hyperparameters) is kept only as corroborating evidence, explicitly labelled
as using a different config from PR InfiniteZeroFoundation#263's pinned numbers. BL-23's status
cell on develop (set by PR InfiniteZeroFoundation#263) is left untouched.

Renamed from dp-clipping-norm-verification.md to
dp-noise-multiplier-verification.md to match the finding this doc now
centers on.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@Abidoyesimze
Abidoyesimze force-pushed the feat/issue-279-clipping-norm-verification branch from c7c5777 to 082fd4c Compare October 31, 2026 10:34
@Abidoyesimze Abidoyesimze changed the title fix(dp): retune clipping_norm against real trained weights (issue #279 Track C) docs(dp): verify noise_multiplier is independently destructive (BL-23 follow-on) Oct 31, 2026
@Abidoyesimze

Copy link
Copy Markdown
Collaborator Author

Pushed 082fd4c, going with option 1:

  1. Committed the sweep script — scripts/dp_noise_multiplier_sweep.py reproduces Finding 2 using the exact same fixture as fix(dp): retune clipping_norm for post_training mechanisms and add real-weight tests (BL-23) #263's _build_and_train_reference() (committed cache_model_0/dataset/test/test_dataset.pt, seed=42, 2000 samples, 20 epochs, lr=0.01). Running it prints post-training norm 9.44 and delta norm 3.62 — matching fix(dp): retune clipping_norm for post_training mechanisms and add real-weight tests (BL-23) #263's pinned values exactly, so there's no network dependency and no config to reconcile for this part.
  2. Finding 1 reconciled against fix(dp): retune clipping_norm for post_training mechanisms and add real-weight tests (BL-23) #263's pinned numbers — the doc now cites fix(dp): retune clipping_norm for post_training mechanisms and add real-weight tests (BL-23) #263's 9.44/3.62 directly as the primary, reproducible answer to BL-23's original question, and keeps my earlier exploratory sweep (different config: 400 samples/10 epochs/Adam lr=0.001, matching train_client_model's actual hyperparameters) only as corroborating evidence, explicitly labelled as a different, non-reproduced config rather than presented as equivalent.
  3. BL-23's status cell left untouched — reverted my earlier edit; it stays exactly as fix(dp): retune clipping_norm for post_training mechanisms and add real-weight tests (BL-23) #263 set it ("✅ Fixed — PR No. 263").
  4. Retitled the PR to docs(dp): verify noise_multiplier is independently destructive (BL-23 follow-on), and renamed the doc file itself (dp-clipping-norm-verification.md → dp-noise-multiplier-verification.md) to match what it's actually about now.

Rebased onto current develop; mergeable is clean. pytest tests/test_cache_client_dp.py (31 passed) and check_doc_links.py (573 links) both still pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants