This documentation contains the steps necessary to evaluate the artifacts for our paper (#266) titled "In-Depth Exploration of Differentially Private Quantization for Privacy-Preserving and Communication-Efficient Federated Learning".
Note: The original artifacts hosts at here.
- cpu-smoke: x86_64 CPU; ≥ 16 GB RAM (8 GB can be tight); ~15 GB disk for env + Zenodo assets (more if you keep checkpoints—see below).
- gpu-full: NVIDIA GPU with a working driver (PyTorch wheels ship CUDA libs). VRAM: ≥ 8 GB minimum, 12–16 GB more comfortable for default batches (CIFAR-10/ResNet + ROG). If you OOM, try lower
TRAIN_BATCH_SIZEin yaml configs.
- OS: Linux x86_64 is the primary target (macOS/Windows are not covered here).
- Shell:
bashfor the provided scripts. - Python environment: Miniconda so you can create an isolated env with Python 3.9 (section 2).
- Package manager:
pipinside that env (used to installpp_compression_eval/requirements.txtin section 3). - Network: Internet access to download the Zenodo archive, optional Hugging Face or other URLs used by the download script, and PyPI wheels unless you mirror them offline.
- Setup: approx. 10-20 mins (from downloading to extracting and setting up large artifacts)
- Sweep tests: approx. < 10 mins (including both CPU and GPU path)
- Utility eval: from 5 to 45 mins per task (depending on the task)
- Defense eval: 5 to 20 mins per task (depending on the task)
- Compressibility eval takes < 5 mins per task
Note: the above time was measured on a HPC node (AMD EPYC 7532 32-Core Processor with NVIDIA A100-PCIE-40GB)
Download the code tarball from the Zenodo record and extract it under any parent path <repo_root> on your machine (e.g., your home directory):
cd <repo_root>
tar -xzf pp_compression_eval-code-v1.0.1.tar.gzThis produces the repository root:
<repo_root>/pp_compression_eval/
Create a directory for datasets and other large files (use a disk with enough space if your home quota is tight):
mkdir -p <repo_root>/dataExample layout:
<repo_root>/pp_compression_eval/
<repo_root>/data/
<repo_root>/conda-envs/<env_name>/
Run Python as modules from <repo_root>:
cd <repo_root>
python -m pp_compression_eval.<module>eval "$(/path/to/miniconda3/bin/conda shell.bash hook)"
conda create -y -p <repo_root>/conda-envs/dp_quant_eval python=3.9 pip
conda activate <repo_root>/conda-envs/dp_quant_evalConfirm pip points at this env:
pip --versionYou will see pip 26.0.1 from /<repo_root>/conda-envs/dp_quant_eval/lib/python3.9/site-packages/pip (python 3.9)
From <repo_root> (the directory that contains pp_compression_eval):
pip install -r pp_compression_eval/requirements.txtQuick checks:
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"You will see 2.8.0+cu128 True
Set direct download URLs from your Zenodo record:
export ROG_MODEL_ZOOS_URL="https://zenodo.org/records/.../files/rog_model_zoos.tar.gz?download=1"
export ROG_DATA_URL="https://zenodo.org/records/.../files/rog_data.tar.gz?download=1"
export LPIPS_ALEX_PTH_URL="https://zenodo.org/records/.../files/alex.pth?download=1"
export GRADIENT_DUMP_TAR_URL="https://zenodo.org/records/.../files/gradient_dump.tar.gz?download=1"Download and verify:
bash pp_compression_eval/scripts/download_and_extract_assets.sh --verifyFiles land under pp_compression_eval/zenodo_assets/ and are checked against SHA256SUMS.txt. The download script uses sha256sum --ignore-missing when available, so checksum lines for files you have not downloaded yet (for example gradient_dump.tar.gz or the optional code tarball) do not fail verification.
Optional: manual download and hash check:
Save the Zenodo files next to the checksum file pp_compression_eval/zenodo_assets/SHA256SUMS.txt in the same directory (e.g., rog_model_zoos.tar.gz, rog_data.tar.gz, alex.pth, gradient_dump.tar.gz). Then verify (same check the script runs):
cd pp_compression_eval/zenodo_assets
sha256sum --ignore-missing -c SHA256SUMS.txtYou should see OK for each file present. To print a single file’s SHA-256 for comparison: sha256sum rog_data.tar.gz.
Small datasets like MNIST and CIFAR-10 are downloaded and initialized on-the-fly. This step unpacks the large Zenodo archives needed for the ROG attack:
bash pp_compression_eval/scripts/prepare_assets_for_smoke.sh \
--pp-dir <repo_root>/pp_compression_evalOptions:
--clean— remove previous extractions under the target paths, then redo (use when you want a fresh tree).--skip-rog— skip ROG tar extraction if already in place.
Extract the gradient archive for compressibility task:
bash pp_compression_eval/scripts/download_and_extract_assets.sh --no-download --no-verify \
--extract-gradient-dump --gradient-dump-dir <repo_root>/gradient_dumpMNIST and CIFAR-10 are fetched automatically into PyTorch’s default cache. If ROG or other assets expect paths under $HOME but you keep <repo_root> elsewhere, run eval commands with:
export HOME=<repo_root>(or prefix each command: HOME=<repo_root> bash ...).
bash pp_compression_eval/scripts/run_artifact_eval.sh cpu-smokeCHECK_GPU_CAPABILITY=1 bash pp_compression_eval/scripts/run_artifact_eval.sh gpu-fullInstead of using wrapper, call the harness directly to skip steps:
bash pp_compression_eval/scripts/sweep_smoke.sh --skip-train --skip-cp --skip-rogOnce the output shows
[sweep_smoke] Finished. Logs are in: <repo_root>/pp_compression_eval/runtime_log/sweep_smoke_YYMMDD_HHMMSSIt indicates sweep test completes and you are ready for full reproduction.
- CPU smoke path uses
run_mia_attack --smoke(train and obatin signals then checks that pickle artifacts exist; does not run inference attack). - GPU full path runs full
run_mia_attackwith settings compatible with the attack code (epoch checkpoints andcosine_attackenabled in the generated smoke config).
Adhering to the DSN Artifact Evaluation guidelines for minimizing hardware overhead and software dependencies, we propose a consolidated reproduction plan centered on CIFAR-10 with ResNet-18. While MNIST/LeNet allows for rapid execution, it lacks the architectural complexity required to demonstrate the stability of our proposed insights. Heavier vision baselines are omitted from this artifact bundle to keep runtimes within a reasonable review window. The CIFAR-10/ResNet-18 baseline serves as the optimal balance for validating our four core insights within the artifact evaluation time frame.
The reproduction consists three parts: utility, privacy and compressibility, totally 56 individual test cases.
Table 1. Compression Ratio Comparison
| Bit-Width | QDPSGD-1 | QDPSGD-2 | cpSGD | RQM |
|---|---|---|---|---|
| 2 | 17.82 | 21.33 | 21.13 | 20.31 |
| 8 | 4.81 | 13.60 | 11.20 | 17.85 |
From <repo_root> (directory that contains pp_compression_eval/), run cp_measure once per bit width and per DP-Quantizer (Table 1 columns map to --compressor values: QDPSGD-1 → qdpsgd1, QDPSGD-2 → qdpsgd2, cpSGD → cpsgd, RQM → rqm). Use dp-quantizer below as a placeholder for whichever of those you are sweeping:
cd <repo_root>
python -m pp_compression_eval.compression.cp_measure \
--config pp_compression_eval/configs/artifact_eval_compress/cifar10_resnet18_l2.yaml \
--mode cr \
--grad_dir <path_to_gradient_dump> \
--compressor dp-quantizer \
--bit_width 2Repeat with --bit_width 8 for the second row. Replace dp-quantizer with qdpsgd1, qdpsgd2, cpsgd, or rqm to fill each column. Other settings (e.g. epsilon, clipping) should match your paper / YAML defaults unless you override them on the CLI.
Table 2. DP Noise Norm Change at Low vs High Bit-Width (epsilon = 3.5, 10 from top to bottom)
| Bit-Width | QDPSGD-1 | QDPSGD-2 | cpSGD |
|---|---|---|---|
| 2 | 0.61 | 85.79 | 57.21 |
| 2 | 0.55 | 38.97 | 25.99 |
| 8 | 0.61 | 2.74 | 1.62 |
| 8 | 0.55 | 0.37 | 0.34 |
How to reproduce (Table 2 — DP noise norm):
Run training task with train.py and enable --verbose. That turns on compressor debug output, including DP noise ℓ₂ norms (along with clipping / bin-width diagnostics) each epoch. Sweep --compressor (e.g. qdpsgd1, qdpsgd2, cpsgd_v3) and --bit_width (2 and 8). Each config YAML already encodes the epsilon value (3.5 or 10) and clipping settings. Use dp-quantizer as a placeholder:
cd <repo_root>
# ε = 3.5, L2 clip
python -m pp_compression_eval.training.train \
--config pp_compression_eval/configs/artifact_eval_training/cifar10_resnet18_l2_eps3.5.yaml \
--compressor dp-quantizer --bit_width 2 \
--verbose
# ε = 10, L2 clip
python -m pp_compression_eval.training.train \
--config pp_compression_eval/configs/artifact_eval_training/cifar10_resnet18_l2_eps10.yaml \
--compressor dp-quantizer --bit_width 2 \
--verboseRepeat for each column and row setting (--bit_width 2 and 8, each --compressor); read the printed noise norm lines from the terminal (or redirect logs to a file). Noise norm only available for external noise method like QDPSGD-1/2 and cpSGD.
Table 3. Accuracy Drop Comparison I (epsilon = 3.5, 10 from top to bottom with L2 clip)
| Bit-Width | QDPSGD-1 | QDPSGD-2 | cpSGD | RQM |
|---|---|---|---|---|
| 2 | -12.42% | -16.36% | -13.98% | -25.42% |
| 2 | -11.48% | -13.52% | -8.15% | -25.11% |
| 8 | -3.70% | -3.96% | -3.86% | -25.04% |
| 8 | -3.67% | -3.66% | -3.81% | -24.89% |
Table 4. Accuracy Drop Comparison II (epsilon = 3.5, 10 from top to bottom with L-inf clip)
| Bit-Width | QDPSGD-1 | QDPSGD-2 | cpSGD | RQM |
|---|---|---|---|---|
| 2 | -33.96% | -36.51% | -28.59% | -27.33% |
| 2 | -31.59% | -33.85% | -26.32% | -26.22% |
| 8 | -23.07% | -25.72% | -24.65% | -27.21% |
| 8 | -19.42% | -23.33% | -23.17% | -25.29% |
How to reproduce (Tables 3 & 4 — accuracy drop):
These come from test accuracy under DP-quantized training vs the non-private / full-precision baseline, for each DP-Quantizer, bit width, and ε (3.5 and 10 — two rows per bit width in each table). Table 3 uses L2 clipping (clip_type: norm in YAML); Table 4 uses L∞ clipping (clip_type: value). Each config YAML encodes the epsilon and clipping settings:
- Table 3 (L2, ε=3.5):
--config pp_compression_eval/configs/artifact_eval_training/cifar10_resnet18_l2_eps3.5.yaml - Table 3 (L2, ε=10):
--config pp_compression_eval/configs/artifact_eval_training/cifar10_resnet18_l2_eps10.yaml - Table 4 (L∞, ε=3.5):
--config pp_compression_eval/configs/artifact_eval_training/cifar10_resnet18_linf_eps3.5.yaml - Table 4 (L∞, ε=10):
--config pp_compression_eval/configs/artifact_eval_training/cifar10_resnet18_linf_eps10.yaml
Example (replace dp-quantizer and --bit_width; add --verbose only if you also want noise-norm prints):
cd <repo_root>
# Table 3, ε = 3.5
python -m pp_compression_eval.training.train \
--config pp_compression_eval/configs/artifact_eval_training/cifar10_resnet18_l2_eps3.5.yaml \
--compressor dp-quantizer --bit_width 2
# Table 4, ε = 10
python -m pp_compression_eval.training.train \
--config pp_compression_eval/configs/artifact_eval_training/cifar10_resnet18_linf_eps10.yaml \
--compressor dp-quantizer --bit_width 2Compute accuracy drop as (final test accuracy − baseline accuracy) / baseline accuracy, expressed as a percentage. Different dp-quantizer follow the same as mentioned in above.
Table 1 supports Insight 1: pre-noise addition reduces compressibility at low bit-widths. Table 2 supports Insight 3: post-noise addition enlarges sensitivity at low bit-widths, requiring more noise under the same privacy budget. Tables 3 and 4 support Insights 2 and 4, showing that RQM narrows the accuracy drop discrepancies between L-2 and L-inf clipping, and that its utility depends mainly on the privacy budget rather than bit-width.
Privacy Eval Reproduction
Our insight for this part of the evaluation is that RQM provides more consistent protection because its effective number of quantization levels stabilizes to a privacy budget–dependent threshold even as bit-width increases, whereas for other DP-quantization mechanisms with external DP noise, that noise is strongly affected by bit-width.
Privacy experiments use the ROG (K. Yue et al.,) and FedMIA (G. Zhu et al.,) pipelines with DP-quantized gradients for GIA and MIA attack correspondingly. Configs live under pp_compression_eval/configs/artifact_eval_defense/. From <repo_root> (the directory that contains pp_compression_eval/), use dp-quantizer as a placeholder for the compressor name (qdpsgd1, qdpsgd2, cpsgd, rqm, …) and set --bit_width / --epsilon to match your run.
Single-run examples (one compressor × bit-width × ε):
cd <repo_root>
# ROG attack (FedAvg client simulation + reconstruction)
python -m pp_compression_eval.attack.rog.attack_fedavg \
--config pp_compression_eval/configs/artifact_eval_defense/rog_fedavg.yaml \
--compressor dp-quantizer --bit_width 2 --epsilon 3.5
# FedMIA — federated training (saves signals for MIA)
python -m pp_compression_eval.attack.fedmia.main \
--config pp_compression_eval/configs/artifact_eval_defense/fedmia_cifar10.yaml \
--defense dpquant --compressor dp-quantizer --bit_width 2 --epsilon 3.5
# FedMIA — membership inference attack (after training has produced logs/artifacts for the same settings)
python -m pp_compression_eval.attack.fedmia.run_mia_attack \
--config pp_compression_eval/configs/artifact_eval_defense/fedmia_cifar10.yaml \
--defense dpquant --compressor dp-quantizer --bit_width 2 --epsilon 3.5Replace dp-quantizer with a valid --compressor value and repeat over bit widths (e.g. 2 and 8) and ε (e.g. 3.5 and 10) as in your tables.
Full grid (all compressors × bit widths × ε in run_attack_grid.sh):
cd <repo_root>
bash pp_compression_eval/scripts/run_attack_grid.sh