Clear-Weighted Codec

Code and retained results for Clear-Weighted Bit Allocation for Satellite Downlinks. The repository covers clear-weighted codec training, real-byte evaluation, the native checkerboard progressive interface, the codec-side semantic readout, deadline-pressure scheduling, and the reported onboard measurement.

The training detector supplies a soft spatial weight. It does not enter the deployed codec or bitstream. The network scheduler receives layer sizes, deadlines, arrival times, and the codec-native value. It receives no image, cloud mask, or reference score.

Contents

• src/clear_weighted_codec contains training, coding, evaluation, and scheduling code.
• src/clear_weighted_visualization builds the active evaluation figures.
• configs fixes the reported training and network protocols.
• results contains compact tables used by the paper figures and verification gates.
• manifests records data, checkpoint, source, and result identities.
• docs/PAPER_TO_CODE.md maps locked paper sections to code.
• docs/REPRODUCTION.md records the experiment contract.

AllClear imagery and trained weights are excluded. The manifests identify the reported checkpoints.

Installation

Python 3.11 or later and uv are required. Real-byte coding uses the commit-pinned RANSKit dependency. Setup installs the frozen uv.lock environment.

./setup.sh

Data

./run.sh prepare-data

The command selects the fixed AllClear ROI subset from the official split lists and extracts s2_toa and cld_shdw. The verified subset contains 25,382 train, 2,039 validation, and 4,052 test pairs.

Codec Training

Each seed starts from the joint RD300 detector bootstrap. Validation RD loss selects the detector checkpoint used as the frozen training mask for later codec runs.

./run.sh bootstrap-detector --set runtime.seed=42
./sweep.sh

The main sweep covers seeds 42 through 45 for the clear-weighted codec, cloud-agnostic controls, and learned baselines. Real-byte evaluation counts the emitted hyperlatent, anchor, and non-anchor streams.

./run.sh evaluate-sweep
./run.sh aggregate-rd

Frozen Transfer

transfer.py aggregates the unchanged codecs and detector on CloudSEN12+. The config pins the three-point four-seed RD record, 975-frame CloudSEN12+ detector record, and paired 501-frame AllClear deletion records by SHA-256. The same command reproduces the reported deletion-resampling curve.

./run.sh transfer \
  --set transfer.rate_distortion_input=/absolute/path/to/xeval-bdrate.json \
  --set transfer.cloudsen_deletion_input=/absolute/path/to/cloudsen-deletion.csv \
  --set transfer.allclear_ground_truth_input=/absolute/path/to/allclear-codec.csv \
  --set transfer.allclear_detector_input=/absolute/path/to/allclear-detector.csv

Progressive Interface

The selected network comparison uses the clear-weighted rate weight 10000 and cloud-agnostic rate weight 4567 at matched clear-region quality.

for seed in 42 43 44 45; do
  ./run.sh train-network-cw \
    --set runtime.seed="$seed" \
    --set method.detector.checkpoint="/absolute/path/to/detector-seed${seed}.pt"
  ./run.sh train-network-ca --set runtime.seed="$seed"
done

The codec emits [anchor, non_anchor, hyperlatent]. The base layer contains the hyperlatent and anchor streams. The refinement layer contains the non-anchor stream. Each layer adds a 16-byte header. Base reconstruction decodes the transmitted streams and fills non-anchor positions with their conditional means.

Train the semantic readout from the side parameters already computed during compression, then emit the progressive test trace.

./run.sh train-readout \
  --set runtime.seed=42 \
  --set model.checkpoint=/absolute/path/to/cw-rd10000-seed42.pt
./run.sh evaluate-progressive \
  --set model.checkpoint=/absolute/path/to/cw-rd10000-seed42.pt \
  --set progressive_evaluation.readout_checkpoint=/absolute/path/to/cw-head-seed42.pt

The readout predicts a 16 by 16 cloud grid and converts its mean to predicted clear pixels. It has 123,137 parameters and 31,473,664 multiply-accumulates per tile.

The default readout and progressive configs select the clear-weighted rate-weight-10000 codec. For the cloud-agnostic rate-weight-4567 traces, set loss.objective=uniform_mse, loss.rate_weight=4567, and distinct readout and trace outputs. Emit both test and valprop splits for every seed before registering the files in configs/network.yaml.

Spatial Analyses

analyses.py computes the reported conditional codelength and scoring-region aggregates from the registered four-seed records. It also contains the quantized-CDF codelength and GT, dilated-GT, and OmniCloudMask region definitions.

./run.sh analyses \
  --set analysis.codelength_input=/absolute/path/to/mechanism-region-bits.csv \
  --set analysis.region_input=/absolute/path/to/region-rescore.json

Network Scheduling

scheduling.py contains DPMW and the registered capacity and heldout controls. DPMW freezes base and full pressure at contact start, resumes partial layers at byte offsets, evicts refinements before bases, unlocks refinement after same-contact base completion, and uses the registered expired-work fallback. Its state contains no evaluator-only fields and it calls no solver. The fixed two-stage comparator uses SciPy MILP.

configs/network.yaml fixes the physical contact shape, capacity grids, deadlines, storage rule, interruption events, seeds, and 720-cell heldout grid. A progressive trace can be sampled to 90% mean cloud cover and run through any scheduler.

./run.sh network-trace \
  --set network.trace_input=/absolute/path/to/progressive.csv \
  --set network.trace_sha256="$TRACE_SHA256" \
  --set network.reference_trace_input=/absolute/path/to/cloud-agnostic-valprop.csv \
  --set network.reference_trace_sha256="$REFERENCE_TRACE_SHA256" \
  --set network.scheduler=dpmw

The command derives the reference capture volume from the registered cloud-agnostic valprop trace. Set network.capacity_bytes to replay an explicit capacity sequence. The output records input identities, contact service, evictions, completion contacts, and deadline metrics.

Fill the four seed-specific trace entries and common-detector entry in configs/network.yaml. Each progressive trace entry includes the trace and generated summary paths with their SHA-256 digests.

./run.sh network-capacity
./run.sh network-heldout
./run.sh network-references

These commands require ordered 501-frame valprop and 4,052-frame test traces from each selected codec. They emit the capacity and heldout grids, the four online reference replays, the offline HiGHS bound, the crossed cluster bootstrap, and compact publication tables under runs/network.

network-heldout can regenerate the 720 cells without the external scheduler audit bundle. Fresh paper_eligibility additionally requires all four paths under network_grid.heldout_audits. The summary keeps that result separate from the retained reported_paper_eligibility value.

Onboard Measurement

./run.sh measure-system \
  --set model.checkpoint=/absolute/path/to/codec-rd300-seed45.pt

The primary cell uses the trained seed-45 RD300 full encoder and the first registered valprop frame on the named Orin Nano configuration in configs/system.yaml. The command checks checkpoint and input identities, hardware state, live and retained entropy-table equality, stream parity, replay parity, timing, and the retained rail-window inputs before writing the measurement.

Figures

./run.sh figures

The command builds the nine active evaluation PDFs under runs/figures.

• fig_deletion_v3.pdf
• fig_progressive_capacity.pdf
• fig_dpmw_capacity.pdf
• fig_dpmw_mechanism.pdf
• fig_survivor_v3.pdf
• fig_energy_v3.pdf
• fig_c1_rd3.pdf
• fig_c2_maps_final.pdf
• fig_c3_perimage3.pdf

Use repeated --figure options to select individual figures. Accepted names are deletion, progressive, dpmw, dpmw-mechanism, survivor, energy, rd, maps, and per-image.

Verification

./run.sh verify
./run.sh report

Verification loads every config and manifest, checks all retained result hashes and paper values, and runs loss, entropy, and scheduler smoke cases.

The code is MIT licensed. The compact AllClear derivative in assets/figures/codelength_maps.npz remains subject to the AllClear CC BY-NC 4.0 terms.

Citation

[Preprint]

@misc{furutanpey2026clearweightedbitallocationsatellite,
      title={Clear-Weighted Bit Allocation for Satellite Downlinks}, 
      author={Alireza Furutanpey and Qiyang Zhang and Yujie Huang and Philipp Raith and Schahram Dustdar},
      year={2026},
      eprint={2608.01457},
      archivePrefix={arXiv},
      primaryClass={cs.NI},
      url={https://arxiv.org/abs/2608.01457}, 
}