Context
Follow-up to #2.
The challenge is planned as two independent open tracks, segmentation and
classification, with participants able to customize local training and server
aggregation. Each track should rank task performance together with communication
efficiency.
The current segmentation prototype does not yet have scoring-grade communication
accounting. Its client and participant aggregator log approximate raw tensor
sizes for diagnostics, but those values omit serialized metadata, transport
behavior, cumulative directional totals, and ranking integration. NVFLARE
already writes message and stream statistics such as
simulator_cell_stats.json; the challenge runtime does not currently consume
them.
Proposed implementation
- Add an organizer-owned, task-neutral communication ledger.
- Read NVFLARE's post-serialization message and stream statistics after
recipe.execute().
- Count each supported FL transfer exactly once:
- server-to-client global models and associated tensor/metadata streams
- client-to-server updates and associated tensor/metadata streams
- Exclude registration, heartbeat, polling, acknowledgements, logging, and
experiment-tracking traffic.
- Reject unsupported auxiliary communication so submissions cannot bypass the
accounting path.
- Report server-to-client, client-to-server, and total byte counts in the JSON
and CSV result artifacts.
- Add a versioned reference-baseline artifact tied to the task, model signature,
initial checkpoint, participating sites, round count, transfer type, NVFLARE
version, and accounting-rule version.
- Preserve segmentation's
overall_public_score and per-site val_dice, while
adding task-neutral performance, communication, and ranking fields that can
also be used by the future classification track.
- Require valid communication evidence and a matching baseline artifact during
official scoring. Fail clearly instead of falling back to raw tensor-size
estimates.
With normalized task performance in [0, 1], calculate:
communication_efficiency = min(1, baseline_bytes / measured_bytes)
ranking_score = 0.95 * task_performance_score + 0.05 * communication_efficiency
Classification data, models, training, and evaluation are outside this issue.
The exact classification primary metric will be selected in the classification
task specification.
Acceptance criteria
- Unit tests cover directional totals, control-traffic exclusion, and avoidance
of sender/receiver double counting.
- Missing or malformed NVFLARE statistics fail clearly.
- An incompatible baseline fingerprint is rejected in official mode.
- Score tests cover baseline-equal, lower, and higher communication totals.
- JSON and CSV outputs retain the existing segmentation metric fields and add
the task-neutral fields.
- A two-client, two-round H100 run reconciles the reported totals with NVFLARE's
transport statistics.
- The reference baseline receives communication efficiency
1.0.
- Adding participant metadata increases measured bytes and lowers the
communication component.
Context
Follow-up to #2.
The challenge is planned as two independent open tracks, segmentation and
classification, with participants able to customize local training and server
aggregation. Each track should rank task performance together with communication
efficiency.
The current segmentation prototype does not yet have scoring-grade communication
accounting. Its client and participant aggregator log approximate raw tensor
sizes for diagnostics, but those values omit serialized metadata, transport
behavior, cumulative directional totals, and ranking integration. NVFLARE
already writes message and stream statistics such as
simulator_cell_stats.json; the challenge runtime does not currently consumethem.
Proposed implementation
recipe.execute().experiment-tracking traffic.
accounting path.
and CSV result artifacts.
initial checkpoint, participating sites, round count, transfer type, NVFLARE
version, and accounting-rule version.
overall_public_scoreand per-siteval_dice, whileadding task-neutral performance, communication, and ranking fields that can
also be used by the future classification track.
official scoring. Fail clearly instead of falling back to raw tensor-size
estimates.
With normalized task performance in
[0, 1], calculate:Classification data, models, training, and evaluation are outside this issue.
The exact classification primary metric will be selected in the classification
task specification.
Acceptance criteria
of sender/receiver double counting.
the task-neutral fields.
transport statistics.
1.0.communication component.