Skip to content

Add scoring-grade bidirectional communication accounting #3

Description

@holgerroth

Context

Follow-up to #2.

The challenge is planned as two independent open tracks, segmentation and
classification, with participants able to customize local training and server
aggregation. Each track should rank task performance together with communication
efficiency.

The current segmentation prototype does not yet have scoring-grade communication
accounting. Its client and participant aggregator log approximate raw tensor
sizes for diagnostics, but those values omit serialized metadata, transport
behavior, cumulative directional totals, and ranking integration. NVFLARE
already writes message and stream statistics such as
simulator_cell_stats.json; the challenge runtime does not currently consume
them.

Proposed implementation

  • Add an organizer-owned, task-neutral communication ledger.
  • Read NVFLARE's post-serialization message and stream statistics after
    recipe.execute().
  • Count each supported FL transfer exactly once:
    • server-to-client global models and associated tensor/metadata streams
    • client-to-server updates and associated tensor/metadata streams
  • Exclude registration, heartbeat, polling, acknowledgements, logging, and
    experiment-tracking traffic.
  • Reject unsupported auxiliary communication so submissions cannot bypass the
    accounting path.
  • Report server-to-client, client-to-server, and total byte counts in the JSON
    and CSV result artifacts.
  • Add a versioned reference-baseline artifact tied to the task, model signature,
    initial checkpoint, participating sites, round count, transfer type, NVFLARE
    version, and accounting-rule version.
  • Preserve segmentation's overall_public_score and per-site val_dice, while
    adding task-neutral performance, communication, and ranking fields that can
    also be used by the future classification track.
  • Require valid communication evidence and a matching baseline artifact during
    official scoring. Fail clearly instead of falling back to raw tensor-size
    estimates.

With normalized task performance in [0, 1], calculate:

communication_efficiency = min(1, baseline_bytes / measured_bytes)
ranking_score = 0.95 * task_performance_score + 0.05 * communication_efficiency

Classification data, models, training, and evaluation are outside this issue.
The exact classification primary metric will be selected in the classification
task specification.

Acceptance criteria

  • Unit tests cover directional totals, control-traffic exclusion, and avoidance
    of sender/receiver double counting.
  • Missing or malformed NVFLARE statistics fail clearly.
  • An incompatible baseline fingerprint is rejected in official mode.
  • Score tests cover baseline-equal, lower, and higher communication totals.
  • JSON and CSV outputs retain the existing segmentation metric fields and add
    the task-neutral fields.
  • A two-client, two-round H100 run reconciles the reported totals with NVFLARE's
    transport statistics.
  • The reference baseline receives communication efficiency 1.0.
  • Adding participant metadata increases measured bytes and lowers the
    communication component.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions