Skip to content

fix: report worker failures during collection - #602

Closed
simone-chen wants to merge 2 commits into
simonec/vllm031-b200-campaign-v2from
simonec/collector-live-failures
Closed

simone-chen wants to merge 2 commits into
simonec/vllm031-b200-campaign-v2from
simonec/collector-live-failures

Conversation

@simone-chen

@simone-chen simone-chen commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor
  • Worker failures were written to retained files while CI received only tqdm counters until collection ended. Forward each structured worker error from all three parent queue drains as a newline-delimited log with module, case ID, worker, group, exception and parameters.
  • Preserve worker output redirection, full traceback artifacts, error counts, checkpoint behavior, dispatch and measured callbacks. No automatic cancellation, skip policy or runner change is introduced.
  • Validation: 36 CPU parallel-run tests pass, including a failing callback followed by a callback blocked until the parent emits the failure. Retained traceback and exact failed/passed IDs are checked. Ruff check/format and git diff --check pass.
  • Stacked review against simonec/vllm031-b200-campaign-v2; implementation is one commit from immutable campaign source 6ec6925fabf863ffc2bfe46442b726448c70ce98. The logging commit changes only collect.py, its README and parallel-run tests. A separate follow-up repairs inherited ownership coverage for four campaign checkpoint files; it adds the existing FPE/maintainer owners through the policy generator. Eventual publication remains main-only; this draft does not merge or alter campaign pins.
  • Integration validation: 111 CODEOWNERS tests pass and strict ownership coverage is 3963/3963 files. Generated CODEOWNERS is committed; contributor metadata is unchanged. The prior head's Full CI failed only because these inherited seed files had catch-all coverage.
  • No GPU collection was launched for this logging change. Full CI is requested for the exact signed head.

Signed-off-by: Simone Chen <simonec@nvidia.com>
@coderabbitai

coderabbitai Bot commented Oct 27, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@simone-chen

Copy link
Copy Markdown
Contributor Author

/ok to test 48dbc99

Signed-off-by: Simone Chen <simonec@nvidia.com>
@simone-chen

Copy link
Copy Markdown
Contributor Author

/ok to test 7c5c519

@simone-chen

Copy link
Copy Markdown
Contributor Author

Closing as outside the requested review scope: retain only attention #604, recurrent/MoE #605, and communication #591 for Tianhao's review. This campaign infrastructure change is not merged or included in the replacements. Its branch and history remain available for separate future work; collection remains stopped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant