Skip to content
This repository was archived by the owner on Oct 25, 2026. It is now read-only.

feat: merge resumed training attempts into logical tasks - #9

Merged
LingyeSoul merged 1 commit into
LingyeSoul:mainfrom
buxinzi2233:codex/logical-training-resume
Aug 12, 2026
Merged

LingyeSoul merged 1 commit into
LingyeSoul:mainfrom
buxinzi2233:codex/logical-training-resume

Conversation

@buxinzi2233

Copy link
Copy Markdown

Summary

  • Model resumed processes as attempts of one logical training task through root_job_id, parent_job_id, and attempt_index.
  • Fix resumable-state discovery for output names containing @ by matching both raw and filesystem-safe layouts while retaining marker and signature validation.
  • Aggregate attempts in the daemon, main WebUI, and run analyzer so metrics, logs, samples, events, and final status remain attached to one logical run.
  • Keep physical job APIs compatible and make record cleanup remove the whole attempt chain without deleting models, checkpoints, or training-state directories.

Implementation

  • Adds lineage persistence and logical-task list/detail/delete/resume behavior to the daemon.
  • Rejects stale-branch resume requests and resumes from the current attempt while preserving per-attempt logs and exit evidence.
  • Merges metrics by global_step with newer attempts taking precedence, concatenates logs with attempt boundaries, and adds attempt-aware sample access.
  • Adds an expandable attempt history to the main task card and switches task actions to the latest physical job.
  • Groups analyzer runs by root task and merges JSONL/TensorBoard curves, checkpoints, samples, warnings, stdout, and event timing.
  • Adds focused coverage for safe output names, persistence/restart, lineage constraints, aggregation, cleanup, and path traversal.

Validation

  • 131 passed, 3 warnings:
    • tests/test_daemon.py
    • tests/test_task_service_recovery.py
    • tests/test_run_analyzer.py
    • tests/test_preview.py
    • tests/test_progress_sink.py
    • tests/test_staged_resolution_plan.py
  • npm run build passed for webui/frontend.
  • git diff --check origin/main..HEAD passed.

Known findings for maintainer decision

These findings are intentionally not fixed in this PR so the maintainers can choose the desired policy and implementation:

  1. [P1] Recovery-state ownership is inferred from compatibility signatures. scripts/daemon/manager.py can associate a compatible state with a later independent job because config/data signatures prove compatibility, not ownership. One option is to persist and validate job_id or root_job_id in the training state.
  2. [P1] Rolling state remains eligible after successful training. library/training/checkpoints.py keeps the rolling -state directory discoverable, so a later fresh run can silently auto-resume stale state, particularly when the target step is not aligned with the rolling save cadence. The state also retains large optimizer artifacts. Maintainers should decide whether completion deletes, retires, or requires explicit selection of rolling state.
  3. [P1] Concurrent resume requests are not atomically reserved per root task. With the threaded daemon server, two requests can both pass the latest/active checks before either registers its attempt, creating branches. A root-level lock or atomic reservation would close the race.
  4. [P2] Task-history pagination and filters share mutable Pinia list state. A filtered or non-default history fetch can affect later no-argument fetchTasks() calls from other views, hiding active tasks and weakening duplicate-submission checks. Separate query state or explicit reset semantics would avoid the cross-view coupling.

Generated acceptance artifacts, local historical job.json updates, and tsconfig.tsbuildinfo are deliberately excluded from the commit.

Comment thread webui/server.py
host = request.headers.get("host")
if not host:
return False
return origin == f"{request.url.scheme}://{host}"

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Critical: Do not derive trusted origins from the request Host

Host is attacker-controlled. I reproduced this with Host: attacker.example:8000, Origin: http://attacker.example:8000, and Sec-Fetch-Site: same-origin; _origin_is_allowed() returns True. When the supported --host 0.0.0.0 mode exposes the WebUI, DNS rebinding can therefore bypass the mutation guard and reach task submission/resume/deletion, preprocessing, updates, and daemon shutdown.

Build the allowed public origins from trusted configuration and enforce a host allowlist (for example TrustedHostMiddleware); the current request's Host must not become a trust root.

Confidence: 95/100 · Impact: 90/100

Code

Comment thread scripts/daemon/manager.py
continue
if job.dataset_signature and data.get("dataset_signature") != job.dataset_signature:
continue
out.append((int(data.get("global_step", 0) or 0), path))

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High: Bind recovery state to the logical task that created it

These checks establish configuration/data compatibility, but not state ownership. Two independent jobs that reuse the same output_name with identical signatures are indistinguishable here, so the later job can resume the earlier job's optimizer/scheduler state and silently become a continuation of the wrong run.

Persist job_id/root_job_id in train_state.json and require an ownership match for modern states; legacy states can remain behind an explicit compatibility path.

Confidence: 95/100 · Impact: 78/100

Code

Comment thread scripts/daemon/manager.py
if source.kind == "command":
argv = list(source.argv)
argv += ["--resume", str(state_dir)]
job = self.submit_command(label=source.method, argv=argv, extra_env=source.extra_env,

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High: Reserve a resumed attempt atomically per root task

The latest/active checks above are not atomic with this registration. A two-thread reproduction calling resume_job() for the same source created two child jobs successfully, both with attempt_index == 1. Under the threaded HTTP server, duplicate clicks/requests can therefore branch one logical task and queue both continuations.

Hold a root-scoped lock or persist an in-progress reservation across the latest/active check, attempt-index allocation, and _register_and_queue() call.

Confidence: 100/100 · Impact: 72/100

Code

rolling_suffix = ROLLING_STATE_NAME.format(
default_if_none(args.output_name, DEFAULT_LAST_OUTPUT_NAME)
)
suffixes.append(rolling_suffix)

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High: Retire rolling state after a successful run

The rolling directory is always an auto-resume candidate, while normal completion only removes the epoch checkpoint state. For example, a successful 120-step run with the default cadence leaves a complete step-100 rolling state; a later fresh invocation with the same output name silently resumes from step 100 and skips most of the requested training.

Delete/retire the rolling state after success, or require an explicit continuation policy before considering it during a fresh invocation.

Confidence: 98/100 · Impact: 75/100

Code

if (options.state !== undefined) taskFilter.value = options.state
if (options.page !== undefined) page.value = Math.max(1, options.page)
if (options.pageSize !== undefined) pageSize.value = Math.max(1, options.pageSize)
const params = new URLSearchParams({

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium: Do not reuse the history view's page/filter for global task polling

A no-argument fetchTasks() now reads the mutable global taskFilter, page, and pageSize. The dashboard, config, preprocess, adapter, merge, and other views all call it without options, so after the Task Monitor selects success or a later page, those views keep polling only that historical slice. Active training can disappear from the dashboard and duplicate-submission checks can miss it.

Keep history query state local to TaskMonitorView, or make no-argument calls explicitly mean all + page 1 and pass pagination only from the history view.

Confidence: 95/100 · Impact: 55/100

Code

@LingyeSoul
LingyeSoul merged commit 548d800 into LingyeSoul:main Aug 12, 2026
10 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants