Repository navigation
feat: merge resumed training attempts into logical tasks - #9
Conversation
| host = request.headers.get("host") | ||
| if not host: | ||
| return False | ||
| return origin == f"{request.url.scheme}://{host}" |
There was a problem hiding this comment.
🔴 Critical: Do not derive trusted origins from the request Host
Host is attacker-controlled. I reproduced this with Host: attacker.example:8000, Origin: http://attacker.example:8000, and Sec-Fetch-Site: same-origin; _origin_is_allowed() returns True. When the supported --host 0.0.0.0 mode exposes the WebUI, DNS rebinding can therefore bypass the mutation guard and reach task submission/resume/deletion, preprocessing, updates, and daemon shutdown.
Build the allowed public origins from trusted configuration and enforce a host allowlist (for example TrustedHostMiddleware); the current request's Host must not become a trust root.
Confidence: 95/100 · Impact: 90/100
| continue | ||
| if job.dataset_signature and data.get("dataset_signature") != job.dataset_signature: | ||
| continue | ||
| out.append((int(data.get("global_step", 0) or 0), path)) |
There was a problem hiding this comment.
🟠 High: Bind recovery state to the logical task that created it
These checks establish configuration/data compatibility, but not state ownership. Two independent jobs that reuse the same output_name with identical signatures are indistinguishable here, so the later job can resume the earlier job's optimizer/scheduler state and silently become a continuation of the wrong run.
Persist job_id/root_job_id in train_state.json and require an ownership match for modern states; legacy states can remain behind an explicit compatibility path.
Confidence: 95/100 · Impact: 78/100
| if source.kind == "command": | ||
| argv = list(source.argv) | ||
| argv += ["--resume", str(state_dir)] | ||
| job = self.submit_command(label=source.method, argv=argv, extra_env=source.extra_env, |
There was a problem hiding this comment.
🟠 High: Reserve a resumed attempt atomically per root task
The latest/active checks above are not atomic with this registration. A two-thread reproduction calling resume_job() for the same source created two child jobs successfully, both with attempt_index == 1. Under the threaded HTTP server, duplicate clicks/requests can therefore branch one logical task and queue both continuations.
Hold a root-scoped lock or persist an in-progress reservation across the latest/active check, attempt-index allocation, and _register_and_queue() call.
Confidence: 100/100 · Impact: 72/100
| rolling_suffix = ROLLING_STATE_NAME.format( | ||
| default_if_none(args.output_name, DEFAULT_LAST_OUTPUT_NAME) | ||
| ) | ||
| suffixes.append(rolling_suffix) |
There was a problem hiding this comment.
🟠 High: Retire rolling state after a successful run
The rolling directory is always an auto-resume candidate, while normal completion only removes the epoch checkpoint state. For example, a successful 120-step run with the default cadence leaves a complete step-100 rolling state; a later fresh invocation with the same output name silently resumes from step 100 and skips most of the requested training.
Delete/retire the rolling state after success, or require an explicit continuation policy before considering it during a fresh invocation.
Confidence: 98/100 · Impact: 75/100
| if (options.state !== undefined) taskFilter.value = options.state | ||
| if (options.page !== undefined) page.value = Math.max(1, options.page) | ||
| if (options.pageSize !== undefined) pageSize.value = Math.max(1, options.pageSize) | ||
| const params = new URLSearchParams({ |
There was a problem hiding this comment.
🟡 Medium: Do not reuse the history view's page/filter for global task polling
A no-argument fetchTasks() now reads the mutable global taskFilter, page, and pageSize. The dashboard, config, preprocess, adapter, merge, and other views all call it without options, so after the Task Monitor selects success or a later page, those views keep polling only that historical slice. Active training can disappear from the dashboard and duplicate-submission checks can miss it.
Keep history query state local to TaskMonitorView, or make no-argument calls explicitly mean all + page 1 and pass pagination only from the history view.
Confidence: 95/100 · Impact: 55/100
Summary
root_job_id,parent_job_id, andattempt_index.@by matching both raw and filesystem-safe layouts while retaining marker and signature validation.Implementation
global_stepwith newer attempts taking precedence, concatenates logs with attempt boundaries, and adds attempt-aware sample access.Validation
131 passed, 3 warnings:tests/test_daemon.pytests/test_task_service_recovery.pytests/test_run_analyzer.pytests/test_preview.pytests/test_progress_sink.pytests/test_staged_resolution_plan.pynpm run buildpassed forwebui/frontend.git diff --check origin/main..HEADpassed.Known findings for maintainer decision
These findings are intentionally not fixed in this PR so the maintainers can choose the desired policy and implementation:
scripts/daemon/manager.pycan associate a compatible state with a later independent job because config/data signatures prove compatibility, not ownership. One option is to persist and validatejob_idorroot_job_idin the training state.library/training/checkpoints.pykeeps the rolling-statedirectory discoverable, so a later fresh run can silently auto-resume stale state, particularly when the target step is not aligned with the rolling save cadence. The state also retains large optimizer artifacts. Maintainers should decide whether completion deletes, retires, or requires explicit selection of rolling state.fetchTasks()calls from other views, hiding active tasks and weakening duplicate-submission checks. Separate query state or explicit reset semantics would avoid the cross-view coupling.Generated acceptance artifacts, local historical
job.jsonupdates, andtsconfig.tsbuildinfoare deliberately excluded from the commit.