Repository navigation
Feat/stig upload tus sse - #26
Merged
JeremyDGreene merged 16 commits intoOct 27, 2026
Merged
Conversation
Replace the single-request STIG library upload with a resumable tus upload plus out-of-band processing, so large compilations survive nginx/HTTP2 request cutoffs. A tus server accepts the upload, an ImportJob row tracks it, and processing runs detached from the request. Progress streams back over SSE (replayed on reconnect), and a startup sweep marks jobs interrupted by a restart as errored. Removes the old synchronous /api/stigLibrary/upload endpoint. Signed-off-by: Mark Rivera <mcrivera@gmail.com>
Stop trusting the compilation filename. Derive classification from the inner per-STIG zip prefixes (highest of U/FOUO/CUI wins) and the library date from the newest benchmark date, reject collisions on classification and date, and skip non-STIG XMLs and unmarked packages instead of failing. The filename column now stores the real uploaded name (no longer unique, now TEXT), and an audit log records who imported which library. The library table shows classification, library date, STIG count, and file, sorted by date. The boundary STIG baseline selector shows classification and date, and the upload progress shows an ETA. Signed-off-by: Mark Rivera <mcrivera@gmail.com>
Widen StigData.severity to the full XCCDF Rule/@Severity domain (unknown|info|low|medium|high), default unknown, so SRG rules marked "info" no longer fail the import. Make benchmark release-info parsing null-safe and handle full month names ("27 July 2018"), and skip non-STIG XMLs (no Benchmark element) instead of erroring. Log concise Sequelize errors (name/message/column) instead of the full SQL and bind parameters, with an over-255-char field diagnostic, and drop the per-STIG success/console noise. Signed-off-by: Mark Rivera <mcrivera@gmail.com>
Integrate Kevin Jablonski's migration-fix patch onto the tus/SSE upload rework. checkBoundary() + checkStigLibrary endpoint surface STIGs missing from the target baseline; migrateBoundary gets null-safety and seeds an Evaluation per migrated STIG; EvaluationItem columns become nullable with FK associations; a warnings modal lets users confirm lossy migrations. Signed-off-by: Mark Rivera <mcrivera@gmail.com>
Signed-off-by: Mark Rivera <mcrivera@gmail.com>
…artbeat One import at a time across all users: job creation claims the lock via an in-process chain frees the lock within minutes, and stale jobs are removed on lookup as well as on startup. A blocked import gets a 409 naming the file that holds the lock, surfaced as a notification in the UI. Signed-off-by: Mark Rivera <mcrivera@gmail.com>
…self-detection Detection was classification + newest stig date, which cannot tell a Q3 from a Q3v2 whose only changes doesn't touch the newest date. Here a legitimate re-release would be rejected. Drop the date-based rejection. Byte identical re-uploads still hit the zip-hash UNIQUE (naming the existing library), and anything else imports. A duplicate that differs only in packaging self-identifies instead of being blocked. Same-date siblings get revision labels ordered by per STIG content dominance (date, then version/release), with average benchmark date as the tiebreaker. The newer set wins the higher revision. Ordering is dynamically derived regardless of upload order and recomputed on every change. So uploading Q3 after Q3v2 retroactively relabels Q3v2 as rev 2. Libraries are rev 1 from the start. Labels are display-only. Admins pin a label or reset it to automatic. DELETE /api/stigLibrary/:id removes a library, detaching shared STIGs and deleting exclusive ones plus their orphaned rules. Refuses while a boundary is using the library or assessments/evaluations reference its exclusive STIGs. Import/delete exclusion is symmetric through the import lock: deletion registers in the same in-process claim chain stiglibrary uploads uses, so neither can start while the other runs. Per-STIG import failures now surface to the UI. processLibrary tracks failed imports UI: revision column and pencil/trash icon actions in the library table; the STIG baseline dropdown includes the revision label so same-date siblings are distinguishable. Also fixes parseXmlStig returning the wrong shape, which zeroed the new/updated/unchanged counters in every import summary. Signed-off-by: Mark Rivera <mcrivera@gmail.com>
Signed-off-by: Mark Rivera <mcrivera@gmail.com>
…ialog Signed-off-by: Mark Rivera <mcrivera@gmail.com>
Signed-off-by: Mark Rivera <mcrivera@gmail.com>
Sort the added and removed STIG ids in diffBenchmarks with localeCompare to match the changed list, instead of the default code-unit sort (S2871). Give highestClassification an initial value of U, the lowest rank, so reduce cannot throw on an empty list (S6959). Signed-off-by: Mark Rivera <mcrivera@gmail.com>
Use node: specifiers for fs and path, Number.isFinite and Number.parseInt, Array.at for the last two ordered revisions, and RegExp.exec for the classification prefix. Drop a redundant continue, un-nest the StigData error log template, stringify non-Error throwables as JSON, re-export ImportJob directly, and remove a commented-out table cell left from the baseline migration patch. Signed-off-by: Mark Rivera <mcrivera@gmail.com>
Attempt to address the four code smells opened by sonarqube cloud on processLibrary, the library delete handler and migrateBoundary, and S6551 on errorSummary. - Move the role check, id validation and 404 lookup shared by the delete and label endpoints into requireLibraryManager (named for futurue role permission configuration.) - Split the delete handler into stigIdsInLibrary, assertLibraryUnused, deleteLibraryRows and removeWorkingFiles. Responses and the order of the 409 checks are unchanged. The STIG id sets are named libraryStigIds and exclusiveStigIds so they do not read as boundary membership. - Give checkBoundary and migrateBoundary one async generator, currentAssessments, that yields each current assessment with its old and new STIG. migrateAssessment and copyCheckResults hold the per assessment work. An assessment whose STIG row is missing is now skipped instead of failing the whole migration on an undefined where clause. - Pull saveLibraryRow, librarySaveError, importXmlFile, tallyParseResult and reportRevision out of processLibrary. - errorSummary returns strings as they are and JSON for any other non-Error value. - Drop the leftover console.log in checkBoundary. Signed-off-by: Mark Rivera <mcrivera@gmail.com>
The tus hooks trusted the jobId metadata from the client. Any authenticated user could start an import without a job row, which skipped the single-import lock, and could truncate or terminate another user's upload by reusing its id. Every tus request now resolves the authenticated user from the Node request, loads the ImportJob by upload id, and requires that it belongs to the caller and is still queued or uploading. A second POST for an existing upload gets a 409 instead of resetting the file, and the finish hook only starts the import when it can claim the job row. Errors that tus would otherwise send with their raw message are now logged and returned as a plain 500. Signed-off-by: Mark Rivera <mcrivera@gmail.com>
The uploaded compilation and its extraction were kept under the temp folder after every import, and abandoned or interrupted uploads left their partial files behind. Only library deletion removed anything. Add tempFiles.ts with a path guard, best-effort removal and a scoped temp directory helper, and importJobFiles.ts for the three paths an import job owns. The import runner, the stale-job path, the startup sweep and the tus finish hook now remove them, so the delete handler no longer needs to. Signed-off-by: Mark Rivera <mcrivera@gmail.com>
…ranch The branch carried the STIG baseline migration fixes alongside the upload rework: the pre-migration check endpoint, the warnings modal, the null-safe migrateBoundary, the evaluation seeding and the EvaluationItem model changes. None of that depends on the upload transport, so it now lives on fix/stig-baseline-migration. This restores main's versions of those files and keeps only the library label changes in CompanyBoundaries.vue. Signed-off-by: Mark Rivera <mcrivera@gmail.com>
|
Deployed d4c49c7 to https://pr-26.tirtest.com |
JeremyDGreene
merged commit Oct 27, 2026
f103629
into
fix/22-stig-library-import-right-after-first-boot-on-sqlite-fails-with-sqlite_busy
8 of 9 checks passed
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
Testing
Related