Skip to content

Preserve projects and scale review for large collections - #7

Merged
Yariminal merged 15 commits into
masterfrom
codex/scalable-organization
Sep 29, 2026
Merged

Yariminal merged 15 commits into
masterfrom
codex/scalable-organization

Conversation

@Yariminal

@Yariminal Yariminal commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner

Mixed collections need to preserve project structure, expose unsupported files honestly, and keep review and execution scoped to the intended session. This change adds those safeguards and improves extraction, provider reliability, and progress reporting for local organization runs.

  • Preserve linked resources, named project folders, and numbered sequences; organize supported loose files by evidence. MOVE retains its filename and cannot implicitly apply an unselected RENAME.
  • Reconcile physical inventory, explicit exclusions, and analysis outcomes with a streaming read-only ledger. Read Word tables and signature-checked CAD log/shape metadata without claiming opaque CAD resources are understood.
  • Avoid unsupported filename date prefixes. Recover bounded model JSON key-separator errors while rejecting duplicate keys and incomplete outer-container fragments; required-field/semantic response validation remains follow-up work.
  • Scope review queues, saved snapshots, and late responses to their session. Require paired non-exact comparison and a separate approval action.
  • Show relation progress with unknown totals and retained heartbeat counts. Omit unsupported SHX from semantic prompts while retaining its indexed records and structural handling. Numeric progress cannot end a job; cancellation and generator close retain the worker lease until the worker exits.

Validation:

  • Latest code revision 2801240: 451 Python tests passed, one Windows platform-dependent skip; all six Node UI state checks and JavaScript syntax passed. Independent review found no remaining blocker in the final focused fixes. All ten CI checks passed at final head 33324de (nine OS/Python combinations plus UI state).
  • Full, no-cache combined run at b0212d5: 1,337 physical files / 4.43 GB; 1,330 indexed plus seven explicit OS metadata exclusions; 720 content-verified, 45 context-only, 12 metadata-only, 553 unsupported; zero errors/unprocessed. Runtime 35.6 minutes on local gemma4:26b / RTX 3090. Physical paths, bytes, directories and all six source ZIP hashes reconciled.
  • Separate copied-database follow-ups: four CAD files became metadata-only with zero model calls; both unsafe project MOVE proposals were removed and could not survive regeneration. These narrower results are not a fresh full run of the latest revision.
  • Selected disposable operations passed dry run, commit and undo: six moves in the constructed document/project replay and one reviewed rename in the combined copy. Original paths/bytes were restored; approval history intentionally remains.

Follow-up acceptance gates: actual visual/keyboard acceptance is blocked by saved browser permission and a rejected live-server restart. The live UI remains on the prior build. Current evidence does not establish useful naming accuracy, non-exact interchangeability, consuming-app compatibility, or physical 100–500 GB readiness. No bulk disposal was approved.

See validation evidence and next verification plan.

Follow-up PR plan

This PR is a scoped safety and validation increment. The following acceptance work is separate from merging it.

Order Proposed PR Completion criteria
1 Validate typed analysis responses Required fields and types validated; malformed, empty, duplicate, missing and wrong-type response regressions cover retry exhaustion and honest no-evidence outcomes; document the trailing-text policy.
2 Complete visual review acceptance Desktop, narrow layouts and keyboard flows verified; actual keeper/candidate media and provenance shown; session switches and stale requests remain isolated; saved snapshots stay read-only. Fix observed defects after browser access is restored.
3 Measure and improve organization quality Freeze independent labels and reviewer criteria before tuning a new mixed corpus; report useful, supported, redundant and unsafe suggestions with denominators; no invented event dates or protected-project splits; selected operations pass dry run, commit and undo.
4 Calibrate duplicate review and capability reporting Labeled keeper-relative positives, negatives, derivatives and sequences establish measured retrieval quality; examined/deferred counts remain visible; publish honest content/metadata/missing-dependency/unsupported capability states. No automatic non-exact disposal.
5 Validate a physically backed 100 GB collection Review the streaming harness and provision source/copy/headroom storage; complete a read-only pipeline run with full inventory/hash reconciliation and measured throughput, RAM/VRAM, temporary, WAL and undo-storage budgets.
6 Qualify recovery and stress behavior before 500 GB Exercise dense 100,000-file cases and interrupted filesystem/DB/journal transitions on disposable copies; verify exact bytes, paths, topology, references and foreign keys after recovery, with latency/memory/deferred-work measurements. Attempt 500 GB only after these checks and sufficient storage.

PR 1 should precede reliance on new model-quality measurements. PR 2 can proceed in parallel when browser control is available. Broad product acceptance requires the UI and quality gates; large write operations require recovery qualification. The current 4.43 GB evidence is not physical 100–500 GB qualification.

@Yariminal
Yariminal marked this pull request as ready for review September 29, 2026 05:20
@Yariminal
Yariminal merged commit 56a4cb8 into master Sep 29, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant