Cross-process lock on Windows: AuditLogger and the approval gate have no lock, so concurrent writers fork the audit chain #1030

Embeddable engine · EMB-03 · Kind: code + docs · Size M · Severity: medium · background: #1027

Summary

On Windows the cross-process lock around the audit log is a no-op. AuditLogger.log_event relies on that lock to re-read the chain head and append in one step, so two processes writing the same log at the same time fork the hash chain. forgelm verify-audit then reports a chain break and exits 6, the code operators are told to wire security alarms to. A benign concurrency defect becomes indistinguishable from tampering. The same no-op sits under the approval gate, so two concurrent approve / reject calls for one run can both succeed.

This issue replaces the three copies of the POSIX-only lock with one cross-platform helper and proves it with a multi-process test on all three platforms.

What exists today (2059b59)

• compliance.py#L140-L158: _flock_ex / _flock_un call fcntl.flock, and on ImportError (Windows) are empty functions. The comment says so: "on Windows there is NO cross-process lock … use a distinct output_dir per run."
• AuditLogger.log_event, compliance.py#L518-L611 opens the log a+b, takes the lock (L551), re-reads the chain head from disk under the lock (_read_chain_head, L264), appends one line, fsyncs and unlocks (L600). The docstring names the purpose: "two writers sharing the same log file cannot both append against a stale self._prev_hash (which would silently fork the chain)." Without a lock that is exactly what happens.
• cli/subcommands/_approve.py#L27-L46 is a second copy of the same helper, behind _approval_decision_lock (L52-L91). Its docstring describes the race it closes: two reviewers, or a retried CI script overlapping a human decision, "could otherwise both pass the 'no decision yet' check … leaving two contradictory terminal decisions for one run." On Windows that guarantee does not exist.
• The limitation is documented (library_api_reference.md#L82: "do not share an output_dir across concurrent processes on Windows"), but the workaround of one directory per run does not cover approve / reject / verify-* / purge against a directory a run is still writing, torch DDP ranks on Windows (No rank-0 gating: every distributed rank runs the whole post-train pipeline with its own random run_id, writing final_path, model_integrity.json, compliance files, audit events an… (CR-A07-018) #470), or a front-end that calls the CLI while a run is active.

Evidence: reproduction

Run against the published package (git archive 2059b59 forgelm), with spawn workers (the start method Windows uses). The "no-op" mode replaces the two helpers with empty lambdas, which is exactly what the except ImportError branch installs on Windows. This is an emulation on macOS, not a run on Windows: the repository has no Windows job at pull-request time (#962).

No line is lost; the chain forks. The log_event docstring also names interleaved lines from concurrent writers as a second risk the lock prevents; the emulation above did not produce one, but nothing on Windows rules it out.

import multiprocessing as mp, os, sys, tempfile
N_WORKERS, N_EVENTS = int(sys.argv[2]), int(sys.argv[3])

def worker(out_dir, idx, disable_lock):
    import forgelm.compliance as c
    if disable_lock:
        c._flock_ex = lambda f: None   # what the Windows branch installs
        c._flock_un = lambda f: None
    log = c.AuditLogger(out_dir, run_id=f"w{idx}")
    for i in range(N_EVENTS):
        log.log_event("repro.tick", worker=idx, i=i)

def main():
    disable = sys.argv[1] == "nolock"           # "locked" | "nolock"
    ctx = mp.get_context("spawn")
    with tempfile.TemporaryDirectory() as d:
        import forgelm.compliance as c
        c.AuditLogger(d, run_id="seed").log_event("repro.seed")
        ps = [ctx.Process(target=worker, args=(d, i, disable)) for i in range(N_WORKERS)]
        [p.start() for p in ps]; [p.join() for p in ps]
        res = c.verify_audit_log(os.path.join(d, "audit_log.jsonl"))
        print(sys.argv[1], res.valid, res.entries_count, res.first_invalid_index)

if __name__ == "__main__":
    main()

Windows semantics that shape the design (verified in the vendor documentation)

• msvcrt.locking(fd, mode, nbytes) locks nbytes from the current file position and may extend beyond end of file. LK_LOCK retries once a second and raises OSError after 10 attempts, so it is not an unbounded blocking lock; LK_NBLCK raises at once (Python docs).
• A Windows byte-range lock is mandatory: "locking a portion of a file for exclusive access denies all other processes both read and write access to the specified region" (LockFileEx). If we locked a region of the audit log itself, a reader such as verify-audit, a front-end tailing the log, or export_evidence_bundle (which walks the output directory and zips every file, compliance.py#L2364) could fail with a lock violation. POSIX flock is advisory and never has this effect.
• Locking a region beyond end of file is not an error, and the operating system releases locks when a process ends, although "the time it takes … depends upon available system resources".

Proposal

One helper, forgelm/_filelock.py (new module). A context manager exclusive_lock(path, timeout_s=…) used by AuditLogger, _approval_decision_lock and, when it lands, the pipeline lock proposed in #571 (CR-A04-038).

• POSIX: unchanged behaviour. fcntl.flock on the handle the caller already holds, exactly as today, so there is no regression risk on the platform that works.
• Windows: a sidecar lock file (<log>.lock) instead of the data file, with msvcrt.locking on one byte placed beyond end of file (the file stays empty) so no reader or zipfile read can ever overlap the locked region. Acquire with LK_NBLCK in a loop with a short sleep up to a deadline, instead of relying on LK_LOCK's fixed 10 × 1 s. Release in finally.
• Fail closed. If the deadline passes, raise (proposal: a dedicated LockTimeoutError(OSError)) and do not append. log_event already contract-promises that a write failure surfaces and leaves the chain un-advanced (L518-L611). An unlocked append is the defect being fixed.
• Reader rules. Readers (verify-audit, tailing front-ends) never take the lock. How verify_audit_log treats a last line that a writer has not finished yet has to be checked and, if it reports a truncation, fixed so a concurrent read is not mistaken for a crash (the writer-side "does not end with a newline" error is the right response to a crash only). Document the reader contract. This also answers CR-A07-125 in [Wave 4] Low/info — Module sweep: Compliance core: Annex IV, audit log, integrity manifest, provenance (part 3/5) #658: today readers do not take the writer's lock, so a live append can be reported as tampering (exit 6) or as an empty log (exit 1).
• The helper removes the duplicated block from compliance.py and _approve.py. compliance.py is at its no-growth budget (its split is Split forgelm/compliance.py into a sub-package (_audit_log, _annex_iv, _provenance, _gdpr) — deferred until Phase 16 S9 has landed #906), so the swap must leave it smaller, never larger.

Alternatives considered

Out of scope

NFS, SMB or FAT semantics beyond what the Windows API documents; distributed ranks (#470 owns rank-0 gating); changing the audit-log format or the hash chain.

Acceptance criteria

• forgelm/_filelock.py provides the helper; compliance.py and cli/subcommands/_approve.py import it and their private copies are deleted.
• POSIX behaviour is unchanged (existing tests pass untouched).
• Windows uses a sidecar lock on a byte beyond end of file, a bounded retry loop and a fail-closed timeout.
• Multi-process test (spawn, ≥ 4 processes × ≥ 40 events) asserts verify_audit_log(...).valid and the exact entry count; it runs on ubuntu-latest, macos-latest and windows-latest.
• A test holds the lock in a child process and asserts the timeout error, an unchanged chain and an unchanged _prev_hash.
• A Windows test reads the log and calls export_evidence_bundle while another process holds the lock.
• The reader contract is checked and tested: verify_audit_log on a log whose last line is still being written is either accepted or reported distinctly from a crash-truncated log.
• A concurrency test for forgelm approve / reject on one run_id proves that exactly one terminal decision is recorded on every platform.
• The sidecar file is accepted by every output-directory walker (evidence bundle, integrity and purge code) and does not appear as an unexpected artefact; the .approval.lock precedent is checked.
• A Windows job runs these tests at pull-request time (the minimum of Tests run on Windows and macOS only when a release tag is pushed #962, scoped to these test modules if the full matrix is too slow).
• Documentation updated in English and Turkish: the AuditLogger entry in the library API reference (and the "do not share an output_dir" note), the audit-log and approval guides, and a CHANGELOG Fixed entry; the known-limitation wording is removed everywhere it appears.

Sequencing and dependencies

Stage A of #1027: a new module plus a two-line import swap in files that Phase 16 steps own, so coordinate the compliance.py change with S9 (#895). The Windows pull-request job depends on, or can start as the first slice of, #962.

Related

#962 (Windows and macOS tests run only at release), #543 (CR-A07-038: the flock path is untested), #659 (CR-A07-139: Windows degradations are silent), #571 (CR-A04-038: no lock for concurrent pipeline invocations), #470 (no rank-0 gating), #895 (S9), #679 (CR-A10-063: the approval lock's Windows no-op is undisclosed — this issue implements the msvcrt alternative that finding names; CR-A10-062: approve / reject wait on .approval.lock with no timeout — decide in the same pull request whether the helper's deadline also applies on POSIX), #658 (CR-A07-125: readers do not take the lock; see Reader rules), #906 (the compliance.py split).

_{Raised on 2026-10-20 as part of the embeddable-engine decision. Code links point at 2059b59; line numbers may have moved. The reproduction ran against the extracted 2059b59 package on macOS with the lock helpers replaced by the Windows no-op. Tracking ID: EMB-03.}