Repository navigation
purge --check-policy ends in a traceback, with no JSON envelope, when --config or the audit log is not valid UTF-8 #1070
Description
Summary
forgelm purge --check-policy is documented to exit EXIT_CONFIG_ERROR (1) with a {"success": false, "error": ...} envelope when its --config cannot be loaded. A config file, or an audit_log.jsonl, that is not valid UTF-8 raises an uncaught UnicodeDecodeError instead: Python prints a traceback, stdout stays empty, and the exit status is 1 only because that is the interpreter's default for an uncaught exception.
Evidence
Reproduced on the head of #1068; that pull request does not touch either path.
forgelm purge --check-policy --config bad.yaml --output-dir out --output-format json, wherebad.yamlis valid YAML followed by a0xffbyte:UnicodeDecodeError, nothing on stdout, exit 1._load_retention_and_config(forgelm/cli/_config_load.py) runsyaml.safe_loadon a text-mode file and lets the error through, and_run_purge_check_policy(forgelm/cli/subcommands/_purge.py) catches only(ConfigError, OSError, yaml.YAMLError).- The same command with a valid retention-only config and an
out/audit_log.jsonlthat has a non-UTF-8 line:_build_audit_age_lookupcatchesOSErroronly, so reading the log raisesUnicodeDecodeErrorin the middle of the report. Same result: traceback, empty stdout, exit 1. - For contrast, a
--configpath that does not exist gives the documented envelope:{"success": false, "error": "--check-policy could not load --config: Configuration file not found: ..."}, exit 1.
Why it matters
--check-policy is meant for CI and scheduled jobs: the purge reference documents gating on jq -e '.success == true and has("count") and .count == 0' under set -o pipefail. A consumer that parses stdout gets an empty stream instead of success: false, and the failure reads as a crash in ForgeLM rather than as a bad input. The command is read-only, so no data is at risk.
The row-erasure path in the same file, which runs after the row is already gone, was hardened against the same error in #1068 (_scan_run_ids_from_granted_events, _maybe_load_config); that is how this gap was noticed.
Acceptance criteria
- A
--configthat is not valid UTF-8 makes--check-policyexitEXIT_CONFIG_ERRORwith thesuccess: falseenvelope under--output-format json, and an operator-actionable error in text mode, like any other unloadable config. - A non-UTF-8
audit_log.jsonldoes not abort the report. Triage decision: do not fail and do not drop the whole log. Read it withiter_audit_events(strict=False)(forgelm/cli/subcommands/_audit_log_reader.py), which decodes line by line, skips undecodable lines and logs one WARNING with the skip count. Intact lines still anchor the per-run and genesis ages; age frommtime(age_source=mtime) only when no usable event is left, as the report already does when the log is absent. State in the docs that when the skipped line carried the earliest timestamp, ages anchor on the next intact event. - Tests cover both inputs through
_run_purge_cmd(..., output_format="json")and assert the exit code and the envelope. -
docs/reference/purge_subcommand.mdand its TR mirror still describe the behaviour; update both if the chosen behaviour differs from what the exit-code table says.
Context
Found while fixing #102 in #1068. Related: #1054 (retention-only config for --check-policy).
Tracked finding IDs this overlaps (name them in the pull request so they can be ticked): the audit-log half is the unfixed remainder of CR-A10-040 (#555, _build_audit_age_lookup); the config half shares its root cause with CR-A01-094 (#661) and CR-A01-137 (#664), which describe load_config / _load_config_or_exit, not this loader.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsTodo
Activity
cemililik commentedon Oct 23, 2026
Triaged on
developmente6bf28d, after #1068 merged during this triage. Each finding was first checked on 83e62f6 and on #1068's head (ab5ce5f), whose code is identical to the merge.Accepted: bug, severity low, wave 4, under the W4 module-sweep epic #875.
Reproduced with a
0xffbyte, on 83e62f6, on #1068's head and again on the mergeddevelopment:--configthat is valid YAML followed by0xffUnicodeDecodeErrortracebackaudit_log.jsonlwith one non-UTF-8 line--configpath that does not exist{"success": false, "error": ...}So the defect is on
developmentand is not introduced by #1068 (it changed the loader's return shape, not its decoding).Why low: the command is read-only, exit 1 happens to be the documented config-error code, and the CI recipe fails closed (
printf '' | jq -e '.success == true'exits 4), so there is no false pass. What is missing is the envelope and an actionable message.Overlap with tracked findings (so the ticks are not lost)
_scan_run_ids_from_granted_events;_build_audit_age_lookupstill reads in text mode and aborts at the first undecodable line.load_configand_load_config_or_exit. This issue is the--check-policyloader, a separate code path. A fix at the shared level would resolve those too; if so, name them in the PR.Decision on acceptance criterion 2: neither fail nor fall back to mtime for the whole log. Read the log with
iter_audit_events(strict=False), which decodes per line, skips undecodable lines and logs one WARNING with the skip count. The lookup only needs the first timestamp per run and the genesis, so intact lines still anchor the ages; mtime stays the fallback when no usable event is left. Documented limitation: if the skipped line carried the earliest timestamp, ages anchor on the next intact event. The body is updated.Implementation notes
UnicodeDecodeErrortoEXIT_CONFIG_ERRORtranslation can live inforgelm/cli/_config_load.py, which has no size budget;config.pyand_purge.pyboth have 0 LOC of headroom (see purge --check-policy: add a retention horizon forcheckpoint-*directories #1069 for the budget note).docs/reference/purge_subcommand.mdalready lists a "missing / unreadable" config as exit 1, so the EN and TR pages need a change only for the audit-log WARNING behaviour.Not open for contribution yet: this needs a maintainer decision first. If you would like to work on it, please comment with your proposed approach before opening a pull request.