Repository navigation
Description
Problem — conversation content is written to logs today
JudgeClassifier::verdict logs every unavailable verdict with the error's raw Display:
tracing::warn!(target: "libsy", judge_model, error = %error, "judge verdict unavailable; ...");That is not safe here, and the leak is concrete rather than theoretical:
LibsyError::ClientCallrenders as"client call to target {target:?} failed: {source}";- its source
LlmClientError::UpstreamHttprenders as"upstream returned HTTP {status}: {body}".
body is the upstream error body verbatim. Providers routinely quote the offending request back
in it, so any 4xx from a judge target writes conversation text into the operator's logs. Boxed
transport, decode, and FFI sources can carry the same.
This runs on a per-turn path, in the component most likely to be pointed at a self-hosted endpoint
with unfamiliar error formats.
Proposal
A robustness module with safe_error_summary / safe_client_error, used at the judge's warn site.
Each case contributes only vetted operational detail — the failure's class, the target or model it
concerns, and a status code where one exists:
| Error | Logged as |
|---|---|
UpstreamHttp { status, body } |
upstream HTTP 400 — body dropped |
ContextWindowExceeded { model, message } |
context window exceeded for model weak — message dropped |
Transport / InvalidResponse / Timeout / Ffi |
class only; boxed source dropped |
External { operation, source } |
{operation} failed — static label kept, source dropped |
AlgorithmError { message } |
verbatim — libsy builds it from static text |
libsy's own AlgorithmError is reported in full deliberately: it is built from a static string plus a
serde position and is the one message an operator needs to diagnose a bad judge reply. This mirrors
the safe_types idea from the Python implementation this is ported from.
Design note — the exhaustive match earned its keep
The match over LibsyError is deliberately exhaustive rather than _ => error.to_string(), so a new
variant cannot start leaking by omission. That paid off immediately: the compiler caught
External { operation, source }, which pairs a safe static label with an unvetted boxed source from a
user extension, and which a catch-all would have silently rendered in full.
Test approach
Each test asserts both that the summary omits the sensitive text and that the error's
Display still contains it — so the tests fail if the redaction is ever bypassed, not merely if it
changes shape.
Scope
3 files, +177/−8. New crates/libsy/src/algorithms/util/robustness.rs (with 6 tests), its module
declaration, and 20 lines at the judge's log site. No public API change, no config surface.
Why this is its own PR
Independent of the other two classifier changes: it is a data-hygiene defect on the logging path,
with no overlap in code path or surface. It is also the most clearly severity-driven of the three,
so it should not be queued behind a prompt-tuning discussion or a breaking-config debate.
Validation
| Gate | Result |
|---|---|
cargo test --workspace |
646 passed (baseline 640 + 6 new) |
cargo clippy --workspace --all-targets -- -D warnings |
clean |
cargo fmt --check |
clean |
uv run pytest tests/ |
unchanged from baseline |
Follow-up worth raising separately
Only the judge path is fixed here. Other %error log sites in the workspace may have the same
exposure and are worth an audit — deliberately out of scope so this stays reviewable.
Activity
gburachas commentedon Aug 11, 2026
Refreshing this, because two things have landed nearby that look like they close it and do not.
#205 added
report_fail_open(judge_model, error, reason)with a boundedreasontaxonomy and arecord_classifier_fail_openmetric. That fixed the label cardinality problem. #271 redacts base-url userinfo and header values fromDebug, and caps retained error bodies at 64 KB.The leak this issue is about survives both.
report_fail_openstill does:and
LlmClientError::UpstreamHttp'sDisplayis#[error("upstream returned HTTP {status}: {body}")]— the raw upstream body, interpolated verbatim into a warning that fires on every turn whose judge is unavailable. Providers routinely echo the offending request back in a 400, and the judge's request is a condensed view of the user's conversation. A 64 KB cap bounds how much leaks, not whether it does.The fix keeps everything #205 built and changes one parameter:
report_fail_opentakes an already-redacted summary rather than&dyn Display, produced by an exhaustive match overLibsyError/LlmClientErrorthat reports each failure's class, the target or model it concerns, and a status code — never content sourced from outside the process.Two properties I would want preserved in review:
to_string(), so a new error variant cannot start leaking by omission. Resolving a future compile error there with a_ =>arm would silently discard the whole guarantee.error.to_string()still contains it. The second assertion is not redundant — it is what documents that the hazard is still live inDisplay, and it will fail loudly if anyone "fixes" the redaction by changingDisplayinstead.Rebased onto current
mainand green.