Skip to content

build(worker): pypdf floor tracks only advisories; pin lags two SEC fixes #187

Closed

Description

@waterbro-8

Summary

The Worker's pypdf pin is set by the advisory stream, and the advisory stream
cannot see the two newest upstream security fixes.

worker/pyproject.toml:31 is pypdf>=4.0 and worker/uv.lock pins 6.15.0,
so that is what a Worker build installs today. Three public advisories are
first-patched at 6.16.0 / 6.16.1 and #156 (Dependabot) moves the lock there.
Upstream has since shipped two more releases whose only Security (SEC) entry
is the same class of fix — bound an unbounded parser input — and neither has
a GHSA allocated
:

  • 6.17.0 (2026-09-04) — Limit value for Roman numerals (#4047)
  • 6.18.0 (2026-09-07) — Limit allowed length of indirect object tokens (#4055)

Measured just now against the advisory database:
GET /advisories?affects=pypdf returns 43 entries, all type: reviewed,
and the newest first_patched_version anywhere in that set is 6.16.1. The
most recently published one (GHSA-fc8x-2rww-xw9m, 2026-09-02) already patches
at 6.15.0.

So an advisory-driven floor — >=4.0 today, >=6.16.1 after #156 — covers
everything the database knows about, and no alert will ever schedule 6.17.0 or
6.18.0
. The surface those two harden is live here and it is an untrusted one:
worker/mem_worker/processors/pdf.py:79 runs PdfReader(BytesIO(file.data)) on
user-uploaded files.

All references are main @ 2986fe38175f54d99f15dd38a498708c6ecd88cd.

Reproduction (deterministic; run by the author of this issue)

With the tool CI pins — uv 0.9.27 (ci.yml:185-187), sanitized env, no
mirror/proxy variables — on a tarball of that exact main commit whose tree was
verified equal to commits/2986fe38175f (.commit.tree.sha
f447ca554377e85ba26090d59882b8d4f2b78731, obtained with
git init && git add -A && git write-tree):

  1. uv lock on the untouched tree → 0 diff lines against the committed
    worker/uv.lock. The pin is reproducible, so the numbers below are about the
    resolve, not about a stale tool.
  2. Edit only the constraint and run uv lock again → Resolved 90 packages,
    Updated pypdf v6.15.0 -> v6.18.0. uv resolves the newest release
    satisfying a floor, not the floor, which is why the manifest text is what
    carries the intent.
  3. uv lock --locked on the result → rc=0, so uv sync --frozen
    (ci.yml:193, with UV_FROZEN: "1" at :170) accepts it.
  4. Name/version pairs across both locks (85 packages each) → exactly one
    difference: pypdf 6.15.0 → 6.18.0. The remaining 68 of the 76 changed lock
    lines are uv's own marker renormalisation (python_full_version >= '3.15' and sys_platform == 'darwin' moving between torch 2.13.0 and 2.13.0+cpu,
    likewise torchvision 0.28.0 / +cpu, python_full_version narrowing
    dropping off numpy / scipy / tifffile, and the resolution-markers
    list reordering).
  5. Behaviour on the repository's own fixture — the single-page PDF built by
    worker/tests/test_processor_logic.py:419, extracted under Python 3.11 (the
    version the Worker leg uses, ci.yml:182) at 6.15.0 / 6.16.1 / 6.18.0:
    identical page_count, identical sha256(extracted_text)
    (46e7c072b2684841…, 214 chars), "1800 RMB" in text true on all three, and
    the malformed input raising the same PdfStreamError on all three.
  6. The one behavioural change 6.18.0 makes (DEP: Rework configuration value handling (#4044)) has no consumer here: grep over the whole tree for
    overwrite_configuration / apply_configuration /
    disable_legacy_handling / pypdf.constants → 0 hits; pypdf appears in
    worker/pyproject.toml, worker/mem_worker/processors/pdf.py, and
    scripts/seed_demo_data.sh only.

Per docs/maintainers/triage.md, E3 is for reproduction by someone other than
the author, so this issue carries evidence:e2-source even though the steps
above were executed. A non-author re-running them is the fastest way to raise
it.

Acceptance criteria

  • AC-001 worker/pyproject.toml declares a floor at or above the newest
    upstream release carrying a Security (SEC) entry — >=6.18.0 as of today —
    so a from-scratch resolve cannot land below it.
  • AC-002 worker/uv.lock pins that version and uv lock --locked passes
    with the CI-pinned uv, i.e. the Worker leg's uv sync --frozen accepts it.
  • AC-003 No other package's locked version moves. Measurable: the
    name/version pair set of both locks differs only for pypdf.
  • AC-004 The Worker test leg passes on the exact head, which is what
    executes the 6.18.0 extraction path (test_processor_logic.py:457).
  • AC-005 The dist hashes in the new lock match what the registry publishes
    for 6.18.0 (sdist ae58b7d93c22c169ffb02c3b06321c45c4f223b4916536568adb57d789d95d01,
    wheel 05b762b77bcb9dcb4a7c91fcf5dded585b25bee7269ab3d3001d7c55fa1b324b).

Non-goals

Activity

  1. added
    type:maintenanceMaintenance, tooling, refactoring, or repository work
    evidence:e2-sourceSource or log evidence identifies the likely cause
    area:workerPython processing worker
    dependenciesPull requests that update a dependency file
    on Sep 8, 2026
  2. added
    status:readyScope and acceptance criteria are ready for development
    and removed on Sep 10, 2026
  3. PeterGuy326 commented on Sep 10, 2026

    @PeterGuy326
    Collaborator

    Triage: readiness gate passed — applying status:ready

    Checked against every bullet of the readiness gate in docs/maintainers/triage.md:

    Gate requirement Verdict
    not a duplicate, belongs in this repo Pass. Distinct from #156, which is the Dependabot record for the three GHSA-patched releases; this one is about the two upstream SEC releases no advisory will ever schedule. Non-goal 2 states the separation explicitly.
    no sensitive/security details exposed publicly Pass. Non-goal 1: the 6.16.x advisories are already public and the 6.17.0/6.18.0 entries have no CVE or GHSA, so there is nothing to route through SECURITY.md.
    type, area, status, evidence labels accurate Pass. type:maintenance, area:worker, dependencies are right. evidence:e2-source is the strongest honest level and is being left as-is — the author executed the seven steps, but the policy reserves E3 for reproduction by someone other than the author. Confidence is not a substitute for that.
    bugs have a defensible severity N/A. type:maintenance; the policy does not normally apply severity to maintenance.
    problem / desired outcome clear Pass. The mechanism is stated precisely: an advisory-driven floor cannot express a pin that no advisory schedules, and uv resolves to the newest release satisfying a floor rather than to the floor, so the lockfile alone does not carry the intent.
    acceptance criteria objective and testable Pass. AC-001…AC-005 are each measurable without judgement: a floor value, uv lock --locked exit code, a name/version pair-set diff, the Worker test leg on the exact head, and two literal SHA-256 dist hashes.
    scope and explicit non-goals recorded Pass. Three non-goals, including the pre-existing darwin + Python >= 3.15 clip extra marker geometry, which is correctly excluded rather than silently absorbed.
    dependencies, compatibility constraints, material risks identified Pass. The 68-of-76 lock lines that are uv marker renormalisation rather than pypdf are called out as unavoidable churn with the reason, and DEP: Rework configuration value handling (#4044) — 6.18.0's only behavioural change — is traced to zero consumers in this tree.
    a validation approach exists Pass. Seven steps, each with the CI-pinned tool version (uv 0.9.27 at ci.yml:185-187), a sanitized env, and a tree equality proof against main @ 2986fe38.
    small enough for a reviewable PR Pass. #188 is 2 files, +39/-39.

    Nothing here needed status:waiting-feedback; no bullet is missing information.

    Note on the attack surface, since it is what makes this more than housekeeping: worker/mem_worker/processors/pdf.py:79 runs PdfReader(BytesIO(file.data)) on user-uploaded files, and both 6.17.0 (Limit value for Roman numerals) and 6.18.0 (Limit allowed length of indirect object tokens) bound an unbounded parser input on exactly that path.

    Effect on #188

    #188 named this label as the only thing standing between it and being review-ready, and asked that undrafting follow the label rather than precede it. That ordering is now satisfied, so #188 is being marked ready for review.

    Leaving #188's Refs #187 as written rather than rewriting it to Closes — the author chose that keyword and it is their PR body. The consequence is only that this issue will not auto-close on merge, so it will be closed by hand against AC-001…AC-005 once #188 lands.

    Evidence stays at E2. Raising it to E3 needs a non-author to re-run the seven steps, and to E4 needs a non-author to verify AC-001…AC-005 on the delivered outcome.

  4. PeterGuy326 commented on Sep 10, 2026

    @PeterGuy326
    Collaborator

    Closing as completed. Delivered by #188, merged as 55d09b203ba9 (ordinary squash, no admin bypass).

    Every acceptance criterion re-checked against main after the merge rather than taken from the PR body:

    AC Requirement Verified on main
    AC-001 worker/pyproject.toml declares a floor at or above the newest upstream SEC release Pass. worker/pyproject.toml:31 = "pypdf>=6.18.0". The floor is in the manifest, not only the lock, so a from-scratch resolve cannot land below it.
    AC-002 worker/uv.lock pins that version and uv lock --locked passes with the CI-pinned uv Pass. worker/uv.lock:1646-1648 = name = "pypdf" / version = "6.18.0" / registry pypi.org/simple. The Worker leg runs uv sync --frozen with UV_FROZEN: "1", so a self-inconsistent lock would have failed it.
    AC-003 No other package's locked version moves Pass on the author's measurement (85 name/version pairs each side, exactly one difference) and consistent with the merged diff of +39/-39 across 2 files. The other 68 lock lines are uv marker renormalisation, not version changes. I did not independently re-derive the pair sets.
    AC-004 The Worker test leg passes on the exact head Pass. Worker check conclusion success on merge SHA 55d09b203ba9. This is the leg that actually executes the 6.18.0 extraction path at test_processor_logic.py:457.
    AC-005 The dist hashes in the lock match what the registry publishes for 6.18.0 Pass. worker/uv.lock:1649 sdist sha256:ae58b7d93c22c169ffb02c3b06321c45c4f223b4916536568adb57d789d95d01 and :1651 wheel sha256:05b762b77bcb9dcb4a7c91fcf5dded585b25bee7269ab3d3001d7c55fa1b324b — both byte-identical to the values recorded in this issue.

    Outcome on the alert stream, which is the thing this issue existed to get right:

    #188 used Refs rather than Closes, so this did not auto-close; that is why it is being closed by hand.

    On the evidence label: this stays at evidence:e2-source and is not being raised. AC-001, AC-002, AC-004 and AC-005 above were verified by a non-author against the merged default branch, which is verification of the delivered outcome, but the seven reproduction steps in this issue were never independently re-run by a non-author, and docs/maintainers/triage.md reserves e3-reproduced for exactly that. Closing a delivered issue does not retroactively manufacture reproduction evidence.

    Non-goal 3 remains unaddressed and correctly so: the darwin + Python >= 3.15 unsatisfiable clip extra marker geometry is pre-existing on both sides of this bump and has no CI coverage. It needs its own issue if anyone wants it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:workerPython processing workerdependenciesPull requests that update a dependency fileevidence:e2-sourceSource or log evidence identifies the likely causestatus:readyScope and acceptance criteria are ready for developmenttype:maintenanceMaintenance, tooling, refactoring, or repository work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions