Request action and full_text_xml_url; rank proposals by length

[abigailhaddad]

Three lines, two of which are the field list. Against v0.8.0 (a379c2b).

find_documents doesn't request action or full_text_xml_url

_pick reads d.get("action"), and load() reads final.get("full_text_xml_url"), but neither field is in the fields[] list, so both are absent from every document the live pipeline sees. Two consequences:

• every action test in _pick is inert — _NOT_OPERATIVE, _INTERIM and _SUPPLEMENTAL all search "", so nothing is ever skipped and selection falls through to the ordering rules
• xml_text is always None, so parse_structural never runs from load() and the parse always takes the regex path

load() is the only caller that passes XML, and it is what both __main__.py:130 and server.py:235 invoke, so this is the whole live path. The stress checks pass because they hand _pick document dicts that carry action.

On the live path:

_pick('Proposed Rule') -> chose 2024-03460 (1pp, no action text)
_pick('Rule')          -> chose 2024-22905 (146pp, no action text)
load(): {'convention': 'plain', 'structural': None, 'pairs': 112}

Both fields are available on the same list endpoint, so this is just the two names.

Proposals were ranked by date before length

With action restored, DOD-2023-OS-0063 still selects the wrong proposal. The docket carries a second document typed Proposed Rule — 2024-03460, one page, "Notification of posting of informational video on CMMC." — published two months after the NPRM. No _NOT_OPERATIVE keyword covers it, and max(pool, key=(publication_date, pages)) takes the later one.

That regressed textdiff on the validation docket, which diffed a one-page notice against the final rule:

0 sections in the proposal, 24 in the final rule
error: no section headings matched in one of the documents

Ranking by length first, as the Rule branch already does, fixes it. The supplemental preference still runs first, so EPA is unaffected.

Length is a better backstop than the keyword list because the keyword list can't be completed. In 400 rule and proposed-rule documents published since January 2025 there are 97 distinct action strings, and the ancillary ones take forms no fixed list anticipates — Notification of petition., Notification of availability., Proposed rule; public hearing., Notification of enforcement discretion. Two near-misses in the current pattern, left alone here since length ranking covers them: Final rule; technical amendments. doesn't match technical amendment because \b fails before the plural, and Notice of proposed rulemaking; extension of public comment period. doesn't match extension of (the )?comment period because of the intervening word. The EPA stress fixture carries that same construction, and _pick does not skip it — the "comment-period extension is not treated as a proposal" check passes because the supplemental preference selects 2022-24675 before the extension is ever compared.

After

DOD-2023-OS-0063: NPRM=2023-27280 (81pp)   final=2024-22905
EPA-HQ-OAR-2021-0317: NPRM=2022-24675 (146pp)  final=2024-00366

textdiff.run('DOD-2023-OS-0063'): sections=24 changed=24
load('DOD-2023-OS-0063'): convention='structural' pairs=112
  structural: {'paragraphs': 2079, 'comment_labels': 114, 'usable': True}

Structural parsing, once reachable, does what #5 hoped. Against the XML label counts on five documents, both paths as of a379c2b:

document     mode         true  found  recall   starts inside a Response
2026-13420   plain          71     69     97%          0/69     SSA
2026-13420   structural     71     69     97%          0/69
2024-22905   plain         114    112     98%          0/112    DoD CMMC
2024-22905   structural    114    112     98%          0/112
2026-14439   plain         214    213    100%          0/213    DHS
2026-14439   structural    214    213    100%          0/213
2026-12069   plain         224    219     98%          0/219    CMS
2026-12069   structural    224    219     98%          0/219
2026-14633   numbered       25     25    100%          0/25     FWS
2026-14633   structural     25     25    100%          0/25

SSA 2026-13420 was 37% recall with 17 of 26 stored pairs beginning inside a Response before v0.8.0. The anchored regex alone accounts for that recovery; structural matches it and is equal or slightly better on exact label-boundary matches (112/112 against 111/112 on CMMC, 219/219 against 216/219 on CMS). Those last differences are small enough that they may be an artifact of comparing flattened text against XML paragraph boundaries rather than a defect in either path — the recall and in-Response columns are the ones that carry weight.

Tests

Three checks added to the existing block. They fail on a379c2b and pass here:

FAIL  find_documents requests the action field
FAIL  find_documents requests the XML url, without which structural parsing never runs
FAIL  a later one-page notice does not displace the NPRM  — chose 2024-03460

The first two capture the params find_documents sends rather than asserting on the source, so the field list can't drift back without failing. The third is the real CMMC docket with the action text stripped, which is how find_documents currently returns it.

Full suite otherwise unchanged — baseline 75 pass / 4 fail, patched 78 pass / 4 fail, no check that passed on a379c2b changed state. The four failures are flask missing in my environment, identical in both runs.