Corpus-Driven Real-Paper Testing Guide

Use this guide for the next hardening phase:

Corpus-Driven Hardening Of Extraction, Normalization, Semantics, Footnotes, And References

The goal is to work through real papers in a reproducible order, identify the first artifact where structure goes wrong, and fix the earliest parser stage that owns the problem. The current priority is to keep the layout-aware bibliography and caption/footer footnote baselines stable while continuing failed-status and mixed-family review.

Before committing parser changes, update this guide with the current real-paper run, output directory, observed failures, and any changed interpretation of the review checklist. Treat the guide as part of commit hygiene, not as an optional post-commit note.

All paper paths below are relative to:

/Users/robert/Projects/Epiconnector/testpapers

Full current-corpus runs must also include the package example:

/Users/robert/Projects/Epiconnector/parseTable1/inst/extdata/NutritionEx.pdf

This is the only copy of NutritionEx.pdf; do not duplicate it under the external testpapers directory. The complete corpus therefore contains 28 PDFs: 27 external papers plus this package example. R can locate the installed copy with system.file("extdata", "NutritionEx.pdf", package = "parseTable1").

Current Reference Baseline

Current continuation-header checkpoint:

outputs/testpapers_batch_header_inheritance_recovered_20260714

Its physical comparison baseline is outputs/testpapers_batch_canonical_axis_validation_final_20260714. That historical checkpoint is superseded by the occupancy-only canonical- geometry checkpoint below.

The equal-count cell-bbox check rejects only printed Table 2 on PDF page 4 and printed Table 4 on PDF page 6 of GOLD BioAge and depression- Associations with mortality among depressed NHANES participants (2005–2018).pdf; occupancy restores both exactly to their clean committed cells. Printed Table 2 on PDF page 5 of mdpi-The Relationship Between a Mediterranean Diet and Frailty in Older Adults- NHANES 2007–2017.pdf remains six columns, rejecting an indented stub lane that appears on only 5 of 13 measured body lines. All five tables in cobaltpaper.pdf, PDF pages 3-5, exactly match commit 6db4bb1 in dimensions, text, and bboxes. Of the 83 table IDs shared with that commit, 46 are exact physical matches, up from 14 in the preceding retained checkpoint, and no table moves away from a prior exact match. Printed Table 1 on PDF pages 5-6 of Systemic inflammation markers and the prevalence of hypertension- A NHANES cross-sectional study.pdf remains 48 x 5 plus 14 x 5 and merges with matching headers as one 61-row, 5-column logical table.

The preceding incomplete/multilevel header-geometry checkpoint is outputs/testpapers_batch_header_structure_geometry_20260714. Its candidate reconstruction reduced the 23 cross-band header-run concerns to two. Those survivors were caption/table-ownership symptoms in cobaltpaper.pdf, PDF page 3; the current extraction restores the five real Cobalt tables. The current candidate corpus has no cross-band header-run or Cobalt header concern. Its only non-stub missing-header case was the eight blank local child labels in printed Table 2 (continued) on PDF page 13 of periodontis2.pdf. Only that table now inherits labels, with local blanks and parent-leaf provenance retained. It keeps its 22 x 11 physical grid and resolves with PDF page 12 as one 38 x 11 table. All other non-timestamp artifacts change only within periodontis2.pdf as a consequence of that integration.

Current numeric edge-furniture checkpoint:

outputs/testpapers_batch_numeric_edge_furniture_refined_20260713

All 28 parse commands completed. Forty-four printed page-number lines across five PDFs are newly recognized through top/bottom all-page or odd/even recurrence. No prior ignored region is removed, and all 91 previously extracted tables retain identical dimensions, cell text, and cell coordinates. The Systemic inflammation paper adds one 1-row, 5-column page-5 candidate with collapsed_grid_unrecovered; its page-6 continuation retains the existing no_variables_for_descriptive_table failure. These failures isolate the next rule-region and column-grid problems and are not furniture regressions.

Current accepted canonical-geometry checkpoint:

outputs/testpapers_batch_geometry_phase_h_closed_20260715

This run covers all 28 PDFs and 92 physical tables with no command failure. Positioned geometry and exact body occupancy establish every selected axis; token-start evidence is no longer part of the parser output or selection path. All physical IDs, pages, shapes, cells, and bboxes match outputs/testpapers_batch_gated_sparse_body_occupancy_first_edge_20260715.

The same run retains the accepted preliminary header-structure checkpoint. Eighty-nine tables have usable header rows; their 627 preliminary leaves match their 627 occupancy bands one-to-one. The non-target text table without header rows retains header_rows_missing and emits no leaves for its three bands. The artifact contains 109 partial-rule groups and 74 uniquely linked header markers: 66 leaf links and eight group links. Twenty intact header runs have words crossing occupancy-band boundaries, 13 non-stub bands lack attached header text, and 31 blank stub labels remain explicit for Phase F review.

Compared with the preceding Phase G run, six tables gain one font-qualified body band. Selected table-region rows and semantic data are unchanged; two competing-model diagnostics record the new separator count and propagate that text through normalized and resolved provenance.

Current accepted body-interval and provisional leaf-column checkpoint:

outputs/testpapers_batch_phase_f_font_space_gap_20260713

This run covers all 28 PDFs and 90 tables. Every table produces a maximal font-qualified exact-gap candidate grid with no occupancy diagnostics or leaf-candidate concerns. All bands are ordered and contiguous; all separators are exact zero-occupancy gaps at least two observed table-font spaces wide; and all attached unmerged rule endpoints lie inside their separator. Sixty-eight candidate counts agree with the extracted grid, while 22 disagreements remain diagnostic.

At the historical Phase F checkpoint, boundary evidence defined the provisional body before occupancy: eight tables used one supported body/footer model and 18 compared multiple intervals. That selection path has since been retired. The current unified footer work leaves boundary proposals as evidence and makes one bottom-of-table ownership decision in build_table_region().

Current body-occupancy checkpoint:

outputs/testpapers_batch_phase_d_complete_20260712

This run covers all 28 PDFs and 90 physical tables with no command failure. Every body_occupancy.json record has a valid raw line-by-bin matrix, aligned count and proportion vectors, zero diagnostics, and zero unlinked marker occurrences. The artifact is diagnostic and is not consumed downstream.

The accompanying furniture change adds one cluster relative to Phase C: the manuscript footer repeated on 16 of 18 pages in periodontis2.pdf. Exact block/line provenance removes that line without masking crossing rotated-table characters. On page 10, four footer-created rows disappear; the cleaned positioned grid exposes a separate 13-versus-14 column ambiguity in the stub, which is retained for occupancy-driven column work. No other physical table grid changes.

Current structural-boundary checkpoint:

outputs/testpapers_batch_phase_c_complete_20260712

This run covers all 28 current corpus PDFs and 90 tables with no failed status. Raw extraction is unchanged from the preceding boundary-proposal checkpoint apart from the relative source path recorded for inst/extdata/NutritionEx.pdf. The selected TableRegion boundaries now use the shared structural header detector, sparse body rows are protected from false footer cuts, direct row-spanning header leaves are preserved, and compatible continuation pages may inherit an omitted parent group row. The proposal artifact retains 54 diagnostic alternatives but no longer claims that a selected edge is unsupported merely because the compact proposal omitted it. Twenty ending-rule footer candidates now carry following text line IDs, canonical bounds, and span-derived dominant font styles. This remains a diagnostic-only change; all other artifacts are byte-identical apart from parse-quality timestamps. All 90 proposals also report credible rule geometry and a coherent positioned grid. No current table triggers the final Phase C fail-closed guard. If both facts are false in a future input, TableRegion and normalization preserve empty header/body bands rather than manufacturing a split.

Prior canonical table-local evidence checkpoint:

outputs/testpapers_batch_canonical_evidence_stage3_final_20260712

Prior reference run:

outputs/testpapers_batch_20260709_bib_region_mask

The prior refreshed baseline was outputs/testpapers_batch_20260709_bib_region_mask. It was produced after the PyMuPDF layout-aware text stream became the source of document order for sections and reference lists, and after table-local caption/footer note blocks began splitting footnote definitions from structured marker evidence, including raised superscript markers in PDF character geometry and confirmed footer-cell marker prefixes. Page text is ordered page, column, then y-position, with page furniture removed before bibliography and table extraction, and before cell text annotation or text-stream footer detection consume page geometry. The bibliography extractor reads numbered and hanging-indent reference lists through one positioned stream, supports references split across columns and pages, treats numbered offset labels as the same entry-start structure as unnumbered hanging-indent entries, and does not require bibliographies to be numbered. The purpose-built bibliography pass now owns the reference-region evidence used by table extraction: when it finds a bibliography, extraction receives bibliography entry source-line IDs and bboxes and removes bibliography-owned words/chars before table candidate construction; if it finds no bibliography, extraction does not run a separate raw-text References scan. Credible ruled table candidates can also be rebuilt from PyMuPDF word positions and stroked horizontal rules instead of using the PyMuPDF4LLM grid as the sole row/column structure. The current run also adds a pre-extraction paper_table_mentions.json pass that classifies Table N lines as caption candidates, continuation labels, or prose references from the page-furniture-filtered text stream. Text-position fallback now consumes that artifact and rejects numeric-anchor grids whose value region is mostly prose fragments. Explicit table extraction now uses shared PyMuPDF caption geometry directly: line-initial caption regions are bound one-to-one in the canonical orientation-group frame, and strong uncaptioned fragments can integrate with a following below-captioned fragment when the column-header schema matches. Caption-aligned multi-column pages can be split into per-column extraction bands before text-position candidate building, so side-by-side tables in the same page region are not interleaved into one full-page stream. The backend JSON grid is no longer an emitted extraction fallback: PyMuPDF4LLM may still provide a rough table box, but rows, columns, cell boxes, row bounds, and header geometry must come from positioned PyMuPDF words, characters, and rules. If positioned reconstruction cannot build a credible grid from a rough box, that backend candidate is not emitted as an extracted table. Ruled-table extraction and text-position fallback also share the same header-span repair: sparse upper-header word clusters become spanning group cells only when their start-column spans are ordered, non-overlapping, and cover multiple lower columns; dense or tight wrapped header clusters remain separate leaf columns. Leaf-header text runs are now grouped by small visual between-word spacing before assignment to body-derived column extents, so large gaps between adjacent leaf columns are not glued together by nearby value anchors. If geometry roles cover only an upper prefix of declared header rows, ColumnHeaderSchema may use the lower declared rows as the leaf band, but it does not promote rows with body-value evidence into leaf headers. Comma-containing leaf headers are split only when adjacent body value columns repeatedly show the corresponding structure: the left body cell ends with a comma and the right body cell is populated. This fixes estimate/p-value headers such as OR (95% CI), P-value without splitting unit labels such as Mean AL, mm or interval values that keep commas inside one body cell. The old low-quality page rescue, page-wide sideways text replacement, and model/estimate-specific word_positions_with_horizontal_rules refinement have been removed; the current corpus run keeps the same table count and footnote / bibliography baseline without those paths.

Older generated outputs/ runs should not be treated as current.

Current header/body and multicolumn-header inspection note:

docs/implementation/header_body_cleanup_inspection.md

Current bibliography summary:

PDFs: 27
parse command failures: 0
paper_bibliography:
  papers with bibliography entries: 27
  papers with empty bibliographies: 0
  papers with bibliography diagnostics: 0
  total bibliography entries: 1308
  numbered entries: 1197
  unnumbered entries: 111
  numbered bibliography papers: 25
  unnumbered bibliography papers: 2
  mixed numbering-style papers: 0

Current footnote summary:

PDFs: 27
parse command failures: 0
paper_footnotes:
  anchors: 464
  definitions: 108
  links: 464
  resolved links: 398
  inferred links: 0
  ambiguous links: 2
  unresolved links: 64
  math/unit anchors suppressed before footnote linking: 33
  subscript anchors suppressed before footnote linking: 7
  word-like subscript anchors suppressed before footnote linking: 0
  citation-like anchors suppressed before footnote linking: 0
  non-footnote symbol anchors suppressed before footnote linking: 2
  document line groups classified as table footers: 104
  extracted-table footer records: 10
  page-furniture filter stage: before_paper_document_footer_detection (27 papers)
extraction page-furniture mask:
  extracted tables with mask metadata: 76
  page words removed before extraction/refinement: 1134
  page chars removed before extraction/refinement: 8989
  explicit-grid rows removed by page-furniture mask: 0

Current papers with unresolved or ambiguous footnote links:

None in outputs/testpapers_batch_20260709_bib_region_mask.

Current extraction/status summary:

PDFs: 27
parse command failures: 0
extracted tables: 82
extraction geometry:
  hline_word_positions: 24
  collapsed_explicit_grid_word_positions: 2
  pymupdf_positioned_bbox_words: 35
  value_matrix_word_positions: 8
  text_position_column_geometry: 4
  rotated_word_positions_with_rules: 9
canonical extraction layer:
  pymupdf_positioned_geometry: 82
table_processing_status:
  ok: 21
  rescued: 53
  failed: 0
empty-grid invariant:
  extracted tables with all-empty rows: 0
  extracted tables with trailing all-empty columns: 0
  normalized tables with all-empty rows: 0
  normalized tables with trailing all-empty columns: 0
previous backend-grid survivor:
  periodontitis-p11-t0 is no longer emitted

Current table-mention summary:

paper_table_mentions:
  total mentions: 318
  prose_reference: 244
  caption_candidate: 66
  continuation_label: 8
  Table S listings under Supplementary Information: 4 prose_reference
caption candidates without same-page extracted tables:
  periodontis2 Table 1, Table 3, Table 4
  Uses of NHANES Biomarker Data for Chemical Risk Assessment- Trends, Challenges, and Opportunities Table 1
periodontis2 page 6:
  Table 4 shows... -> prose_reference, same_line_prose_verb_after
  ... is shown in / Table 5. At ... -> prose_reference, previous_line_prose_cue_before
  extracted page-6 table candidates: 0
periodontis2 post-reference tables:
  Table 2 and Table 5 are extracted after the bibliography because extraction no
  longer suppresses every page at or after the first bibliography page.
Supplementary Information listings:
  metabolic Table S2 -> prose_reference, supplementary_information_table_listing
  periodontitis Table S1/S2/S3 -> prose_reference, supplementary_information_table_listing
known weak bucket:
  line_initial_table_label without heading evidence can still include
  false positives such as Eke `Table 2 also shows...`; current policy is to
  monitor this bucket rather than add another rule until it causes extraction
  harm.

Current open structural issues:

No current open leaf-header extraction defect is known in the retained baseline. The prior hypertension-p5-t0 failure is resolved by grouping leaf-header text into small-gap visual runs before assigning those runs to body-derived column extents. Anchor-start grouping was rejected because it regressed MDPI frailty and Eke headers by gluing visually separated leaf columns.
papers_from_laha/Science-Advanaced-Planetary Health Diet and risk of mortality and chronic diseases- Results from US NHANES, UK Biobank, and a meta-analysis.pdf still needs a path-consistency audit for Table 1. The p2 -> p3 continuation is accepted and the grouped column-header schemas now match, but the fragments reach that result through different candidate paths: p2 enters through whole-page PyMuPDF text-position extraction, while p3 still starts from a PyMuPDF4LLM rough table box and then uses hline word-position refinement. This is not currently a semantic failure, but it remains unresolved extraction-path debt because two fragments of one visual table should ideally follow the same canonical positioned-geometry path.

Resolved since the prior baseline:

cardiovascular Table 1 page 4 now resolves 20 double-dagger body-cell anchors to the local ‡ table footer definition.
Association between anthropometric indices and chronic kidney disease now resolves its 2 dagger links and converts 160 formerly conventional inferred p-value-star links into explicit same-table footer links. During the marker-evidence cleanup, this paper briefly regressed to 29 ambiguous Table 1 * links because the footer parser found the raised † marker and dropped the ordinary upright * p < 0.05 marker in the same block. The current parser merges structured marker evidence with ordinary symbol marker evidence before splitting one footer block.
Science-Advanaced-Planetary Health Diet and risk of mortality and chronic diseases now resolves its Table 1 *, †, ‡, and § links from the complete footer block on the continued page. CO₂, I², P_Begg, and P_Egger are suppressed as notation or word-like subscripts rather than unresolved footnotes.
mdpi-The Relationship Between a Mediterranean Diet and Frailty in Older Adults no longer has unresolved or ambiguous footnote links in the current baseline.
Early page-furniture masking, trailing-trim retirement, footer-block detection, and marker-evidence cleanup reduce the current full-corpus footnote issue count to 0 unresolved / 0 ambiguous links.
Eke drops from 529 unresolved / 11 ambiguous links to 0 unresolved / 0 ambiguous links in the current full-corpus baseline.
Rotated Ethnic table annotation cleanup and bibliography/citation suppression resolved the prior Ethnic false-marker issue. Caption/footer footnote parsing now resolves the remaining metabolic, Systemic inflammation, Planetary Health, and Ethnic Differences caption/footer markers through general table-local marker-definition parsing rather than paper-specific link patches.
OR/CI estimate routing, matrix-like table status guards, front-matter/prose suppression, and backend-grid survival removal keep current table-processing failures at 0 in the retained baseline.
Asthma NHANES below-table captions are now attached to the correct visual tables: p5-t0 carries Table 1, p5-t1 carries Table 2, and p6-t0 carries Table 3. The strong uncaptained p4-t0 fragment integrates with p5-t0 as resolved Table 1 after column-schema matching, and the prior false Table 1 continuation candidate involving p5-t1/p6-t0 is gone.
The caption-contaminated backend-row-drop path has been retired. The same class of issue should now be solved by positioned PyMuPDF reconstruction and TableRegion ownership, not by mutating backend rows.
Shared PyMuPDF word/rule header-span repair improves real multicolumn headers by splitting collapsed upper group labels in PRISm, PAD, gallstones, MDPI frailty, Lead, Cobalt, periodontitis, and Planetary Health. The current header-grid rule uses word start columns rather than glyph right edges when deciding whether an upper header cluster spans multiple lower columns, which preserves tight wrapped leaf headers such as periodontitis page 6 while allowing Planetary Health p2-t0 to expose the same grouped schema as p3-t0.
Small-gap visual-run grouping for leaf headers resolves the remaining header-leaf issues from the prior retained baseline without changing table counts or processing statuses. hypertension-p5-t0 now has the expected Healthy diet ... / Unhealthy diet ... leaf columns, cardiovascular p5-t0 uses All-cause mortality and Cardiovascular mortality as leaves with the AUC text as upper groups, Systemic inflammation p5/p6 no longer splits the hypertension cohort labels across leaves, and EWAS p6-t0 keeps OR (95% CI) together. The full run has 27 parsed PDFs, 82 extracted tables, 0 backend JSON grid survivors, 442 resolved / 0 unresolved footnote links, and 1308 bibliography entries with 0 empty bibliographies. The current empty-grid cleanup check found 0 all-empty extracted or normalized rows and 0 trailing all-empty extracted or normalized columns.
The METS-IR/hypertension paper now keeps Reference as ordinary table-body text in Table 2 while extracting the real 33-entry bibliography from pages 9 and later. Its page 7 side-by-side Tables 3 and 4 are split through caption-aligned column-band extraction instead of one interleaved full-page text stream.
Association between metabolic score for insulin resistance (METS-IR) and hypertension- a cross-sectional study based on NHANES 2007-2018 now exposes the estimate/p-value column pairs in Tables 2, 3, and 4 through ColumnHeaderSchema: Table 2 has Model 1/Model 2/Model 3 groups over OR (95% CI) and P-value leaves, Table 3 has Adjusted over those two leaves, and Table 4 has separate OR (95% CI), P-value, and P-interaction leaves. This is driven by adjacent body cells where the estimate cell ends with a comma and the p-value cell is populated.

Review Loop

For each checklist item:

Inspect the current artifacts under the reference baseline first.
Identify the first bad artifact in parser order: extracted_tables.json, normalized_tables.json, column_header_schemas.json, resolved_tables.json, table_definitions.json, parsed_cell_values.json, parsed_tables.json, paper_footnotes.json, paper_bibliography.json, paper_table_mentions.json, paper_style_profile.json, paper_references.json.
Record whether the issue belongs to extraction, normalization, continuation, table semantics, footnote detection/linking, or visual-reference resolution.
Fix only the earliest responsible stage.
Add focused regression coverage only for a known failure or stable artifact contract.
Re-run the reviewed chunk, then re-run the full corpus after parser behavior changes.

Ordered Review Checklist

C0. Footnotes And References First

Acceptance for C0: footnote anchors are separated into true table-note markers, bibliographic citations, statistical-significance markers, and false positives. Reference resolution should clearly distinguish unresolved in-paper tables from supplement-only or out-of-scope references.

C1. Actual Failed Table Statuses

C1.1 Review failed statuses that may be correct non-target tables.
1. papers_from_laha/An environment-wide association study (EWAS) on type 2 diabetes mellitus.pdf
  - An environment-wide association study (EWAS) on type 2 diabetes mellitus-p6-t0
  - Current status in outputs/testpapers_batch_20260709_bib_region_mask: ok, categorized as analysis_outputs.
  - Review result: structurally plausible ENWAS analysis-output table with sparse left descriptor columns; not a Table 1 descriptive failure.
2. papers_from_laha/mdpi-The Relationship Between a Mediterranean Diet and Frailty in Older Adults- NHANES 2007–2017.pdf
  - mdpi-The Relationship Between a Mediterranean Diet and Frailty in Older Adults- NHANES 2007–2017-p3-t0
  - Current status in outputs/testpapers_batch_20260709_bib_region_mask: ok.
  - Review result: text/reference table comparing frailty definitions; a correct non-target table for the current Table 1 descriptive parser.
C1.2 Review non_table_layout_candidate failures.
1. papers_from_laha/Asthma prevalence among United States population insights from NHANES data analysis.pdf
  - Asthma prevalence among United States population insights from NHANES data analysis-p6-t0
  - Current status: ok; TableProfile.table_family is estimate_results, and paper_table_inventory.table_category is analysis_outputs.
  - Fix: repeated OR/CI headers and estimate-CI range cells now provide enough deterministic evidence for estimate-result routing.
2. papers_from_laha/GOLD BioAge and depression- Associations with mortality among depressed NHANES participants (2005–2018).pdf
  - GOLD BioAge and depression- Associations with mortality among depressed NHANES participants (2005–2018)-p1-t0
  - Current status: no longer extracted as a table candidate in outputs/testpapers_batch_20260709_bib_region_mask.
  - Review result: abstract/title-page text laid out as article front matter, not a table artifact to route semantically. The front-matter guard now suppresses it before the table pipeline.
3. papers_from_laha/Helicobacter pylori infection in the United States beyond NHANES- a scoping review of seroprevalence estimates by racial and ethnic groups.pdf
  - Helicobacter pylori infection in the United States beyond NHANES- a scoping review of seroprevalence estimates by racial and ethnic groups-p5-t0
  - Helicobacter pylori infection in the United States beyond NHANES- a scoping review of seroprevalence estimates by racial and ethnic groups-p6-t0
  - Helicobacter pylori infection in the United States beyond NHANES- a scoping review of seroprevalence estimates by racial and ethnic groups-p7-t0
  - Current status: ok with matrix_like_table_without_supported_semantic_route; categorized as data_presentation.
  - Review result: real continued data/reference matrix, not a non-table layout. It still needs future data-matrix semantics rather than Table 1 descriptive parsing.
4. papers_from_laha/periodontis2.pdf
  - Prior artifacts: false candidates on PDF pages 5 and 6, including periodontis2-p6-t0 and periodontis2-p6-t1.
  - Current status in outputs/testpapers_batch_prose_bbox_fix_20260712: no page-5 or page-6 table candidates are extracted.
  - Review result: pages 5 and 6 contain prose references to Tables 3, 4, and 5, not visual tables. paper_table_mentions.json correctly classifies Table 3. Nearly..., Table 4 shows..., and the split-line ... is shown in / Table 5. At ... as prose references. Mention records now preserve the source-line bbox, and text-position fallback rejects a proposed caption line by bbox overlap rather than normalized-text equality. This prevents a raw ≥ glyph and normalized >= text from representing the same source line differently during caption rejection. Genuine tables on PDF pages 10-18 remain unchanged.
5. papers_from_johnny/periodontitis.pdf
  - periodontitis-p11-t0
  - Current status in outputs/testpapers_batch_20260709_bib_region_mask: no longer emitted as an extracted table.
  - Review result: abbreviation glossary/reference block, not a table artifact. The retired backend-grid survival path is no longer allowed to emit this rough box as a table.
C1.3 Review the collapsed-grid failure.
- PDF path: papers_from_laha/Ethnic Differences in the Relationship Between Insulin Sensitivity and Insulin Response.pdf
- Table: Ethnic Differences in the Relationship Between Insulin Sensitivity and Insulin Response-p5-t0
- Current status: rescued; the prior collapsed_grid_unrecovered failure is no longer present in outputs/testpapers_batch_20260709_bib_region_mask.
- Current footnote result: 9 resolved links and 0 unresolved links. The prior residual letter:i, letter:x, and letter:g false-marker issue is no longer present as unresolved footnote evidence; the remaining letter:x marker resolves through confirmed footer-cell marker evidence for the local xP < ... table note.

Acceptance for C1: every failed table is classified as correct non-target table, extraction failure, normalization failure, unsupported table family, or ambiguous pending review. The current baseline has no failed table-processing statuses.

C2. Caption, Header/Body, And Extraction Geometry

C2 is table-structure review, and should run before continuation review. It covers table captions, continuation labels, title/preamble rows, column-header rows, body rows, footers, and the extraction geometry that separates those regions. Figure extraction and figure-caption interpretation are future work, not part of this pass. However, figure captions should still be kept as distinct non-table document components when encountered, so they do not contaminate table candidates now and can support later figure extraction.

C2.1 Review caption and region ownership before header/body or continuation decisions.
- Confirm table captions and continuation captions are attached to the correct table candidate.
- If a table has no caption above it, check for a caption below it before treating it as uncaptioned; a strong uncaptained adjacent fragment is continuation evidence, not a reason to reassign a later caption by page order.
- Confirm caption text, title/preamble rows, body rows, and footer/note blocks are not being merged into the wrong table region.
- Confirm figure captions are excluded from table candidates while preserving their identity as separate document components for future figure/caption extraction.
- For each paper, check artifacts in this order:
  1. paper_visual_inventory.json: confirm each visual component has the correct visual_kind, label, page, caption, and source_table_id. Tables should be table visuals; figure captions may appear as figure visuals but should not point at table candidates.
  2. extracted_tables.json: confirm title, caption, page_num, n_rows, n_cols, metadata.bbox, metadata.row_bounds, metadata.horizontal_rules, metadata.trailing_non_table_rows, metadata.table_orientation, metadata.is_continuation, and metadata.continuation_of_table_number. The caption/continuation label should not also appear as a body row, footer definition, or unrelated table candidate.
  3. table_boundary_proposals.json: inspect canonical rule candidates, stub/value coverage, adjacent row edges, font changes, and any current TableRegion boundary reported as unsupported.
  4. paper_document.json joined to paper_positioned_document.json: inspect canonical blocks and positioned lines on the relevant page around the table bbox. The expected order is caption/title, header, body, footer/note block, then surrounding prose or other document components. Column and rotation evidence should explain that order.
  5. normalized_tables.json: confirm header_rows, body_rows, row_views, metadata.header_detection, metadata.header_body_split_rule_comparison, preamble rows, post-header note rows, and continuation-note rows match the visual table structure.
  6. column_header_schemas.json: confirm the selected header rows produce the expected leaf columns and that caption/title/footer text is not being promoted into column headers.
  7. paper_footnotes.json and cell_text_annotations.json: confirm footer/note text is represented as table-local definitions or suppressed non-footnote notation, not as caption text or body rows. Superscript marker evidence should come from positioned glyphs where possible.
  8. table1_continuation_groups.json and table_continuation_column_checks.json: only after the earlier checks pass, confirm continuation decisions use the correct caption, continuation label, row provenance, and column schema.
1. papers_from_laha/Journal of Periodontology - 2015 - Eke - Update on Prevalence of Periodontitis in Adults in the United States NHANES 2009.pdf
  - Current footnote pollution issue appears addressed: continued-page caption text is no longer being written into footnote definitions, and all footnote links are resolved.
  - p4 -> p5 for Table 1. (continued) and p6 -> p7 for Table 2. (continued) now pass continuation column/header checks and are accepted as integrated continuations in resolved_tables.json. This remains a C2 review case because the normalized header/body split still selects row 9 where hline evidence proposes row 2; decide whether the value-anchor selection is correct for these multirow headers before changing continuation behavior.
2. papers_from_johnny/metabolic.pdf
  - Current artifacts appear clean for this check: below-table footer parsing keeps the table caption separate, raised a and b markers link body cells to definitions, and Table 2 * links resolve through the local statistical-significance footer.
3. papers_from_laha/Ethnic Differences in the Relationship Between Insulin Sensitivity and Insulin Response.pdf
  - Current artifacts appear clean for this check: rotated Table 1 is extracted as a table, the caption/footer are separated, and the nine detected footnote links resolve.
4. papers_from_laha/Science-Advanaced-Planetary Health Diet and risk of mortality and chronic diseases- Results from US NHANES, UK Biobank, and a meta-analysis.pdf
  - Current footnote links resolve. The p2 -> p3 Table 1 continuation is accepted in resolved_tables.json after the base fragment's PyMuPDF-positioned header grid records the same two grouped upper headers as the continuation fragment. This remains a C2.4 path-review case because p2 currently enters through pymupdf_text_positions while p3 still enters through a PyMuPDF4LLM rough box plus hline word refinement. Figure-caption components in paper_visual_inventory.json remain noisy; confirm whether that is only future figure-extraction debt or whether it is contaminating table candidates, captions, or rows.
C2.2 Review current header/body split disagreements.
1. papers_from_laha/GOLD BioAge and depression- Associations with mortality among depressed NHANES participants (2005–2018).pdf
  - p4-t1: hline body start 2, value-anchor body start 4, selected 2. This appears to be an acceptable disagreement caused by body parent and wrapped-value rows.
  - p5-t0 / Table 3 is no longer a C2 extraction/header failure: grid_refinement_source = "hline_word_positions", extracted shape is 5 rows by 3 columns, header rows are 0-1, body rows are 2-4, and Adjusted Model spans OR (95%CI) plus p-value. The remaining issue is downstream TableDefinition semantics for a small estimate result table, not extraction geometry.
2. papers_from_laha/Asthma prevalence among United States population insights from NHANES data analysis.pdf
  - p5-t0 and p6-t0.
  - p5-t0: hline body start 3, value-anchor body start 1, selected 1.
  - p6-t0: hline body start 2, value-anchor body start 1, selected 1.
3. papers_from_laha/Journal of Periodontology - 2015 - Eke - Update on Prevalence of Periodontitis in Adults in the United States NHANES 2009.pdf
  - p4-t0 and p5-t0: hline body start 2, value-anchor body start 9, selected 9.
4. papers_from_laha/Science-Advanaced-Planetary Health Diet and risk of mortality and chronic diseases- Results from US NHANES, UK Biobank, and a meta-analysis.pdf
  - p2-t0 is no longer a continuation-blocking header schema failure: PyMuPDF text-position geometry now records header_row_geometry_roles = group_header, leaf_header, leaf_header, leaf_header, so p2-t0 and p3-t0 expose the same column header schema. The only schema change versus the prior no-backend-grid baseline is the intended p2-t0 split into two upper groups over cols 1-4 and 5-8; the periodontitis p6-t0 wrapped leaf headers remain flat leaf headers.
  - p5-t0: hline body start 1, value-anchor body start 0, selected 1.
5. papers_from_laha/Helicobacter pylori infection in the United States beyond NHANES- a scoping review of seroprevalence estimates by racial and ethnic groups.pdf
  - p6-t0: hline body start 8, value-anchor body start 18, selected 7.
  - p7-t0: hline body start 8, value-anchor body start 7, selected 7.
6. papers_from_johnny/cardiovascular.pdf
  - p5-t0.
  - hline body start 4, value-anchor body start 7, selected 7.
  - Current result: All-cause mortality and Cardiovascular mortality are leaf labels, while the AUC text is represented as upper spanning groups over training and testing cohort columns.
7. papers_from_johnny/hypertension.pdf
  - hypertension-p5-t0.
  - Current result: the tight wrapped lifestyle leaves are reconstructed from small-gap visual text runs and stacked across the leaf-header band: Healthy diet and physically active, Healthy diet but physically inactive, Unhealthy diet and physically inactive, Unhealthy diet but physically active, and P value.
C2.3 Review explicit hline and preamble behavior in papers with title/preamble rows above true column headers.
1. papers_from_johnny/fld.pdf
2. papers_from_johnny/pad.pdf
3. papers_from_laha/cobaltpaper.pdf
4. papers_from_johnny/cardiovascular.pdf
5. papers_from_laha/Journal of Periodontology - 2015 - Eke - Update on Prevalence of Periodontitis in Adults in the United States NHANES 2009.pdf
C2.4 Decide whether raw PyMuPDF geometry should replace PyMuPDF4LLM-derived grids for specific failure classes.
- Start only with ruled or rotated cases where words, rules, and table bbox provide strong structural evidence.
- Do not add downstream continuation/schema hacks to compensate for bad extraction geometry.
- Initial ruled-table path is implemented: when credible full-width stroked horizontal rules exist, extraction rebuilds the grid from positioned PyMuPDF words and hline geometry, using PyMuPDF4LLM only as a candidate region source. Keep this item open for broader replacement decisions, especially rotated and non-ruled layouts.
- Current path-consistency audit target: papers_from_laha/Science-Advanaced-Planetary Health Diet and risk of mortality and chronic diseases- Results from US NHANES, UK Biobank, and a meta-analysis.pdf. Table 1 p2 and p3 now produce compatible grouped schemas and integrate, but p2 enters through whole-page PyMuPDF text-position extraction while p3 starts from a PyMuPDF4LLM rough table box plus hline word-position refinement. Determine why the same visual continued table does not use one canonical positioned-geometry path before adding any continuation or schema compensation.

Acceptance for C2: each disagreement is classified as correct caption ownership, caption/body/footer ownership defect, figure-caption contamination of a table artifact, correct hline split, correct value-anchor split, extraction hline defect, normalization candidate defect, or unsupported/misrouted table family.

C3. Continuation Decisions

C3 should run after C2 because continuation decisions depend on correct caption, header/body, footer, and column-schema evidence.

C3.1 Review accepted continuations and confirm one visual continued table becomes one semantic resolved table.
1. papers_from_laha/Association between anthropometric indices and chronic kidney disease- Insights from NHANES 2009–2018.pdf
  - p7 -> p8 and p11 -> p12 accepted.
2. papers_from_laha/Helicobacter pylori infection in the United States beyond NHANES- a scoping review of seroprevalence estimates by racial and ethnic groups.pdf
  - p5 -> p6 and p6 -> p7 accepted.
3. papers_from_laha/Systemic inflammation markers and the prevalence of hypertension- A NHANES cross-sectional study.pdf
  - p5 -> p6 accepted.
4. papers_from_johnny/gallstones.pdf
  - p5 -> p6 accepted.
5. papers_from_laha/mdpi-The Relationship Between a Mediterranean Diet and Frailty in Older Adults- NHANES 2007–2017.pdf
  - p5 -> p6 accepted.
6. papers_from_laha/Journal of Periodontology - 2015 - Eke - Update on Prevalence of Periodontitis in Adults in the United States NHANES 2009.pdf
  - p4 -> p5 accepted for Table 1, and p6 -> p7 accepted for Table 2.
7. papers_from_laha/Asthma prevalence among United States population insights from NHANES data analysis.pdf
  - p4 -> p5 accepted as a prefix fragment integrated with the below-captioned terminal Table 1 fragment after column-schema matching.
8. papers_from_laha/Science-Advanaced-Planetary Health Diet and risk of mortality and chronic diseases- Results from US NHANES, UK Biobank, and a meta-analysis.pdf
  - p2 -> p3 accepted after grouped column-header schemas match.
C3.2 Review rejected continuation candidates that may expose header or column-schema defects.
1. papers_from_laha/Asthma prevalence among United States population insights from NHANES data analysis.pdf
  - Prior p5 -> p6 false continuation is no longer present; p5-t1 and p6-t0 pass through as singleton Table 2 and Table 3.
2. papers_from_laha/periodontis2.pdf
  - Prior p5 -> p6 rejection is no longer present because page 6 no longer emits table candidates from prose references.

Acceptance for C3: continuation decisions are explainable from resolved_tables.json, source table IDs, row provenance, captions or continued-table labels, and column_header_schemas.json.

C4. Mixed Table Families And Regression Baseline

C4.1 Review estimate-result and data-presentation tables without forcing them into descriptive Table 1 semantics.
1. papers_from_johnny/cardiovascular.pdf
2. papers_from_johnny/stroke.pdf
3. papers_from_laha/cobaltpaper.pdf
4. papers_from_laha/An atlas of exposome–phenome associations in health and disease risk.pdf
5. papers_from_laha/An environment-wide association study (EWAS) on type 2 diabetes mellitus.pdf
C4.2 Re-run representative mostly-successful descriptive papers after fixes from C0-C4.1.
1. papers_from_laha/Systemic inflammation markers and the prevalence of hypertension- A NHANES cross-sectional study.pdf
2. papers_from_laha/Lead exposure as a contributor to the Black–White racial disparity in blood pressure- evidence from NHANES 1988–1994 and 2017–2020.pdf
3. papers_from_johnny/hypertension.pdf
4. papers_from_johnny/pad.pdf
5. papers_from_johnny/fld.pdf
C4.3 Run the complete 27-PDF corpus after each substantial parser change set and compare against the latest accepted baseline. Report command failures, missing artifacts, table status counts, failure reasons, resolved continuation counts, footnote link statuses, reference resolution statuses, parsed cell-value counts, and header/body split disagreements.

Acceptance for C4: corpus comparison explains any movement in table-level status, artifact counts, footnote/reference counts, or structural split disagreements. Command success alone is not sufficient.

Issue Note Template

Use this short note for each reviewed chunk or paper:

Review ID:
PDF path:
output directory:
table_id/table_index:
current status:
first bad artifact:
earliest responsible stage:
observed evidence:
expected behavior:
fix decision:
regression decision:

Scope Guardrails

Do not add paper-specific vocabulary rules to make one paper pass.
Do not judge success from CLI exit status alone.
Do not debug downstream semantics before confirming extraction and normalization are structurally correct.
Do not treat row highlighting or background fills as horizontal-rule separators.
Do not force data/result tables into descriptive Table 1 semantics.
Do not create broad unit-test expansion; add focused regressions for known failures or artifact contracts.
Do not add R helpers before repeated review use shows what is needed.
Do not add word-level header reassignment across column boundaries. Header reconstruction should first group positioned words into visual text runs using small between-word spacing, then assign those runs to column bands or spans.

Navigation Menu

Repository navigation

real_paper_testing_guide.md

real_paper_testing_guide.md

Corpus-Driven Real-Paper Testing Guide

Current Reference Baseline

Review Loop

Ordered Review Checklist

C0. Footnotes And References First

C1. Actual Failed Table Statuses

C2. Caption, Header/Body, And Extraction Geometry

C3. Continuation Decisions

C4. Mixed Table Families And Regression Baseline

Issue Note Template

Scope Guardrails

FilesExpand file tree

real_paper_testing_guide.md

Latest commit

History

real_paper_testing_guide.md

File metadata and controls

Corpus-Driven Real-Paper Testing Guide

Current Reference Baseline

Review Loop

Ordered Review Checklist

C0. Footnotes And References First

C1. Actual Failed Table Statuses

C2. Caption, Header/Body, And Extraction Geometry

C3. Continuation Decisions

C4. Mixed Table Families And Regression Baseline

Issue Note Template

Scope Guardrails