diff --git a/.gitignore b/.gitignore index 37c69a0..ff7dc2e 100644 --- a/.gitignore +++ b/.gitignore @@ -4,6 +4,7 @@ dist/ build/ .omx/ .longtable/team/ +.worktrees/ .codex/ .claude/ .env diff --git a/README.md b/README.md index 65093af..817e3d7 100644 --- a/README.md +++ b/README.md @@ -147,6 +147,20 @@ longtable search --query "" longtable panel --prompt "review this measurement plan" --json ``` +## Institutional Research + +`longtable research` is the single surface for reproducible institutional-database work. It provides pilot and protocol checkpoints, immutable academic-database exports, append-only screening decisions, external Research PDF Vault intake, audited reports, and APA 7 or journal-profile Word packages. + +```bash +longtable research init --cwd "" +longtable research pilot --run-id pilot-1 --protocol-reference draft-1 --cwd "" +longtable research freeze --protocol-file protocol.json --cwd "" +longtable research ingest-export --file export.ris --database scopus --format ris --cwd "" +longtable research status --cwd "" +``` + +MCP is the preferred checkpoint transport and numbered text is the recorded fallback. Authentication, MFA, CAPTCHA, terms decisions, and ambiguous methods remain researcher-owned. Bulk production is unavailable until an approved institution profile passes a 5-20 record live smoke gate. See [the institutional research workflow](docs/INSTITUTIONAL-RESEARCH-WORKFLOW.md) for artifact locations, resume rules, reporting-to-manuscript mapping, PDF boundaries, and all commands. + Provider skill installation is explicit: ```bash diff --git a/Spec.md b/Spec.md index bbf261b..6f862b9 100644 --- a/Spec.md +++ b/Spec.md @@ -10,6 +10,10 @@ Repository framing and migration boundary are defined in `README.md`, `RELATIONS Active supporting docs are: +- `docs/superpowers/specs/2026-08-06-longtable-research-institutional-workflow-design.md` +- `docs/superpowers/plans/2026-08-06-longtable-research-institutional-workflow.md` +- `docs/INSTITUTIONAL-RESEARCH-WORKFLOW.md` +- `docs/INSTITUTIONAL-RESEARCH-ACCEPTANCE-AUDIT.md` - `docs/ARCHITECTURE.md` - `docs/ACTIVE-DECISIONS.md` - `docs/CHECKPOINTING.md` @@ -94,6 +98,8 @@ Current code-level scaffolds now cover: - question runtime contracts now separate shared checkpoint judgment from Claude/Codex-specific presentation surfaces - panel orchestration is now scoped as provider-neutral `InvocationIntent` and `PanelPlan` semantics with provider-specific execution surfaces and sequential fallback - `@longtable/mcp` now exposes `longtable-state` as optional structured transport over `.longtable/` state, not as a replacement source of truth +- `@longtable/scholar-research` now exposes one receipt-backed institutional workflow for protocol freezing, export intake, screening, Research PDF Vault manifests, audited reporting, and declarative Word render profiles +- institutional production eligibility requires profile approval, deterministic replay, a researcher-approved 5-20 record live smoke run, and one or two permitted PDF intakes ## Core Domain Objects diff --git a/docs/INSTITUTIONAL-RESEARCH-ACCEPTANCE-AUDIT.md b/docs/INSTITUTIONAL-RESEARCH-ACCEPTANCE-AUDIT.md new file mode 100644 index 0000000..92a7392 --- /dev/null +++ b/docs/INSTITUTIONAL-RESEARCH-ACCEPTANCE-AUDIT.md @@ -0,0 +1,30 @@ +# Institutional Research Acceptance Audit + +This matrix maps every criterion in the approved institutional-workflow design to a fresh automated assertion and a canonical artifact contract. Test names refer to scripts included in `npm test`; artifact paths are relative to a generated research project unless they point to a source contract. + +| Acceptance criterion | Automated assertion | Canonical artifact or contract | +|---|---|---| +| One LongTable Research surface | `e2e-institutional-review.mjs` asserts the exact command contract; CLI build and pilot smoke exercise `longtable research` | `packages/longtable-scholar-research/src/coordinator.ts`; CLI `research` route | +| MCP preferred for required checkpoints | `smoke-institutional-checkpoints.mjs` checks every hard stop has `mcp_elicitation` then `numbered`; E2E records MCP decline and numbered fallback | `audit/checkpoints.jsonl`; checkpoint delivery record | +| No hard-stop resume without DecisionRecord | `smoke-institutional-checkpoints.mjs` loops over every declared hard-stop code, rejects empty decisions, and resumes only with linked IDs | blocked `ResearchRun`, `QuestionRecord`, linked `DecisionRecord` | +| Research variables are extracted independently and frozen before analysis | `smoke-data-extraction.mjs` tests profile hashing, required evidence, double extraction, conflict detection, adjudication, reliability, idempotent dataset hashing, and freeze rejection | `protocol/extraction-profiles/`, extraction ledgers, `audit/extracted-data-freezes.jsonl`, `data/08_analysis-ready/` | +| Approved protocol revisions are immutable | `smoke-institutional-workflow.mjs` rejects different content at an existing revision; E2E freezes revisions 1 and 2 with different hashes | `protocol/amendments/protocol-revision-NNNN.json` | +| Mechanical stages use zero model calls | `smoke-research-failures.mjs` loops over `ZERO_MODEL_CALL_STAGES` and rejects one observed call | model-budget record and deterministic stage receipt | +| Interrupted runs resume safely without duplicate artifacts | E2E compares repeated resume results and replays export, screening, PDF intake, and Word rendering | `latestSafeCursor`, SHA raw export, screening ledger, PDF manifest, content hash | +| Count invariants pass before rendering | `smoke-research-reporting.mjs` renders balanced counts and rejects a modified final-included count | `reports/review-flow-counts.md`, `reports/review-flow.svg` | +| Material records, exclusions, acquisitions, and manuscript counts have provenance | corpus, screening, PDF, reporting, and manuscript smokes check source links, evidence IDs, SHA/access basis, artifact provenance, and every rendered Word object ID | `corpus/papers.jsonl`, screening ledger, PDF manifest, report provenance, manuscript provenance sidecar | +| No credentials, sessions, secrets, or licensed PDFs enter state or Git | `smoke-research-failures.mjs` seeds six violation classes and verifies redacted findings; repository audit scans canonical research artifact roots | `scripts/audit-research-artifacts.mjs` report | +| Research PDF Vault is the only canonical PDF store | `smoke-pdf-vault.mjs` rejects a vault inside the project and verifies one SHA-addressed copy with manifest-only project state | external Research PDF Vault; `corpus/pdf-manifest.jsonl` | +| Final reports and Word artifacts derive from canonical state | reporting and manuscript smokes reject invariant, database-execution, full-text, and analysis-approval failures | `reports/`, `manuscript/*.docx`, provenance sidecars | +| Same approved state rerenders identically | reporting smoke deep-compares all outputs; manuscript smoke and E2E compare content hashes | deterministic report artifacts and DOCX content hash | +| New institutions require calibration, tests, live smoke, and approval | `smoke-institution-adapters.mjs` tests each missing gate and complete eligibility; E2E enforces 5-20 records, one or two PDFs, and four approval flags | institution calibration checklist and live-smoke `StageReceipt` | + +## Completion-specific cross-checks + +- Every declared hard-stop code traverses creation, MCP preference, numbered fallback, decision rejection, and linked resume in one parameterized test. +- An eligible corpus is not called analysis-ready until its extraction profile, independent extraction records, reliability result, adjudications, and extracted-data freeze all pass. +- All stages declared in `ZERO_MODEL_CALL_STAGES` reject any observed model call. +- Every Word paragraph role generated by the renderer has a provenance object: title, author lines, abstract heading/body, keywords, every section heading/body, readiness table, references heading, and every reference. +- Raw exports are content-addressed and immutable after intake; identical replay is idempotent. +- PDF bytes remain outside the project and are referenced only by a verified manifest. +- The included institution fixture remains non-production until login verification, resolver discovery, per-database calibration, fixture replay, limited live smoke, and the profile DecisionRecord are all present. diff --git a/docs/INSTITUTIONAL-RESEARCH-WORKFLOW.md b/docs/INSTITUTIONAL-RESEARCH-WORKFLOW.md new file mode 100644 index 0000000..a4d756d --- /dev/null +++ b/docs/INSTITUTIONAL-RESEARCH-WORKFLOW.md @@ -0,0 +1,162 @@ +# Institutional Research Workflow + +This is the researcher-facing operating contract for the integrated `longtable research` surface. It covers reproducible academic-database retrieval, document screening, lawful full-text intake, reporting, and manuscript rendering. It does not automate authentication, decide an ambiguous inclusion rule, or convert a pilot into production without a recorded researcher decision. + +## Method boundary + +Use two related but distinct evidence streams for a demand-supply mismatch study: + +1. **Demand corpus:** procurement notices, commissioned-research topics, policy documents, and employer demand documents. Treat this as a document-analysis population with an explicit sampling frame, inclusion criteria, extraction matrix, pilot coding, agreement/adjudication, and missing-document audit. Do not call this corpus a PRISMA literature search. +2. **Supply and scholarly corpus:** training-course metadata and scholarly records from selected academic databases. The academic search can use the workflow below and can produce review-flow artifacts when its count invariants pass. + +The two streams meet only after each has its own stable unit of analysis and codebook. A conceptual framework may then map demanded competencies and problem types to supplied curricula, learning protocols, and measured competency outcomes. This separation prevents procurement demand from being mistaken for scholarly evidence and prevents course availability from being mistaken for effectiveness. + +## One research surface + +Initialize the canonical folders before the pilot: + +```bash +longtable research init --cwd "" +longtable research pilot --run-id --protocol-reference draft-1 --cwd "" +``` + +The pilot creates one compact search-strategy checkpoint. MCP elicitation is attempted first. Decline, timeout, unsupported transport, or transport error records the failed attempt and renders the same checkpoint as numbered text; it does not infer an answer. + +After the researcher approves a protocol revision: + +```bash +longtable research freeze --protocol-file protocol-revision.json --cwd "" +longtable research ingest-export --file export.csv --database web-of-science --format csv --cwd "" +longtable research screen --decisions-file screening-decisions.json --cwd "" +longtable research acquire --candidate-path paper.pdf --vault-root "" --paper-id \ + --acquisition-method researcher_manual_upload --access-basis researcher_provided \ + --version accepted-manuscript --project-inclusion included --screening-state fulltext_included \ + --cwd "" +longtable research freeze-extraction --profile-file extraction-profile.json --cwd "" +longtable research extract --profile-file extraction-profile.json --records-file extraction-records.json --cwd "" +longtable research adjudicate-extraction --profile-file extraction-profile.json \ + --adjudications-file adjudications.json --cwd "" +longtable research freeze-data --profile-file extraction-profile.json --freeze-file data-freeze.json --cwd "" +longtable research report --input-file report-input.json --cwd "" +longtable research package --input-file manuscript-input.json --profile-file apa7.json \ + --output-path manuscript/review.docx --cwd "" +``` + +`freeze` writes immutable, numbered revisions. The same revision and content is idempotent; different content cannot overwrite an existing revision number. `ingest-export` stores the original export under its SHA-256, then produces normalized and deduplicated derivatives. Re-ingesting identical bytes does not create a second raw artifact. `screen` appends versioned decisions and ignores an identical decision ID on replay. + +Metadata identifies research units; it is not the coded research dataset. `freeze-extraction` locks the unit of analysis, variables, value types, evidence requirements, double-extraction rule, and reliability threshold. `extract` validates independent records and detects field-level conflicts. `adjudicate-extraction` records a human resolution with evidence. `freeze-data` refuses unresolved conflict, incomplete double extraction, below-threshold reliability, or missing required evidence, then writes a deterministic analysis-ready dataset hash. + +## Mandatory checkpoints and resume rule + +| Event | Immediate action | Resume evidence | +|---|---|---| +| Login, MFA, CAPTCHA, or expired session | Stop at the last safe cursor | Researcher restores access and records a linked DecisionRecord | +| Terms or access basis is unclear | Stop before retrieval | Researcher records the permitted route or excludes the route | +| Query drift, result cap, export mismatch, or changed download pattern | Stop production | Researcher approves a protocol amendment or corrective action | +| Ambiguous screening rule or human-AI disagreement | Preserve both decisions; do not collapse | Adjudication decision linked to the QuestionRecord | +| Missing full-text threshold exceeded | Stop full-text screening | Revised acquisition plan or approved limitation | +| Analysis method changes | Stop synthesis | Approved analysis-method DecisionRecord | +| Corpus freeze is absent | Refuse analysis | Frozen corpus receipt | +| Extraction profile is unfrozen or a coding rule is ambiguous | Stop extraction | Approved profile or rule revision linked to a DecisionRecord | +| Pilot reliability is below threshold | Stop production extraction | Revised pilot/codebook or approved threshold decision | +| Double extraction conflicts remain | Preserve both records | Evidence-backed adjudication | +| Extracted-data freeze is absent | Refuse analysis and manuscript rendering | Deterministic dataset freeze linked to a DecisionRecord | + +Every stage advances only with a `StageReceipt`. A blocked run resumes only when the DecisionRecord references both the blocking checkpoint key and QuestionRecord ID. Replaying resume from the same blocked state with the same timestamp and decision produces the same state. + +## Canonical project artifacts + +```text +/ + protocol/ + research-question.yaml + search-protocol.yaml + inclusion-exclusion.yaml + database-profiles/ + extraction-profiles/ + amendments/protocol-revision-0001.json + data/ + 00_raw-exports/ + 01_normalized/ + 02_deduplicated/ + 03_title-abstract-screening/ + 04_fulltext-screening/ + 05_extraction-pilot/ + 06_extracted/ + 07_adjudicated/ + 08_analysis-ready/ + corpus/ + papers.jsonl + acquisition-ledger.jsonl + pdf-manifest.jsonl + screening-decisions.jsonl + extraction-records.jsonl + extraction-conflicts.jsonl + extraction-adjudications.jsonl + manual-action-queue.csv + audit/ + stage-receipts.jsonl + checkpoints.jsonl + decisions.jsonl + exclusions.jsonl + deviations.jsonl + failures.jsonl + extracted-data-freezes.jsonl + reports/ + manuscript/ + analysis/ + references/ +``` + +The Research PDF Vault must be outside the project. The project retains only a Vault-relative manifest locator, SHA-256, access basis, acquisition method, version, and screening state; the approved institution storage profile resolves the external Vault root at runtime. Placeholder, partial, corrupt, symbolic-link, absolute-path, or token-bearing PDF intake is rejected. + +## Final researcher report and paper mapping + +The report command generates a researcher audit report, count table, deterministic review-flow SVG, systematic-review Methods/Results text, and provenance records. + +| Research artifact | Direct manuscript use | +|---|---| +| Frozen protocol and exact database queries | Methods: information sources, search strategy, eligibility criteria | +| Database yields and raw-export hashes | Methods/Results: search dates, yields, export reconciliation | +| Screening ledger and exclusion reasons | Methods: selection process; Results: study selection and exclusions | +| PDF manifest and missing-text audit | Methods: retrieval routes; Results/Limitations: reports not retrieved | +| Count-invariant report and review-flow SVG | Results table and study-flow figure | +| Deviation and DecisionRecord ledger | Protocol amendments and deviations | +| Analysis-readiness table | Analysis plan, missing statistics, dependency and author-contact needs | +| Extraction profile, reliability, conflict/adjudication, and data-freeze ledger | Methods: data items and coding; Results: reliability, missingness, and final analytic dataset | +| Paragraph/table provenance map | Internal audit before submission | + +Planned work renders future-tense protocol language. Completed work renders past-tense Methods and observed Results. Rendering fails when count equations do not balance or when database execution, full-text verification, extracted-data freeze, or analysis approval is absent. + +The review-flow output is an internal deterministic representation, not a claim that every document corpus is a systematic review. Final reporting should be checked against the [PRISMA 2020 checklist and flow materials](https://www.prisma-statement.org/prisma-2020) and, for literature-search reporting, the [PRISMA-S extension](https://doi.org/10.1186/s13643-020-01542-z). + +## Word profiles + +The included `apa7-memory` profile fixes a working-manuscript layout: US Letter, one-inch margins, Times New Roman 12 pt, double spacing, top-right page numbers, title page, 0.5-inch first-line indents, and 0.5-inch hanging reference indents. It is a declared working profile, not a substitute for a named journal's current author instructions. + +The Korean profile is explicitly `provisional_until_journal_rules_verified`. A named target-journal profile should be approved only after recording its aims and scope, author instructions, article type, recent reference-paper pattern, table/figure conventions, and template hash. User templates are accepted only with a matching SHA-256. + +## Live institution smoke gate + +Production eligibility requires a researcher-owned login, an approved institution profile, deterministic recipe replay, and a live smoke run: + +```bash +longtable research live-smoke --run-id --record-count 5 --pdf-count 1 \ + --researcher-approved --profile-approved --calibrated --replay-passed --cwd "" +``` + +The gate accepts 5-20 metadata records and one or two permitted PDFs. Anything larger is not a smoke run. The receipt records the counts and eligibility decision; it never stores credentials, cookies, session tokens, MFA material, or authentication headers. + +## Researcher closeout + +Before manuscript submission, the researcher receives and reviews: + +1. frozen protocol revision and amendments; +2. database-by-database yield and export reconciliation; +3. unresolved issues, recovered failures, exclusions, and deviations; +4. corpus and PDF-manifest invariants; +5. analysis-readiness and required-action list; +6. manuscript profile status and unresolved citation fields; +7. artifact provenance map and security-audit result. + +This closeout is the final checkpoint. The automation prepares the evidence package; the researcher remains responsible for methodological interpretation, journal fit, and submission. diff --git a/docs/LONGTABLE-COMMAND-SURFACE.md b/docs/LONGTABLE-COMMAND-SURFACE.md index 231beba..24336da 100644 --- a/docs/LONGTABLE-COMMAND-SURFACE.md +++ b/docs/LONGTABLE-COMMAND-SURFACE.md @@ -180,6 +180,24 @@ Tmux is not a LongTable core requirement. If an OMX-style Codex popup transport is added, it must be documented as optional, attached-tmux-only, and backed by the standard fallback path. +## LongTable Research Surface + +`longtable research` is the single public command surface for the integrated +institutional-database workflow. It owns project initialization, readiness, +protocol checkpoints, corpus processing, Research PDF Vault references, +screening, reports, and manuscript packaging behind one capability tree. + +```bash +longtable research doctor --json +longtable research init --cwd "" --json +longtable research status --cwd "" --json +``` + +`longtable scholar-research` is retained only as a deprecated compatibility +alias. Required decisions use MCP elicitation first and numbered fallback +second. A fallback reuses the same pending QuestionRecord by checkpoint key; +it never creates an implicit answer. + ## Supporting Commands ```bash diff --git a/docs/superpowers/plans/2026-08-06-longtable-research-institutional-workflow.md b/docs/superpowers/plans/2026-08-06-longtable-research-institutional-workflow.md new file mode 100644 index 0000000..7ef6acb --- /dev/null +++ b/docs/superpowers/plans/2026-08-06-longtable-research-institutional-workflow.md @@ -0,0 +1,625 @@ +# LongTable Research Institutional Workflow Implementation Plan + +> **For Codex:** REQUIRED SUB-SKILL: Use `superpowers:executing-plans` to implement this plan task-by-task. Use `superpowers:test-driven-development` for every behavior change and `superpowers:verification-before-completion` before claiming completion. + +**Goal:** Implement the approved, reproducible institutional-database research workflow behind one LongTable Research surface, with durable hard stops, immutable production protocols, deterministic corpus processing, one canonical Research PDF Vault, and manuscript-ready outputs. + +**Architecture:** Extend `@longtable/scholar-research` into the provider-neutral execution core. Keep database and institution behavior behind typed adapters; route required decisions through existing LongTable QuestionRecord/DecisionRecord and MCP elicitation; expose the integrated workflow as `longtable research` while retaining `scholar-research` as a compatibility alias. Machine-readable protocol, corpus, audit, and provenance files are canonical; reports and Word files are deterministic renderings. + +**Tech Stack:** TypeScript 5.6, Node.js 18+, MCP SDK/Zod, existing LongTable core/checkpoint/CLI packages, deterministic CSV/RIS/NBIB/JSON parsers, SHA-256, JSONL/YAML/Markdown/HTML, and DOCX packaging through the bundled document runtime selected during Task 14. + +**Global Constraints:** Preserve user edits in `.longtable/state.json` and `CURRENT.md`. Never persist credentials, cookies, session tokens, MFA material, authentication headers, or licensed PDF bytes in project state or Git. No CAPTCHA/paywall/access-control bypass. Use deterministic code for pagination, parsing, deduplication, counts, PRISMA, and rendering. A production run requires an approved InstitutionProfile and frozen ProtocolRevision. Every hard stop must persist a QuestionRecord and reject resume until a linked DecisionRecord exists. Each task begins with a failing smoke/contract test, implements the smallest passing behavior, runs targeted verification, and creates an atomic commit. + +--- + +## Tranche 1 — Core run, protocol, receipt, adapter, and checkpoint contracts + +### Task 1: Define institutional-research domain contracts + +**Files:** +- Create: `packages/longtable-scholar-research/src/workflow-types.ts` +- Modify: `packages/longtable-scholar-research/src/index.ts` +- Test: `scripts/smoke-institutional-workflow.mjs` +- Modify: `package.json` + +**Step 1: Write the failing contract test** + +Assert that the package exports the exact stage sequence, hard-stop codes, retryable codes, run/protocol/profile constructors, and safe profile validator. The test must instantiate this minimal boundary: + +```ts +const revision = research.createProtocolRevision({ + id: "protocol-1", + revision: 1, + decisionRecordId: "decision-1", + databases: ["wos"], + queries: { wos: "TS=(hackathon AND competency)" }, + filters: { years: [2015, 2026], languages: ["en", "ko"], publicationTypes: ["article"] } +}); +assert(Object.isFrozen(revision)); +assertEqual(research.nextResearchStage("PILOT"), "PROTOCOL_CHECKPOINT"); +``` + +**Step 2: Run the test to verify it fails** + +Run: `npm run build --workspace @longtable/scholar-research && node scripts/smoke-institutional-workflow.mjs` +Expected: FAIL because `workflow-types.js` exports do not exist. + +**Step 3: Implement the contracts** + +Define and export: + +```ts +export const RESEARCH_STAGES = [ + "SETUP", "PILOT", "PROTOCOL_CHECKPOINT", "PRODUCTION_SEARCH", + "EXPORT_AUDIT", "TITLE_ABSTRACT_SCREENING", "FULLTEXT_PLAN_CHECKPOINT", + "FULLTEXT_ACQUISITION", "FULLTEXT_SCREENING", "CORPUS_FREEZE_CHECKPOINT", + "ANALYSIS", "SYNTHESIS", "RESEARCHER_REPORT", "MANUSCRIPT_PACKAGE" +] as const; + +export type ResearchRunStatus = "planned" | "running" | "blocked" | "completed" | "failed"; +export interface ProtocolRevision { readonly id: string; readonly revision: number; readonly frozenAt: string; readonly decisionRecordId: string; readonly protocolHash: string; readonly databases: readonly string[]; readonly queries: Readonly>; readonly filters: Readonly; } +export interface StageReceipt { readonly id: string; readonly runId: string; readonly stage: ResearchStage; readonly protocolRevisionId: string; readonly inputArtifactIds: readonly string[]; readonly outputArtifactIds: readonly string[]; readonly cursor: string; readonly createdAt: string; } +``` + +Also define ResearchRun, InstitutionProfile, ArtifactProvenance, capability names, database-adapter interfaces, hard-stop codes, retryable-event codes, canonical constructors, SHA-256 protocol hashing, deep freezing, and a validator that rejects forbidden secret-like profile keys. + +**Step 4: Register and run the smoke test** + +Add `smoke:institutional-workflow` before `typecheck` in root `npm test`. + +Run: `npm run smoke:institutional-workflow && npm run typecheck --workspace @longtable/scholar-research` +Expected: PASS. + +**Step 5: Commit** + +```bash +git add package.json packages/longtable-scholar-research/src/index.ts packages/longtable-scholar-research/src/workflow-types.ts scripts/smoke-institutional-workflow.mjs +git commit -m "feat(research): define institutional workflow contracts" +``` + +### Task 2: Implement receipt-backed state transitions and real hard stops + +**Files:** +- Create: `packages/longtable-scholar-research/src/workflow-state.ts` +- Modify: `packages/longtable-scholar-research/src/index.ts` +- Test: `scripts/smoke-institutional-workflow.mjs` + +**Step 1: Add failing transition tests** + +Test valid advancement, invalid stage skipping, safe-cursor persistence, hard-stop creation, and resume rejection without an answer-linked decision. Include this assertion: + +```ts +const blocked = research.blockResearchRun(run, { + code: "EXPORT_COUNT_MISMATCH", + questionRecordId: "question-1", + safeCursor: "wos:page:3" +}); +assertThrows(() => research.resumeResearchRun(blocked, []), /DecisionRecord/); +``` + +**Step 2: Verify failure** + +Run: `npm run smoke:institutional-workflow` +Expected: FAIL because transition functions are absent. + +**Step 3: Implement state-machine guards** + +Implement `advanceResearchRun`, `blockResearchRun`, and `resumeResearchRun`. Advancement requires a StageReceipt for the current stage. Resume requires a DecisionRecord whose `id` matches the blocking QuestionRecord's `decisionRecordId` and whose checkpoint key matches the stop. Keep the last safe cursor stable across repeated identical failures. + +**Step 4: Verify** + +Run: `npm run smoke:institutional-workflow && npm run typecheck --workspace @longtable/scholar-research` +Expected: PASS. + +**Step 5: Commit** + +```bash +git add packages/longtable-scholar-research/src/index.ts packages/longtable-scholar-research/src/workflow-state.ts scripts/smoke-institutional-workflow.mjs +git commit -m "feat(research): enforce receipt-backed hard stops" +``` + +### Task 3: Create the canonical local project scaffold and append-only stores + +**Files:** +- Create: `packages/longtable-scholar-research/src/project-store.ts` +- Modify: `packages/longtable-scholar-research/src/protocol.ts` +- Modify: `packages/longtable-scholar-research/src/index.ts` +- Test: `scripts/smoke-institutional-workflow.mjs` + +**Step 1: Add a failing filesystem test** + +Create a temporary project; assert the complete `protocol/`, `data/`, `corpus/`, `audit/`, `reports/`, `manuscript/`, `analysis/`, `references/`, and `.longtable/research-runs/` contract. Verify that append helpers write newline-delimited records and refuse mutation of an already frozen protocol revision. + +**Step 2: Verify failure** + +Run: `npm run smoke:institutional-workflow` +Expected: FAIL on missing directories/store functions. + +**Step 3: Implement deterministic stores** + +Implement `buildResearchProjectLayout`, `writeResearchProjectScaffold`, `appendJsonlRecord`, `readJsonlRecords`, `writeFrozenProtocolRevision`, and `readLatestProtocolRevision`. Write atomically through a same-directory temporary file plus rename for snapshots; use append-only JSONL for receipts, checkpoints, decisions, exclusions, deviations, and failures. + +**Step 4: Verify and commit** + +Run: `npm run smoke:institutional-workflow && npm run typecheck` + +```bash +git add packages/longtable-scholar-research/src/index.ts packages/longtable-scholar-research/src/project-store.ts packages/longtable-scholar-research/src/protocol.ts scripts/smoke-institutional-workflow.mjs +git commit -m "feat(research): scaffold canonical research projects" +``` + +--- + +## Tranche 2 — MCP-first search-strategy and freeze UX + +### Task 4: Build adaptive checkpoint cards and QuestionRecord mappings + +**Files:** +- Create: `packages/longtable-scholar-research/src/checkpoints.ts` +- Modify: `packages/longtable-scholar-research/src/index.ts` +- Modify: `packages/longtable-scholar-research/package.json` +- Test: `scripts/smoke-institutional-checkpoints.mjs` +- Modify: `package.json` + +**Step 1: Add failing checkpoint tests** + +Test one compact search-strategy card, progressive database multi-select only after `modify_databases`, pilot-freeze, full-text plan, corpus freeze, access ambiguity, screening conflict, and analysis-method questions. Required records must set `hardStop: true`, the correct `hardStopScope`, and `preferredSurfaces: ["mcp_elicitation", "numbered"]`. + +**Step 2: Verify failure** + +Run: `npm run build && node scripts/smoke-institutional-checkpoints.mjs` +Expected: FAIL because builders do not exist. + +**Step 3: Implement pure checkpoint builders** + +Add `@longtable/core` as a workspace dependency and depend on its types only. Generate stable checkpoint keys from run ID, protocol revision, stage, and stop code. Never choose a default response after cancel, decline, timeout, or unsupported MCP transport. + +**Step 4: Register, verify, and commit** + +Run: `npm run smoke:institutional-checkpoints && npm run typecheck` + +```bash +git add package.json package-lock.json packages/longtable-scholar-research/package.json packages/longtable-scholar-research/src/checkpoints.ts packages/longtable-scholar-research/src/index.ts scripts/smoke-institutional-checkpoints.mjs +git commit -m "feat(research): add MCP-first research checkpoint cards" +``` + +### Task 5: Expose one `longtable research` surface with compatibility aliases + +**Files:** +- Modify: `packages/longtable/src/cli.ts` +- Modify: `packages/longtable-mcp/src/server.ts` +- Modify: `packages/longtable/README.md` +- Modify: `docs/LONGTABLE-COMMAND-SURFACE.md` +- Test: `scripts/smoke-institutional-checkpoints.mjs` + +**Step 1: Add failing CLI/MCP routing tests** + +Assert `longtable research doctor|init|status|checkpoint` works, `longtable scholar-research ...` delegates to the same handlers with a deprecation field, and MCP `elicit_question` persists then reuses the same pending QuestionRecord on fallback. + +**Step 2: Verify failure** + +Run: `npm run build && node scripts/smoke-institutional-checkpoints.mjs` +Expected: FAIL on the `research` command. + +**Step 3: Implement routing** + +Rename the internal handler to `runLongTableResearch`; map both command names to it. Add `research` to command parsing/help. Do not expose database-specific commands as new top-level surfaces. Reuse existing MCP elicitation rather than creating a second question transport. + +**Step 4: Verify and commit** + +Run: `npm run smoke:institutional-checkpoints && npm run smoke:scholar-research && npm run typecheck` + +```bash +git add packages/longtable/src/cli.ts packages/longtable-mcp/src/server.ts packages/longtable/README.md docs/LONGTABLE-COMMAND-SURFACE.md scripts/smoke-institutional-checkpoints.mjs +git commit -m "feat(research): expose one integrated research surface" +``` + +--- + +## Tranche 3 — Deterministic corpus/export pipeline + +### Task 6: Implement sanitized export parsers and golden fixtures + +**Files:** +- Create: `packages/longtable-scholar-research/src/export-parsers.ts` +- Create: `packages/longtable-scholar-research/fixtures/exports/crossref.json` +- Create: `packages/longtable-scholar-research/fixtures/exports/pubmed.nbib` +- Create: `packages/longtable-scholar-research/fixtures/exports/scopus.ris` +- Create: `packages/longtable-scholar-research/fixtures/exports/wos.csv` +- Create: `packages/longtable-scholar-research/fixtures/exports/expected-records.json` +- Modify: `packages/longtable-scholar-research/src/index.ts` +- Test: `scripts/smoke-export-parsers.mjs` +- Modify: `package.json` + +**Step 1: Write failing golden tests** + +Assert identical canonical bibliographic records for DOI, title, authors, year, abstract, database source, source record ID, and export artifact hash. Include quoted CSV fields, folded RIS lines, multi-valued NBIB tags, missing DOI, Korean text, and empty export rejection. + +**Step 2: Verify failure** + +Run: `npm run build --workspace @longtable/scholar-research && node scripts/smoke-export-parsers.mjs` +Expected: FAIL on parser imports. + +**Step 3: Implement parsers without model calls** + +Implement `parseCsvExport`, `parseRisExport`, `parseNbibExport`, `parseJsonExport`, and `parseExportArtifact`. Return records plus parse rejections with explicit reasons; never discard malformed rows silently. + +**Step 4: Register, verify, and commit** + +Run: `npm run smoke:export-parsers && npm run typecheck --workspace @longtable/scholar-research` + +```bash +git add package.json packages/longtable-scholar-research/src packages/longtable-scholar-research/fixtures/exports scripts/smoke-export-parsers.mjs +git commit -m "feat(research): parse scholarly exports deterministically" +``` + +### Task 7: Normalize, deduplicate, and enforce corpus invariants + +**Files:** +- Create: `packages/longtable-scholar-research/src/corpus.ts` +- Create: `packages/longtable-scholar-research/src/invariants.ts` +- Modify: `packages/longtable-scholar-research/src/index.ts` +- Test: `scripts/smoke-corpus-pipeline.mjs` +- Modify: `package.json` + +**Step 1: Add failing deterministic and property-style tests** + +Test DOI normalization, exact DOI merge, exact normalized title/year fallback, version links, duplicate provenance, stable IDs, order-independent output, and all four approved count equations. Test that PRISMA eligibility is false when any equation fails. + +**Step 2: Verify failure** + +Run: `npm run build --workspace @longtable/scholar-research && node scripts/smoke-corpus-pipeline.mjs` +Expected: FAIL on corpus imports. + +**Step 3: Implement the pipeline** + +Implement pure `normalizeRecords`, `deduplicateRecords`, `calculateCorpusCounts`, and `verifyCorpusInvariants`. Keep every source link and rejection/exclusion basis. Sort canonical output by stable paper ID. + +**Step 4: Register, verify, and commit** + +Run: `npm run smoke:corpus-pipeline && npm run typecheck` + +```bash +git add package.json packages/longtable-scholar-research/src/corpus.ts packages/longtable-scholar-research/src/invariants.ts packages/longtable-scholar-research/src/index.ts scripts/smoke-corpus-pipeline.mjs +git commit -m "feat(research): build auditable canonical corpora" +``` + +--- + +## Tranche 4 — Research PDF Vault integration + +### Task 8: Intake lawful PDFs into one canonical vault + +**Files:** +- Create: `packages/longtable-scholar-research/src/pdf-vault.ts` +- Modify: `packages/longtable-scholar-research/src/index.ts` +- Test: `scripts/smoke-pdf-vault.mjs` +- Modify: `package.json` + +**Step 1: Add failing vault tests** + +Use synthetic PDF bytes. Assert SHA-256 identity, one canonical file per hash, project manifest references rather than copies, access-basis requirement, placeholder/partial/corrupt PDF rejection, and no licensed bytes beneath the project directory. + +**Step 2: Verify failure** + +Run: `npm run build --workspace @longtable/scholar-research && node scripts/smoke-pdf-vault.mjs` +Expected: FAIL on vault imports. + +**Step 3: Implement safe intake** + +Implement `inspectPdfCandidate`, `intakeResearchPdf`, `buildPdfManifestRecord`, and `verifyPdfManifest`. The caller supplies a configured absolute vault root; reject vault roots inside the project and canonical paths outside the root. Record paperId, SHA, source URL, acquisition method, access basis, version, inclusion state, and screening state—never auth material. + +**Step 4: Register, verify, and commit** + +Run: `npm run smoke:pdf-vault && npm run typecheck` + +```bash +git add package.json packages/longtable-scholar-research/src/pdf-vault.ts packages/longtable-scholar-research/src/index.ts scripts/smoke-pdf-vault.mjs +git commit -m "feat(research): integrate the canonical Research PDF Vault" +``` + +--- + +## Tranche 5 — Database adapters, supervised browser execution, and institution calibration + +### Task 9: Implement capability contracts and institution-profile approval + +**Files:** +- Create: `packages/longtable-scholar-research/src/adapters.ts` +- Create: `packages/longtable-scholar-research/src/institutions.ts` +- Create: `packages/longtable-scholar-research/fixtures/institutions/psu-sanitized.json` +- Modify: `packages/longtable-scholar-research/src/index.ts` +- Test: `scripts/smoke-institution-adapters.mjs` +- Modify: `package.json` + +**Step 1: Add failing contract tests** + +Run a fake adapter through open, authentication verification, query, filters, result count, export, export verification, full-text resolution, permitted download, suspend, and resume. Assert unsupported capabilities are explicit. Assert an institution remains ineligible until calibration receipt, fixture replay, live smoke receipt, and profile-approval DecisionRecord all exist. + +**Step 2: Verify failure** + +Run: `npm run build && node scripts/smoke-institution-adapters.mjs` +Expected: FAIL on adapter/profile imports. + +**Step 3: Implement typed layers** + +Add adapter registry, capability result union, sanitized InstitutionProfile loader/validator, and `assessProductionEligibility`. The PSU fixture contains database IDs, access modes, resolver patterns, export defaults, result caps, storage references, and verification dates only. + +**Step 4: Register, verify, and commit** + +Run: `npm run smoke:institution-adapters && npm run typecheck` + +```bash +git add package.json packages/longtable-scholar-research/src/adapters.ts packages/longtable-scholar-research/src/institutions.ts packages/longtable-scholar-research/src/index.ts packages/longtable-scholar-research/fixtures/institutions scripts/smoke-institution-adapters.mjs +git commit -m "feat(research): add database and institution adapter contracts" +``` + +### Task 10: Implement accessibility-first browser recipes and calibration replay + +**Files:** +- Create: `packages/longtable-scholar-research/src/browser-recipes.ts` +- Create: `packages/longtable-scholar-research/fixtures/browser/wos-results.html` +- Create: `packages/longtable-scholar-research/fixtures/browser/wos-export.html` +- Create: `packages/longtable-scholar-research/fixtures/browser/signatures.json` +- Modify: `packages/longtable-scholar-research/src/index.ts` +- Test: `scripts/smoke-browser-recipes.mjs` +- Modify: `package.json` + +**Step 1: Add failing replay tests** + +Assert semantic landmark matching, page-signature hashing, bounded coordinate fallback only after a matching signature, export-count extraction, and hard stops for login, MFA/CAPTCHA, session expiry, changed signatures, changed download patterns, terms ambiguity, and result caps. + +**Step 2: Verify failure** + +Run: `npm run build && node scripts/smoke-browser-recipes.mjs` +Expected: FAIL on recipe imports. + +**Step 3: Implement replayable recipes** + +Implement pure recipe validation/replay over sanitized accessibility snapshots. Add executor action/result types without embedding a browser vendor. Production execution accepts only approved recipe version + matching signature; generic exploration is flagged `pilot_or_calibration_only`. + +**Step 4: Register, verify, and commit** + +Run: `npm run smoke:browser-recipes && npm run typecheck` + +```bash +git add package.json packages/longtable-scholar-research/src/browser-recipes.ts packages/longtable-scholar-research/src/index.ts packages/longtable-scholar-research/fixtures/browser scripts/smoke-browser-recipes.mjs +git commit -m "feat(research): add supervised browser recipe replay" +``` + +--- + +## Tranche 6 — Screening and adjudication + +### Task 11: Implement title/abstract and full-text screening ledgers + +**Files:** +- Create: `packages/longtable-scholar-research/src/screening.ts` +- Modify: `packages/longtable-scholar-research/src/index.ts` +- Test: `scripts/smoke-screening.mjs` +- Modify: `package.json` + +**Step 1: Add failing screening tests** + +Test decisions `include`, `exclude`, `pending`, and `full_text_needed_for_screening`; rule/codebook versions; human and AI actor attribution; exclusion reasons; weak/missing abstract routing; batch cache keys; and deterministic candidate counts. + +**Step 2: Verify failure** + +Run: `npm run build && node scripts/smoke-screening.mjs` +Expected: FAIL on screening imports. + +**Step 3: Implement screening records and reducers** + +Implement stable screening decision IDs, append-only decision validation, latest-decision reduction, candidate selection, and full-text missing-threshold calculation. Do not call an LLM from these reducers. + +**Step 4: Register, verify, and commit** + +Run: `npm run smoke:screening && npm run typecheck` + +```bash +git add package.json packages/longtable-scholar-research/src/screening.ts packages/longtable-scholar-research/src/index.ts scripts/smoke-screening.mjs +git commit -m "feat(research): add reproducible screening ledgers" +``` + +### Task 12: Enforce human–AI conflict and method-change checkpoints + +**Files:** +- Modify: `packages/longtable-scholar-research/src/screening.ts` +- Modify: `packages/longtable-scholar-research/src/checkpoints.ts` +- Modify: `packages/longtable-scholar-research/src/workflow-state.ts` +- Test: `scripts/smoke-screening.mjs` + +**Step 1: Add failing conflict tests** + +Assert that human–AI disagreement, ambiguous rules, excess missing full text, analysis-method changes, and corpus freeze requirements create the correct hard stop and cannot advance or resume without the linked decision. + +**Step 2: Verify failure** + +Run: `npm run smoke:screening` +Expected: FAIL on conflict resolution behavior. + +**Step 3: Implement conflict aggregation** + +Bundle repeated conflicts by checkpoint key, retain all affected paper IDs in the audit record, and ask one MCP question. Implement explicit adjudication application that writes a new screening decision rather than mutating prior decisions. + +**Step 4: Verify and commit** + +Run: `npm run smoke:screening && npm run smoke:institutional-checkpoints && npm run typecheck` + +```bash +git add packages/longtable-scholar-research/src/screening.ts packages/longtable-scholar-research/src/checkpoints.ts packages/longtable-scholar-research/src/workflow-state.ts scripts/smoke-screening.mjs +git commit -m "feat(research): hard-stop ambiguous screening decisions" +``` + +--- + +## Tranche 7 — Reporting, PRISMA, references, Word, and journal renderers + +### Task 13: Render deterministic researcher and PRISMA reports + +**Files:** +- Create: `packages/longtable-scholar-research/src/reporting.ts` +- Create: `packages/longtable-scholar-research/src/prisma.ts` +- Modify: `packages/longtable-scholar-research/src/index.ts` +- Test: `scripts/smoke-research-reporting.mjs` +- Modify: `package.json` + +**Step 1: Add failing snapshot tests** + +Build a synthetic completed review and assert researcher report sections, exact database yields, recovered failures, unresolved actions, protocol deviations, analysis readiness, PRISMA count table, and SVG flow. Assert planned runs use future tense and completed runs use past tense. Assert rendering is blocked when invariants fail. + +**Step 2: Verify failure** + +Run: `npm run build && node scripts/smoke-research-reporting.mjs` +Expected: FAIL on renderer imports. + +**Step 3: Implement deterministic renderers** + +Render Markdown, JSON, CSV count table, and SVG from canonical artifacts only. Add ArtifactProvenance entries for every report/table/figure and stable sorting so reruns match byte-for-byte except where the caller explicitly changes the rendering timestamp. + +**Step 4: Register, verify, and commit** + +Run: `npm run smoke:research-reporting && npm run typecheck` + +```bash +git add package.json packages/longtable-scholar-research/src/reporting.ts packages/longtable-scholar-research/src/prisma.ts packages/longtable-scholar-research/src/index.ts scripts/smoke-research-reporting.mjs +git commit -m "feat(research): render auditable researcher and PRISMA reports" +``` + +### Task 14: Render references, manuscript sections, and APA 7 Word packages + +**Files:** +- Create: `packages/longtable-scholar-research/src/references.ts` +- Create: `packages/longtable-scholar-research/src/manuscript.ts` +- Create: `packages/longtable-scholar-research/src/word-renderer.ts` +- Create: `packages/longtable-scholar-research/src/render-profiles.ts` +- Create: `packages/longtable-scholar-research/fixtures/render-profiles/apa7.json` +- Create: `packages/longtable-scholar-research/fixtures/render-profiles/korean-journal.json` +- Modify: `packages/longtable-scholar-research/package.json` +- Modify: `packages/longtable-scholar-research/src/index.ts` +- Test: `scripts/smoke-manuscript-package.mjs` +- Modify: `package.json` + +**Step 1: Inspect and lock the local document runtime** + +Run the workspace dependency loader and `scripts/generate_longtable_bjet_apa7.py` inspection. Record the selected maintained DOCX library and exact package version in `packages/longtable-scholar-research/package.json`; do not invent OOXML when a supported local runtime exists. + +**Step 2: Add failing reference and Word tests** + +Assert deterministic CSL JSON, RIS, BibTeX, APA 7 references, unresolved-citation report, systematic-review Methods/Results, meta-analysis readiness tables, paragraph/table provenance map, valid DOCX ZIP structure, APA 7 memory/template profile, Korean-journal profile, and user-supplied template-profile validation. + +**Step 3: Verify failure** + +Run: `npm run build && node scripts/smoke-manuscript-package.mjs` +Expected: FAIL on renderer imports. + +**Step 4: Implement renderers** + +Generate content only from approved state. Refuse to claim an unexecuted database, unavailable full text, or unapproved analysis. Store hashes and source artifact IDs for every paragraph, table, and figure. Keep the source profile declarative so a target journal can be added without changing research state. + +**Step 5: Register, verify, and commit** + +Run: `npm run smoke:manuscript-package && npm run typecheck` + +```bash +git add package.json package-lock.json packages/longtable-scholar-research/package.json packages/longtable-scholar-research/src packages/longtable-scholar-research/fixtures/render-profiles scripts/smoke-manuscript-package.mjs +git commit -m "feat(research): render APA7 and journal manuscript packages" +``` + +--- + +## Tranche 8 — Failure, security, token-budget, E2E, and live-smoke gates + +### Task 15: Add failure injection, security scans, and model-call budgets + +**Files:** +- Create: `packages/longtable-scholar-research/src/retry.ts` +- Create: `packages/longtable-scholar-research/src/model-budget.ts` +- Create: `scripts/smoke-research-failures.mjs` +- Create: `scripts/audit-research-artifacts.mjs` +- Modify: `package.json` + +**Step 1: Add failing fault tests** + +Inject network timeout, delayed export, partial download, corrupt PDF, empty export, count mismatch, placeholder file, interrupted run, missing DOI, session expiry, result cap, and OneDrive hydration delay. Assert bounded retries for approved retryable events, one aggregated advisory, no retry for hard stops, idempotent resume, and zero model calls for mechanical stages. + +**Step 2: Add failing security tests** + +Seed synthetic credential/cookie/token/auth-header keys, licensed byte markers, and absolute path leaks. Assert audit failure. Assert clean fixtures pass and report exact artifact IDs without printing secret values. + +**Step 3: Implement retry and budget guards** + +Use deterministic capped exponential schedules returned as data rather than sleeping in tests. Implement stage budgets and cache keys from artifact hash + rule/codebook version + prompt version + model version. + +**Step 4: Register, verify, and commit** + +Run: `npm run smoke:research-failures && node scripts/audit-research-artifacts.mjs --fixtures && npm run typecheck` + +```bash +git add package.json packages/longtable-scholar-research/src/retry.ts packages/longtable-scholar-research/src/model-budget.ts scripts/smoke-research-failures.mjs scripts/audit-research-artifacts.mjs +git commit -m "test(research): enforce failure security and token budgets" +``` + +### Task 16: Complete the synthetic review and live institution smoke gate + +**Files:** +- Create: `scripts/e2e-institutional-review.mjs` +- Create: `docs/INSTITUTIONAL-RESEARCH-WORKFLOW.md` +- Modify: `README.md` +- Modify: `Spec.md` +- Modify: `package.json` +- Test: `scripts/e2e-institutional-review.mjs` + +**Step 1: Write the failing end-to-end scenario** + +Run a synthetic request through setup, pilot, protocol checkpoint, frozen production, export audit, title/abstract screening, full-text plan, PDF Vault intake, full-text screening, corpus freeze, analysis readiness, researcher report, PRISMA, references, and Word package. Simulate MCP cancellation followed by numbered fallback, process interruption/resume, and a protocol amendment. Assert no duplicate artifact and deterministic rerender. + +**Step 2: Verify failure** + +Run: `npm run build && node scripts/e2e-institutional-review.mjs` +Expected: FAIL until all layers are connected. + +**Step 3: Wire the integrated coordinator and CLI commands** + +Add `research pilot`, `research freeze`, `research ingest-export`, `research screen`, `research acquire`, `research report`, and `research package` subcommands behind the single surface. The live-smoke command permits 5–20 records, one metadata export, and one or two permitted PDFs only; it writes a receipt and requires researcher approval before setting production eligibility. + +**Step 4: Document the researcher workflow** + +Document login ownership, MCP checkpoints, numbered fallback, artifact locations, resume behavior, PDF Vault configuration, profile calibration, live-smoke limits, researcher final report, manuscript package, and the security boundary. Link the approved design and this implementation plan from `Spec.md`. + +**Step 5: Run full verification** + +Run: + +```bash +npm test +npm run pack:check +node scripts/audit-research-artifacts.mjs --repo . +git diff --check +git status --short +``` + +Expected: all tests pass; no secrets/licensed PDFs/path leaks; only the user's pre-existing `.longtable/state.json` and `CURRENT.md` modifications remain outside implementation commits. + +**Step 6: Commit** + +```bash +git add README.md Spec.md package.json docs/INSTITUTIONAL-RESEARCH-WORKFLOW.md packages/longtable packages/longtable-mcp packages/longtable-scholar-research scripts/e2e-institutional-review.mjs +git commit -m "feat(research): complete institutional review workflow" +``` + +--- + +## Completion Audit + +Before marking the implementation complete: + +1. Map every acceptance criterion in design section 20 to at least one passing automated assertion and one canonical artifact. +2. Confirm every hard-stop code has a creation, MCP/fallback, decision, and resume test. +3. Confirm every deterministic stage has a zero-model-call assertion. +4. Confirm every rendered count, paragraph, table, figure, and reference has ArtifactProvenance. +5. Confirm protocol revisions are frozen, raw exports are immutable after intake, and repeated resume/rerender operations are idempotent. +6. Confirm the Research PDF Vault is the only PDF byte store and project manifests contain references only. +7. Confirm a new institution cannot enter production without calibration, replay fixtures, live smoke, and profile approval. +8. Run `rg -n "TODO|TBD|FIXME|placeholder" packages/longtable-scholar-research packages/longtable-mcp packages/longtable scripts docs/INSTITUTIONAL-RESEARCH-WORKFLOW.md` and resolve any implementation placeholders introduced by this plan. +9. Review `git diff --stat`, `git diff --check`, and all commits while excluding the user's existing `.longtable/state.json` and `CURRENT.md` edits. diff --git a/docs/superpowers/plans/2026-08-07-research-data-extraction.md b/docs/superpowers/plans/2026-08-07-research-data-extraction.md new file mode 100644 index 0000000..33b6d34 --- /dev/null +++ b/docs/superpowers/plans/2026-08-07-research-data-extraction.md @@ -0,0 +1,45 @@ +# Research Data Extraction Implementation Plan + +**Goal:** Make research-variable extraction a first-class, reproducible phase between corpus freeze and analysis. + +**Constraints:** Keep the engine corpus-neutral; persist no credentials or licensed full text; require evidence provenance and DecisionRecords at protected checkpoints; use deterministic code for validation, comparison, conflict detection, adjudication, hashing, and readiness. + +## Task 1: Lock the public contract with failing tests + +- Extend the institutional workflow smoke test with the approved stage sequence and five new hard-stop codes. +- Add `scripts/smoke-data-extraction.mjs` covering profile freeze/hash, required evidence, double extraction, conflict detection, adjudication, dataset freeze, idempotent replay, and pre-analysis blocking. +- Add the targeted smoke command to `package.json` and run it once to record the expected red state. + +## Task 2: Implement the extraction domain + +- Add typed extraction profiles, records, evidence, conflicts, adjudications, reliability summaries, and dataset freezes. +- Canonicalize and SHA-256 hash profiles and frozen datasets. +- Validate field types, categorical values, required fields, evidence provenance, profile hashes, and independent extractor identity. +- Detect field-level conflicts deterministically, adjudicate them, and reject freeze while required work or conflicts remain. +- Export the module and pass the targeted smoke test. + +## Task 3: Integrate stages, checkpoints, and storage + +- Insert extraction plan, pilot, extraction, adjudication, and data-freeze stages after corpus freeze. +- Add the five extraction hard stops to checkpoint and resume coverage. +- Extend the project scaffold with extraction-profile, pilot, extracted, adjudicated, analysis-ready, record, conflict, adjudication, and freeze paths. +- Update model-budget policy so mechanical extraction validation/comparison does not consume model calls. +- Run institutional workflow, checkpoint, project-store/E2E, and data-extraction tests. + +## Task 4: Add one researcher-facing command surface + +- Extend `longtable research` with `freeze-extraction`, `extract`, `adjudicate-extraction`, and `freeze-data` commands using JSON inputs and JSON outputs. +- Ensure each command writes or validates only declared project artifacts and is safe to replay. +- Add CLI/E2E assertions for help text, artifacts, rejection paths, and successful freeze. + +## Task 5: Make reports and manuscripts extraction-aware + +- Add extraction profile/hash, design, reliability, conflicts/adjudications, missingness, and data-freeze provenance to report/manuscript inputs and rendered outputs. +- Update workflow, acceptance-audit, and command documentation. +- Run reporting and manuscript package tests. + +## Task 6: Verify and amend PR #24 + +- Run `npm test`, `npm run pack:check`, `git diff --check`, and secret/path audits. +- Commit cohesive changes, push the existing branch, update the draft PR body with the extraction amendment, and verify CI success. +- Return to the empirical research project only after the product workflow is green. diff --git a/docs/superpowers/specs/2026-08-06-longtable-research-institutional-workflow-design.md b/docs/superpowers/specs/2026-08-06-longtable-research-institutional-workflow-design.md new file mode 100644 index 0000000..f4df05f --- /dev/null +++ b/docs/superpowers/specs/2026-08-06-longtable-research-institutional-workflow-design.md @@ -0,0 +1,571 @@ +# LongTable Research Institutional Workflow Design + +Status: researcher-approved design + +Date: 2026-08-06 + +Product surface: `LongTable Research` + +Repository: `/Users/hosung/dev/LongTable` + +## 1. Executive Decision + +LongTable exposes one researcher-facing scholarly research surface. Internally it routes a capability tree for protocol design, scholarly discovery, institution-aware access, supervised browser execution, local corpus management, screening, systematic review, meta-analysis readiness, synthesis, and publication rendering. + +One surface does not mean one monolithic `SKILL.md`. LongTable core owns research state, Researcher Checkpoints, DecisionRecords, stage transitions, and audit semantics. Database adapters, institution profiles, parsers, browser recipes, Research PDF Vault integration, and renderers remain isolated and testable internal modules. + +The default access model is: + +1. the researcher completes institution login and MFA; +2. LongTable uses the active authenticated browser to perform normal, permitted research actions; +3. LongTable never extracts or persists credentials, cookies, session tokens, or MFA material; +4. access ambiguity, session expiry, CAPTCHA, material protocol change, or failed invariants cause a real hard stop; +5. execution resumes only after a durable researcher decision. + +## 2. Goals + +- Let a researcher request one LongTable Research workflow from question formation through manuscript-ready outputs. +- Support APIs, publisher TDM routes, public scholarly services, institution link resolvers, and supervised computer use without bypassing access controls. +- Adapt to multiple institutions through reusable database adapters and declarative institution profiles. +- Preserve systematic review and meta-analysis reproducibility through pilot separation, protocol freeze, revision history, immutable manifests, and provenance. +- Minimize researcher monitoring through exception-based supervision and deterministic stage audits. +- Minimize model-token use by reserving LLM calls for interpretive research decisions. +- Store PDFs once in Research PDF Vault and reference them from research projects by stable identity and hash. +- Generate researcher reports, PRISMA artifacts, Methods, Results, supplements, reference files, and analysis-readiness packages from the same canonical artifacts. + +## 3. Non-Goals + +- Automating password entry, MFA completion, CAPTCHA solving, cookie export, session extraction, proxy/VPN bypass, or access-control circumvention. +- Treating metadata or abstracts as verified full-text evidence. +- Downloading every identified PDF before title/abstract screening. +- Allowing the model to change a frozen production query without a protocol amendment and researcher approval. +- Making a browser automation transcript, generated Word file, or provider-specific UI the source of truth. +- Requiring continuous researcher observation of normal pagination, export, parsing, download, or audit operations. +- Building a separate always-on autonomous research daemon in the first implementation. + +## 4. Existing Product Boundaries + +The design extends existing LongTable boundaries rather than introducing a parallel product: + +- `@longtable/core` owns provider-neutral research and state contracts. +- `@longtable/mcp` provides structured elicitation transport over durable QuestionRecords and DecisionRecords. +- `@longtable/scholar-research` provides scholarly search and evidence-recovery capabilities. +- `@longtable/research-search` remains a deprecated compatibility package. +- `packages/longtable` owns the unified researcher-facing command surface. +- `.longtable/` remains the canonical project state. +- Research PDF Vault is an external local-first canonical PDF service, not a replacement LongTable state store. + +The current `longtable-research` blanket prohibition on institution-login automation must be refined during implementation. The replacement boundary permits supervised actions inside a researcher-authenticated active browser while continuing to forbid credential automation, cookie reuse, session extraction, CAPTCHA bypass, proxy/VPN bypass, and unapproved access circumvention. + +## 5. Approved Researcher Journey + +```text +research request + -> project and Research Specification recovery + -> adaptive search-strategy proposal + -> MCP Researcher Checkpoint + -> small institution-aware pilot + -> pilot yield and quality report + -> MCP protocol-freeze checkpoint + -> production metadata search and export + -> export audit, normalization, and deduplication + -> title/abstract screening + -> potential-inclusion corpus + -> full-text acquisition plan checkpoint + -> lawful PDF acquisition and Research PDF Vault intake + -> full-text screening and adjudication + -> final-corpus freeze checkpoint + -> systematic review or meta-analysis-readiness workflow + -> synthesis + -> researcher report + -> manuscript and reproducibility package +``` + +Search results are not bulk-downloaded before screening. Records with absent or insufficient abstracts are sent to `full_text_needed_for_screening`; they are not silently excluded. + +## 6. Architecture + +```text +LongTable Research +| ++-- Research Protocol Controller +| +-- question, scope, eligibility, information sources +| +-- pilot and production separation +| +-- protocol revisions and freeze +| ++-- Checkpoint and Hook Engine +| +-- durable QuestionRecord and DecisionRecord +| +-- MCP elicitation first +| +-- blocking state transitions +| ++-- Discovery Capability +| +-- APIs and official TDM +| +-- metadata import/export +| +-- supervised computer use +| ++-- Adapter Registry +| +-- database adapters +| +-- institution profiles +| +-- browser recipes and page signatures +| ++-- Corpus Controller +| +-- parsing, normalization, and deduplication +| +-- acquisition and screening ledgers +| +-- Research PDF Vault integration +| ++-- Review and Analysis +| +-- PRISMA selection flow +| +-- systematic review +| +-- human validation and adjudication +| +-- meta-analysis readiness +| ++-- Reporting and Rendering + +-- researcher final report + +-- Methods, Results, tables, figures, supplements + +-- APA 7 and target-journal renderers +``` + +Internal capabilities are registry entries, not separate researcher-facing skills. The public skill and command surface remains LongTable Research; legacy scholarly-search commands remain compatibility aliases only. + +## 7. Core Domain Records + +### 7.1 ResearchRun + +- `runId` +- `projectId` +- `researchType` +- `stage` +- `status`: `active | blocked | completed | failed | cancelled` +- `protocolRevisionId` +- `institutionProfileId` +- `lastSuccessfulCursor` +- `openExceptions` +- `createdAt` +- `updatedAt` + +### 7.2 ProtocolRevision + +- exact research question and review type +- databases and their roles +- exact queries +- date, language, publication-type, and source filters +- inclusion and exclusion criteria +- retrieval and screening policies +- model-assisted screening policy +- human-validation policy +- analysis intent +- protocol hash +- amendment rationale and linked DecisionRecord + +### 7.3 StageReceipt + +- `runId` +- `stage` +- institution and database +- adapter and recipe versions +- protocol revision and hash +- expected and observed counts +- input and output artifact IDs +- SHA-256 values +- invariant results +- retry summary +- completion status and next stage + +### 7.4 InstitutionProfile + +- institution identifier and locale +- enabled databases +- access modes +- library link resolver configuration +- permitted and prohibited actions +- export formats and result caps +- storage destinations +- checkpoint policy +- profile approval DecisionRecord +- calibration and last-verified dates + +The profile never includes credentials, cookies, authentication headers, session tokens, or MFA material. + +### 7.5 ArtifactProvenance + +- stable artifact ID +- source stage and input IDs +- protocol revision +- generator/parser/renderer version +- local path or canonical paper ID +- SHA-256 +- access basis when applicable +- creation time + +## 8. Execution State Machine + +```text +SETUP + -> PILOT + -> PROTOCOL_CHECKPOINT + -> PRODUCTION_SEARCH + -> EXPORT_AUDIT + -> TITLE_ABSTRACT_SCREENING + -> FULLTEXT_PLAN_CHECKPOINT + -> FULLTEXT_ACQUISITION + -> FULLTEXT_SCREENING + -> CORPUS_FREEZE_CHECKPOINT + -> ANALYSIS + -> SYNTHESIS + -> RESEARCHER_REPORT + -> MANUSCRIPT_PACKAGE +``` + +Every completed transition writes a StageReceipt. A hard-stop event writes the latest safe cursor, changes the run to `blocked`, creates a QuestionRecord, attempts MCP elicitation, and rejects resume until a linked DecisionRecord authorizes the next action. + +### Immediate hard stops + +- `LOGIN_REQUIRED` +- `MFA_OR_CAPTCHA_REQUIRED` +- `SESSION_EXPIRED` +- `TERMS_OR_ACCESS_UNCLEAR` +- `QUERY_DRIFT_DETECTED` +- `DATABASE_RESULT_CAP_REACHED` +- `EXPORT_COUNT_MISMATCH` +- `DOWNLOAD_PATTERN_CHANGED` +- `UI_SIGNATURE_CHANGED` +- `SCREENING_RULE_AMBIGUOUS` +- `HUMAN_AI_SCREENING_CONFLICT` +- `FULLTEXT_MISSING_THRESHOLD_EXCEEDED` +- `ANALYSIS_METHOD_CHANGE` +- `CORPUS_FREEZE_REQUIRED` + +### Locally retryable events + +- transient network timeout +- delayed export generation +- incomplete download +- temporary page-render failure +- OneDrive/File Provider hydration delay + +Retryable events use a bounded retry policy and one summarized advisory record. Repeated identical failures are bundled into one checkpoint rather than interrupting the researcher for every record. + +## 9. MCP-First Researcher Checkpoints + +Required checkpoints use the existing `longtable-state` MCP `elicit_question` surface first. Advisory notices and ordinary progress remain nonblocking logs. If MCP elicitation is unavailable, declined, cancelled, or times out, the same pending QuestionRecord is rendered through the numbered fallback. There is no implicit default answer. + +### Search-strategy card + +LongTable presents one compact proposal containing: + +- review type and objective; +- recommended databases grouped as core, domain, complementary, normalization, and full-text resolution; +- time, language, publication-type, and source filters; +- recall/precision posture; +- known access limitations; +- actions: pilot, modify databases, modify scope, open detailed wizard, or cancel. + +If the researcher requests database modification, a second MCP multi-select question lists databases exposed by the current institution profile and available public adapters. The detailed wizard is progressive disclosure, not the default path. + +### Pilot-freeze card + +After pilot execution, LongTable reports per-database yield, estimated overlap, missing-abstract rate, relevance sample, result caps, query problems, and recommended amendments. The researcher freezes the production protocol or selects a material revision. Production cannot begin without the freeze decision. + +Other required MCP checkpoints cover access ambiguity, full-text acquisition planning, corpus freeze, screening conflicts, and analysis-method commitments. + +## 10. Pilot and Production Separation + +Pilot artifacts and production artifacts are separate. A pilot may explore UI behavior, access routes, synonyms, result caps, and small relevance samples. Pilot results cannot be silently merged into the production corpus. + +Production execution requires an approved ProtocolRevision. Its databases, queries, filters, time boundaries, languages, target types, and result handling are immutable for that revision. Any material change causes a hard stop and creates a new revision linked to a DecisionRecord. + +## 11. Capability, Database Adapter, and Institution Layers + +### 11.1 Provider-neutral research capabilities + +- `open` +- `verifyAuthenticated` +- `submitQuery` +- `applyFilters` +- `readResultCount` +- `exportMetadata` +- `verifyExport` +- `resolveFulltext` +- `downloadPermittedPdf` +- `suspend` +- `resume` + +Unsupported capabilities are explicit; adapters do not improvise an unapproved alternative. + +### 11.2 Database adapters + +Each adapter may include: + +- official API or TDM client; +- export parser; +- browser recipe; +- accessibility-first page signatures; +- result-cap and pagination rules; +- permitted full-text resolution paths; +- versioned fixtures and contract tests. + +API and official TDM paths take precedence. Supervised computer use is used for normal user actions that lack an adequate supported API. + +### 11.3 Institution profiles + +Institution profiles select available database adapters, access modes, library link resolvers, export defaults, institution-specific result caps, storage paths, and checkpoint rules. They do not duplicate database logic. + +### 11.4 New-institution calibration + +```text +researcher login + -> database and resolver discovery + -> small supervised dry run + -> draft profile and recipes + -> researcher MCP approval + -> local fixture and replay tests + -> small live smoke test + -> production eligibility +``` + +Generic computer-use exploration is allowed only in calibration or pilot. Production requires an approved profile and compatible recipe signature. + +## 12. Supervised Computer-Use Contract + +The researcher owns login and MFA. LongTable verifies that the intended database and authenticated page are present, loads the frozen protocol, performs the approved query and filters, reads the result count, exports metadata, verifies the file and parsed row count, writes a StageReceipt, and advances. + +Recipes prefer accessibility roles, labels, headings, and stable semantic landmarks. Coordinates may be a bounded fallback inside a verified page signature; they are not the primary selector. + +LongTable must not guess when a required control, result count, export dialog, full-text notice, access warning, or page signature differs from the approved recipe. It suspends and asks the researcher. + +## 13. Corpus and PDF Ownership + +PDFs are stored once in Research PDF Vault as canonical artifacts. Each research project stores a `pdf-manifest.jsonl` reference containing: + +- `paperId` +- SHA-256 +- canonical Vault-relative local path (resolved from the approved institution storage profile, without leaking a user-home absolute path) +- acquisition method and access basis +- version and source URL identifiers +- project inclusion and screening state + +Licensed PDFs are not copied into each project, committed to Git, embedded in LongTable logs, or redistributed in a reproducibility package. + +The default acquisition sequence is: + +```text +all metadata + -> normalize and deduplicate + -> title/abstract screening + -> potential-inclusion set + -> lawful full-text resolution + -> Research PDF Vault intake + -> full-text screening + -> final corpus +``` + +## 14. Local Project Contract + +```text +research-project/ ++-- protocol/ +| +-- research-question.yaml +| +-- search-protocol.yaml +| +-- inclusion-exclusion.yaml +| +-- database-profiles/ +| +-- amendments/ ++-- data/ +| +-- 00_raw-exports/ +| +-- 01_normalized/ +| +-- 02_deduplicated/ +| +-- 03_title-abstract-screening/ +| +-- 04_fulltext-screening/ +| +-- 05_analysis-ready/ ++-- corpus/ +| +-- papers.jsonl +| +-- acquisition-ledger.jsonl +| +-- pdf-manifest.jsonl +| +-- screening-decisions.jsonl +| +-- manual-action-queue.csv ++-- audit/ +| +-- stage-receipts.jsonl +| +-- checkpoints.jsonl +| +-- decisions.jsonl +| +-- exclusions.jsonl +| +-- deviations.jsonl +| +-- failures.jsonl ++-- reports/ ++-- manuscript/ ++-- analysis/ ++-- references/ ++-- .longtable/research-runs/ +``` + +Raw export files are immutable after intake. Derived layers point to upstream artifact IDs and hashes. + +## 15. Low-Audit and Low-Token Execution + +### Deterministic execution + +Pagination, export, file verification, CSV/RIS/XLS/NBIB parsing, DOI normalization, exact deduplication, checksums, PRISMA counting, tables, and flow rendering use deterministic code. They must not invoke an LLM. + +### Interpretive execution + +LLM use is reserved for query-concept expansion, relevance calibration, ambiguous screening, construct mapping, synthesis, and method reasoning. Material decisions still require the configured human-validation or Researcher Checkpoint policy. + +### Context minimization + +Each model call receives a stage-specific state slice: + +- protocol hash and relevant revision fields; +- current-stage summary; +- bounded record or evidence batch; +- open exceptions; +- required output schema. + +The full browser log, full run history, and unrelated PDFs are not injected. Results are cached by record/PDF hash, rule or codebook version, prompt version, and model version. Unchanged inputs are not reprocessed. + +### Exception-based supervision + +Normal execution records compact receipts without notifying the researcher. Recoverable errors use bounded local retry and advisory aggregation. Only hard-stop events and final reports request attention. + +## 16. Deterministic Audit Invariants + +At minimum: + +```text +identified = normalized + parse_rejected +normalized = unique + duplicate_links +title_abstract_screened = fulltext_candidates + excluded + pending +fulltext_sought = retrieved + not_retrieved +fulltext_assessed = final_included + fulltext_excluded + unresolved +``` + +Additional invariants require every rejection, exclusion, unresolved item, duplicate link, acquired PDF, and protocol deviation to have an explicit basis. PRISMA rendering is blocked if count invariants fail. + +## 17. Researcher Reporting and Publication Rendering + +Machine-readable protocol, corpus, audit, decision, and receipt artifacts are the source of truth. Researcher-facing and publication-facing files are renderings. + +### Researcher report + +- databases and protocol revisions executed; +- per-database yields and export verification; +- normalization, deduplication, screening, retrieval, and final inclusion counts; +- automatic retries and recovered failures; +- unresolved access and screening items; +- protocol deviations and linked decisions; +- analysis readiness; +- required researcher actions; +- generated manuscript and reproducibility artifacts. + +### Systematic-review package + +- information sources; +- exact search strategy; +- selection process; +- automation and human validation; +- full-text access and missing-text handling; +- study-selection Results; +- PRISMA flow and count table; +- database yield, included-study, and full-text exclusion tables; +- PRISMA 2020 and PRISMA-S materials; +- exact search strings and amendments. + +Planned runs render future-tense protocol language. Completed runs render past-tense methods and observed Results. No renderer may claim an unexecuted database, unavailable full text, or unapproved analysis. + +### Meta-analysis readiness package + +- study-level records; +- effect-size candidates and required statistics; +- construct mapping and evidence; +- sample and publication dependency clusters; +- missing-data and author-contact needs; +- extraction codebook; +- analysis-readiness report. + +Analysis-model selection remains a Researcher Checkpoint. + +### Reference and format renderers + +Canonical bibliographic records generate CSL JSON, RIS, BibTeX, APA 7 references, and unresolved-citation reports. Content is then rendered through APA 7, Korean journal, international journal, or user-supplied Word-template profiles. Paragraph and table provenance maps link rendered claims and counts to source artifacts and protocol revisions. + +## 18. Security and Rights Boundary + +- No credential, cookie, session-token, MFA, or authentication-header persistence. +- No CAPTCHA solving or access-control bypass. +- No production run on an unapproved institution profile or incompatible page signature. +- No licensed full-text redistribution. +- No model upload of restricted/Red-lane content unless a separate explicit policy and access decision permits it. +- Secrets stay in approved credential storage and never enter `.longtable`, manifests, transcripts, fixtures, or repository history. +- Access basis and terms reference are recorded without recording authentication material. + +## 19. Verification Strategy + +### Local automated tests + +1. provider-neutral capability contract tests; +2. export-parser golden tests using sanitized XLS, CSV, RIS, NBIB, and JSON fixtures; +3. browser-recipe replay against accessibility snapshots and local HTML fixtures; +4. hard-stop, MCP cancellation, fallback, decision, and resume tests; +5. count-invariant and property tests; +6. network, partial-download, corrupt-PDF, empty-export, count-mismatch, placeholder-file, interrupted-run, DOI-missing, session-expiry, and result-cap failure injection; +7. secret, licensed-content, and path-leak scans; +8. model-call budget tests; +9. complete synthetic review from request through Word package; +10. deterministic rerender and provenance checks. + +### Model-call budgets + +- pagination: zero model calls; +- export and parsing: zero; +- DOI deduplication: zero; +- PRISMA counting: zero; +- search-strategy proposal: at most one per unchanged protocol draft; +- pilot interpretation: at most one per completed pilot snapshot; +- screening: only configured batches and ambiguous items, with cache reuse. + +### Live institution smoke gate + +After local tests pass, the researcher logs in and approves a small smoke run: 5-20 records per database, one metadata export, one or two permitted PDFs, Research PDF Vault intake, and report rendering. Bulk production is prohibited until the institution profile and recipe receive explicit approval. + +## 20. Acceptance Criteria + +- Only one LongTable Research surface is exposed for the integrated workflow. +- Existing MCP elicitation is the preferred transport for required checkpoints. +- No hard-stop run can resume without a DecisionRecord. +- Approved protocol revisions are immutable during production. +- Repetitive mechanical stages consume zero model calls. +- Interrupted runs resume at the last safe cursor without duplicating artifacts. +- PRISMA and corpus count invariants pass before rendering. +- Every material record, exclusion, acquisition, and manuscript count has provenance. +- No credentials, browser sessions, secrets, or licensed PDFs enter LongTable state or Git. +- Research PDF Vault is the only canonical PDF store. +- Researcher final reports and manuscript-ready Word artifacts are generated from canonical machine-readable state. +- Rerendering the same approved state produces identical counts and traceable content. +- A new institution cannot enter production without calibration, tests, a live smoke run, and researcher approval. + +## 21. Researcher-Approved Decisions + +- `decision_msh9nbsk_fc52od`: researcher login followed by supervised automation. +- `decision_msh9pddd_zrc6lj`: pilot, checkpoint, then frozen production protocol. +- `decision_msh9rfum_kjz6v1`: screen metadata before potential-inclusion PDF acquisition and generate manuscript-ready outputs. +- `decision_msh9up1f_r65s1d`: Research PDF Vault is the single canonical PDF store. +- `decision_msh9wf2w_c8ovx7`: one surface with an internal capability tree. +- `decision_msh9zxqn_h63enn`: receipt-based state machine with real hard stops. +- `decision_msha2raa_0ewqpq`: adaptive single-card MCP search UX. +- `decision_msha6r03_qtd2gx`: required checkpoints use MCP first; advisory information is nonblocking. +- `decision_msha91su_rc5qiu`: capability, database, and institution adapter layers with calibration. +- `decision_mshaayqg_4jul4u`: deterministic, exception-supervised, low-token execution. +- `decision_mshahg8d_ryn4tx`: canonical artifacts with multi-profile researcher and manuscript rendering. +- `decision_mshancvj_tz6i1l`: fixture-first verification with a live institution smoke gate. + +## 22. Implementation Decomposition Boundary + +This design is intentionally broader than one safe implementation change. The implementation plan must decompose it into independently verifiable tranches while preserving the single product surface: + +1. core run, protocol, receipt, adapter, and checkpoint contracts; +2. MCP search-strategy and freeze UX; +3. deterministic corpus/export pipeline and local artifact contract; +4. Research PDF Vault integration; +5. browser executor, database recipes, and institution calibration; +6. screening and adjudication workflow; +7. reporting, PRISMA, reference, Word, and journal renderers; +8. fixture, failure, security, token-budget, E2E, and live-smoke gates. + +Each tranche requires its own implementation plan and evidence-based verification. No tranche may weaken the approved access, checkpoint, provenance, or token-budget contracts to simplify delivery. diff --git a/docs/superpowers/specs/2026-08-07-research-data-extraction-design.md b/docs/superpowers/specs/2026-08-07-research-data-extraction-design.md new file mode 100644 index 0000000..21e5cd4 --- /dev/null +++ b/docs/superpowers/specs/2026-08-07-research-data-extraction-design.md @@ -0,0 +1,72 @@ +# Research Data Extraction Design + +**Status:** Approved by the researcher on 2026-08-07 +**Scope:** Extend the institutional research workflow in PR #24 so metadata acquisition and research-variable extraction are distinct, auditable phases. + +## Problem + +The current workflow freezes the eligible corpus and moves directly to analysis. That preserves source acquisition but does not prove how study- or document-level variables were extracted, compared between extractors, adjudicated, or frozen. Metadata is not research data: citation, procurement, and course metadata identify units; research data are the coded variables used in tables, graphs, models, and manuscript claims. + +## Approved workflow + +```text +metadata import and source-document acquisition +-> corpus screening and corpus freeze +-> extraction profile/codebook checkpoint +-> extraction pilot +-> reliability checkpoint +-> double extraction +-> conflict detection and adjudication +-> extracted-data freeze +-> analysis and synthesis +``` + +The engine is corpus-neutral. An `ExtractionProfile` declares the unit of analysis, field definitions, allowed values, evidence requirements, version, approving DecisionRecord, and deterministic hash. A profile may represent scholarly studies, procurement documents, training courses, or another declared corpus; current-study vocabulary must not be hard-coded into the runtime. + +## Data model + +- `ExtractionFieldDefinition`: stable field ID, label, value type, required flag, allowed values, and whether source evidence is mandatory. +- `ExtractionProfile`: stable ID/version, corpus type, unit of analysis, fields, approval provenance, frozen timestamp, and SHA-256 profile hash. +- `ExtractionRecord`: unit ID, profile/hash, extractor identity and human/AI type, field values, evidence spans, source artifact IDs, timestamp, and lifecycle status. +- `ExtractionConflict`: unit/field pair, compared record IDs, conflicting values, and open/resolved state. +- `ExtractionAdjudication`: conflict ID, final value, rationale, adjudicator, evidence, timestamp, and DecisionRecord when the resolution changes a protected rule. +- `ExtractedDatasetFreeze`: profile hash, included adjudicated records, unresolved-conflict count, completeness and reliability results, artifact hash, approving DecisionRecord, and freeze timestamp. + +Raw licensed full text, browser credentials, cookies, and session tokens remain outside `.longtable`. Extraction records keep bounded evidence locators or short evidence spans and artifact provenance. + +## Invariants and hard stops + +Analysis cannot begin unless: + +1. the extraction profile is frozen and linked to a DecisionRecord; +2. the pilot meets the declared reliability threshold; +3. required double extraction is complete; +4. required fields and evidence are present; +5. every material conflict is adjudicated; and +6. the extracted dataset is frozen with a deterministic hash. + +The workflow adds hard stops for an unfrozen schema, ambiguous extraction rule, below-threshold reliability, unresolved double-extraction conflict, and missing extracted-data freeze. Resume requires an answer-linked DecisionRecord through the existing checkpoint system. + +## Storage layout + +The scaffold adds durable extraction paths without silently treating screened documents as analysis-ready data: + +```text +protocol/extraction-profiles/ +data/05_extraction-pilot/ +data/06_extracted/ +data/07_adjudicated/ +data/08_analysis-ready/ +corpus/extraction-records.jsonl +corpus/extraction-conflicts.jsonl +corpus/extraction-adjudications.jsonl +audit/extracted-data-freezes.jsonl +``` + +## Researcher-facing surface + +The single `longtable research` surface gains commands to freeze an extraction profile, validate/import extraction records, adjudicate conflicts, and freeze the extracted dataset. Reports and manuscript packages state the extraction profile/hash, extractor design, reliability statistic/threshold, conflicts, adjudications, missingness, and data-freeze identifier so the Methods and Results sections can reuse the audit trail. + +## Verification + +Tests must prove deterministic profile and dataset hashes; required evidence validation; independent record comparison; conflict and adjudication behavior; hard-stop coverage and decision-linked resume; idempotent replay; scaffold paths; and rejection of analysis before extracted-data freeze. Existing workflow, credential-safety, export, screening, reporting, packaging, and full E2E tests must remain green. diff --git a/package-lock.json b/package-lock.json index 2918447..f82195d 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1397,6 +1397,9 @@ "name": "@longtable/scholar-research", "version": "0.1.72", "license": "MIT", + "dependencies": { + "@longtable/core": "0.1.72" + }, "devDependencies": { "@types/node": "^22.10.1", "typescript": "^5.6.0" diff --git a/package.json b/package.json index fdb27e6..c7005db 100644 --- a/package.json +++ b/package.json @@ -21,7 +21,20 @@ "smoke:grilling-interview": "node scripts/e2e-grilling-interview-flow.mjs", "smoke:search": "node scripts/smoke-search.mjs", "smoke:scholar-research": "node scripts/smoke-scholar-research.mjs", - "test": "npm run build && npm run smoke:setup && npm run smoke:checkpoints && npm run smoke:hooks && npm run smoke:panel-handoff && npm run smoke:panel-workers && npm run smoke:research-spec-audit && npm run smoke:question-audit && npm run smoke:role-audit && npm run smoke:human-persona && npm run smoke:grilling-interview && npm run smoke:search && npm run smoke:scholar-research && npm run typecheck", + "smoke:institutional-workflow": "node scripts/smoke-institutional-workflow.mjs", + "smoke:institutional-checkpoints": "node scripts/smoke-institutional-checkpoints.mjs", + "smoke:export-parsers": "node scripts/smoke-export-parsers.mjs", + "smoke:corpus-pipeline": "node scripts/smoke-corpus-pipeline.mjs", + "smoke:pdf-vault": "node scripts/smoke-pdf-vault.mjs", + "smoke:institution-adapters": "node scripts/smoke-institution-adapters.mjs", + "smoke:browser-recipes": "node scripts/smoke-browser-recipes.mjs", + "smoke:screening": "node scripts/smoke-screening.mjs", + "smoke:data-extraction": "node scripts/smoke-data-extraction.mjs", + "smoke:research-reporting": "node scripts/smoke-research-reporting.mjs", + "smoke:manuscript-package": "node scripts/smoke-manuscript-package.mjs", + "smoke:research-failures": "node scripts/smoke-research-failures.mjs", + "e2e:institutional-review": "node scripts/e2e-institutional-review.mjs", + "test": "npm run build && npm run smoke:setup && npm run smoke:checkpoints && npm run smoke:hooks && npm run smoke:panel-handoff && npm run smoke:panel-workers && npm run smoke:research-spec-audit && npm run smoke:question-audit && npm run smoke:role-audit && npm run smoke:human-persona && npm run smoke:grilling-interview && npm run smoke:search && npm run smoke:scholar-research && npm run smoke:institutional-workflow && npm run smoke:institutional-checkpoints && npm run smoke:export-parsers && npm run smoke:corpus-pipeline && npm run smoke:pdf-vault && npm run smoke:institution-adapters && npm run smoke:browser-recipes && npm run smoke:screening && npm run smoke:data-extraction && npm run smoke:research-reporting && npm run smoke:manuscript-package && npm run smoke:research-failures && npm run e2e:institutional-review && npm run typecheck", "pack:check": "npm pack --workspaces --dry-run", "release:check": "npm run test && npm run pack:check" }, diff --git a/packages/longtable-mcp/src/server.ts b/packages/longtable-mcp/src/server.ts index aa95d37..1bf8d4e 100644 --- a/packages/longtable-mcp/src/server.ts +++ b/packages/longtable-mcp/src/server.ts @@ -12,6 +12,7 @@ import { classifyCheckpointTrigger } from "@longtable/checkpoints"; import { renderQuestionRecordInput } from "@longtable/provider-claude"; import { renderQuestionRecordPrompt } from "@longtable/provider-codex"; import type { + HardStopScope, ProviderKind, QuestionCommitmentFamily, QuestionEpistemicBasis, @@ -259,6 +260,15 @@ const epistemicBasisSchema = z.enum([ "mixed" ]); +const hardStopScopeSchema = z.enum([ + "research_question", + "scope", + "construct", + "method", + "evidence", + "protected_decision" +]); + const firstResearchShapeSchema = z.object({ handle: z.string().min(1), currentGoal: z.string().min(1), @@ -2236,11 +2246,13 @@ export function createLongTableMcpServer(): McpServer { displayReason: z.string().optional(), provider: z.enum(["codex", "claude"]).optional(), required: z.boolean().optional(), + hardStop: z.boolean().optional(), + hardStopScope: hardStopScopeSchema.optional(), commitmentFamily: commitmentFamilySchema.optional(), epistemicBasis: epistemicBasisSchema.optional() }) }, - async ({ cwd: inputCwd, prompt, title, question, type, checkpointKey, options, allowOther, otherLabel, displayReason, provider, required, commitmentFamily, epistemicBasis }) => { + async ({ cwd: inputCwd, prompt, title, question, type, checkpointKey, options, allowOther, otherLabel, displayReason, provider, required, hardStop, hardStopScope, commitmentFamily, epistemicBasis }) => { try { const context = await requireContext(inputCwd); const result = await createWorkspaceQuestion({ @@ -2256,6 +2268,8 @@ export function createLongTableMcpServer(): McpServer { displayReason, provider, required, + hardStop, + hardStopScope: hardStopScope as HardStopScope | undefined, commitmentFamily: commitmentFamily as QuestionCommitmentFamily | undefined, epistemicBasis: epistemicBasis as QuestionEpistemicBasis | undefined }); @@ -2286,12 +2300,14 @@ export function createLongTableMcpServer(): McpServer { displayReason: z.string().optional(), provider: z.enum(["codex", "claude"]).default("codex"), required: z.boolean().optional(), + hardStop: z.boolean().optional(), + hardStopScope: hardStopScopeSchema.optional(), commitmentFamily: commitmentFamilySchema.optional(), epistemicBasis: epistemicBasisSchema.optional(), fallbackOnly: z.boolean().default(false).describe("Create and render the checkpoint without calling MCP elicitation.") }) }, - async ({ cwd: inputCwd, prompt, title, question, type, checkpointKey, options, allowOther, otherLabel, displayReason, provider, required, commitmentFamily, epistemicBasis, fallbackOnly }) => { + async ({ cwd: inputCwd, prompt, title, question, type, checkpointKey, options, allowOther, otherLabel, displayReason, provider, required, hardStop, hardStopScope, commitmentFamily, epistemicBasis, fallbackOnly }) => { try { const context = await requireContext(inputCwd); const created = await createWorkspaceQuestion({ @@ -2307,6 +2323,8 @@ export function createLongTableMcpServer(): McpServer { displayReason, provider, required, + hardStop, + hardStopScope: hardStopScope as HardStopScope | undefined, commitmentFamily: commitmentFamily as QuestionCommitmentFamily | undefined, epistemicBasis: epistemicBasis as QuestionEpistemicBasis | undefined }); @@ -2460,7 +2478,15 @@ export async function runStdioServer(): Promise { export async function runLongTableMcpCli(argv = process.argv): Promise { if (argv.includes("--self-test")) { - console.log(JSON.stringify({ name: SERVER_NAME, version: SERVER_VERSION, tools: TOOL_NAMES }, null, 2)); + console.log(JSON.stringify({ + name: SERVER_NAME, + version: SERVER_VERSION, + tools: TOOL_NAMES, + institutionalResearch: { + hardStopInputs: true, + fallbackSurfaces: ["mcp_elicitation", "numbered"] + } + }, null, 2)); return; } diff --git a/packages/longtable-scholar-research/fixtures/browser/signatures.json b/packages/longtable-scholar-research/fixtures/browser/signatures.json new file mode 100644 index 0000000..46c0b49 --- /dev/null +++ b/packages/longtable-scholar-research/fixtures/browser/signatures.json @@ -0,0 +1,22 @@ +{ + "wos_results_v1": { + "databaseId": "wos", + "pageKind": "results", + "titlePattern": "Web of Science Results", + "requiredLandmarks": [ + { "role": "heading", "name": "Search Results" }, + { "role": "status", "name": "Result count" }, + { "role": "button", "name": "Export" } + ] + }, + "wos_export_v1": { + "databaseId": "wos", + "pageKind": "export_dialog", + "titlePattern": "Web of Science Export", + "requiredLandmarks": [ + { "role": "dialog", "name": "Export records" }, + { "role": "combobox", "name": "File format" }, + { "role": "button", "name": "Export records" } + ] + } +} diff --git a/packages/longtable-scholar-research/fixtures/browser/wos-export.html b/packages/longtable-scholar-research/fixtures/browser/wos-export.html new file mode 100644 index 0000000..904ec63 --- /dev/null +++ b/packages/longtable-scholar-research/fixtures/browser/wos-export.html @@ -0,0 +1,11 @@ + + +Web of Science Export + +
+ + + +
+ + diff --git a/packages/longtable-scholar-research/fixtures/browser/wos-results.html b/packages/longtable-scholar-research/fixtures/browser/wos-results.html new file mode 100644 index 0000000..3f4647c --- /dev/null +++ b/packages/longtable-scholar-research/fixtures/browser/wos-results.html @@ -0,0 +1,12 @@ + + +Web of Science Results + +
Institution access active
+
+

Search Results

+
1,234 results
+ +
+ + diff --git a/packages/longtable-scholar-research/fixtures/exports/crossref.json b/packages/longtable-scholar-research/fixtures/exports/crossref.json new file mode 100644 index 0000000..32623f0 --- /dev/null +++ b/packages/longtable-scholar-research/fixtures/exports/crossref.json @@ -0,0 +1,17 @@ +{ + "message": { + "items": [ + { + "DOI": "10.1000/CHALLENGE.4", + "title": ["Challenge weeks as accelerated workplace learning"], + "author": [ + { "given": "Sora", "family": "Choi" } + ], + "published": { "date-parts": [[2021, 6, 1]] }, + "abstract": "An evaluation of a challenge-week protocol.", + "container-title": ["Studies in Continuing Education"], + "URL": "https://doi.org/10.1000/CHALLENGE.4" + } + ] + } +} diff --git a/packages/longtable-scholar-research/fixtures/exports/expected-records.json b/packages/longtable-scholar-research/fixtures/exports/expected-records.json new file mode 100644 index 0000000..1bf0588 --- /dev/null +++ b/packages/longtable-scholar-research/fixtures/exports/expected-records.json @@ -0,0 +1,8 @@ +[ + { "sourceRecordId": "WOS:0001", "title": "Hackathons, capability building, and work", "year": 2024, "doi": "10.1000/hack.1", "authorCount": 2 }, + { "sourceRecordId": "WOS:0002", "title": "직무역량 해커톤 연구", "year": 2025, "authorCount": 2 }, + { "sourceRecordId": "SCOPUS:002", "title": "Design sprints for professional learning", "year": 2023, "doi": "10.1000/sprint.2", "authorCount": 2 }, + { "sourceRecordId": "39000001", "title": "Health hackathons and interprofessional competence", "year": 2022, "doi": "10.1000/health.3", "authorCount": 2 }, + { "sourceRecordId": "10.1000/challenge.4", "title": "Challenge weeks as accelerated workplace learning", "year": 2021, "doi": "10.1000/challenge.4", "authorCount": 1 }, + { "sourceRecordId": "WOS:0005", "title": "Innovation camps and employee capability", "year": 2020, "doi": "10.1000/camp.5", "authorCount": 2 } +] diff --git a/packages/longtable-scholar-research/fixtures/exports/pubmed.nbib b/packages/longtable-scholar-research/fixtures/exports/pubmed.nbib new file mode 100644 index 0000000..a219bef --- /dev/null +++ b/packages/longtable-scholar-research/fixtures/exports/pubmed.nbib @@ -0,0 +1,12 @@ +PMID- 39000001 +TI - Health hackathons and interprofessional competence +FAU - Lee, Hana +FAU - Garcia, Mateo +DP - 2022 Jan +AB - A simulation-based hackathon was evaluated across health professions. + Team competence was assessed before and after the event. +JT - Medical Education +LID - 10.1000/HEALTH.3 [doi] +OT - hackathon +OT - interprofessional competence + diff --git a/packages/longtable-scholar-research/fixtures/exports/scopus.ris b/packages/longtable-scholar-research/fixtures/exports/scopus.ris new file mode 100644 index 0000000..0d73e13 --- /dev/null +++ b/packages/longtable-scholar-research/fixtures/exports/scopus.ris @@ -0,0 +1,13 @@ +TY - JOUR +TI - Design sprints for professional learning +AU - Park, Jisoo +AU - Smith, Alex +PY - 2023 +AB - This study examines intensive design challenges + as a workplace learning protocol. +DO - https://doi.org/10.1000/SPRINT.2 +JO - Human Resource Development Review +AN - SCOPUS:002 +KW - design sprint +KW - professional learning +ER - diff --git a/packages/longtable-scholar-research/fixtures/exports/wos.csv b/packages/longtable-scholar-research/fixtures/exports/wos.csv new file mode 100644 index 0000000..a219155 --- /dev/null +++ b/packages/longtable-scholar-research/fixtures/exports/wos.csv @@ -0,0 +1,3 @@ +Title,Authors,Publication Year,Abstract,DOI,Source Title,Accession Number,Author Keywords +"Hackathons, capability building, and work","Kim, Mina; You, Hosung",2024,"A mixed-method study of workplace competency development.",10.1000/HACK.1,Journal of Workplace Learning,WOS:0001,"hackathon; competency" +"직무역량 해커톤 연구","유호성; 김민아",2025,"한국 조직의 집중형 문제해결 프로그램을 분석하였다.",,한국직업교육연구,WOS:0002,"직무역량; 해커톤" diff --git a/packages/longtable-scholar-research/fixtures/exports/wos.xls b/packages/longtable-scholar-research/fixtures/exports/wos.xls new file mode 100644 index 0000000..7710357 --- /dev/null +++ b/packages/longtable-scholar-research/fixtures/exports/wos.xls @@ -0,0 +1,7 @@ + + + + TitleAuthorsPublication YearAbstractDOISource TitleAccession Number + Innovation camps and employee capabilityNguyen, Linh; Brown, Taylor2020A longitudinal evaluation of an innovation camp.10.1000/CAMP.5Adult Education QuarterlyWOS:0005 +
+ diff --git a/packages/longtable-scholar-research/fixtures/institutions/psu-sanitized.json b/packages/longtable-scholar-research/fixtures/institutions/psu-sanitized.json new file mode 100644 index 0000000..9768f17 --- /dev/null +++ b/packages/longtable-scholar-research/fixtures/institutions/psu-sanitized.json @@ -0,0 +1,33 @@ +{ + "id": "psu-sanitized", + "institutionName": "Pennsylvania State University", + "profileVersion": 1, + "calibrationStatus": "draft_fixture", + "approvedDecisionRecordId": "decision-profile-psu", + "databases": [ + { + "databaseId": "wos", + "accessMode": "browser_sso", + "exportFormats": ["csv", "ris", "spreadsheet_xml"], + "resultCap": 100000, + "recipeId": "wos-accessibility-v1" + }, + { + "databaseId": "scopus", + "accessMode": "browser_sso", + "exportFormats": ["csv", "ris"], + "resultCap": 20000, + "recipeId": "scopus-accessibility-v1" + } + ], + "resolverUrls": ["https://resolver.example.test/openurl"], + "storage": { + "pdfVaultRoot": "/configured/research-pdf-vault", + "exportRoot": "/configured/research-exports" + }, + "checkpointPolicy": "mcp_first", + "permittedActions": ["search", "apply_filters", "export_metadata", "download_permitted_pdf"], + "prohibitedActions": ["persist_credentials", "persist_browser_session", "solve_captcha", "bypass_access_control"], + "calibratedAt": "2026-08-06T00:00:00.000Z", + "lastVerifiedAt": "2026-08-06T00:00:00.000Z" +} diff --git a/packages/longtable-scholar-research/fixtures/render-profiles/apa7.json b/packages/longtable-scholar-research/fixtures/render-profiles/apa7.json new file mode 100644 index 0000000..f493c0b --- /dev/null +++ b/packages/longtable-scholar-research/fixtures/render-profiles/apa7.json @@ -0,0 +1,19 @@ +{ + "id": "apa7-memory", + "label": "APA 7 working manuscript (memory/template tokens)", + "version": 1, + "status": "verified", + "sourceBasis": "User-requested APA 7 memory format; exact journal rules remain external to this profile.", + "page": { "size": "Letter", "marginInches": 1, "headerInches": 0.5, "footerInches": 0.5 }, + "body": { + "latinFont": "Times New Roman", + "eastAsiaFont": "Times New Roman", + "fontSizePt": 12, + "lineSpacing": 2, + "paragraphAfterPt": 0, + "firstLineIndentInches": 0.5 + }, + "references": { "hangingIndentInches": 0.5 }, + "pageNumber": "top_right", + "titlePage": true +} diff --git a/packages/longtable-scholar-research/fixtures/render-profiles/korean-journal.json b/packages/longtable-scholar-research/fixtures/render-profiles/korean-journal.json new file mode 100644 index 0000000..33be3ac --- /dev/null +++ b/packages/longtable-scholar-research/fixtures/render-profiles/korean-journal.json @@ -0,0 +1,19 @@ +{ + "id": "korean-journal-provisional", + "label": "General Korean journal working manuscript", + "version": 1, + "status": "provisional_until_journal_rules_verified", + "sourceBasis": "Generic Korean-language working profile; replace only after journal author instructions and sample articles are verified.", + "page": { "size": "A4", "marginInches": 1, "headerInches": 0.5, "footerInches": 0.5 }, + "body": { + "latinFont": "Times New Roman", + "eastAsiaFont": "Batang", + "fontSizePt": 10.5, + "lineSpacing": 1.6, + "paragraphAfterPt": 0, + "firstLineIndentInches": 0.4 + }, + "references": { "hangingIndentInches": 0.4 }, + "pageNumber": "bottom_center", + "titlePage": false +} diff --git a/packages/longtable-scholar-research/package.json b/packages/longtable-scholar-research/package.json index 0bd1182..7c5abfc 100644 --- a/packages/longtable-scholar-research/package.json +++ b/packages/longtable-scholar-research/package.json @@ -14,12 +14,17 @@ }, "files": [ "dist", + "resources", + "fixtures/render-profiles", "README.md" ], "scripts": { "build": "tsc -p tsconfig.json", "typecheck": "tsc -p tsconfig.json --noEmit" }, + "dependencies": { + "@longtable/core": "0.1.72" + }, "devDependencies": { "@types/node": "^22.10.1", "typescript": "^5.6.0" @@ -43,5 +48,10 @@ }, "engines": { "node": ">=18.0.0" + }, + "longtableDocumentRuntime": { + "library": "python-docx", + "version": "1.2.0", + "pythonModule": "docx" } } diff --git a/packages/longtable-scholar-research/resources/render_word.py b/packages/longtable-scholar-research/resources/render_word.py new file mode 100644 index 0000000..184f632 --- /dev/null +++ b/packages/longtable-scholar-research/resources/render_word.py @@ -0,0 +1,190 @@ +import json +import re +import sys +from pathlib import Path + +from docx import Document +from docx.enum.section import WD_SECTION +from docx.enum.text import WD_ALIGN_PARAGRAPH +from docx.oxml import OxmlElement +from docx.oxml.ns import qn +from docx.shared import Inches, Pt + + +def set_run_font(run, latin, east_asia, size): + run.font.name = latin + run.font.size = Pt(size) + run._element.get_or_add_rPr().rFonts.set(qn("w:ascii"), latin) + run._element.get_or_add_rPr().rFonts.set(qn("w:hAnsi"), latin) + run._element.get_or_add_rPr().rFonts.set(qn("w:eastAsia"), east_asia) + + +def add_page_number(paragraph): + run = paragraph.add_run() + begin = OxmlElement("w:fldChar") + begin.set(qn("w:fldCharType"), "begin") + instruction = OxmlElement("w:instrText") + instruction.set(qn("xml:space"), "preserve") + instruction.text = " PAGE " + separate = OxmlElement("w:fldChar") + separate.set(qn("w:fldCharType"), "separate") + text = OxmlElement("w:t") + text.text = "1" + end = OxmlElement("w:fldChar") + end.set(qn("w:fldCharType"), "end") + for item in (begin, instruction, separate, text, end): + run._r.append(item) + + +def style_document(doc, profile): + section = doc.sections[0] + section.page_width = Inches(8.5 if profile["page"]["size"] == "Letter" else 8.27) + section.page_height = Inches(11 if profile["page"]["size"] == "Letter" else 11.69) + margin = Inches(profile["page"]["marginInches"]) + section.top_margin = section.bottom_margin = section.left_margin = section.right_margin = margin + section.header_distance = Inches(profile["page"]["headerInches"]) + section.footer_distance = Inches(profile["page"]["footerInches"]) + body = profile["body"] + normal = doc.styles["Normal"] + normal.font.name = body["latinFont"] + normal.font.size = Pt(body["fontSizePt"]) + normal._element.get_or_add_rPr().rFonts.set(qn("w:ascii"), body["latinFont"]) + normal._element.get_or_add_rPr().rFonts.set(qn("w:hAnsi"), body["latinFont"]) + normal._element.get_or_add_rPr().rFonts.set(qn("w:eastAsia"), body["eastAsiaFont"]) + normal.paragraph_format.line_spacing = body["lineSpacing"] + normal.paragraph_format.space_after = Pt(body["paragraphAfterPt"]) + for style_name in ("Heading 1", "Heading 2"): + style = doc.styles[style_name] + style.font.name = body["latinFont"] + style.font.size = Pt(body["fontSizePt"]) + style.font.bold = True + style.font.color.rgb = None + style._element.get_or_add_rPr().rFonts.set(qn("w:eastAsia"), body["eastAsiaFont"]) + style.paragraph_format.line_spacing = body["lineSpacing"] + style.paragraph_format.space_before = Pt(12) + style.paragraph_format.space_after = Pt(0) + style.paragraph_format.keep_with_next = True + page_paragraph = section.header.paragraphs[0] if profile["pageNumber"] == "top_right" else section.footer.paragraphs[0] + page_paragraph.alignment = WD_ALIGN_PARAGRAPH.RIGHT if profile["pageNumber"] == "top_right" else WD_ALIGN_PARAGRAPH.CENTER + add_page_number(page_paragraph) + + +def add_centered(doc, text, bold=False): + paragraph = doc.add_paragraph() + paragraph.alignment = WD_ALIGN_PARAGRAPH.CENTER + run = paragraph.add_run(text) + run.bold = bold + return paragraph + + +def add_body(doc, text, profile, indent=True): + paragraph = doc.add_paragraph() + if indent: + paragraph.paragraph_format.first_line_indent = Inches(profile["body"]["firstLineIndentInches"]) + paragraph.add_run(text) + return paragraph + + +def set_table_geometry(table, widths): + table.autofit = False + tbl_pr = table._tbl.tblPr + tbl_w = tbl_pr.first_child_found_in("w:tblW") + tbl_w.set(qn("w:type"), "dxa") + tbl_w.set(qn("w:w"), str(sum(widths))) + tbl_ind = OxmlElement("w:tblInd") + tbl_ind.set(qn("w:type"), "dxa") + tbl_ind.set(qn("w:w"), "120") + tbl_pr.append(tbl_ind) + grid = table._tbl.tblGrid + for child in list(grid): + grid.remove(child) + for width in widths: + col = OxmlElement("w:gridCol") + col.set(qn("w:w"), str(width)) + grid.append(col) + for row in table.rows: + for cell, width in zip(row.cells, widths): + cell.width = Inches(width / 1440) + tc_w = cell._tc.get_or_add_tcPr().first_child_found_in("w:tcW") + tc_w.set(qn("w:type"), "dxa") + tc_w.set(qn("w:w"), str(width)) + + +def add_readiness_table(doc, manuscript): + rows = [ + ("점검 항목", "상태"), + ("포함 연구", f'{len(manuscript["references"])}편'), + ("효과크기 산출 가능성", "승인된 분석계획에 따라 별도 추출"), + ("표본 및 출판물 의존성", "코딩 필요"), + ("결측 및 저자 연락", "코딩 필요"), + ] + table = doc.add_table(rows=len(rows), cols=2) + table.style = "Table Grid" + for row_index, values in enumerate(rows): + for column_index, value in enumerate(values): + table.cell(row_index, column_index).text = value + if row_index == 0: + for run in table.cell(row_index, column_index).paragraphs[0].runs: + run.bold = True + set_table_geometry(table, [3000, 6360]) + + +def add_markdown_italic_runs(paragraph, text): + for index, part in enumerate(re.split(r"\*([^*]+)\*", text)): + if not part: + continue + run = paragraph.add_run(part) + run.italic = index % 2 == 1 + + +def build(payload, output_path): + manuscript = payload["manuscript"] + profile = payload["profile"] + template_path = profile.get("templatePath") + doc = Document(template_path) if template_path else Document() + style_document(doc, profile) + if profile["titlePage"]: + for _ in range(5): + doc.add_paragraph() + add_centered(doc, manuscript["title"], bold=True) + doc.add_paragraph() + for line in manuscript["authorLines"]: + add_centered(doc, line) + doc.add_page_break() + else: + add_centered(doc, manuscript["title"], bold=True) + for line in manuscript["authorLines"]: + add_centered(doc, line) + add_centered(doc, "초록" if manuscript["language"] == "ko" else "Abstract", bold=True) + add_body(doc, manuscript["abstract"], profile, indent=False) + keywords = doc.add_paragraph() + label = keywords.add_run("주제어: " if manuscript["language"] == "ko" else "Keywords: ") + label.italic = True + keywords.add_run(", ".join(manuscript["keywords"])) + keywords.paragraph_format.space_after = Pt(12) + for section in manuscript["sections"]: + heading = doc.add_paragraph(style="Heading 1") + heading.alignment = WD_ALIGN_PARAGRAPH.CENTER + heading.paragraph_format.space_before = Pt(12) + heading.paragraph_format.space_after = Pt(0) + heading.paragraph_format.keep_with_next = True + heading.add_run(section["heading"]) + add_body(doc, section["content"], profile) + if section["id"] == "methods": + add_readiness_table(doc, manuscript) + add_centered(doc, "참고문헌" if manuscript["language"] == "ko" else "References", bold=True) + hanging = profile["references"]["hangingIndentInches"] + for reference in manuscript["references"]: + paragraph = doc.add_paragraph() + paragraph.paragraph_format.left_indent = Inches(hanging) + paragraph.paragraph_format.first_line_indent = Inches(-hanging) + add_markdown_italic_runs(paragraph, reference) + doc.core_properties.title = manuscript["title"] + doc.core_properties.subject = f'LongTable protocol {manuscript["protocol"]["id"]}' + doc.core_properties.comments = f'Render profile {profile["id"]}; generated {manuscript["generatedAt"]}' + Path(output_path).parent.mkdir(parents=True, exist_ok=True) + doc.save(output_path) + + +if __name__ == "__main__": + build(json.load(sys.stdin), sys.argv[1]) diff --git a/packages/longtable-scholar-research/src/adapters.ts b/packages/longtable-scholar-research/src/adapters.ts new file mode 100644 index 0000000..0812a7e --- /dev/null +++ b/packages/longtable-scholar-research/src/adapters.ts @@ -0,0 +1,79 @@ +import { + INSTITUTIONAL_RESEARCH_CAPABILITIES, + type CapabilityResult, + type DatabaseAdapter, + type InstitutionalResearchCapability +} from "./workflow-types.js"; + +export type AdapterCapabilityInput = Readonly>; +export type AdapterCapabilityHandler = ( + input: AdapterCapabilityInput +) => Promise>; + +export type ExecutableDatabaseAdapter = DatabaseAdapter & + Partial>; + +export interface DatabaseAdapterRegistry { + readonly get: (id: string) => ExecutableDatabaseAdapter | undefined; + readonly list: () => readonly ExecutableDatabaseAdapter[]; +} + +export function validateDatabaseAdapter(adapter: ExecutableDatabaseAdapter): string[] { + const issues: string[] = []; + if (!adapter.id?.trim()) issues.push("Database adapter ID is required."); + if (!adapter.version?.trim()) issues.push(`Database adapter ${adapter.id || ""} version is required.`); + const seen = new Set(); + for (const capability of adapter.capabilities ?? []) { + if (!INSTITUTIONAL_RESEARCH_CAPABILITIES.includes(capability)) { + issues.push(`Database adapter ${adapter.id} declares unknown capability ${capability}.`); + continue; + } + if (seen.has(capability)) issues.push(`Database adapter ${adapter.id} declares duplicate capability ${capability}.`); + seen.add(capability); + if (typeof adapter[capability] !== "function") { + issues.push(`Database adapter ${adapter.id} declares ${capability} without an implementation.`); + } + } + return issues; +} + +export async function executeAdapterCapability( + adapter: ExecutableDatabaseAdapter, + capability: InstitutionalResearchCapability, + input: AdapterCapabilityInput +): Promise> { + if (!adapter.capabilities.includes(capability)) { + return { + status: "unsupported", + reason: `Database adapter ${adapter.id} does not declare capability ${capability}.` + }; + } + const handler = adapter[capability]; + if (typeof handler !== "function") { + return { + status: "unsupported", + reason: `Database adapter ${adapter.id} declares ${capability} but has no implementation.` + }; + } + return handler(input); +} + +export function createDatabaseAdapterRegistry( + adapters: readonly ExecutableDatabaseAdapter[] +): DatabaseAdapterRegistry { + const byId = new Map(); + for (const adapter of adapters) { + const issues = validateDatabaseAdapter(adapter); + if (issues.length > 0) { + throw new Error(issues.join(" ")); + } + if (byId.has(adapter.id)) { + throw new Error(`Duplicate database adapter ID: ${adapter.id}.`); + } + byId.set(adapter.id, adapter); + } + return { + get: (id) => byId.get(id), + list: () => [...byId.values()].sort((left, right) => left.id.localeCompare(right.id)) + }; +} diff --git a/packages/longtable-scholar-research/src/browser-recipes.ts b/packages/longtable-scholar-research/src/browser-recipes.ts new file mode 100644 index 0000000..be0b277 --- /dev/null +++ b/packages/longtable-scholar-research/src/browser-recipes.ts @@ -0,0 +1,323 @@ +import { createHash } from "node:crypto"; +import type { + CapabilityResult, + InstitutionalResearchCapability, + InstitutionalResearchHardStop +} from "./workflow-types.js"; + +export interface BrowserAccessibilityElement { + readonly role: string; + readonly name: string; + readonly value: string; +} + +export interface BrowserPageSnapshot { + readonly url: string; + readonly title: string; + readonly authenticated: boolean; + readonly text: string; + readonly elements: readonly BrowserAccessibilityElement[]; +} + +export interface BrowserSignatureLandmark { + readonly role: string; + readonly name: string; +} + +export interface BrowserSignatureProfile { + readonly databaseId: string; + readonly pageKind: string; + readonly titlePattern: string; + readonly requiredLandmarks: readonly BrowserSignatureLandmark[]; +} + +export interface BrowserPageSignature { + readonly hash: string; + readonly compatible: boolean; + readonly missingLandmarks: readonly BrowserSignatureLandmark[]; +} + +export interface SemanticBrowserSelector { + readonly role: string; + readonly name: string; +} + +export interface CoordinateFallback { + readonly x: number; + readonly y: number; + readonly requiredSignatureHash: string; +} + +export interface BrowserRecipeStep { + readonly capability: InstitutionalResearchCapability; + readonly selector?: SemanticBrowserSelector; + readonly coordinateFallback?: CoordinateFallback; +} + +export interface BrowserRecipe { + readonly id: string; + readonly version: string; + readonly databaseId: string; + readonly pageKind: string; + readonly approvedSignatureHash: string; + readonly productionEligible: boolean; + readonly steps: readonly BrowserRecipeStep[]; +} + +export interface BrowserReplayOptions { + readonly mode: "calibration" | "pilot" | "production"; + readonly resultCap: number; +} + +export interface BrowserReplayAction { + readonly capability: InstitutionalResearchCapability; + readonly selector?: SemanticBrowserSelector; + readonly coordinateFallback?: CoordinateFallback; +} + +export interface BrowserReplayResult { + readonly status: "ready" | "blocked"; + readonly actions: readonly BrowserReplayAction[]; + readonly observations: { readonly resultCount?: number }; + readonly hardStop?: Extract, { status: "hard_stop" }>; +} + +export interface DownloadObservation { + readonly filename: string; + readonly mimeType: string; +} + +export interface DownloadPattern { + readonly filenamePattern: RegExp; + readonly mimeTypes: readonly string[]; +} + +function decodeEntities(value: string): string { + return value + .replace(/&/gi, "&") + .replace(/</gi, "<") + .replace(/>/gi, ">") + .replace(/"/gi, "\"") + .replace(/'|'/gi, "'"); +} + +function cleanText(value: string): string { + return decodeEntities(value) + .replace(/]*>[\s\S]*?<\/script>/gi, " ") + .replace(/]*>[\s\S]*?<\/style>/gi, " ") + .replace(/<[^>]+>/g, " ") + .replace(/\s+/g, " ") + .trim(); +} + +function attribute(attributes: string, name: string): string | undefined { + const match = attributes.match(new RegExp(`\\b${name}\\s*=\\s*["']([^"']*)["']`, "i")); + return match ? decodeEntities(match[1]).trim() : undefined; +} + +function implicitRole(tag: string, attributes: string): string | undefined { + const explicit = attribute(attributes, "role"); + if (explicit) return explicit.toLowerCase(); + if (/^h[1-6]$/i.test(tag)) return "heading"; + if (tag.toLowerCase() === "button") return "button"; + if (tag.toLowerCase() === "select") return "combobox"; + if (tag.toLowerCase() === "input") return "textbox"; + return undefined; +} + +function accessibleElements(html: string): BrowserAccessibilityElement[] { + const elements: BrowserAccessibilityElement[] = []; + const tags = ["button", "h1", "h2", "h3", "h4", "h5", "h6", "div", "span", "select", "input"]; + for (const tag of tags) { + const paired = new RegExp(`<${tag}\\b([^>]*)>([\\s\\S]*?)<\\/${tag}>`, "gi"); + for (const match of html.matchAll(paired)) { + const role = implicitRole(tag, match[1]); + if (!role) continue; + const name = attribute(match[1], "aria-label") ?? cleanText(match[2]); + if (!name) continue; + elements.push({ role, name, value: cleanText(match[2]) }); + } + if (tag === "input") { + const single = /]*)\/?\s*>/gi; + for (const match of html.matchAll(single)) { + const role = implicitRole(tag, match[1]); + const name = attribute(match[1], "aria-label") ?? attribute(match[1], "name") ?? attribute(match[1], "type") ?? "input"; + if (role) elements.push({ role, name, value: attribute(match[1], "value") ?? "" }); + } + } + } + return elements.sort((left, right) => + left.role.localeCompare(right.role) || left.name.localeCompare(right.name) || left.value.localeCompare(right.value) + ); +} + +export function buildBrowserPageSnapshot(input: { readonly url: string; readonly html: string }): BrowserPageSnapshot { + const title = cleanText(input.html.match(/]*>([\s\S]*?)<\/title>/i)?.[1] ?? ""); + const authenticated = /data-authenticated\s*=\s*["']true["']/i.test(input.html) || /aria-label\s*=\s*["']Authenticated institution["']/i.test(input.html); + return { + url: input.url, + title, + authenticated, + text: cleanText(input.html), + elements: accessibleElements(input.html) + }; +} + +function structuralHash(snapshot: BrowserPageSnapshot): string { + const parsedUrl = new URL(snapshot.url); + const canonical = { + origin: parsedUrl.origin, + path: parsedUrl.pathname.replace(/\/+$/, "") || "/", + title: snapshot.title, + authenticated: snapshot.authenticated, + elements: snapshot.elements.map(({ role, name, value }) => ({ role, name, value })) + }; + return createHash("sha256").update(JSON.stringify(canonical)).digest("hex"); +} + +function hasLandmark(snapshot: BrowserPageSnapshot, landmark: BrowserSignatureLandmark): boolean { + return snapshot.elements.some((element) => + element.role.toLowerCase() === landmark.role.toLowerCase() && + element.name.toLowerCase() === landmark.name.toLowerCase() + ); +} + +export function computeBrowserPageSignature( + snapshot: BrowserPageSnapshot, + profile: BrowserSignatureProfile +): BrowserPageSignature { + const missingLandmarks = profile.requiredLandmarks.filter((landmark) => !hasLandmark(snapshot, landmark)); + const titleCompatible = snapshot.title.toLowerCase().includes(profile.titlePattern.toLowerCase()); + return { + hash: structuralHash(snapshot), + compatible: titleCompatible && missingLandmarks.length === 0, + missingLandmarks + }; +} + +export function validateBrowserRecipe(recipe: BrowserRecipe): string[] { + const issues: string[] = []; + if (!recipe.id.trim() || !recipe.version.trim() || !recipe.databaseId.trim()) { + issues.push("Browser recipe requires stable id, version, and database id."); + } + if (!/^[a-f0-9]{64}$/i.test(recipe.approvedSignatureHash)) { + issues.push("Browser recipe requires an approved SHA-256 page signature."); + } + for (const [index, step] of recipe.steps.entries()) { + if (!step.selector && !step.coordinateFallback) { + issues.push(`Browser recipe step ${index + 1} needs a semantic selector or bounded coordinate fallback.`); + } + if (step.coordinateFallback && step.coordinateFallback.requiredSignatureHash !== recipe.approvedSignatureHash) { + issues.push(`Browser recipe step ${index + 1} coordinate fallback signature must match the approved page signature.`); + } + } + return issues; +} + +function hardStop(code: InstitutionalResearchHardStop, reason: string): Extract, { status: "hard_stop" }> { + return { status: "hard_stop", code, reason }; +} + +function detectedBlocker(snapshot: BrowserPageSnapshot): Extract, { status: "hard_stop" }> | undefined { + const text = snapshot.text.toLowerCase(); + if (/session expired|session has expired|timed out/.test(text)) { + return hardStop("SESSION_EXPIRED", "The authenticated database session expired."); + } + if (/multi-factor|verification code|captcha|verify you are human/.test(text)) { + return hardStop("MFA_OR_CAPTCHA_REQUIRED", "Researcher action is required for MFA or CAPTCHA."); + } + if (/updated terms|terms of use|accept.*terms/.test(text)) { + return hardStop("TERMS_OR_ACCESS_UNCLEAR", "Terms or access conditions require researcher review."); + } + if (/type=['\"]password|\bsign in\b|\blog in\b/.test(text)) { + return hardStop("LOGIN_REQUIRED", "The researcher must complete database login."); + } + return undefined; +} + +function elementForSelector( + snapshot: BrowserPageSnapshot, + selector: SemanticBrowserSelector +): BrowserAccessibilityElement | undefined { + return snapshot.elements.find((element) => + element.role.toLowerCase() === selector.role.toLowerCase() && + element.name.toLowerCase() === selector.name.toLowerCase() + ); +} + +function parseResultCount(value: string): number | undefined { + const match = value.match(/[\d,]+/); + return match ? Number(match[0].replace(/,/g, "")) : undefined; +} + +export function replayBrowserRecipe( + recipe: BrowserRecipe, + snapshot: BrowserPageSnapshot, + options: BrowserReplayOptions +): BrowserReplayResult { + const blocker = detectedBlocker(snapshot); + if (blocker) return { status: "blocked", actions: [], observations: {}, hardStop: blocker }; + if (options.mode === "production" && !recipe.productionEligible) { + return { status: "blocked", actions: [], observations: {}, hardStop: hardStop("UI_SIGNATURE_CHANGED", "Exploration recipes are calibration/pilot-only and cannot run in production.") }; + } + const currentHash = structuralHash(snapshot); + if (currentHash !== recipe.approvedSignatureHash) { + return { status: "blocked", actions: [], observations: {}, hardStop: hardStop("UI_SIGNATURE_CHANGED", "The current accessibility page signature differs from the approved browser recipe.") }; + } + const actions: BrowserReplayAction[] = []; + let resultCount: number | undefined; + for (const step of recipe.steps) { + const element = step.selector ? elementForSelector(snapshot, step.selector) : undefined; + if (step.selector && !element && !step.coordinateFallback) { + return { status: "blocked", actions, observations: {}, hardStop: hardStop("UI_SIGNATURE_CHANGED", `Required control ${step.selector.role}:${step.selector.name} was not found.`) }; + } + if (step.coordinateFallback && step.coordinateFallback.requiredSignatureHash !== currentHash) { + return { status: "blocked", actions, observations: {}, hardStop: hardStop("UI_SIGNATURE_CHANGED", "Coordinate fallback is not authorized for the current page signature.") }; + } + actions.push({ + capability: step.capability, + ...(step.selector ? { selector: step.selector } : {}), + ...(step.coordinateFallback ? { coordinateFallback: step.coordinateFallback } : {}) + }); + if (step.capability === "readResultCount" && element) { + resultCount = parseResultCount(element.value); + } + } + if (resultCount !== undefined && resultCount > options.resultCap) { + return { + status: "blocked", + actions, + observations: { resultCount }, + hardStop: hardStop("DATABASE_RESULT_CAP_REACHED", `Observed ${resultCount} results, above the approved cap ${options.resultCap}.`) + }; + } + return { status: "ready", actions, observations: { ...(resultCount !== undefined ? { resultCount } : {}) } }; +} + +function normalizedQuery(value: string): string { + return value.replace(/\s+/g, " ").trim(); +} + +export function verifySubmittedQuery(expected: string, observed: string): CapabilityResult { + return normalizedQuery(expected) === normalizedQuery(observed) + ? { status: "supported", value: observed } + : hardStop("QUERY_DRIFT_DETECTED", "The submitted database query differs from the frozen protocol query."); +} + +export function verifyExportObservation(expectedCount: number, parsedCount: number): CapabilityResult { + return expectedCount === parsedCount + ? { status: "supported", value: parsedCount } + : hardStop("EXPORT_COUNT_MISMATCH", `Database reported ${expectedCount} records but the export parser recovered ${parsedCount}.`); +} + +export function verifyDownloadPattern( + observed: DownloadObservation, + expected: DownloadPattern +): CapabilityResult { + expected.filenamePattern.lastIndex = 0; + const filenameMatches = expected.filenamePattern.test(observed.filename); + const mimeMatches = expected.mimeTypes.includes(observed.mimeType); + return filenameMatches && mimeMatches + ? { status: "supported", value: observed } + : hardStop("DOWNLOAD_PATTERN_CHANGED", `Observed download ${observed.filename} (${observed.mimeType}) does not match the approved pattern.`); +} diff --git a/packages/longtable-scholar-research/src/checkpoint-delivery.ts b/packages/longtable-scholar-research/src/checkpoint-delivery.ts new file mode 100644 index 0000000..c046922 --- /dev/null +++ b/packages/longtable-scholar-research/src/checkpoint-delivery.ts @@ -0,0 +1,41 @@ +import type { QuestionRecord, QuestionTransportStatus } from "@longtable/core"; + +export interface CheckpointDeliveryAttempt { + readonly surface: "mcp_elicitation" | "numbered"; + readonly status: QuestionTransportStatus; + readonly attemptedAt: string; + readonly message?: string; +} + +export interface CheckpointDeliveryRecord { + readonly questionRecordId: string; + readonly checkpointKey: string; + readonly attempts: readonly CheckpointDeliveryAttempt[]; +} + +export function deliverCheckpointWithFallback( + question: QuestionRecord, + input: { + readonly status: "declined" | "unsupported" | "timeout" | "error"; + readonly message: string; + readonly attemptedAt: string; + readonly fallbackAt: string; + } +): { readonly question: QuestionRecord; readonly delivery: CheckpointDeliveryRecord } { + if (question.status !== "pending" || !question.prompt.checkpointKey) { + throw new Error("Only a pending checkpoint with a stable key can use transport fallback."); + } + const attempts: CheckpointDeliveryAttempt[] = [ + { surface: "mcp_elicitation", status: input.status, attemptedAt: input.attemptedAt, message: input.message }, + { surface: "numbered", status: "fallback_rendered", attemptedAt: input.fallbackAt, message: "MCP did not yield a decision; the same checkpoint was rendered as numbered text." } + ]; + return { + question: { + ...question, + updatedAt: input.fallbackAt, + transportStatus: { surface: "numbered", status: "fallback_rendered", updatedAt: input.fallbackAt, message: attempts[1].message } + }, + delivery: { questionRecordId: question.id, checkpointKey: question.prompt.checkpointKey, attempts } + }; +} + diff --git a/packages/longtable-scholar-research/src/checkpoints.ts b/packages/longtable-scholar-research/src/checkpoints.ts new file mode 100644 index 0000000..34a4917 --- /dev/null +++ b/packages/longtable-scholar-research/src/checkpoints.ts @@ -0,0 +1,437 @@ +import { createHash } from "node:crypto"; +import type { + HardStopScope, + QuestionCommitmentFamily, + QuestionEpistemicBasis, + QuestionOption, + QuestionPromptType, + QuestionRecord +} from "@longtable/core"; +import type { InstitutionalResearchHardStop, InstitutionalResearchStage } from "./workflow-types.js"; + +interface CheckpointIdentity { + readonly runId: string; + readonly protocolRevisionId?: string; + readonly protocolReference?: string; + readonly createdAt?: string; +} + +export interface SearchStrategyCheckpointInput extends CheckpointIdentity { + readonly reviewType: string; + readonly objective: string; + readonly databaseGroups: { + readonly core: readonly string[]; + readonly domain: readonly string[]; + readonly complementary: readonly string[]; + readonly normalization: readonly string[]; + readonly fulltextResolution: readonly string[]; + }; + readonly filters: { + readonly years?: readonly [number, number]; + readonly languages: readonly string[]; + readonly publicationTypes: readonly string[]; + }; + readonly recallPrecisionPosture: "recall_first" | "balanced" | "precision_first"; + readonly accessLimitations: readonly string[]; +} + +export interface AvailableDatabase { + readonly id: string; + readonly label: string; + readonly group: "core" | "domain" | "complementary" | "normalization" | "fulltext_resolution"; + readonly available: boolean; + readonly limitation?: string; +} + +export interface DatabaseSelectionCheckpointInput extends CheckpointIdentity { + readonly availableDatabases: readonly AvailableDatabase[]; + readonly selectedDatabaseIds: readonly string[]; +} + +export interface PilotFreezeCheckpointInput extends CheckpointIdentity { + readonly databaseYields: Readonly>; + readonly estimatedOverlap: number; + readonly missingAbstractRate: number; + readonly relevanceSample: { readonly relevant: number; readonly reviewed: number }; + readonly resultCaps: readonly string[]; + readonly queryProblems: readonly string[]; + readonly recommendedAmendments: readonly string[]; +} + +export interface AccessAmbiguityCheckpointInput extends CheckpointIdentity { + readonly databaseId: string; + readonly issue: string; +} + +export interface FulltextPlanCheckpointInput extends CheckpointIdentity { + readonly candidateCount: number; + readonly lawfulRoutes: readonly string[]; + readonly unresolvedCount: number; +} + +export interface CorpusFreezeCheckpointInput extends CheckpointIdentity { + readonly includedCount: number; + readonly unresolvedCount: number; +} + +export interface ScreeningConflictCheckpointInput extends CheckpointIdentity { + readonly paperIds: readonly string[]; + readonly conflict: string; +} + +export interface AnalysisMethodCheckpointInput extends CheckpointIdentity { + readonly proposedMethod: string; + readonly rationale: string; +} + +export interface ScreeningRuleAmbiguityCheckpointInput extends CheckpointIdentity { + readonly ruleVersion: string; + readonly ambiguity: string; + readonly affectedPaperIds: readonly string[]; +} + +export interface OperationalHardStopCheckpointInput extends CheckpointIdentity { + readonly stage: InstitutionalResearchStage; + readonly code: InstitutionalResearchHardStop; + readonly issue: string; + readonly safeCursor: string; +} + +interface RequiredCheckpointInput extends CheckpointIdentity { + readonly stage: string; + readonly code: string; + readonly title: string; + readonly question: string; + readonly type: QuestionPromptType; + readonly options: readonly QuestionOption[]; + readonly scope: HardStopScope; + readonly family: QuestionCommitmentFamily; + readonly epistemicBasis: QuestionEpistemicBasis; + readonly displayReason: string; + readonly rationale: readonly string[]; +} + +function requiredText(value: string | undefined, label: string): string { + if (!value?.trim()) { + throw new Error(`${label} is required for a research checkpoint.`); + } + return value.trim(); +} + +function checkpointReference(input: CheckpointIdentity): string { + return requiredText(input.protocolRevisionId ?? input.protocolReference, "Protocol reference"); +} + +export function buildInstitutionalCheckpointKey( + input: CheckpointIdentity & { readonly stage: string; readonly code: string } +): string { + return [ + "institutional", + requiredText(input.runId, "Run ID"), + checkpointReference(input), + requiredText(input.stage, "Research stage"), + requiredText(input.code, "Checkpoint code") + ].join(":"); +} + +function stableRecordId(prefix: string, checkpointKey: string): string { + return `${prefix}_${createHash("sha256").update(checkpointKey).digest("hex").slice(0, 20)}`; +} + +function buildRequiredCheckpoint(input: RequiredCheckpointInput): QuestionRecord { + const checkpointKey = buildInstitutionalCheckpointKey(input); + const createdAt = input.createdAt ?? new Date().toISOString(); + const recordId = stableRecordId("question_record", checkpointKey); + return { + id: recordId, + createdAt, + updatedAt: createdAt, + status: "pending", + hardStop: true, + hardStopScope: input.scope, + commitmentFamily: input.family, + epistemicBasis: input.epistemicBasis, + prompt: { + id: stableRecordId("question_prompt", checkpointKey), + checkpointKey, + title: input.title, + question: input.question, + type: input.type, + options: [...input.options], + allowOther: false, + required: true, + source: "checkpoint", + displayReason: input.displayReason, + rationale: [...input.rationale], + preferredSurfaces: ["mcp_elicitation", "numbered"] + }, + transportStatus: { + surface: "mcp_elicitation", + status: "not_attempted", + updatedAt: createdAt + } + }; +} + +export function buildOperationalHardStopCheckpoint(input: OperationalHardStopCheckpointInput): QuestionRecord { + return buildRequiredCheckpoint({ + ...input, + title: `Research hard stop: ${input.code}`, + question: [ + `Issue: ${requiredText(input.issue, "Hard-stop issue")}`, + `Last safe cursor: ${requiredText(input.safeCursor, "Safe cursor")}`, + "No automated action will be taken. Which recorded resolution should authorize a resume?" + ].join("\n"), + type: "single_choice", + options: [ + { value: "researcher_resolved", label: "Researcher resolved", description: "Record the researcher-owned resolution and resume from the safe cursor." }, + { value: "amend_protocol", label: "Amend protocol", description: "Freeze a new protocol revision before resuming." }, + { value: "exclude_route", label: "Exclude route", description: "Exclude the affected database, record, or acquisition route with rationale." }, + { value: "cancel", label: "Keep blocked", description: "Preserve the checkpoint without an implicit decision." } + ], + scope: "protected_decision", + family: "evidence", + epistemicBasis: "mixed", + displayReason: "An institutional hard stop requires an explicit researcher resolution.", + rationale: ["The last safe cursor is preserved.", "A linked DecisionRecord is required before resume."] + }); +} + +function groupLine(label: string, values: readonly string[]): string { + return `${label}: ${values.length > 0 ? values.join(", ") : "none"}`; +} + +function listLine(label: string, values: readonly string[]): string { + return `${label}: ${values.length > 0 ? values.join(", ") : "none"}`; +} + +export function buildSearchStrategyCheckpoint(input: SearchStrategyCheckpointInput): QuestionRecord { + const yearText = input.filters.years ? `${input.filters.years[0]}–${input.filters.years[1]}` : "not limited"; + const question = [ + `Review type: ${input.reviewType}`, + `Objective: ${input.objective}`, + groupLine("Core databases", input.databaseGroups.core), + groupLine("Domain databases", input.databaseGroups.domain), + groupLine("Complementary databases", input.databaseGroups.complementary), + groupLine("Normalization", input.databaseGroups.normalization), + groupLine("Full-text resolution", input.databaseGroups.fulltextResolution), + `Time: ${yearText}`, + listLine("Languages", input.filters.languages), + listLine("Publication types", input.filters.publicationTypes), + `Recall/precision posture: ${input.recallPrecisionPosture}`, + listLine("Known access limitations", input.accessLimitations), + "How should LongTable proceed?" + ].join("\n"); + return buildRequiredCheckpoint({ + ...input, + stage: "SETUP", + code: "SEARCH_STRATEGY", + title: "Search strategy", + question, + type: "single_choice", + options: [ + { value: "pilot", label: "Run the pilot", description: "Test yield, overlap, missing abstracts, result caps, and relevance before freezing production.", recommended: true }, + { value: "modify_databases", label: "Modify databases", description: "Open a focused multi-select using databases available through the institution profile." }, + { value: "modify_scope", label: "Modify scope", description: "Revise time, language, publication type, or review objective." }, + { value: "open_detailed_wizard", label: "Open detailed wizard", description: "Inspect all strategy fields through progressive disclosure." }, + { value: "cancel", label: "Cancel", description: "Leave the run blocked without an implicit decision." } + ], + scope: "method", + family: "method", + epistemicBasis: "mixed", + displayReason: "The database and scope strategy determines the reproducible pilot frame.", + rationale: ["One compact card is the default researcher surface.", "Production is not authorized by this pilot decision."] + }); +} + +export function buildDatabaseSelectionCheckpoint(input: DatabaseSelectionCheckpointInput): QuestionRecord { + const unavailable = input.availableDatabases.filter((database) => !database.available); + const options = input.availableDatabases + .filter((database) => database.available) + .map((database): QuestionOption => ({ + value: database.id, + label: database.label, + description: `${database.group.replaceAll("_", " ")} database exposed by the current profile.`, + recommended: input.selectedDatabaseIds.includes(database.id) + })); + return buildRequiredCheckpoint({ + ...input, + stage: "SETUP", + code: "DATABASE_SELECTION", + title: "Database selection", + question: `Select the databases for the pilot. Unavailable: ${unavailable.map((database) => `${database.label} (${database.limitation ?? "unavailable"})`).join(", ") || "none"}.`, + type: "multi_choice", + options, + scope: "method", + family: "method", + epistemicBasis: "mixed", + displayReason: "The researcher requested database modification.", + rationale: ["This multi-select is progressive disclosure after modify_databases, not the default search screen."] + }); +} + +function percent(value: number): string { + return `${(value * 100).toFixed(1)}%`; +} + +export function buildPilotFreezeCheckpoint(input: PilotFreezeCheckpointInput): QuestionRecord { + const yields = Object.entries(input.databaseYields) + .sort(([left], [right]) => left.localeCompare(right)) + .map(([database, count]) => `${database}: ${count}`) + .join(", "); + return buildRequiredCheckpoint({ + ...input, + stage: "PROTOCOL_CHECKPOINT", + code: "PILOT_FREEZE", + title: "Pilot results and production freeze", + question: [ + `Database yields: ${yields || "none"}`, + `Estimated overlap: ${percent(input.estimatedOverlap)}`, + `Missing-abstract rate: ${percent(input.missingAbstractRate)}`, + `Relevance sample: ${input.relevanceSample.relevant}/${input.relevanceSample.reviewed}`, + listLine("Result caps", input.resultCaps), + listLine("Query problems", input.queryProblems), + listLine("Recommended amendments", input.recommendedAmendments), + "Freeze this revision for production or revise it?" + ].join("\n"), + type: "single_choice", + options: [ + { value: "freeze_production_protocol", label: "Freeze for production", description: "Make database, query, filter, language, time, and result handling immutable for this revision.", recommended: true }, + { value: "revise_protocol", label: "Revise protocol", description: "Create a material amendment and rerun the pilot checkpoint." }, + { value: "cancel", label: "Cancel", description: "Keep production blocked." } + ], + scope: "method", + family: "method", + epistemicBasis: "mixed", + displayReason: "Production cannot begin until the pilot evidence is reviewed and the protocol is frozen.", + rationale: ["Pilot artifacts remain separate from the production corpus."] + }); +} + +export function buildAccessAmbiguityCheckpoint(input: AccessAmbiguityCheckpointInput): QuestionRecord { + return buildRequiredCheckpoint({ + ...input, + stage: "PRODUCTION_SEARCH", + code: "TERMS_OR_ACCESS_UNCLEAR", + title: "Access or terms clarification", + question: `${input.databaseId} reported an access or terms ambiguity: ${input.issue}`, + type: "single_choice", + options: [ + { value: "researcher_reviewed_and_permitted", label: "Reviewed and permitted", description: "Continue only after the researcher confirms the lawful route." }, + { value: "use_metadata_only", label: "Use metadata only", description: "Do not attempt licensed full-text acquisition." }, + { value: "suspend", label: "Suspend", description: "Keep the run blocked." } + ], + scope: "evidence", + family: "evidence", + epistemicBasis: "researcher_knowledge", + displayReason: "LongTable must not infer access rights or terms acceptance.", + rationale: ["Credentials and browser session material are never requested or persisted."] + }); +} + +export function buildFulltextPlanCheckpoint(input: FulltextPlanCheckpointInput): QuestionRecord { + return buildRequiredCheckpoint({ + ...input, + stage: "FULLTEXT_PLAN_CHECKPOINT", + code: "FULLTEXT_PLAN", + title: "Full-text acquisition plan", + question: `${input.candidateCount} records need full-text resolution through ${input.lawfulRoutes.join(", ") || "no configured route"}; ${input.unresolvedCount} currently have no route.`, + type: "single_choice", + options: [ + { value: "approve_lawful_routes", label: "Approve lawful routes", description: "Acquire only permitted PDFs into Research PDF Vault.", recommended: true }, + { value: "modify_routes", label: "Modify routes", description: "Change resolver or manual-action handling." }, + { value: "suspend", label: "Suspend", description: "Keep acquisition blocked." } + ], + scope: "evidence", + family: "evidence", + epistemicBasis: "mixed", + displayReason: "Potential-inclusion records are screened before PDF acquisition.", + rationale: ["Research PDF Vault is the only canonical PDF store."] + }); +} + +export function buildCorpusFreezeCheckpoint(input: CorpusFreezeCheckpointInput): QuestionRecord { + return buildRequiredCheckpoint({ + ...input, + stage: "CORPUS_FREEZE_CHECKPOINT", + code: "CORPUS_FREEZE_REQUIRED", + title: "Corpus freeze", + question: `${input.includedCount} studies are included and ${input.unresolvedCount} remain unresolved. Freeze the corpus for analysis?`, + type: "single_choice", + options: [ + { value: "freeze_corpus", label: "Freeze corpus", description: "Authorize analysis using the recorded included set.", recommended: input.unresolvedCount === 0 }, + { value: "resolve_remaining", label: "Resolve remaining records", description: "Return to full-text screening." }, + { value: "revise_screening_rules", label: "Revise screening rules", description: "Create a material protocol amendment before rescreening." } + ], + scope: "evidence", + family: "evidence", + epistemicBasis: "mixed", + displayReason: "Analysis requires an explicit, auditable corpus commitment.", + rationale: ["No unresolved record is silently dropped."] + }); +} + +export function buildScreeningConflictCheckpoint(input: ScreeningConflictCheckpointInput): QuestionRecord { + return buildRequiredCheckpoint({ + ...input, + stage: "FULLTEXT_SCREENING", + code: "HUMAN_AI_SCREENING_CONFLICT", + title: "Screening conflict", + question: `${input.conflict}. Affected records: ${input.paperIds.join(", ")}.`, + type: "single_choice", + options: [ + { value: "open_adjudication", label: "Open adjudication", description: "Review the evidence and write a new human adjudication decision.", recommended: true }, + { value: "revise_rule", label: "Revise screening rule", description: "Create a protocol amendment before applying a changed rule." }, + { value: "suspend", label: "Suspend", description: "Keep affected records unresolved." } + ], + scope: "construct", + family: "coding", + epistemicBasis: "mixed", + displayReason: "Human–AI disagreement cannot be resolved by an implicit model default.", + rationale: ["Repeated conflicts are bundled into one researcher checkpoint."] + }); +} + +export function buildAnalysisMethodCheckpoint(input: AnalysisMethodCheckpointInput): QuestionRecord { + return buildRequiredCheckpoint({ + ...input, + stage: "ANALYSIS", + code: "ANALYSIS_METHOD_CHANGE", + title: "Analysis-method commitment", + question: `Proposed method: ${input.proposedMethod}. Rationale: ${input.rationale}.`, + type: "single_choice", + options: [ + { value: "approve_method", label: "Approve method", description: "Record the method commitment and continue.", recommended: true }, + { value: "revise_method", label: "Revise method", description: "Return to the analysis plan without changing the frozen corpus." }, + { value: "cancel", label: "Cancel", description: "Keep analysis blocked." } + ], + scope: "method", + family: "method", + epistemicBasis: "mixed", + displayReason: "Analysis-model selection is a protected researcher decision.", + rationale: ["The renderer cannot claim an unapproved analysis."] + }); +} + +export function buildScreeningRuleAmbiguityCheckpoint( + input: ScreeningRuleAmbiguityCheckpointInput +): QuestionRecord { + return buildRequiredCheckpoint({ + ...input, + stage: "TITLE_ABSTRACT_SCREENING", + code: "SCREENING_RULE_AMBIGUOUS", + title: "Screening-rule ambiguity", + question: `Rule ${input.ruleVersion} is ambiguous: ${input.ambiguity}. Affected records: ${[...input.affectedPaperIds].sort().join(", ")}.`, + type: "single_choice", + options: [ + { value: "clarify_rule", label: "Clarify rule", description: "Create a versioned rule clarification before screening affected records.", recommended: true }, + { value: "open_record_review", label: "Review affected records", description: "Inspect the evidence without silently changing the rule." }, + { value: "suspend", label: "Suspend", description: "Keep affected records pending." } + ], + scope: "construct", + family: "coding", + epistemicBasis: "mixed", + displayReason: "An ambiguous inclusion or exclusion rule can change the final corpus.", + rationale: ["Affected records remain pending until the researcher records a decision."] + }); +} diff --git a/packages/longtable-scholar-research/src/coordinator.ts b/packages/longtable-scholar-research/src/coordinator.ts new file mode 100644 index 0000000..a7487b5 --- /dev/null +++ b/packages/longtable-scholar-research/src/coordinator.ts @@ -0,0 +1,343 @@ +import { constants } from "node:fs"; +import { copyFile, mkdir, readFile, writeFile } from "node:fs/promises"; +import { basename, extname, join, resolve } from "node:path"; +import { buildSearchStrategyCheckpoint } from "./checkpoints.js"; +import { createResearchRun, createStageReceipt } from "./workflow-state.js"; +import { createProtocolRevision } from "./workflow-types.js"; +import { + appendJsonlRecord, + buildResearchProjectLayout, + readJsonlRecords, + writeFrozenProtocolRevision, + writeResearchProjectScaffold +} from "./project-store.js"; +import { parseExportArtifact, type ExportFormat } from "./export-parsers.js"; +import { deduplicateBibliographicRecords, normalizeBibliographicRecords } from "./corpus.js"; +import { createScreeningDecision, type ScreeningDecisionInput } from "./screening.js"; +import { intakeResearchPdf, type PdfAccessBasis, type PdfAcquisitionMethod } from "./pdf-vault.js"; +import { renderResearchOutputs, type ResearchReportInput } from "./reporting.js"; +import { buildApprovedManuscript, type BuildApprovedManuscriptInput } from "./manuscript.js"; +import { validateRenderProfile } from "./render-profiles.js"; +import { renderWordManuscript } from "./word-renderer.js"; +import { + adjudicateExtractionConflict, + createExtractionProfile, + detectExtractionConflicts, + freezeExtractedDataset, + validateExtractionRecord, + type ExtractedDatasetFreezeInput, + type ExtractionAdjudication, + type ExtractionAdjudicationInput, + type ExtractionConflict, + type ExtractionProfile, + type ExtractionProfileInput, + type ExtractionRecord +} from "./extraction.js"; + +export type InstitutionalResearchCommand = "pilot" | "freeze" | "ingest-export" | "screen" | "acquire" | "freeze-extraction" | "extract" | "adjudicate-extraction" | "freeze-data" | "report" | "package" | "live-smoke"; +export const INSTITUTIONAL_RESEARCH_COMMANDS: readonly InstitutionalResearchCommand[] = [ + "pilot", "freeze", "ingest-export", "screen", "acquire", "freeze-extraction", "extract", "adjudicate-extraction", "freeze-data", "report", "package", "live-smoke" +]; +export type InstitutionalResearchCommandArgs = Readonly>; + +function text(args: InstitutionalResearchCommandArgs, key: string): string { + const value = args[key]; + if (typeof value !== "string" || !value.trim()) throw new Error(`Research command requires --${key}.`); + return value.trim(); +} + +function optionalText(args: InstitutionalResearchCommandArgs, key: string): string | undefined { + const value = args[key]; + return typeof value === "string" && value.trim() ? value.trim() : undefined; +} + +async function jsonFile(path: string): Promise { + return JSON.parse(await readFile(resolve(path), "utf8")) as T; +} + +async function writeJson(path: string, value: unknown): Promise { + await mkdir(resolve(path, ".."), { recursive: true }); + await writeFile(path, `${JSON.stringify(value, null, 2)}\n`, "utf8"); +} + +async function writeJsonl(path: string, values: readonly object[]): Promise { + await mkdir(resolve(path, ".."), { recursive: true }); + await writeFile(path, values.length ? `${values.map((value) => JSON.stringify(value)).join("\n")}\n` : "", "utf8"); +} + +async function writeImmutableJson(path: string, value: unknown): Promise { + const content = `${JSON.stringify(value, null, 2)}\n`; + await mkdir(resolve(path, ".."), { recursive: true }); + try { + await writeFile(path, content, { encoding: "utf8", flag: "wx" }); + return true; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error; + if (await readFile(path, "utf8") !== content) throw new Error(`Immutable extraction-profile collision at ${path}.`); + return false; + } +} + +function projectRoot(args: InstitutionalResearchCommandArgs): string { + return resolve(optionalText(args, "cwd") ?? process.cwd()); +} + +async function pilot(args: InstitutionalResearchCommandArgs) { + const root = projectRoot(args); + const layout = await writeResearchProjectScaffold(root); + const createdAt = optionalText(args, "createdAt") ?? new Date().toISOString(); + const runId = text(args, "runId"); + const protocolReference = optionalText(args, "protocolReference") ?? "draft-1"; + const run = createResearchRun({ id: runId, createdAt }); + const checkpoint = buildSearchStrategyCheckpoint({ + runId, + protocolReference, + createdAt, + reviewType: "systematic document analysis", + objective: "Pilot the frozen demand-supply mismatch review frame before production.", + databaseGroups: { + core: ["web-of-science", "scopus"], + domain: ["ERIC", "RISS"], + complementary: [], + normalization: ["Crossref"], + fulltextResolution: ["institutional resolver", "Research PDF Vault"] + }, + filters: { languages: ["Korean", "English"], publicationTypes: ["Article", "Conference Paper"] }, + recallPrecisionPosture: "recall_first", + accessLimitations: ["Researcher owns login, MFA, CAPTCHA, and terms decisions."] + }); + await appendJsonlRecord(layout.audit.checkpoints, checkpoint); + await writeJson(join(layout.researchRuns, `${runId}.json`), run); + return { surface: "longtable-research", run, checkpoint, layout }; +} + +async function freeze(args: InstitutionalResearchCommandArgs) { + const root = projectRoot(args); + const layout = await writeResearchProjectScaffold(root); + const protocol = createProtocolRevision(await jsonFile(text(args, "protocolFile"))); + const path = await writeFrozenProtocolRevision(layout, protocol); + return { protocol, path }; +} + +async function immutableRawExport(path: string, target: string): Promise { + await mkdir(resolve(target, ".."), { recursive: true }); + try { + await copyFile(path, target, constants.COPYFILE_EXCL); + return true; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error; + const [source, existing] = await Promise.all([readFile(path), readFile(target)]); + if (!source.equals(existing)) throw new Error(`Immutable raw-export collision at ${target}.`); + return false; + } +} + +async function ingestExport(args: InstitutionalResearchCommandArgs) { + const root = projectRoot(args); + const layout = await writeResearchProjectScaffold(root); + const path = resolve(text(args, "file")); + const database = text(args, "database"); + const format = text(args, "format") as ExportFormat; + const content = await readFile(path); + const parsed = parseExportArtifact({ filename: basename(path), format, database, content }); + if (parsed.records.length === 0) throw new Error(`Export intake produced no records: ${parsed.rejections.map((entry) => entry.reason).join(", ")}.`); + const extension = extname(path) || `.${format}`; + const rawPath = join(layout.data.rawExports, `${parsed.artifactSha256}${extension}`); + const rawArtifactCreated = await immutableRawExport(path, rawPath); + const normalized = normalizeBibliographicRecords(parsed.records); + const corpus = deduplicateBibliographicRecords(normalized); + await writeJson(join(layout.data.normalized, `${parsed.artifactSha256}.json`), normalized); + await writeJson(join(layout.data.deduplicated, `${parsed.artifactSha256}.json`), corpus); + await writeFile(layout.corpus.papers, corpus.papers.map((paper) => JSON.stringify(paper)).join("\n") + "\n", "utf8"); + return { parsed, corpus, rawPath, rawArtifactCreated }; +} + +async function screen(args: InstitutionalResearchCommandArgs) { + const root = projectRoot(args); + const layout = await writeResearchProjectScaffold(root); + const inputs = await jsonFile(text(args, "decisionsFile")); + const existing = await readJsonlRecords(layout.corpus.screeningDecisions) as unknown as { id: string }[]; + const ids = new Set(existing.map((entry) => entry.id)); + let appended = 0; + for (const input of inputs) { + const decision = createScreeningDecision(input); + if (ids.has(decision.id)) continue; + await appendJsonlRecord(layout.corpus.screeningDecisions, decision); + ids.add(decision.id); + appended += 1; + } + return { appended, total: ids.size }; +} + +async function acquire(args: InstitutionalResearchCommandArgs) { + const root = projectRoot(args); + const layout = await writeResearchProjectScaffold(root); + return intakeResearchPdf({ + projectRoot: root, + vaultRoot: text(args, "vaultRoot"), + projectManifestPath: layout.corpus.pdfManifest, + candidatePath: text(args, "candidatePath"), + paperId: text(args, "paperId"), + acquisitionMethod: text(args, "acquisitionMethod") as PdfAcquisitionMethod, + accessBasis: text(args, "accessBasis") as PdfAccessBasis, + version: text(args, "version"), + projectInclusion: text(args, "projectInclusion"), + screeningState: text(args, "screeningState"), + ...(optionalText(args, "sourceUrl") ? { sourceUrl: optionalText(args, "sourceUrl") } : {}), + ...(optionalText(args, "acquiredAt") ? { acquiredAt: optionalText(args, "acquiredAt") } : {}) + }); +} + +async function freezeExtraction(args: InstitutionalResearchCommandArgs) { + const layout = await writeResearchProjectScaffold(projectRoot(args)); + const profile = createExtractionProfile(await jsonFile(text(args, "profileFile"))); + const path = join(layout.protocol.extractionProfiles, `${profile.id}.json`); + const created = await writeImmutableJson(path, profile); + return { profile, path, created }; +} + +async function loadExtractionProfile(args: InstitutionalResearchCommandArgs): Promise { + const input = await jsonFile(text(args, "profileFile")); + return createExtractionProfile(input); +} + +async function extract(args: InstitutionalResearchCommandArgs) { + const layout = await writeResearchProjectScaffold(projectRoot(args)); + const profile = await loadExtractionProfile(args); + const inputs = await jsonFile(text(args, "recordsFile")); + const existing = await readJsonlRecords(layout.corpus.extractionRecords) as unknown as ExtractionRecord[]; + const byId = new Map(existing.map((record) => [record.id, validateExtractionRecord(profile, record)])); + let appended = 0; + for (const input of inputs) { + const record = validateExtractionRecord(profile, input); + const current = byId.get(record.id); + if (current && JSON.stringify(current) !== JSON.stringify(record)) throw new Error(`Immutable extraction-record collision: ${record.id}.`); + if (!current) { + byId.set(record.id, record); + appended += 1; + } + } + const records = [...byId.values()].sort((a, b) => a.id.localeCompare(b.id)); + const conflicts = detectExtractionConflicts(profile, records); + await Promise.all([ + writeJsonl(layout.corpus.extractionRecords, records), + writeJsonl(layout.corpus.extractionConflicts, conflicts) + ]); + return { appended, total: records.length, conflicts: conflicts.length }; +} + +type AdjudicationFileEntry = ExtractionAdjudicationInput & { readonly conflictId: string }; + +async function adjudicateExtraction(args: InstitutionalResearchCommandArgs) { + const layout = await writeResearchProjectScaffold(projectRoot(args)); + const profile = await loadExtractionProfile(args); + const inputs = await jsonFile(text(args, "adjudicationsFile")); + const conflicts = await readJsonlRecords(layout.corpus.extractionConflicts) as unknown as ExtractionConflict[]; + const existing = await readJsonlRecords(layout.corpus.extractionAdjudications) as unknown as ExtractionAdjudication[]; + const byId = new Map(existing.map((item) => [item.id, item])); + let appended = 0; + for (const { conflictId, ...input } of inputs) { + const conflict = conflicts.find((item) => item.id === conflictId); + if (!conflict) throw new Error(`Unknown extraction conflict: ${conflictId}.`); + const adjudication = adjudicateExtractionConflict(profile, conflict, input); + const current = byId.get(adjudication.id); + if (current && JSON.stringify(current) !== JSON.stringify(adjudication)) throw new Error(`Immutable extraction-adjudication collision: ${adjudication.id}.`); + if (!current) { + byId.set(adjudication.id, adjudication); + appended += 1; + } + } + const adjudications = [...byId.values()].sort((a, b) => a.id.localeCompare(b.id)); + await writeJsonl(layout.corpus.extractionAdjudications, adjudications); + return { appended, total: adjudications.length }; +} + +type FreezeDataFile = Pick; + +async function freezeData(args: InstitutionalResearchCommandArgs) { + const layout = await writeResearchProjectScaffold(projectRoot(args)); + const profile = await loadExtractionProfile(args); + const metadata = await jsonFile(text(args, "freezeFile")); + const records = await readJsonlRecords(layout.corpus.extractionRecords) as unknown as ExtractionRecord[]; + const conflicts = await readJsonlRecords(layout.corpus.extractionConflicts) as unknown as ExtractionConflict[]; + const adjudications = await readJsonlRecords(layout.corpus.extractionAdjudications) as unknown as ExtractionAdjudication[]; + const freeze = freezeExtractedDataset({ ...metadata, profile, records, conflicts, adjudications }); + const existing = await readJsonlRecords(layout.audit.extractedDataFreezes) as unknown as { id: string; datasetHash: string }[]; + const current = existing.find((item) => item.id === freeze.id); + if (current && current.datasetHash !== freeze.datasetHash) throw new Error(`Immutable extracted-data freeze collision: ${freeze.id}.`); + if (!current) await appendJsonlRecord(layout.audit.extractedDataFreezes, freeze); + await writeJson(join(layout.data.analysisReady, `${freeze.datasetHash}.json`), { profile, records, adjudications, freeze }); + return { freeze, created: !current }; +} + +async function report(args: InstitutionalResearchCommandArgs) { + const root = projectRoot(args); + const layout = await writeResearchProjectScaffold(root); + const input = await jsonFile(text(args, "inputFile")); + const outputs = renderResearchOutputs(input); + await Promise.all([ + writeFile(join(layout.reports, "researcher-report.md"), outputs.researcherReportMarkdown, "utf8"), + writeFile(join(layout.reports, "review-flow-counts.md"), outputs.prismaCountTableMarkdown, "utf8"), + writeFile(join(layout.reports, "review-flow.svg"), outputs.prismaFlowSvg, "utf8"), + writeFile(join(layout.reports, "systematic-review.md"), outputs.systematicReviewMarkdown, "utf8"), + writeJson(join(layout.reports, "artifact-provenance.json"), outputs.artifactProvenance) + ]); + return { outputs, reportsDirectory: layout.reports }; +} + +async function packageManuscript(args: InstitutionalResearchCommandArgs) { + const root = projectRoot(args); + const layout = await writeResearchProjectScaffold(root); + const input = await jsonFile(text(args, "inputFile")); + const profile = validateRenderProfile(await jsonFile(text(args, "profileFile"))); + const manuscript = buildApprovedManuscript(input); + const outputPath = resolve(optionalText(args, "outputPath") ?? join(layout.manuscript, "manuscript.docx")); + const rendered = renderWordManuscript({ manuscript, profile, outputPath }); + await writeJson(join(layout.manuscript, `${basename(outputPath, extname(outputPath))}.provenance.json`), manuscript.provenanceMap); + return { manuscript, rendered }; +} + +export function validateLiveSmokeGate(input: { readonly recordCount: number; readonly pdfCount: number; readonly researcherApproved: boolean }): void { + if (!Number.isInteger(input.recordCount) || input.recordCount < 5 || input.recordCount > 20) throw new Error("Live smoke requires 5–20 metadata records."); + if (!Number.isInteger(input.pdfCount) || input.pdfCount < 1 || input.pdfCount > 2) throw new Error("Live smoke requires one or two permitted PDFs."); + if (!input.researcherApproved) throw new Error("Live smoke requires explicit researcher approval."); +} + +async function liveSmoke(args: InstitutionalResearchCommandArgs) { + const root = projectRoot(args); + const layout = await writeResearchProjectScaffold(root); + const recordCount = Number(text(args, "recordCount")); + const pdfCount = Number(text(args, "pdfCount")); + const researcherApproved = args.researcherApproved === true; + validateLiveSmokeGate({ recordCount, pdfCount, researcherApproved }); + const productionEligible = researcherApproved && args.profileApproved === true && args.calibrated === true && args.replayPassed === true; + const completedAt = optionalText(args, "completedAt") ?? new Date().toISOString(); + const receipt = createStageReceipt({ + id: `live-smoke-${text(args, "runId")}`, + runId: text(args, "runId"), + stage: "PILOT", + protocolRevisionId: "institution-profile-calibration", + inputArtifactIds: [], + outputArtifactIds: [`metadata:${recordCount}`, `permitted-pdfs:${pdfCount}`, `production-eligible:${productionEligible}`], + cursor: `live-smoke:${completedAt}`, + createdAt: completedAt + }); + await appendJsonlRecord(layout.audit.stageReceipts, receipt); + return { productionEligible, receipt }; +} + +export async function executeInstitutionalResearchCommand(command: InstitutionalResearchCommand, args: InstitutionalResearchCommandArgs): Promise { + if (command === "pilot") return pilot(args); + if (command === "freeze") return freeze(args); + if (command === "ingest-export") return ingestExport(args); + if (command === "screen") return screen(args); + if (command === "acquire") return acquire(args); + if (command === "freeze-extraction") return freezeExtraction(args); + if (command === "extract") return extract(args); + if (command === "adjudicate-extraction") return adjudicateExtraction(args); + if (command === "freeze-data") return freezeData(args); + if (command === "report") return report(args); + if (command === "package") return packageManuscript(args); + if (command === "live-smoke") return liveSmoke(args); + throw new Error(`Unknown institutional research command: ${command}`); +} diff --git a/packages/longtable-scholar-research/src/corpus.ts b/packages/longtable-scholar-research/src/corpus.ts new file mode 100644 index 0000000..b38e844 --- /dev/null +++ b/packages/longtable-scholar-research/src/corpus.ts @@ -0,0 +1,201 @@ +import { createHash } from "node:crypto"; +import type { CanonicalBibliographicRecord } from "./export-parsers.js"; + +export interface BibliographicSourceLink { + readonly sourceDatabase: string; + readonly sourceRecordId: string; + readonly sourceRow: number; + readonly exportArtifactSha256: string; +} + +export interface NormalizedBibliographicRecord extends CanonicalBibliographicRecord { + readonly normalizedTitle: string; + readonly recordFingerprint: string; + readonly sourceLink: BibliographicSourceLink; +} + +export interface CanonicalPaper { + readonly paperId: string; + readonly title: string; + readonly normalizedTitle: string; + readonly authors: readonly string[]; + readonly year?: number; + readonly abstract?: string; + readonly doi?: string; + readonly venue?: string; + readonly keywords: readonly string[]; + readonly sourceLinks: readonly BibliographicSourceLink[]; + readonly provenanceBasis: readonly string[]; +} + +export interface DuplicateLink { + readonly paperId: string; + readonly primarySource: BibliographicSourceLink; + readonly duplicateSource: BibliographicSourceLink; + readonly basis: "exact_doi" | "exact_normalized_title_year"; +} + +export interface VersionLink { + readonly leftPaperId: string; + readonly rightPaperId: string; + readonly reason: "same_title_year_different_identifier"; +} + +export interface DeduplicatedBibliographicCorpus { + readonly papers: readonly CanonicalPaper[]; + readonly duplicateLinks: readonly DuplicateLink[]; + readonly versionLinks: readonly VersionLink[]; +} + +function sha256(value: string): string { + return createHash("sha256").update(value).digest("hex"); +} + +function normalizeDoi(value?: string): string | undefined { + if (!value?.trim()) return undefined; + return value + .replace(/^https?:\/\/(?:dx\.)?doi\.org\//i, "") + .replace(/^doi:\s*/i, "") + .trim() + .toLowerCase() || undefined; +} + +export function normalizeBibliographicTitle(value: string): string { + return value + .normalize("NFKC") + .toLocaleLowerCase("en") + .replace(/&/g, " ") + .replace(/[^\p{L}\p{N}]+/gu, " ") + .replace(/\s+/g, " ") + .trim(); +} + +function sourceLink(record: CanonicalBibliographicRecord): BibliographicSourceLink { + return { + sourceDatabase: record.sourceDatabase, + sourceRecordId: record.sourceRecordId, + sourceRow: record.sourceRow, + exportArtifactSha256: record.exportArtifactSha256 + }; +} + +export function normalizeBibliographicRecords( + records: readonly CanonicalBibliographicRecord[] +): NormalizedBibliographicRecord[] { + return records.map((record) => { + const doi = normalizeDoi(record.doi); + const normalizedTitle = normalizeBibliographicTitle(record.title); + const link = sourceLink(record); + return { + ...record, + ...(doi ? { doi } : {}), + normalizedTitle, + sourceLink: link, + recordFingerprint: sha256(JSON.stringify(link)) + }; + }); +} + +function deduplicationKey(record: NormalizedBibliographicRecord): string { + if (record.doi) return `doi:${record.doi}`; + return `title-year:${record.normalizedTitle}:${record.year ?? "unknown"}`; +} + +function compareSourceLinks(left: BibliographicSourceLink, right: BibliographicSourceLink): number { + return left.sourceDatabase.localeCompare(right.sourceDatabase) || + left.sourceRecordId.localeCompare(right.sourceRecordId) || + left.exportArtifactSha256.localeCompare(right.exportArtifactSha256) || + left.sourceRow - right.sourceRow; +} + +function compareRepresentatives(left: NormalizedBibliographicRecord, right: NormalizedBibliographicRecord): number { + return Number(Boolean(right.abstract)) - Number(Boolean(left.abstract)) || + (right.abstract?.length ?? 0) - (left.abstract?.length ?? 0) || + right.authors.length - left.authors.length || + right.title.length - left.title.length || + left.title.localeCompare(right.title) || + compareSourceLinks(left.sourceLink, right.sourceLink); +} + +function uniqueSorted(values: readonly string[]): string[] { + return [...new Set(values.map((value) => value.trim()).filter(Boolean))] + .sort((left, right) => left.localeCompare(right)); +} + +function paperFromGroup(key: string, group: readonly NormalizedBibliographicRecord[]): CanonicalPaper { + const sorted = [...group].sort(compareRepresentatives); + const representative = sorted[0]; + const sourceLinks = group.map((record) => record.sourceLink).sort(compareSourceLinks); + const paperId = `paper_${sha256(key).slice(0, 24)}`; + return { + paperId, + title: representative.title, + normalizedTitle: representative.normalizedTitle, + authors: [...representative.authors], + ...(representative.year ? { year: representative.year } : {}), + ...(representative.abstract ? { abstract: representative.abstract } : {}), + ...(representative.doi ? { doi: representative.doi } : {}), + ...(representative.venue ? { venue: representative.venue } : {}), + keywords: uniqueSorted(group.flatMap((record) => record.keywords)), + sourceLinks, + provenanceBasis: [ + representative.doi ? `canonical exact DOI ${representative.doi}` : `canonical exact normalized title/year ${representative.normalizedTitle}/${representative.year ?? "unknown"}`, + ...sourceLinks.map((link) => `${link.sourceDatabase}:${link.sourceRecordId}@${link.exportArtifactSha256}`) + ] + }; +} + +function versionLinks(papers: readonly CanonicalPaper[]): VersionLink[] { + const links: VersionLink[] = []; + for (let leftIndex = 0; leftIndex < papers.length; leftIndex += 1) { + for (let rightIndex = leftIndex + 1; rightIndex < papers.length; rightIndex += 1) { + const left = papers[leftIndex]; + const right = papers[rightIndex]; + if ( + left.normalizedTitle === right.normalizedTitle && + left.year === right.year && + left.doi && right.doi && left.doi !== right.doi + ) { + links.push({ + leftPaperId: left.paperId, + rightPaperId: right.paperId, + reason: "same_title_year_different_identifier" + }); + } + } + } + return links.sort((left, right) => + left.leftPaperId.localeCompare(right.leftPaperId) || left.rightPaperId.localeCompare(right.rightPaperId) + ); +} + +export function deduplicateBibliographicRecords( + records: readonly NormalizedBibliographicRecord[] +): DeduplicatedBibliographicCorpus { + const groups = new Map(); + for (const record of records) { + const key = deduplicationKey(record); + groups.set(key, [...(groups.get(key) ?? []), record]); + } + + const papers: CanonicalPaper[] = []; + const duplicateLinks: DuplicateLink[] = []; + for (const [key, group] of [...groups.entries()].sort(([left], [right]) => left.localeCompare(right))) { + const paper = paperFromGroup(key, group); + papers.push(paper); + const sources = [...paper.sourceLinks]; + for (const duplicateSource of sources.slice(1)) { + duplicateLinks.push({ + paperId: paper.paperId, + primarySource: sources[0], + duplicateSource, + basis: key.startsWith("doi:") ? "exact_doi" : "exact_normalized_title_year" + }); + } + } + papers.sort((left, right) => left.paperId.localeCompare(right.paperId)); + duplicateLinks.sort((left, right) => + left.paperId.localeCompare(right.paperId) || compareSourceLinks(left.duplicateSource, right.duplicateSource) + ); + return { papers, duplicateLinks, versionLinks: versionLinks(papers) }; +} diff --git a/packages/longtable-scholar-research/src/export-parsers.ts b/packages/longtable-scholar-research/src/export-parsers.ts new file mode 100644 index 0000000..32815ed --- /dev/null +++ b/packages/longtable-scholar-research/src/export-parsers.ts @@ -0,0 +1,429 @@ +import { createHash } from "node:crypto"; + +export const EXPORT_FORMATS = ["csv", "ris", "nbib", "json", "spreadsheet_xml"] as const; +export type ExportFormat = typeof EXPORT_FORMATS[number]; + +export interface CanonicalBibliographicRecord { + readonly title: string; + readonly authors: readonly string[]; + readonly year?: number; + readonly abstract?: string; + readonly doi?: string; + readonly venue?: string; + readonly keywords: readonly string[]; + readonly sourceDatabase: string; + readonly sourceRecordId: string; + readonly sourceRow: number; + readonly exportArtifactSha256: string; +} + +export type ExportParseRejectionReason = + | "missing_title" + | "malformed_row" + | "malformed_export" + | "empty_export"; + +export interface ExportParseRejection { + readonly reason: ExportParseRejectionReason; + readonly sourceRow?: number; + readonly sourceRecordId?: string; + readonly detail: string; +} + +export interface ParsedExport { + readonly format: ExportFormat; + readonly artifactSha256: string; + readonly records: readonly CanonicalBibliographicRecord[]; + readonly rejections: readonly ExportParseRejection[]; +} + +export interface ParserContext { + readonly database: string; + readonly artifactSha256: string; +} + +export interface ParseExportArtifactInput { + readonly filename: string; + readonly format: ExportFormat | string; + readonly database: string; + readonly content: string | Buffer; +} + +interface RecordFields { + readonly title?: string; + readonly authors?: readonly string[]; + readonly year?: string | number; + readonly abstract?: string; + readonly doi?: string; + readonly venue?: string; + readonly keywords?: readonly string[]; + readonly sourceRecordId?: string; +} + +function decodeEntities(value: string): string { + return value + .replace(/&/g, "&") + .replace(/</g, "<") + .replace(/>/g, ">") + .replace(/"/g, "\"") + .replace(/'|'/g, "'") + .replace(/&#(\d+);/g, (_match, code: string) => String.fromCodePoint(Number(code))); +} + +function cleanText(value: unknown): string | undefined { + if (typeof value !== "string" && typeof value !== "number") { + return undefined; + } + const cleaned = decodeEntities(String(value)) + .replace(/<[^>]+>/g, " ") + .replace(/\s+/g, " ") + .trim(); + return cleaned || undefined; +} + +function normalizeDoi(value: unknown): string | undefined { + const cleaned = cleanText(value); + if (!cleaned) return undefined; + const normalized = cleaned + .replace(/^https?:\/\/(?:dx\.)?doi\.org\//i, "") + .replace(/^doi:\s*/i, "") + .replace(/\s*\[doi\]\s*$/i, "") + .trim() + .toLowerCase(); + return normalized || undefined; +} + +function parseYear(value: unknown): number | undefined { + const match = cleanText(value)?.match(/(?:19|20)\d{2}/); + return match ? Number(match[0]) : undefined; +} + +function cleanList(values: readonly unknown[] | undefined): string[] { + return (values ?? []).map(cleanText).filter((value): value is string => Boolean(value)); +} + +function splitList(value: unknown): string[] { + const cleaned = cleanText(value); + return cleaned ? cleaned.split(/\s*;\s*/).map((entry) => entry.trim()).filter(Boolean) : []; +} + +function buildRecord( + fields: RecordFields, + context: ParserContext, + sourceRow: number +): { readonly record?: CanonicalBibliographicRecord; readonly rejection?: ExportParseRejection } { + const title = cleanText(fields.title); + const sourceRecordId = cleanText(fields.sourceRecordId) ?? `${context.database}:row:${sourceRow}`; + if (!title) { + return { + rejection: { + reason: "missing_title", + sourceRow, + sourceRecordId, + detail: "The export row has no usable title." + } + }; + } + const year = parseYear(fields.year); + const abstract = cleanText(fields.abstract); + const doi = normalizeDoi(fields.doi); + const venue = cleanText(fields.venue); + return { + record: { + title, + authors: cleanList(fields.authors), + ...(year ? { year } : {}), + ...(abstract ? { abstract } : {}), + ...(doi ? { doi } : {}), + ...(venue ? { venue } : {}), + keywords: cleanList(fields.keywords), + sourceDatabase: context.database, + sourceRecordId, + sourceRow, + exportArtifactSha256: context.artifactSha256 + } + }; +} + +function resultFromFields( + format: ExportFormat, + fields: readonly RecordFields[], + context: ParserContext, + firstSourceRow = 1 +): ParsedExport { + const records: CanonicalBibliographicRecord[] = []; + const rejections: ExportParseRejection[] = []; + fields.forEach((entry, index) => { + const parsed = buildRecord(entry, context, firstSourceRow + index); + if (parsed.record) records.push(parsed.record); + if (parsed.rejection) rejections.push(parsed.rejection); + }); + return { format, artifactSha256: context.artifactSha256, records, rejections }; +} + +function parseCsvRows(content: string): string[][] { + const rows: string[][] = []; + let row: string[] = []; + let field = ""; + let quoted = false; + for (let index = 0; index < content.length; index += 1) { + const character = content[index]; + if (quoted) { + if (character === "\"") { + if (content[index + 1] === "\"") { + field += "\""; + index += 1; + } else { + quoted = false; + } + } else { + field += character; + } + continue; + } + if (character === "\"") { + quoted = true; + } else if (character === ",") { + row.push(field); + field = ""; + } else if (character === "\n") { + row.push(field.replace(/\r$/, "")); + rows.push(row); + row = []; + field = ""; + } else { + field += character; + } + } + if (quoted) { + throw new Error("CSV ended inside a quoted field."); + } + if (field || row.length > 0) { + row.push(field.replace(/\r$/, "")); + rows.push(row); + } + return rows.filter((entry) => entry.some((cell) => cell.trim())); +} + +const HEADER_ALIASES: Record = { + title: ["title", "article title", "document title", "ti"], + authors: ["authors", "author", "author full names", "au"], + year: ["publication year", "year", "py"], + abstract: ["abstract", "ab"], + doi: ["doi", "digital object identifier"], + venue: ["source title", "publication name", "journal", "venue", "so"], + sourceRecordId: ["accession number", "record id", "eid", "ut", "pmid", "an"], + keywords: ["author keywords", "keywords", "kw"] +}; + +function normalizedHeader(value: string): string { + return value.replace(/^\uFEFF/, "").trim().toLowerCase(); +} + +function headerIndex(headers: readonly string[], field: keyof typeof HEADER_ALIASES): number { + return headers.findIndex((header) => HEADER_ALIASES[field].includes(normalizedHeader(header))); +} + +function cell(row: readonly string[], index: number): string | undefined { + return index >= 0 ? row[index] : undefined; +} + +function fieldsFromRows(rows: readonly string[][]): RecordFields[] { + const headers = rows[0] ?? []; + const indexes = { + title: headerIndex(headers, "title"), + authors: headerIndex(headers, "authors"), + year: headerIndex(headers, "year"), + abstract: headerIndex(headers, "abstract"), + doi: headerIndex(headers, "doi"), + venue: headerIndex(headers, "venue"), + sourceRecordId: headerIndex(headers, "sourceRecordId"), + keywords: headerIndex(headers, "keywords") + }; + return rows.slice(1).map((row) => ({ + title: cell(row, indexes.title), + authors: splitList(cell(row, indexes.authors)), + year: cell(row, indexes.year), + abstract: cell(row, indexes.abstract), + doi: cell(row, indexes.doi), + venue: cell(row, indexes.venue), + sourceRecordId: cell(row, indexes.sourceRecordId), + keywords: splitList(cell(row, indexes.keywords)) + })); +} + +export function parseCsvExport(content: string, context: ParserContext): ParsedExport { + return resultFromFields("csv", fieldsFromRows(parseCsvRows(content)), context, 2); +} + +function parseTaggedRecords( + content: string, + linePattern: RegExp, + recordEnd: (tag: string) => boolean +): Array> { + const records: Array> = []; + let current = new Map(); + let lastTag: string | undefined; + const finish = () => { + if (current.size > 0) records.push(current); + current = new Map(); + lastTag = undefined; + }; + for (const line of content.split(/\r?\n/)) { + const match = line.match(linePattern); + if (match) { + const tag = match[1]; + const value = match[2].trim(); + if (recordEnd(tag)) { + finish(); + continue; + } + current.set(tag, [...(current.get(tag) ?? []), value]); + lastTag = tag; + } else if (/^\s+\S/.test(line) && lastTag) { + const values = current.get(lastTag) ?? []; + const lastIndex = values.length - 1; + values[lastIndex] = `${values[lastIndex]} ${line.trim()}`; + current.set(lastTag, values); + } else if (!line.trim() && current.size > 0) { + finish(); + } + } + finish(); + return records; +} + +function firstTag(record: Map, tags: readonly string[]): string | undefined { + for (const tag of tags) { + const value = record.get(tag)?.[0]; + if (value) return value; + } + return undefined; +} + +function allTags(record: Map, tags: readonly string[]): string[] { + return tags.flatMap((tag) => record.get(tag) ?? []); +} + +export function parseRisExport(content: string, context: ParserContext): ParsedExport { + const records = parseTaggedRecords(content, /^([A-Z0-9]{2}) -\s?(.*)$/, (tag) => tag === "ER"); + const fields = records.map((record): RecordFields => ({ + title: firstTag(record, ["TI", "T1"]), + authors: allTags(record, ["AU", "A1"]), + year: firstTag(record, ["PY", "Y1"]), + abstract: firstTag(record, ["AB", "N2"]), + doi: firstTag(record, ["DO"]), + venue: firstTag(record, ["JO", "JF", "T2"]), + sourceRecordId: firstTag(record, ["AN", "ID"]), + keywords: allTags(record, ["KW"]) + })); + return resultFromFields("ris", fields, context); +} + +export function parseNbibExport(content: string, context: ParserContext): ParsedExport { + const records = parseTaggedRecords(content, /^([A-Z0-9]{2,4})\s*-\s?(.*)$/, () => false); + const fields = records.map((record): RecordFields => ({ + title: firstTag(record, ["TI"]), + authors: allTags(record, ["FAU", "AU"]), + year: firstTag(record, ["DP"]), + abstract: firstTag(record, ["AB"]), + doi: firstTag(record, ["LID", "AID"]), + venue: firstTag(record, ["JT", "TA"]), + sourceRecordId: firstTag(record, ["PMID"]), + keywords: allTags(record, ["OT", "MH"]) + })); + return resultFromFields("nbib", fields, context); +} + +function asObject(value: unknown): Record { + return value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +} + +function asArray(value: unknown): unknown[] { + return Array.isArray(value) ? value : []; +} + +function crossrefYear(item: Record): number | undefined { + const date = asObject(item.published ?? item.issued); + const parts = asArray(date["date-parts"]); + return parseYear(asArray(parts[0])[0]); +} + +function crossrefAuthor(value: unknown): string | undefined { + const author = asObject(value); + return cleanText([author.given, author.family].map(cleanText).filter(Boolean).join(" ") || author.name); +} + +export function parseJsonExport(content: string, context: ParserContext): ParsedExport { + const parsed = JSON.parse(content) as unknown; + const root = asObject(parsed); + const message = asObject(root.message); + const items = Array.isArray(parsed) ? parsed : asArray(message.items).length > 0 ? asArray(message.items) : asArray(root.records); + const fields = items.map((value): RecordFields => { + const item = asObject(value); + const doi = cleanText(item.DOI ?? item.doi); + return { + title: cleanText(asArray(item.title)[0] ?? item.title), + authors: asArray(item.author ?? item.authors).map(crossrefAuthor).filter((entry): entry is string => Boolean(entry)), + year: crossrefYear(item) ?? parseYear(item.year), + abstract: cleanText(item.abstract), + doi, + venue: cleanText(asArray(item["container-title"])[0] ?? item.venue), + sourceRecordId: cleanText(item.id ?? item.sourceRecordId) ?? normalizeDoi(doi), + keywords: cleanList(asArray(item.subject ?? item.keywords)) + }; + }); + return resultFromFields("json", fields, context); +} + +export function parseSpreadsheetXmlExport(content: string, context: ParserContext): ParsedExport { + const rows = [...content.matchAll(/]*>([\s\S]*?)<\/Row>/gi)].map((rowMatch) => + [...rowMatch[1].matchAll(/]*>([\s\S]*?)<\/Cell>/gi)].map((cellMatch) => { + const data = cellMatch[1].match(/]*>([\s\S]*?)<\/Data>/i)?.[1] ?? ""; + return cleanText(data) ?? ""; + }) + ); + return resultFromFields("spreadsheet_xml", fieldsFromRows(rows), context, 2); +} + +function withEmptyRejection(result: ParsedExport): ParsedExport { + if (result.records.length > 0 || result.rejections.length > 0) { + return result; + } + return { + ...result, + rejections: [{ reason: "empty_export", detail: "The export contains no bibliographic records." }] + }; +} + +export function parseExportArtifact(input: ParseExportArtifactInput): ParsedExport { + if (!EXPORT_FORMATS.includes(input.format as ExportFormat)) { + throw new Error(`Unsupported export format: ${input.format}`); + } + const buffer = typeof input.content === "string" ? Buffer.from(input.content, "utf8") : input.content; + const artifactSha256 = createHash("sha256").update(buffer).digest("hex"); + const context = { database: input.database, artifactSha256 }; + const content = buffer.toString("utf8"); + try { + const parsed = input.format === "csv" + ? parseCsvExport(content, context) + : input.format === "ris" + ? parseRisExport(content, context) + : input.format === "nbib" + ? parseNbibExport(content, context) + : input.format === "json" + ? parseJsonExport(content, context) + : parseSpreadsheetXmlExport(content, context); + return withEmptyRejection(parsed); + } catch (error) { + return { + format: input.format as ExportFormat, + artifactSha256, + records: [], + rejections: [{ + reason: "malformed_export", + detail: `${input.filename}: ${error instanceof Error ? error.message : String(error)}` + }] + }; + } +} diff --git a/packages/longtable-scholar-research/src/extraction.ts b/packages/longtable-scholar-research/src/extraction.ts new file mode 100644 index 0000000..923cedc --- /dev/null +++ b/packages/longtable-scholar-research/src/extraction.ts @@ -0,0 +1,336 @@ +import { createHash } from "node:crypto"; +import type { InstitutionalResearchHardStop } from "./workflow-types.js"; + +export type ExtractionValueType = "string" | "number" | "boolean" | "categorical" | "multi_categorical" | "verbatim"; +export type ExtractionCorpusType = "scholarly_study" | "document_corpus" | "training_course" | "other"; +export type ExtractorType = "human" | "ai"; + +export interface ExtractionFieldDefinition { + readonly id: string; + readonly label: string; + readonly valueType: ExtractionValueType; + readonly required: boolean; + readonly evidenceRequired: boolean; + readonly allowedValues?: readonly string[]; +} + +export interface ExtractionReliability { + readonly statistic: string; + readonly threshold: number; +} + +export interface ExtractionReliabilityResult { + readonly statistic: string; + readonly observed: number; + readonly sampleSize: number; + readonly assessedAt: string; +} + +export interface ExtractionProfileInput { + readonly id: string; + readonly version: number; + readonly corpusType: ExtractionCorpusType; + readonly unitOfAnalysis: string; + readonly approvedDecisionRecordId: string; + readonly frozenAt: string; + readonly doubleExtractionRequired: boolean; + readonly reliability: ExtractionReliability; + readonly fields: readonly ExtractionFieldDefinition[]; +} + +export interface ExtractionProfile extends ExtractionProfileInput { + readonly profileHash: string; +} + +export interface ExtractionEvidence { + readonly sourceArtifactId: string; + readonly locator: string; + readonly quote?: string; +} + +export interface ExtractedFieldValue { + readonly fieldId: string; + readonly value: unknown; + readonly evidence: readonly ExtractionEvidence[]; +} + +export interface ExtractionRecord { + readonly id: string; + readonly unitId: string; + readonly profileId: string; + readonly profileHash: string; + readonly extractor: { readonly id: string; readonly type: ExtractorType }; + readonly extractedAt: string; + readonly status: "draft" | "submitted" | "adjudicated"; + readonly values: readonly ExtractedFieldValue[]; +} + +export interface ExtractionConflict { + readonly id: string; + readonly unitId: string; + readonly fieldId: string; + readonly recordIds: readonly string[]; + readonly values: readonly unknown[]; + readonly status: "open" | "resolved"; +} + +export interface ExtractionAdjudicationInput { + readonly id: string; + readonly finalValue: unknown; + readonly rationale: string; + readonly adjudicatorId: string; + readonly adjudicatedAt: string; + readonly evidence: readonly ExtractionEvidence[]; + readonly decisionRecordId?: string; +} + +export interface ExtractionAdjudication extends ExtractionAdjudicationInput { + readonly conflictId: string; + readonly unitId: string; + readonly fieldId: string; +} + +export type ExtractionReadiness = + | { readonly status: "ready" } + | { readonly status: "hard_stop"; readonly code: InstitutionalResearchHardStop; readonly reason: string }; + +export interface ExtractedDatasetFreezeInput { + readonly id: string; + readonly profile: ExtractionProfile; + readonly records: readonly ExtractionRecord[]; + readonly conflicts: readonly ExtractionConflict[]; + readonly adjudications: readonly ExtractionAdjudication[]; + readonly reliability: ExtractionReliabilityResult; + readonly decisionRecordId: string; + readonly frozenAt: string; +} + +export interface ExtractedDatasetFreeze { + readonly id: string; + readonly profileId: string; + readonly profileHash: string; + readonly recordIds: readonly string[]; + readonly adjudicationIds: readonly string[]; + readonly unresolvedConflictCount: number; + readonly decisionRecordId: string; + readonly frozenAt: string; + readonly datasetHash: string; +} + +export interface ExtractionAuditSummary { + readonly profileId: string; + readonly profileHash: string; + readonly corpusType: ExtractionCorpusType; + readonly unitOfAnalysis: string; + readonly doubleExtractionRequired: boolean; + readonly recordCount: number; + readonly unitCount: number; + readonly reliabilityStatistic: string; + readonly reliabilityThreshold: number; + readonly reliabilityObserved: number; + readonly conflictCount: number; + readonly adjudicatedCount: number; + readonly missingRequiredValueCount: number; + readonly datasetFreezeId: string; + readonly datasetHash: string; +} + +function canonical(value: unknown): unknown { + if (Array.isArray(value)) return value.map(canonical); + if (value && typeof value === "object") { + return Object.fromEntries(Object.entries(value as Record) + .filter(([, nested]) => nested !== undefined) + .sort(([left], [right]) => left.localeCompare(right)) + .map(([key, nested]) => [key, canonical(nested)])); + } + return value; +} + +function hash(value: unknown): string { + return createHash("sha256").update(JSON.stringify(canonical(value))).digest("hex"); +} + +function cloneFreeze(value: T): T { + const clone = JSON.parse(JSON.stringify(value)) as T; + const freeze = (nested: unknown): void => { + if (nested && typeof nested === "object" && !Object.isFrozen(nested)) { + Object.values(nested as Record).forEach(freeze); + Object.freeze(nested); + } + }; + freeze(clone); + return clone; +} + +function fieldSchema(profile: ExtractionProfileInput): unknown { + return { + corpusType: profile.corpusType, + unitOfAnalysis: profile.unitOfAnalysis, + doubleExtractionRequired: profile.doubleExtractionRequired, + reliability: { statistic: profile.reliability.statistic, threshold: profile.reliability.threshold }, + fields: profile.fields + }; +} + +export function createExtractionProfile(input: ExtractionProfileInput): ExtractionProfile { + if (!input.id.trim() || !input.unitOfAnalysis.trim() || !input.approvedDecisionRecordId.trim() || !input.frozenAt.trim()) { + throw new Error("A frozen extraction profile requires stable IDs, unit of analysis, timestamp, and DecisionRecord approval."); + } + if (!Number.isInteger(input.version) || input.version < 1 || input.fields.length === 0) { + throw new Error("An extraction profile requires a positive version and at least one field."); + } + if (input.reliability.threshold < 0 || input.reliability.threshold > 1) { + throw new Error("Extraction reliability values must be between zero and one."); + } + const ids = new Set(); + for (const field of input.fields) { + if (!field.id.trim() || ids.has(field.id)) throw new Error(`Extraction field IDs must be non-empty and unique: ${field.id}`); + ids.add(field.id); + if ((field.valueType === "categorical" || field.valueType === "multi_categorical") && !field.allowedValues?.length) { + throw new Error(`Categorical extraction field ${field.id} requires allowed values.`); + } + } + return cloneFreeze({ ...input, profileHash: hash(fieldSchema(input)) }); +} + +function valueMatches(field: ExtractionFieldDefinition, value: unknown): boolean { + if (field.valueType === "number") return typeof value === "number" && Number.isFinite(value); + if (field.valueType === "boolean") return typeof value === "boolean"; + if (field.valueType === "multi_categorical") { + return Array.isArray(value) && value.every((entry) => typeof entry === "string" && field.allowedValues?.includes(entry)); + } + if (field.valueType === "categorical") return typeof value === "string" && Boolean(field.allowedValues?.includes(value)); + return typeof value === "string"; +} + +export function validateExtractionRecord(profile: ExtractionProfile, input: ExtractionRecord): ExtractionRecord { + if (!input.id.trim() || !input.unitId.trim() || !input.extractor.id.trim()) throw new Error("An extraction record requires stable record, unit, and extractor IDs."); + if (input.profileId !== profile.id || input.profileHash !== profile.profileHash) throw new Error("Extraction record profile identity or hash does not match the frozen profile."); + const byField = new Map(input.values.map((entry) => [entry.fieldId, entry])); + if (byField.size !== input.values.length) throw new Error("An extraction record cannot repeat a field."); + for (const value of input.values) { + const field = profile.fields.find((candidate) => candidate.id === value.fieldId); + if (!field) throw new Error(`Unknown extraction field: ${value.fieldId}`); + if (!valueMatches(field, value.value)) throw new Error(`Invalid value for extraction field ${field.id}.`); + if (field.evidenceRequired && (!value.evidence.length || value.evidence.some((item) => !item.sourceArtifactId.trim() || !item.locator.trim()))) { + throw new Error(`Extraction field ${field.id} requires evidence provenance.`); + } + } + for (const field of profile.fields) { + if (field.required && !byField.has(field.id)) throw new Error(`Missing required field ${field.id}.`); + } + return cloneFreeze(input); +} + +function comparable(value: unknown): string { + const normalized = Array.isArray(value) ? [...value].sort() : value; + return JSON.stringify(canonical(normalized)); +} + +export function detectExtractionConflicts(profile: ExtractionProfile, records: readonly ExtractionRecord[]): ExtractionConflict[] { + const groups = new Map(); + for (const record of records) { + validateExtractionRecord(profile, record); + const group = groups.get(record.unitId) ?? []; + group.push(record); + groups.set(record.unitId, group); + } + const conflicts: ExtractionConflict[] = []; + for (const [unitId, group] of [...groups].sort(([a], [b]) => a.localeCompare(b))) { + if (profile.doubleExtractionRequired && new Set(group.map((item) => item.extractor.id)).size < 2) continue; + for (const field of profile.fields) { + const entries = group.map((record) => ({ recordId: record.id, value: record.values.find((item) => item.fieldId === field.id)?.value })) + .filter((entry) => entry.value !== undefined); + if (new Set(entries.map((entry) => comparable(entry.value))).size > 1) { + const recordIds = entries.map((entry) => entry.recordId).sort(); + conflicts.push(cloneFreeze({ + id: `conflict-${hash({ unitId, fieldId: field.id, recordIds }).slice(0, 24)}`, + unitId, + fieldId: field.id, + recordIds, + values: entries.sort((a, b) => a.recordId.localeCompare(b.recordId)).map((entry) => entry.value), + status: "open" as const + })); + } + } + } + return conflicts; +} + +export function adjudicateExtractionConflict( + profile: ExtractionProfile, + conflict: ExtractionConflict, + input: ExtractionAdjudicationInput +): ExtractionAdjudication { + const field = profile.fields.find((candidate) => candidate.id === conflict.fieldId); + if (!field || conflict.status !== "open") throw new Error("Only an open conflict in the frozen extraction profile can be adjudicated."); + if (!input.id.trim() || !input.rationale.trim() || !input.adjudicatorId.trim()) throw new Error("Adjudication requires stable identity, adjudicator, and rationale."); + if (!valueMatches(field, input.finalValue)) throw new Error(`Invalid adjudicated value for extraction field ${field.id}.`); + if (field.evidenceRequired && input.evidence.length === 0) throw new Error(`Adjudication for ${field.id} requires evidence.`); + return cloneFreeze({ ...input, conflictId: conflict.id, unitId: conflict.unitId, fieldId: conflict.fieldId }); +} + +export function assessExtractionReadiness( + profile: ExtractionProfile, + records: readonly ExtractionRecord[], + conflicts: readonly ExtractionConflict[], + adjudications: readonly ExtractionAdjudication[], + reliability: ExtractionReliabilityResult +): ExtractionReadiness { + if (!profile.profileHash || !profile.approvedDecisionRecordId) return { status: "hard_stop", code: "EXTRACTION_SCHEMA_UNFROZEN", reason: "The extraction profile is not frozen and approved." }; + if (reliability.statistic !== profile.reliability.statistic || !Number.isInteger(reliability.sampleSize) || reliability.sampleSize < 1 || + reliability.observed < 0 || reliability.observed > 1 || reliability.observed < profile.reliability.threshold) { + return { status: "hard_stop", code: "EXTRACTION_RELIABILITY_BELOW_THRESHOLD", reason: "The extraction pilot has not met its declared reliability threshold." }; + } + if (records.length === 0) { + return { status: "hard_stop", code: "EXTRACTED_DATA_FREEZE_REQUIRED", reason: "No validated research-data extraction records are available to freeze." }; + } + const units = new Map>(); + for (const record of records) { + validateExtractionRecord(profile, record); + const extractors = units.get(record.unitId) ?? new Set(); + extractors.add(record.extractor.id); + units.set(record.unitId, extractors); + } + if (profile.doubleExtractionRequired && [...units.values()].some((extractors) => extractors.size < 2)) { + return { status: "hard_stop", code: "DOUBLE_EXTRACTION_CONFLICT", reason: "At least one unit has not received independent double extraction." }; + } + const expectedConflictIds = detectExtractionConflicts(profile, records).map((conflict) => conflict.id).sort(); + const suppliedConflictIds = conflicts.map((conflict) => conflict.id).sort(); + if (JSON.stringify(expectedConflictIds) !== JSON.stringify(suppliedConflictIds)) { + return { status: "hard_stop", code: "DOUBLE_EXTRACTION_CONFLICT", reason: "The supplied conflict ledger does not match deterministic conflict detection." }; + } + const resolved = new Set(adjudications.map((item) => item.conflictId)); + if (conflicts.some((conflict) => !resolved.has(conflict.id))) { + return { status: "hard_stop", code: "DOUBLE_EXTRACTION_CONFLICT", reason: "At least one extraction conflict remains unresolved." }; + } + return { status: "ready" }; +} + +export function freezeExtractedDataset(input: ExtractedDatasetFreezeInput): ExtractedDatasetFreeze { + if (!input.id.trim() || !input.decisionRecordId.trim() || !input.frozenAt.trim()) throw new Error("An extracted dataset freeze requires stable IDs, timestamp, and DecisionRecord approval."); + const readiness = assessExtractionReadiness(input.profile, input.records, input.conflicts, input.adjudications, input.reliability); + if (readiness.status !== "ready") { + const wording = readiness.code === "DOUBLE_EXTRACTION_CONFLICT" ? `Unresolved conflict: ${readiness.reason}` : readiness.reason; + throw new Error(wording); + } + const recordIds = input.records.map((record) => record.id).sort(); + const adjudicationIds = input.adjudications.map((item) => item.id).sort(); + const datasetHash = hash({ + profileHash: input.profile.profileHash, + records: [...input.records].sort((a, b) => a.id.localeCompare(b.id)), + adjudications: [...input.adjudications].sort((a, b) => a.id.localeCompare(b.id)), + reliability: input.reliability + }); + return cloneFreeze({ + id: input.id, + profileId: input.profile.id, + profileHash: input.profile.profileHash, + recordIds, + adjudicationIds, + unresolvedConflictCount: 0, + decisionRecordId: input.decisionRecordId, + frozenAt: input.frozenAt, + datasetHash + }); +} diff --git a/packages/longtable-scholar-research/src/index.ts b/packages/longtable-scholar-research/src/index.ts index 2d0deed..e3c7c73 100644 --- a/packages/longtable-scholar-research/src/index.ts +++ b/packages/longtable-scholar-research/src/index.ts @@ -5,3 +5,26 @@ export * from "./rank.js"; export * from "./run.js"; export * from "./publisher-access.js"; export * from "./protocol.js"; +export * from "./workflow-types.js"; +export * from "./workflow-state.js"; +export * from "./project-store.js"; +export * from "./checkpoints.js"; +export * from "./export-parsers.js"; +export * from "./corpus.js"; +export * from "./invariants.js"; +export * from "./pdf-vault.js"; +export * from "./adapters.js"; +export * from "./institutions.js"; +export * from "./browser-recipes.js"; +export * from "./screening.js"; +export * from "./extraction.js"; +export * from "./prisma.js"; +export * from "./reporting.js"; +export * from "./references.js"; +export * from "./render-profiles.js"; +export * from "./manuscript.js"; +export * from "./word-renderer.js"; +export * from "./retry.js"; +export * from "./model-budget.js"; +export * from "./checkpoint-delivery.js"; +export * from "./coordinator.js"; diff --git a/packages/longtable-scholar-research/src/institutions.ts b/packages/longtable-scholar-research/src/institutions.ts new file mode 100644 index 0000000..4c01fa2 --- /dev/null +++ b/packages/longtable-scholar-research/src/institutions.ts @@ -0,0 +1,131 @@ +import type { InstitutionProfile } from "./workflow-types.js"; + +export interface InstitutionCalibrationEvidence { + readonly researcherLoginVerified?: boolean; + readonly resolverDiscoveryVerified?: boolean; + readonly calibrationDatabaseIds: readonly string[]; + readonly fixtureReplayDatabaseIds: readonly string[]; + readonly liveSmokeDatabaseIds: readonly string[]; + readonly decisionRecordIds: readonly string[]; +} + +export interface InstitutionProductionEligibility { + readonly eligible: boolean; + readonly missing: readonly string[]; + readonly coveredDatabaseIds: readonly string[]; +} + +export interface InstitutionCalibrationStep { + readonly key: + | "researcher_login" + | "database_and_resolver_discovery" + | "supervised_dry_run" + | "profile_approval" + | "fixture_replay" + | "live_smoke" + | "production_eligibility"; + readonly complete: boolean; + readonly evidence: readonly string[]; +} + +export interface InstitutionCalibrationChecklist { + readonly institutionProfileId: string; + readonly steps: readonly InstitutionCalibrationStep[]; +} + +function missingDatabaseEvidence( + databaseIds: readonly string[], + evidencedIds: readonly string[], + label: string +): string[] { + const present = new Set(evidencedIds); + return databaseIds + .filter((databaseId) => !present.has(databaseId)) + .map((databaseId) => `${label} evidence is missing for database ${databaseId}.`); +} + +function allCovered(databaseIds: readonly string[], evidencedIds: readonly string[]): boolean { + const present = new Set(evidencedIds); + return databaseIds.every((databaseId) => present.has(databaseId)); +} + +export function assessInstitutionProductionEligibility( + profile: InstitutionProfile, + evidence: InstitutionCalibrationEvidence +): InstitutionProductionEligibility { + const databaseIds = profile.databases.map((database) => database.databaseId); + const missing: string[] = []; + if (evidence.researcherLoginVerified !== true) { + missing.push("Researcher login verification is missing; login and MFA remain researcher-owned."); + } + if (evidence.resolverDiscoveryVerified !== true) { + missing.push("Database and resolver discovery evidence is missing."); + } + missing.push(...missingDatabaseEvidence(databaseIds, evidence.calibrationDatabaseIds, "Supervised calibration dry run")); + missing.push(...missingDatabaseEvidence(databaseIds, evidence.fixtureReplayDatabaseIds, "Local fixture replay")); + missing.push(...missingDatabaseEvidence(databaseIds, evidence.liveSmokeDatabaseIds, "Limited live smoke")); + if (!evidence.decisionRecordIds.includes(profile.approvedDecisionRecordId)) { + missing.push(`Profile approval DecisionRecord ${profile.approvedDecisionRecordId} is missing.`); + } + return { + eligible: missing.length === 0, + missing, + coveredDatabaseIds: databaseIds.filter((databaseId) => + evidence.calibrationDatabaseIds.includes(databaseId) && + evidence.fixtureReplayDatabaseIds.includes(databaseId) && + evidence.liveSmokeDatabaseIds.includes(databaseId) + ).sort() + }; +} + +export function buildInstitutionCalibrationChecklist( + profile: InstitutionProfile, + evidence: InstitutionCalibrationEvidence +): InstitutionCalibrationChecklist { + const databaseIds = profile.databases.map((database) => database.databaseId); + const approvalComplete = evidence.decisionRecordIds.includes(profile.approvedDecisionRecordId); + const dryRunComplete = allCovered(databaseIds, evidence.calibrationDatabaseIds); + const fixtureComplete = allCovered(databaseIds, evidence.fixtureReplayDatabaseIds); + const liveSmokeComplete = allCovered(databaseIds, evidence.liveSmokeDatabaseIds); + const eligibility = assessInstitutionProductionEligibility(profile, evidence); + return { + institutionProfileId: profile.id, + steps: [ + { + key: "researcher_login", + complete: evidence.researcherLoginVerified === true, + evidence: [evidence.researcherLoginVerified ? "Researcher confirmed the authenticated database session without sharing credentials." : "Researcher login confirmation pending."] + }, + { + key: "database_and_resolver_discovery", + complete: evidence.resolverDiscoveryVerified === true, + evidence: [evidence.resolverDiscoveryVerified ? `Resolver discovery recorded for ${profile.resolverUrls?.length ?? 0} configured resolver route(s).` : "Database and resolver discovery pending."] + }, + { + key: "supervised_dry_run", + complete: dryRunComplete, + evidence: [evidence.calibrationDatabaseIds.length > 0 ? `Dry-run databases: ${[...evidence.calibrationDatabaseIds].sort().join(", ")}.` : "Supervised dry run pending."] + }, + { + key: "profile_approval", + complete: approvalComplete, + evidence: [approvalComplete ? `DecisionRecord: ${profile.approvedDecisionRecordId}.` : `DecisionRecord ${profile.approvedDecisionRecordId} pending.`] + }, + { + key: "fixture_replay", + complete: fixtureComplete, + evidence: [evidence.fixtureReplayDatabaseIds.length > 0 ? `Fixture replay databases: ${[...evidence.fixtureReplayDatabaseIds].sort().join(", ")}.` : "Fixture replay pending."] + }, + { + key: "live_smoke", + complete: liveSmokeComplete, + evidence: [evidence.liveSmokeDatabaseIds.length > 0 ? `Live-smoke databases: ${[...evidence.liveSmokeDatabaseIds].sort().join(", ")}.` : "Limited live smoke pending."] + }, + { + key: "production_eligibility", + complete: eligibility.eligible, + evidence: [eligibility.eligible ? "All production gates passed." : eligibility.missing.join(" ")] + } + ] + }; +} diff --git a/packages/longtable-scholar-research/src/invariants.ts b/packages/longtable-scholar-research/src/invariants.ts new file mode 100644 index 0000000..eb538a3 --- /dev/null +++ b/packages/longtable-scholar-research/src/invariants.ts @@ -0,0 +1,82 @@ +export interface CorpusCounts { + readonly identified: number; + readonly normalized: number; + readonly parseRejected: number; + readonly unique: number; + readonly duplicateLinks: number; + readonly titleAbstractScreened: number; + readonly fulltextCandidates: number; + readonly titleAbstractExcluded: number; + readonly titleAbstractPending: number; + readonly fulltextSought: number; + readonly fulltextRetrieved: number; + readonly fulltextNotRetrieved: number; + readonly fulltextAssessed: number; + readonly finalIncluded: number; + readonly fulltextExcluded: number; + readonly unresolved: number; +} + +export interface CountEquationAudit { + readonly name: string; + readonly left: number; + readonly right: number; + readonly expression: string; + readonly passed: boolean; +} + +export interface CorpusInvariantAudit { + readonly passed: boolean; + readonly prismaEligible: boolean; + readonly equations: readonly CountEquationAudit[]; +} + +function assertCounts(counts: CorpusCounts): void { + for (const [name, value] of Object.entries(counts)) { + if (!Number.isInteger(value) || value < 0) { + throw new Error(`Corpus count ${name} must be a non-negative integer.`); + } + } +} + +function equation(name: string, left: number, right: number, expression: string): CountEquationAudit { + return { name, left, right, expression, passed: left === right }; +} + +export function verifyCorpusInvariants(counts: CorpusCounts): CorpusInvariantAudit { + assertCounts(counts); + const equations = [ + equation( + "identification", + counts.identified, + counts.normalized + counts.parseRejected, + "identified = normalized + parse_rejected" + ), + equation( + "deduplication", + counts.normalized, + counts.unique + counts.duplicateLinks, + "normalized = unique + duplicate_links" + ), + equation( + "title_abstract_screening", + counts.titleAbstractScreened, + counts.fulltextCandidates + counts.titleAbstractExcluded + counts.titleAbstractPending, + "title_abstract_screened = fulltext_candidates + excluded + pending" + ), + equation( + "fulltext_retrieval", + counts.fulltextSought, + counts.fulltextRetrieved + counts.fulltextNotRetrieved, + "fulltext_sought = retrieved + not_retrieved" + ), + equation( + "fulltext_screening", + counts.fulltextAssessed, + counts.finalIncluded + counts.fulltextExcluded + counts.unresolved, + "fulltext_assessed = final_included + fulltext_excluded + unresolved" + ) + ]; + const passed = equations.every((entry) => entry.passed); + return { passed, prismaEligible: passed, equations }; +} diff --git a/packages/longtable-scholar-research/src/manuscript.ts b/packages/longtable-scholar-research/src/manuscript.ts new file mode 100644 index 0000000..a78c145 --- /dev/null +++ b/packages/longtable-scholar-research/src/manuscript.ts @@ -0,0 +1,127 @@ +import { createHash } from "node:crypto"; +import type { CanonicalPaper } from "./corpus.js"; +import type { CorpusCounts } from "./invariants.js"; +import { verifyCorpusInvariants } from "./invariants.js"; +import { renderReferenceFormats } from "./references.js"; +import type { ProtocolRevision } from "./workflow-types.js"; +import type { ExtractionAuditSummary } from "./extraction.js"; + +export interface ManuscriptSection { + readonly id: string; + readonly heading: string; + readonly content: string; +} + +export interface ManuscriptProvenanceEntry { + readonly objectId: string; + readonly objectType: "paragraph" | "table" | "reference"; + readonly sha256: string; + readonly inputArtifactIds: readonly string[]; + readonly protocolRevisionId: string; +} + +export interface ApprovedManuscript { + readonly title: string; + readonly authorLines: readonly string[]; + readonly language: "ko" | "en"; + readonly abstract: string; + readonly keywords: readonly string[]; + readonly protocol: ProtocolRevision; + readonly sections: readonly ManuscriptSection[]; + readonly references: readonly string[]; + readonly unresolvedCitations: readonly { readonly paperId: string; readonly missingFields: readonly string[] }[]; + readonly metaAnalysisReadinessMarkdown: string; + readonly provenanceMap: readonly ManuscriptProvenanceEntry[]; + readonly generatedAt: string; + readonly generatorVersion: string; +} + +export interface BuildApprovedManuscriptInput { + readonly title: string; + readonly authorLines: readonly string[]; + readonly language: "ko" | "en"; + readonly abstract?: string; + readonly keywords: readonly string[]; + readonly protocol: ProtocolRevision; + readonly counts: CorpusCounts; + readonly papers: readonly CanonicalPaper[]; + readonly inputArtifactIds: readonly string[]; + readonly generatedAt: string; + readonly generatorVersion: string; + readonly approvals: { + readonly databasesExecuted: boolean; + readonly fulltextAvailabilityVerified: boolean; + readonly analysisApproved: boolean; + readonly extractedDataFrozen: boolean; + }; + readonly analysisPlan: string; + readonly extraction: ExtractionAuditSummary; +} + +function hash(value: string): string { + return createHash("sha256").update(value).digest("hex"); +} + +function readinessTable(papers: readonly CanonicalPaper[], analysisPlan: string): string { + const isMeta = /meta/i.test(analysisPlan); + return [ + "| 점검 항목 | 상태 | 근거 |", + "|---|---|---|", + `| 포함 연구 수 | ${papers.length}편 | 동결 코퍼스 |`, + `| 효과크기 산출 가능성 | ${isMeta ? "개별 연구 통계량 추출 필요" : "현재 분석계획의 직접 산출 대상 아님"} | 승인된 분석계획: ${analysisPlan} |`, + "| 표본·출판물 의존성 | 코딩 필요 | 저자·표본·데이터셋 중복 점검표 |", + "| 결측 및 저자 연락 | 코딩 필요 | 전문 추출 ledger |" + ].join("\n"); +} + +export function buildApprovedManuscript(input: BuildApprovedManuscriptInput): ApprovedManuscript { + if (!input.approvals.databasesExecuted) throw new Error("Cannot render a manuscript that claims an unexecuted database search."); + if (!input.approvals.fulltextAvailabilityVerified) throw new Error("Cannot render a manuscript before full-text availability is verified."); + if (!input.approvals.analysisApproved) throw new Error("Cannot render an unapproved analysis."); + if (!input.approvals.extractedDataFrozen || !input.extraction.datasetHash) throw new Error("Cannot render a manuscript before the extracted research dataset is frozen."); + if (!verifyCorpusInvariants(input.counts).passed) throw new Error("Cannot render a manuscript because corpus count invariants failed."); + const references = renderReferenceFormats(input.papers); + const databases = input.protocol.databases.join(", "); + const sections: ManuscriptSection[] = input.language === "ko" ? [ + { id: "introduction", heading: "서론", content: "본 연구는 해커톤형 집중 프로그램과 직무역량 개발을 수요-공급 불일치의 관점에서 검토한다. 연구문제와 개념적 틀은 동결 프로토콜의 범위 안에서 해석한다." }, + { id: "methods", heading: "연구방법", content: `${databases}를 동결된 데이터베이스별 검색식과 포함·배제 기준으로 검색하였다. 메타데이터 내보내기, 정규화, 중복 연결, 제목·초록 선별, 전문 확보 및 전문 선별의 모든 판정을 append-only ledger에 기록하였다. 연구변수는 추출 프로파일 ${input.extraction.profileId}로 이중 추출하고 신뢰도(${input.extraction.reliabilityStatistic}=${input.extraction.reliabilityObserved})를 검증한 뒤 충돌을 합의 판정하여 데이터셋 ${input.extraction.datasetFreezeId}로 동결하였다.` }, + { id: "results", heading: "결과", content: `총 ${input.counts.identified}건을 식별하였고, 중복 연결 후 ${input.counts.unique}건을 선별하였다. 전문 ${input.counts.fulltextAssessed}건을 평가하여 최종 ${input.counts.finalIncluded}편을 포함하였다.` }, + { id: "discussion", heading: "논의", content: "결과 해석은 동결 코퍼스와 승인된 분석계획에 한정한다. 직무역량 수요와 교육 공급의 불일치는 후속 문서 코딩 및 개념 매핑을 통해 검증한다." } + ] : [ + { id: "introduction", heading: "Introduction", content: "This review examines hackathon-like intensive protocols and workplace competency development through a demand-supply mismatch lens." }, + { id: "methods", heading: "Method", content: `${databases} were searched using frozen database-specific queries and eligibility criteria. Export, normalization, duplicate linkage, screening, and full-text acquisition decisions were retained in append-only ledgers. Research variables were independently extracted with profile ${input.extraction.profileId}, reliability was verified (${input.extraction.reliabilityStatistic}=${input.extraction.reliabilityObserved}), conflicts were adjudicated, and dataset ${input.extraction.datasetFreezeId} was frozen before analysis.` }, + { id: "results", heading: "Results", content: `${input.counts.identified} records were identified; ${input.counts.unique} unique records were screened, ${input.counts.fulltextAssessed} reports were assessed in full text, and ${input.counts.finalIncluded} studies were included.` }, + { id: "discussion", heading: "Discussion", content: "Interpretation is limited to the frozen corpus and approved analysis plan. Demand-supply mismatch requires explicit document coding and construct mapping." } + ]; + const metaAnalysisReadinessMarkdown = readinessTable(input.papers, input.analysisPlan); + const inputIds = [...input.inputArtifactIds].sort(); + const provenanceMap: ManuscriptProvenanceEntry[] = [ + { objectId: "title", objectType: "paragraph", sha256: hash(input.title), inputArtifactIds: inputIds, protocolRevisionId: input.protocol.id }, + ...input.authorLines.map((line, index) => ({ objectId: `author-line-${index + 1}`, objectType: "paragraph" as const, sha256: hash(line), inputArtifactIds: inputIds, protocolRevisionId: input.protocol.id })), + { objectId: "abstract-heading", objectType: "paragraph", sha256: hash(input.language === "ko" ? "초록" : "Abstract"), inputArtifactIds: inputIds, protocolRevisionId: input.protocol.id }, + { objectId: "abstract-body", objectType: "paragraph", sha256: hash(input.abstract ?? "[초록은 승인된 연구 결과와 해석을 반영하여 작성해야 합니다.]"), inputArtifactIds: inputIds, protocolRevisionId: input.protocol.id }, + { objectId: "keywords", objectType: "paragraph", sha256: hash(input.keywords.join("\n")), inputArtifactIds: inputIds, protocolRevisionId: input.protocol.id }, + ...sections.flatMap((section) => [ + { objectId: `${section.id}-heading`, objectType: "paragraph" as const, sha256: hash(section.heading), inputArtifactIds: inputIds, protocolRevisionId: input.protocol.id }, + { objectId: `${section.id}-body`, objectType: "paragraph" as const, sha256: hash(section.content), inputArtifactIds: inputIds, protocolRevisionId: input.protocol.id } + ]), + { objectId: "meta-analysis-readiness", objectType: "table", sha256: hash(metaAnalysisReadinessMarkdown), inputArtifactIds: inputIds, protocolRevisionId: input.protocol.id }, + { objectId: "references-heading", objectType: "paragraph", sha256: hash(input.language === "ko" ? "참고문헌" : "References"), inputArtifactIds: inputIds, protocolRevisionId: input.protocol.id }, + ...references.apa7References.map((reference, index) => ({ objectId: `reference-${index + 1}`, objectType: "reference" as const, sha256: hash(reference), inputArtifactIds: inputIds, protocolRevisionId: input.protocol.id })) + ]; + return { + title: input.title, + authorLines: [...input.authorLines], + language: input.language, + abstract: input.abstract ?? "[초록은 승인된 연구 결과와 해석을 반영하여 작성해야 합니다.]", + keywords: [...input.keywords], + protocol: input.protocol, + sections, + references: references.apa7References, + unresolvedCitations: references.unresolvedCitations, + metaAnalysisReadinessMarkdown, + provenanceMap, + generatedAt: input.generatedAt, + generatorVersion: input.generatorVersion + }; +} diff --git a/packages/longtable-scholar-research/src/model-budget.ts b/packages/longtable-scholar-research/src/model-budget.ts new file mode 100644 index 0000000..6945398 --- /dev/null +++ b/packages/longtable-scholar-research/src/model-budget.ts @@ -0,0 +1,69 @@ +import { createHash } from "node:crypto"; +import type { InstitutionalResearchStage } from "./workflow-types.js"; + +export interface ModelCallBudget { + readonly stage: InstitutionalResearchStage; + readonly purpose: string; + readonly maximumCalls: number; +} + +export interface ModelCallCacheKeyInput { + readonly artifactSha256: string; + readonly ruleVersion: string; + readonly codebookVersion: string; + readonly promptVersion: string; + readonly modelVersion: string; +} + +export const ZERO_MODEL_CALL_STAGES: readonly InstitutionalResearchStage[] = [ + "SETUP", + "PRODUCTION_SEARCH", + "EXPORT_AUDIT", + "FULLTEXT_PLAN_CHECKPOINT", + "FULLTEXT_ACQUISITION", + "CORPUS_FREEZE_CHECKPOINT", + "DATA_EXTRACTION_PLAN_CHECKPOINT", + "DATA_EXTRACTION_PILOT", + "DATA_EXTRACTION_ADJUDICATION", + "EXTRACTED_DATA_FREEZE_CHECKPOINT", + "RESEARCHER_REPORT", + "MANUSCRIPT_PACKAGE" +]; + +export function buildModelCallCacheKey(input: ModelCallCacheKeyInput): string { + if (!/^[a-f0-9]{64}$/i.test(input.artifactSha256)) throw new Error("Model cache input requires an artifact SHA-256 hash."); + return createHash("sha256").update(JSON.stringify({ + artifactSha256: input.artifactSha256.toLowerCase(), + codebookVersion: input.codebookVersion, + modelVersion: input.modelVersion, + promptVersion: input.promptVersion, + ruleVersion: input.ruleVersion + })).digest("hex"); +} + +export function modelCallBudgetForStage( + stage: InstitutionalResearchStage, + purpose = "mechanical_stage", + configuredMaximumCalls?: number +): ModelCallBudget { + if (stage === "PROTOCOL_CHECKPOINT" && purpose === "search_strategy_proposal") { + return { stage, purpose, maximumCalls: 1 }; + } + if (stage === "PILOT" && purpose === "pilot_interpretation") { + return { stage, purpose, maximumCalls: 1 }; + } + if ((stage === "TITLE_ABSTRACT_SCREENING" || stage === "FULLTEXT_SCREENING") && purpose === "configured_screening_batch") { + if (!Number.isInteger(configuredMaximumCalls) || (configuredMaximumCalls ?? 0) < 0) { + throw new Error("A configured screening batch requires a non-negative integer model-call limit."); + } + return { stage, purpose, maximumCalls: configuredMaximumCalls as number }; + } + return { stage, purpose, maximumCalls: 0 }; +} + +export function assertModelCallBudget(budget: ModelCallBudget, observedCalls: number): void { + if (!Number.isInteger(observedCalls) || observedCalls < 0) throw new Error("Observed model calls must be a non-negative integer."); + if (observedCalls > budget.maximumCalls) { + throw new Error(`Model-call budget exceeded for ${budget.stage}/${budget.purpose}: observed ${observedCalls}, allowed ${budget.maximumCalls}.`); + } +} diff --git a/packages/longtable-scholar-research/src/pdf-vault.ts b/packages/longtable-scholar-research/src/pdf-vault.ts new file mode 100644 index 0000000..a4ef35b --- /dev/null +++ b/packages/longtable-scholar-research/src/pdf-vault.ts @@ -0,0 +1,248 @@ +import { constants } from "node:fs"; +import { copyFile, lstat, mkdir, readFile } from "node:fs/promises"; +import { basename, dirname, isAbsolute, join, relative, resolve } from "node:path"; +import { createHash } from "node:crypto"; +import { appendJsonlRecord, readJsonlRecords } from "./project-store.js"; + +export type PdfInspectionFailureReason = + | "placeholder_or_empty" + | "not_a_pdf" + | "incomplete_download" + | "corrupt_pdf_structure" + | "symbolic_link_not_allowed"; + +export type PdfInspection = + | { readonly valid: true; readonly path: string; readonly sha256: string; readonly byteLength: number } + | { readonly valid: false; readonly path: string; readonly reason: PdfInspectionFailureReason; readonly byteLength: number }; + +export type PdfAccessBasis = + | "open_access" + | "institutional_subscription" + | "author_copy" + | "repository" + | "researcher_provided" + | "public_domain"; + +export type PdfAcquisitionMethod = + | "official_api" + | "tdm" + | "institutional_browser_download" + | "open_repository_download" + | "researcher_manual_upload"; + +export interface PdfManifestRecord { + readonly paperId: string; + readonly sha256: string; + readonly canonicalVaultPath: string; + readonly acquisitionMethod: PdfAcquisitionMethod; + readonly accessBasis: PdfAccessBasis; + readonly sourceUrl?: string; + readonly version: string; + readonly projectInclusion: string; + readonly screeningState: string; + readonly acquiredAt: string; +} + +export interface IntakeResearchPdfInput { + readonly projectRoot: string; + readonly vaultRoot: string; + readonly projectManifestPath: string; + readonly paperId: string; + readonly candidatePath: string; + readonly acquisitionMethod: PdfAcquisitionMethod; + readonly accessBasis: PdfAccessBasis; + readonly sourceUrl?: string; + readonly version: string; + readonly projectInclusion: string; + readonly screeningState: string; + readonly acquiredAt?: string; +} + +export interface PdfIntakeResult { + readonly created: boolean; + readonly manifestAppended: boolean; + readonly manifest: PdfManifestRecord; +} + +export interface PdfManifestVerification { + readonly passed: boolean; + readonly issues: readonly string[]; +} + +const SENSITIVE_URL_PARAMETER = /(?:access[_-]?token|refresh[_-]?token|session|cookie|auth|authorization|password|passphrase|credential|api[_-]?key|signature|x-amz-signature|mfa)/i; + +function sha256(buffer: Buffer): string { + return createHash("sha256").update(buffer).digest("hex"); +} + +function isInside(parent: string, child: string): boolean { + const path = relative(resolve(parent), resolve(child)); + return path === "" || (!path.startsWith("..") && !isAbsolute(path)); +} + +function requireText(value: string | undefined, label: string): string { + if (!value?.trim()) throw new Error(`${label} is required for Research PDF Vault intake.`); + return value.trim(); +} + +function assertSafeSourceUrl(value?: string): void { + if (!value) return; + let parsed: URL; + try { + parsed = new URL(value); + } catch { + throw new Error("PDF source URL is invalid."); + } + if (parsed.username || parsed.password) { + throw new Error("PDF source URL contains forbidden authentication material."); + } + for (const key of parsed.searchParams.keys()) { + if (SENSITIVE_URL_PARAMETER.test(key)) { + throw new Error("PDF source URL contains forbidden authentication material."); + } + } +} + +export async function inspectPdfCandidate(path: string): Promise { + const resolvedPath = resolve(path); + const metadata = await lstat(resolvedPath); + if (metadata.isSymbolicLink()) { + return { valid: false, path: resolvedPath, reason: "symbolic_link_not_allowed", byteLength: metadata.size }; + } + if (!metadata.isFile() || metadata.size === 0) { + return { valid: false, path: resolvedPath, reason: "placeholder_or_empty", byteLength: metadata.size }; + } + const buffer = await readFile(resolvedPath); + const text = buffer.toString("latin1"); + if (!text.startsWith("%PDF-")) { + return { valid: false, path: resolvedPath, reason: "not_a_pdf", byteLength: buffer.length }; + } + if (!/%%EOF\s*$/.test(text)) { + return { valid: false, path: resolvedPath, reason: "incomplete_download", byteLength: buffer.length }; + } + if (!/\/Type\s*\/Page\b/.test(text) || !/\bxref\b/.test(text) || !/\btrailer\b/.test(text) || !/\bstartxref\b/.test(text)) { + return { valid: false, path: resolvedPath, reason: "corrupt_pdf_structure", byteLength: buffer.length }; + } + return { valid: true, path: resolvedPath, sha256: sha256(buffer), byteLength: buffer.length }; +} + +export function buildPdfManifestRecord( + input: IntakeResearchPdfInput, + inspection: Extract, + canonicalVaultPath: string +): PdfManifestRecord { + assertSafeSourceUrl(input.sourceUrl); + return { + paperId: requireText(input.paperId, "Paper ID"), + sha256: inspection.sha256, + canonicalVaultPath: requireText(canonicalVaultPath, "Canonical Vault-relative PDF path"), + acquisitionMethod: input.acquisitionMethod, + accessBasis: input.accessBasis, + ...(input.sourceUrl ? { sourceUrl: input.sourceUrl } : {}), + version: requireText(input.version, "PDF version"), + projectInclusion: requireText(input.projectInclusion, "Project inclusion state"), + screeningState: requireText(input.screeningState, "Screening state"), + acquiredAt: input.acquiredAt ?? new Date().toISOString() + }; +} + +function canonicalPdfPath(vaultRoot: string, hash: string): string { + return join(resolve(vaultRoot), hash.slice(0, 2), `${hash}.pdf`); +} + +async function ensureCanonicalPdf( + candidatePath: string, + canonicalPath: string, + expectedSha256: string +): Promise { + await mkdir(dirname(canonicalPath), { recursive: true }); + try { + await copyFile(candidatePath, canonicalPath, constants.COPYFILE_EXCL); + return true; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error; + const existing = await inspectPdfCandidate(canonicalPath); + if (!existing.valid || existing.sha256 !== expectedSha256) { + throw new Error(`Research PDF Vault collision or corrupt canonical file at ${canonicalPath}.`); + } + return false; + } +} + +export async function intakeResearchPdf(input: IntakeResearchPdfInput): Promise { + const projectRoot = resolve(input.projectRoot); + const vaultRoot = resolve(input.vaultRoot); + const candidatePath = resolve(input.candidatePath); + const manifestPath = resolve(input.projectManifestPath); + if (isInside(projectRoot, vaultRoot)) { + throw new Error("Research PDF Vault must be outside the research project."); + } + if (isInside(projectRoot, candidatePath)) { + throw new Error("PDF candidate bytes must not be staged inside the research project."); + } + if (!isInside(projectRoot, manifestPath)) { + throw new Error("The PDF manifest must remain inside the research project."); + } + assertSafeSourceUrl(input.sourceUrl); + const inspection = await inspectPdfCandidate(candidatePath); + if (!inspection.valid) { + throw new Error(`PDF intake rejected: ${inspection.reason}.`); + } + const canonicalPath = canonicalPdfPath(vaultRoot, inspection.sha256); + if (!isInside(vaultRoot, canonicalPath)) { + throw new Error("Canonical PDF path escaped the configured Research PDF Vault."); + } + const created = await ensureCanonicalPdf(candidatePath, canonicalPath, inspection.sha256); + const manifest = buildPdfManifestRecord(input, inspection, relative(vaultRoot, canonicalPath)); + const existingRecords = await readJsonlRecords(manifestPath) as unknown as PdfManifestRecord[]; + const alreadyRecorded = existingRecords.some((record) => + record.paperId === manifest.paperId && + record.sha256 === manifest.sha256 && + record.version === manifest.version + ); + if (!alreadyRecorded) { + await appendJsonlRecord(manifestPath, manifest); + } + return { created, manifestAppended: !alreadyRecorded, manifest }; +} + +export async function verifyPdfManifest( + records: readonly PdfManifestRecord[], + vaultRoot: string +): Promise { + const resolvedVaultRoot = resolve(vaultRoot); + const issues: string[] = []; + const seen = new Set(); + for (const record of records) { + const key = `${record.paperId}:${record.sha256}:${record.version}`; + if (seen.has(key)) { + issues.push(`Duplicate PDF manifest record: ${key}.`); + continue; + } + seen.add(key); + if (isAbsolute(record.canonicalVaultPath)) { + issues.push(`PDF manifest contains an absolute path: ${record.paperId}.`); + continue; + } + const canonicalLocalPath = resolve(resolvedVaultRoot, record.canonicalVaultPath); + if (!isInside(resolvedVaultRoot, canonicalLocalPath)) { + issues.push(`PDF path is outside the configured Research PDF Vault: ${record.paperId}.`); + continue; + } + if (basename(canonicalLocalPath) !== `${record.sha256}.pdf`) { + issues.push(`PDF filename does not match its SHA-256: ${record.paperId}.`); + continue; + } + try { + const inspection = await inspectPdfCandidate(canonicalLocalPath); + if (!inspection.valid) { + issues.push(`Canonical PDF is invalid (${inspection.reason}): ${record.paperId}.`); + } else if (inspection.sha256 !== record.sha256) { + issues.push(`Canonical PDF SHA-256 mismatch: ${record.paperId}.`); + } + } catch (error) { + issues.push(`Canonical PDF is unavailable for ${record.paperId}: ${error instanceof Error ? error.message : String(error)}.`); + } + } + return { passed: issues.length === 0, issues }; +} diff --git a/packages/longtable-scholar-research/src/prisma.ts b/packages/longtable-scholar-research/src/prisma.ts new file mode 100644 index 0000000..4956258 --- /dev/null +++ b/packages/longtable-scholar-research/src/prisma.ts @@ -0,0 +1,70 @@ +import type { CorpusCounts } from "./invariants.js"; +import { verifyCorpusInvariants } from "./invariants.js"; + +function requireEligibleCounts(counts: CorpusCounts): void { + if (!verifyCorpusInvariants(counts).prismaEligible) { + throw new Error("Cannot render PRISMA artifacts because corpus count invariants failed."); + } +} + +const COUNT_ROWS: readonly [string, keyof CorpusCounts][] = [ + ["Identified records", "identified"], + ["Parse-rejected records", "parseRejected"], + ["Unique records", "unique"], + ["Duplicate links", "duplicateLinks"], + ["Title/abstract records screened", "titleAbstractScreened"], + ["Title/abstract records excluded", "titleAbstractExcluded"], + ["Reports sought for retrieval", "fulltextSought"], + ["Reports not retrieved", "fulltextNotRetrieved"], + ["Reports assessed for eligibility", "fulltextAssessed"], + ["Full-text reports excluded", "fulltextExcluded"], + ["Studies included in review", "finalIncluded"], + ["Unresolved records", "unresolved"] +]; + +export function renderPrismaCountTable(counts: CorpusCounts): string { + requireEligibleCounts(counts); + return [ + "| Review flow item | Count |", + "|---|---:|", + ...COUNT_ROWS.map(([label, key]) => `| ${label} | ${counts[key]} |`) + ].join("\n"); +} + +function escapeXml(value: string): string { + return value.replaceAll("&", "&").replaceAll("<", "<").replaceAll(">", ">").replaceAll('"', """); +} + +export function renderPrismaFlowSvg(counts: CorpusCounts): string { + requireEligibleCounts(counts); + const nodes = [ + ["Records identified", counts.identified], + ["Unique records screened", counts.titleAbstractScreened], + ["Reports sought for retrieval", counts.fulltextSought], + ["Reports assessed for eligibility", counts.fulltextAssessed], + ["Studies included in review", counts.finalIncluded] + ] as const; + const nodeHeight = 64; + const gap = 36; + const width = 640; + const height = 40 + nodes.length * nodeHeight + (nodes.length - 1) * gap + 40; + const elements: string[] = []; + nodes.forEach(([label, count], index) => { + const y = 40 + index * (nodeHeight + gap); + if (index > 0) { + elements.push(``); + } + elements.push(``); + elements.push(`${escapeXml(label)}`); + elements.push(`n = ${count}`); + }); + return [ + ``, + "Review study flow", + "Deterministic flow generated from the audited corpus counts.", + "", + ...elements, + "" + ].join("\n"); +} + diff --git a/packages/longtable-scholar-research/src/project-store.ts b/packages/longtable-scholar-research/src/project-store.ts new file mode 100644 index 0000000..33a49cf --- /dev/null +++ b/packages/longtable-scholar-research/src/project-store.ts @@ -0,0 +1,282 @@ +import { randomUUID } from "node:crypto"; +import { + appendFile, + mkdir, + open, + readFile, + readdir, + rename, + writeFile +} from "node:fs/promises"; +import { dirname, join, resolve } from "node:path"; +import { createProtocolRevision, type ProtocolRevision } from "./workflow-types.js"; + +export interface ResearchProjectLayout { + readonly root: string; + readonly protocol: { + readonly root: string; + readonly researchQuestion: string; + readonly searchProtocol: string; + readonly inclusionExclusion: string; + readonly databaseProfiles: string; + readonly extractionProfiles: string; + readonly amendments: string; + }; + readonly data: { + readonly root: string; + readonly rawExports: string; + readonly normalized: string; + readonly deduplicated: string; + readonly titleAbstractScreening: string; + readonly fulltextScreening: string; + readonly extractionPilot: string; + readonly extracted: string; + readonly adjudicated: string; + readonly analysisReady: string; + }; + readonly corpus: { + readonly root: string; + readonly papers: string; + readonly acquisitionLedger: string; + readonly pdfManifest: string; + readonly screeningDecisions: string; + readonly extractionRecords: string; + readonly extractionConflicts: string; + readonly extractionAdjudications: string; + readonly manualActionQueue: string; + }; + readonly audit: { + readonly root: string; + readonly stageReceipts: string; + readonly checkpoints: string; + readonly decisions: string; + readonly exclusions: string; + readonly deviations: string; + readonly failures: string; + readonly extractedDataFreezes: string; + }; + readonly reports: string; + readonly manuscript: string; + readonly analysis: string; + readonly references: string; + readonly researchRuns: string; +} + +const JSONL_FILES = [ + "papers", + "acquisitionLedger", + "pdfManifest", + "screeningDecisions", + "extractionRecords", + "extractionConflicts", + "extractionAdjudications" +] as const; + +const AUDIT_JSONL_FILES = [ + "stageReceipts", + "checkpoints", + "decisions", + "exclusions", + "deviations", + "failures", + "extractedDataFreezes" +] as const; + +export function buildResearchProjectLayout(projectRoot: string): ResearchProjectLayout { + const root = resolve(projectRoot); + const protocolRoot = join(root, "protocol"); + const dataRoot = join(root, "data"); + const corpusRoot = join(root, "corpus"); + const auditRoot = join(root, "audit"); + return { + root, + protocol: { + root: protocolRoot, + researchQuestion: join(protocolRoot, "research-question.yaml"), + searchProtocol: join(protocolRoot, "search-protocol.yaml"), + inclusionExclusion: join(protocolRoot, "inclusion-exclusion.yaml"), + databaseProfiles: join(protocolRoot, "database-profiles"), + extractionProfiles: join(protocolRoot, "extraction-profiles"), + amendments: join(protocolRoot, "amendments") + }, + data: { + root: dataRoot, + rawExports: join(dataRoot, "00_raw-exports"), + normalized: join(dataRoot, "01_normalized"), + deduplicated: join(dataRoot, "02_deduplicated"), + titleAbstractScreening: join(dataRoot, "03_title-abstract-screening"), + fulltextScreening: join(dataRoot, "04_fulltext-screening"), + extractionPilot: join(dataRoot, "05_extraction-pilot"), + extracted: join(dataRoot, "06_extracted"), + adjudicated: join(dataRoot, "07_adjudicated"), + analysisReady: join(dataRoot, "08_analysis-ready") + }, + corpus: { + root: corpusRoot, + papers: join(corpusRoot, "papers.jsonl"), + acquisitionLedger: join(corpusRoot, "acquisition-ledger.jsonl"), + pdfManifest: join(corpusRoot, "pdf-manifest.jsonl"), + screeningDecisions: join(corpusRoot, "screening-decisions.jsonl"), + extractionRecords: join(corpusRoot, "extraction-records.jsonl"), + extractionConflicts: join(corpusRoot, "extraction-conflicts.jsonl"), + extractionAdjudications: join(corpusRoot, "extraction-adjudications.jsonl"), + manualActionQueue: join(corpusRoot, "manual-action-queue.csv") + }, + audit: { + root: auditRoot, + stageReceipts: join(auditRoot, "stage-receipts.jsonl"), + checkpoints: join(auditRoot, "checkpoints.jsonl"), + decisions: join(auditRoot, "decisions.jsonl"), + exclusions: join(auditRoot, "exclusions.jsonl"), + deviations: join(auditRoot, "deviations.jsonl"), + failures: join(auditRoot, "failures.jsonl"), + extractedDataFreezes: join(auditRoot, "extracted-data-freezes.jsonl") + }, + reports: join(root, "reports"), + manuscript: join(root, "manuscript"), + analysis: join(root, "analysis"), + references: join(root, "references"), + researchRuns: join(root, ".longtable", "research-runs") + }; +} + +async function ensureFile(path: string, initialContent = ""): Promise { + await mkdir(dirname(path), { recursive: true }); + const handle = await open(path, "a"); + try { + const metadata = await handle.stat(); + if (metadata.size === 0 && initialContent) { + await handle.writeFile(initialContent, "utf8"); + } + } finally { + await handle.close(); + } +} + +export async function writeResearchProjectScaffold(projectRoot: string): Promise { + const layout = buildResearchProjectLayout(projectRoot); + const directories = [ + layout.protocol.root, + layout.protocol.databaseProfiles, + layout.protocol.extractionProfiles, + layout.protocol.amendments, + layout.data.rawExports, + layout.data.normalized, + layout.data.deduplicated, + layout.data.titleAbstractScreening, + layout.data.fulltextScreening, + layout.data.extractionPilot, + layout.data.extracted, + layout.data.adjudicated, + layout.data.analysisReady, + layout.corpus.root, + layout.audit.root, + layout.reports, + layout.manuscript, + layout.analysis, + layout.references, + layout.researchRuns + ]; + await Promise.all(directories.map((directory) => mkdir(directory, { recursive: true }))); + await Promise.all([ + ensureFile(layout.protocol.researchQuestion, "{}\n"), + ensureFile(layout.protocol.searchProtocol, "{}\n"), + ensureFile(layout.protocol.inclusionExclusion, "{}\n"), + ...JSONL_FILES.map((key) => ensureFile(layout.corpus[key])), + ...AUDIT_JSONL_FILES.map((key) => ensureFile(layout.audit[key])), + ensureFile(layout.corpus.manualActionQueue, "paperId,action,reason,status\n") + ]); + return layout; +} + +export async function appendJsonlRecord(path: string, record: Readonly): Promise { + if (!record || typeof record !== "object" || Array.isArray(record)) { + throw new Error("A JSONL record must be an object."); + } + await mkdir(dirname(path), { recursive: true }); + await appendFile(path, `${JSON.stringify(record)}\n`, "utf8"); +} + +export async function readJsonlRecords = Record>( + path: string +): Promise { + let content: string; + try { + content = await readFile(path, "utf8"); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") { + return []; + } + throw error; + } + return content + .split(/\r?\n/) + .map((line) => line.trim()) + .filter(Boolean) + .map((line, index) => { + try { + return JSON.parse(line) as T; + } catch (error) { + throw new Error(`Invalid JSONL at ${path}:${index + 1}: ${error instanceof Error ? error.message : String(error)}`); + } + }); +} + +async function atomicWrite(path: string, content: string): Promise { + await mkdir(dirname(path), { recursive: true }); + const temporaryPath = `${path}.${process.pid}.${randomUUID()}.tmp`; + await writeFile(temporaryPath, content, { encoding: "utf8", flag: "wx" }); + await rename(temporaryPath, path); +} + +function revisionFilename(revision: number): string { + return `protocol-revision-${String(revision).padStart(4, "0")}.json`; +} + +async function existingRevision(path: string): Promise { + try { + const parsed = JSON.parse(await readFile(path, "utf8")) as ProtocolRevision; + return createProtocolRevision(parsed); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") { + return undefined; + } + throw error; + } +} + +export async function writeFrozenProtocolRevision( + layout: ResearchProjectLayout, + revision: ProtocolRevision +): Promise { + const path = join(layout.protocol.amendments, revisionFilename(revision.revision)); + const existing = await existingRevision(path); + if (existing) { + if (existing.protocolHash !== revision.protocolHash) { + throw new Error(`Protocol revision ${revision.revision} is immutable and already contains a different protocol hash.`); + } + return path; + } + + const serialized = `${JSON.stringify(revision, null, 2)}\n`; + await atomicWrite(path, serialized); + await atomicWrite(layout.protocol.searchProtocol, serialized); + return path; +} + +export async function readLatestProtocolRevision( + layout: ResearchProjectLayout +): Promise { + const entries = (await readdir(layout.protocol.amendments)) + .filter((entry) => /^protocol-revision-\d{4}\.json$/.test(entry)) + .sort(); + const latest = entries.at(-1); + if (!latest) { + throw new Error("No frozen ProtocolRevision exists in this research project."); + } + const revision = await existingRevision(join(layout.protocol.amendments, latest)); + if (!revision) { + throw new Error(`Frozen ProtocolRevision disappeared while reading ${latest}.`); + } + return revision; +} diff --git a/packages/longtable-scholar-research/src/references.ts b/packages/longtable-scholar-research/src/references.ts new file mode 100644 index 0000000..ae10f55 --- /dev/null +++ b/packages/longtable-scholar-research/src/references.ts @@ -0,0 +1,109 @@ +import type { CanonicalPaper } from "./corpus.js"; + +export interface UnresolvedCitation { + readonly paperId: string; + readonly missingFields: readonly string[]; +} + +export interface ReferenceFormats { + readonly cslJson: readonly Record[]; + readonly ris: string; + readonly bibtex: string; + readonly apa7References: readonly string[]; + readonly unresolvedCitations: readonly UnresolvedCitation[]; + readonly unresolvedCitationReportMarkdown: string; +} + +function sortedPapers(papers: readonly CanonicalPaper[]): CanonicalPaper[] { + return [...papers].sort((left, right) => + (left.authors[0] ?? "").localeCompare(right.authors[0] ?? "") || + (left.year ?? 0) - (right.year ?? 0) || + left.title.localeCompare(right.title) || + left.paperId.localeCompare(right.paperId) + ); +} + +function authorParts(author: string): { family?: string; given?: string; literal?: string } { + const [family, ...givenParts] = author.split(",").map((part) => part.trim()); + if (givenParts.length === 0) return { literal: author }; + return { family, given: givenParts.join(", ") }; +} + +function initials(value: string): string { + return value.split(/[\s-]+/).filter(Boolean).map((part) => `${part[0]?.toUpperCase() ?? ""}.`).join(" "); +} + +function apaAuthors(authors: readonly string[]): string { + const rendered = authors.map((author) => { + const parts = authorParts(author); + return parts.literal ?? `${parts.family}, ${initials(parts.given ?? "")}`; + }); + if (rendered.length <= 1) return rendered[0] ?? "Unknown author"; + if (rendered.length === 2) return `${rendered[0]}, & ${rendered[1]}`; + return `${rendered.slice(0, -1).join(", ")}, & ${rendered.at(-1)}`; +} + +function bibtexKey(paper: CanonicalPaper): string { + const lead = (paper.authors[0] ?? "unknown").split(",")[0].normalize("NFKD").replace(/[^\p{L}\p{N}]/gu, ""); + return `${lead || "unknown"}${paper.year ?? "nd"}_${paper.paperId.replace(/^paper_?/, "").slice(0, 8)}`; +} + +function escapeBibtex(value: string): string { + return value.replaceAll("{", "\\{").replaceAll("}", "\\}"); +} + +export function renderReferenceFormats(papers: readonly CanonicalPaper[]): ReferenceFormats { + const ordered = sortedPapers(papers); + const cslJson = ordered.map((paper) => ({ + id: paper.paperId, + type: "article-journal", + title: paper.title, + author: paper.authors.map(authorParts), + ...(paper.year ? { issued: { "date-parts": [[paper.year]] } } : {}), + ...(paper.venue ? { "container-title": paper.venue } : {}), + ...(paper.doi ? { DOI: paper.doi } : {}) + })); + const ris = ordered.map((paper) => [ + "TY - JOUR", + `ID - ${paper.paperId}`, + ...paper.authors.map((author) => `AU - ${author}`), + `TI - ${paper.title}`, + ...(paper.year ? [`PY - ${paper.year}`] : []), + ...(paper.venue ? [`JO - ${paper.venue}`] : []), + ...(paper.doi ? [`DO - ${paper.doi}`] : []), + "ER -" + ].join("\n")).join("\n\n"); + const bibtex = ordered.map((paper) => [ + `@article{${bibtexKey(paper)},`, + ` title = {${escapeBibtex(paper.title)}},`, + ` author = {${paper.authors.map(escapeBibtex).join(" and ")}},`, + ...(paper.year ? [` year = {${paper.year}},`] : []), + ...(paper.venue ? [` journal = {${escapeBibtex(paper.venue)}},`] : []), + ...(paper.doi ? [` doi = {${paper.doi}},`] : []), + ` longtable_id = {${paper.paperId}}`, + "}" + ].join("\n")).join("\n\n"); + const apa7References = ordered.map((paper) => { + const year = paper.year ? String(paper.year) : "n.d."; + const venue = paper.venue ? ` *${paper.venue}*.` : ""; + const doi = paper.doi ? ` https://doi.org/${paper.doi}` : ""; + return `${apaAuthors(paper.authors)} (${year}). ${paper.title}.${venue}${doi}`; + }); + const unresolvedCitations = ordered.flatMap((paper) => { + const missingFields = [ + ...(paper.authors.length === 0 ? ["authors"] : []), + ...(!paper.year ? ["year"] : []), + ...(!paper.venue ? ["venue"] : []), + ...(!paper.doi ? ["doi"] : []) + ]; + return missingFields.length > 0 ? [{ paperId: paper.paperId, missingFields }] : []; + }); + const unresolvedCitationReportMarkdown = [ + "# Unresolved citation fields", + "", + ...(unresolvedCitations.length > 0 + ? unresolvedCitations.map((entry) => `- ${entry.paperId}: ${entry.missingFields.join(", ")}`) + : ["- None."]) + ].join("\n"); + return { cslJson, ris, bibtex, apa7References, unresolvedCitations, unresolvedCitationReportMarkdown }; +} diff --git a/packages/longtable-scholar-research/src/render-profiles.ts b/packages/longtable-scholar-research/src/render-profiles.ts new file mode 100644 index 0000000..4738fb9 --- /dev/null +++ b/packages/longtable-scholar-research/src/render-profiles.ts @@ -0,0 +1,54 @@ +export interface WordRenderProfile { + readonly id: string; + readonly label: string; + readonly version: number; + readonly status: "verified" | "provisional_until_journal_rules_verified"; + readonly sourceBasis: string; + readonly page: { + readonly size: "Letter" | "A4"; + readonly marginInches: number; + readonly headerInches: number; + readonly footerInches: number; + }; + readonly body: { + readonly latinFont: string; + readonly eastAsiaFont: string; + readonly fontSizePt: number; + readonly lineSpacing: number; + readonly paragraphAfterPt: number; + readonly firstLineIndentInches: number; + }; + readonly references: { + readonly hangingIndentInches: number; + }; + readonly pageNumber: "top_right" | "bottom_center"; + readonly titlePage: boolean; + readonly templatePath?: string; + readonly templateSha256?: string; +} + +function positive(value: unknown, label: string): number { + if (typeof value !== "number" || !Number.isFinite(value) || value <= 0) throw new Error(`${label} must be a positive number.`); + return value; +} + +export function validateRenderProfile(input: unknown): WordRenderProfile { + if (!input || typeof input !== "object") throw new Error("Render profile must be an object."); + const profile = input as Partial; + if (!profile.id?.trim() || !profile.label?.trim() || !profile.sourceBasis?.trim()) throw new Error("Render profile id, label, and source basis are required."); + if (!Number.isInteger(profile.version) || (profile.version ?? 0) < 1) throw new Error("Render profile version must be a positive integer."); + if (!profile.page || !["Letter", "A4"].includes(profile.page.size)) throw new Error("Render profile page size must be Letter or A4."); + positive(profile.page.marginInches, "Render profile margin"); + positive(profile.page.headerInches, "Render profile header distance"); + positive(profile.page.footerInches, "Render profile footer distance"); + if (!profile.body?.latinFont?.trim() || !profile.body.eastAsiaFont?.trim()) throw new Error("Render profile body fonts are required."); + positive(profile.body.fontSizePt, "Render profile font size"); + positive(profile.body.lineSpacing, "Render profile line spacing"); + if (typeof profile.body.paragraphAfterPt !== "number" || profile.body.paragraphAfterPt < 0) throw new Error("Render profile paragraph spacing must be non-negative."); + if (typeof profile.body.firstLineIndentInches !== "number" || profile.body.firstLineIndentInches < 0) throw new Error("Render profile first-line indent must be non-negative."); + positive(profile.references?.hangingIndentInches, "Render profile reference hanging indent"); + if (!profile.templatePath && profile.templateSha256) throw new Error("A template hash requires a user-supplied template path."); + if (profile.templatePath && !/^[a-f0-9]{64}$/.test(profile.templateSha256 ?? "")) throw new Error("A user-supplied template profile requires a SHA-256 hash."); + return JSON.parse(JSON.stringify(profile)) as WordRenderProfile; +} + diff --git a/packages/longtable-scholar-research/src/reporting.ts b/packages/longtable-scholar-research/src/reporting.ts new file mode 100644 index 0000000..901ca86 --- /dev/null +++ b/packages/longtable-scholar-research/src/reporting.ts @@ -0,0 +1,188 @@ +import { createHash } from "node:crypto"; +import type { CorpusCounts } from "./invariants.js"; +import { verifyCorpusInvariants } from "./invariants.js"; +import { renderPrismaCountTable, renderPrismaFlowSvg } from "./prisma.js"; +import type { ArtifactProvenance, ProtocolRevision, ResearchRun } from "./workflow-types.js"; +import type { ExtractionAuditSummary } from "./extraction.js"; + +export interface DatabaseYield { + readonly databaseId: string; + readonly searchedAt: string; + readonly resultCount: number; + readonly exportedCount: number; +} + +export interface ResearchFailureSummary { + readonly code: string; + readonly count: number; + readonly resolution: string; +} + +export interface ProtocolDeviation { + readonly id: string; + readonly description: string; + readonly decisionRecordId: string; +} + +export interface ResearchReportInput { + readonly run: ResearchRun; + readonly protocol: ProtocolRevision; + readonly counts: CorpusCounts; + readonly databaseYields: readonly DatabaseYield[]; + readonly failures: readonly ResearchFailureSummary[]; + readonly unresolvedIssues: readonly string[]; + readonly deviations: readonly ProtocolDeviation[]; + readonly analysisReadiness: string; + readonly extraction: ExtractionAuditSummary; + readonly requiredActions: readonly string[]; + readonly generatedArtifacts: readonly string[]; + readonly inputArtifactIds: readonly string[]; + readonly renderTimestamp: string; + readonly generatorVersion: string; +} + +export interface SystematicReviewNarrative { + readonly methodsMarkdown: string; + readonly resultsMarkdown?: string; +} + +export interface ResearchOutputs { + readonly researcherReportMarkdown: string; + readonly prismaCountTableMarkdown: string; + readonly prismaFlowSvg: string; + readonly systematicReviewMarkdown: string; + readonly artifactProvenance: readonly ArtifactProvenance[]; +} + +function bullets(values: readonly string[], empty: string): string { + return values.length > 0 ? values.map((value) => `- ${value}`).join("\n") : `- ${empty}`; +} + +function numberWord(value: number): string { + const words = ["Zero", "One", "Two", "Three", "Four", "Five", "Six", "Seven", "Eight", "Nine", "Ten", "Eleven", "Twelve"]; + return words[value] ?? String(value); +} + +export function renderSystematicReviewNarrative(input: { + readonly status: "planned" | "completed"; + readonly protocol: ProtocolRevision; + readonly counts: CorpusCounts; +}): SystematicReviewNarrative { + const databases = input.protocol.databases.join(", "); + if (input.status === "planned") { + return { + methodsMarkdown: `### Planned methods\n\n${databases} will be searched using the frozen database-specific queries and filters recorded in protocol revision ${input.protocol.revision}. Export, normalization, deduplication, screening, full-text acquisition, and synthesis decisions will be retained in append-only audit ledgers.` + }; + } + if (!verifyCorpusInvariants(input.counts).passed) { + throw new Error("Cannot render completed systematic review narrative because corpus count invariants failed."); + } + return { + methodsMarkdown: `### Methods\n\n${databases} were searched using the frozen database-specific queries and filters recorded in protocol revision ${input.protocol.revision}. Export, normalization, deduplication, screening, full-text acquisition, and synthesis decisions were retained in append-only audit ledgers.`, + resultsMarkdown: `### Results\n\n${numberWord(input.counts.identified)} records were identified. After normalization and duplicate linkage, ${input.counts.unique} unique records remained; ${input.counts.fulltextAssessed} reports were assessed in full text and ${input.counts.finalIncluded} studies were included.` + }; +} + +function renderResearcherReport(input: ResearchReportInput): string { + const yields = [...input.databaseYields] + .sort((left, right) => left.databaseId.localeCompare(right.databaseId)) + .map((entry) => `| ${entry.databaseId} | ${entry.searchedAt} | ${entry.resultCount} | ${entry.exportedCount} |`); + const failures = [...input.failures] + .sort((left, right) => left.code.localeCompare(right.code)) + .map((entry) => `- ${entry.code}: ${entry.count} occurrence(s); ${entry.resolution}`); + const deviations = [...input.deviations] + .sort((left, right) => left.id.localeCompare(right.id)) + .map((entry) => `- ${entry.id}: ${entry.description} (DecisionRecord: ${entry.decisionRecordId})`); + return [ + "# Researcher audit report", + "", + `Run: ${input.run.id}`, + `Rendered: ${input.renderTimestamp}`, + "", + "## Protocol and databases", + "", + `- Frozen protocol: ${input.protocol.id}, revision ${input.protocol.revision}, hash ${input.protocol.protocolHash}`, + ...input.protocol.databases.map((database) => `- ${database}: ${input.protocol.queries[database] ?? "query not recorded"}`), + "", + "## Database yields", + "", + "| Database | Searched at | Results | Exported |", + "|---|---|---:|---:|", + ...yields, + "", + "## Failures and recoveries", + "", + failures.length > 0 ? failures.join("\n") : "- None recorded.", + "", + "## Unresolved issues", + "", + bullets(input.unresolvedIssues, "None."), + "", + "## Deviations", + "", + deviations.length > 0 ? deviations.join("\n") : "- None recorded.", + "", + "## Analysis readiness", + "", + `- ${input.analysisReadiness}`, + "", + "## Research-data extraction", + "", + `- Profile: ${input.extraction.profileId} (${input.extraction.profileHash})`, + `- Unit of analysis: ${input.extraction.unitOfAnalysis}; ${input.extraction.unitCount} unit(s), ${input.extraction.recordCount} extraction record(s)`, + `- Double extraction: ${input.extraction.doubleExtractionRequired ? "required" : "not required"}`, + `- Reliability: ${input.extraction.reliabilityStatistic} = ${input.extraction.reliabilityObserved}; threshold ${input.extraction.reliabilityThreshold}`, + `- Conflicts/adjudications: ${input.extraction.conflictCount}/${input.extraction.adjudicatedCount}`, + `- Missing required values: ${input.extraction.missingRequiredValueCount}`, + `- Extracted-data freeze: ${input.extraction.datasetFreezeId} (${input.extraction.datasetHash})`, + "", + "## Required researcher actions", + "", + bullets(input.requiredActions, "None."), + "", + "## Generated artifacts", + "", + bullets(input.generatedArtifacts, "None."), + "" + ].join("\n"); +} + +function provenance(input: ResearchReportInput, kind: string, content: string): ArtifactProvenance { + const sha256 = createHash("sha256").update(content).digest("hex"); + return { + id: `artifact_${kind}_${sha256.slice(0, 16)}`, + sourceStage: "RESEARCHER_REPORT", + inputArtifactIds: [...input.inputArtifactIds].sort(), + protocolRevisionId: input.protocol.id, + generator: `longtable-research-${kind}`, + generatorVersion: input.generatorVersion, + sha256, + createdAt: input.renderTimestamp + }; +} + +export function renderResearchOutputs(input: ResearchReportInput): ResearchOutputs { + const audit = verifyCorpusInvariants(input.counts); + if (!audit.passed) { + throw new Error(`Cannot render research outputs because corpus count invariants failed: ${audit.equations.filter((entry) => !entry.passed).map((entry) => entry.expression).join(", ")}`); + } + const researcherReportMarkdown = renderResearcherReport(input); + const prismaCountTableMarkdown = renderPrismaCountTable(input.counts); + const prismaFlowSvg = renderPrismaFlowSvg(input.counts); + const narrative = renderSystematicReviewNarrative({ status: "completed", protocol: input.protocol, counts: input.counts }); + const extractionMethods = `${narrative.methodsMarkdown} Research variables were coded with extraction profile ${input.extraction.profileId} (${input.extraction.profileHash}); reliability was ${input.extraction.reliabilityObserved} against a ${input.extraction.reliabilityThreshold} threshold, ${input.extraction.conflictCount} conflict(s) were adjudicated, and the extracted dataset was frozen as ${input.extraction.datasetFreezeId}.`; + const systematicReviewMarkdown = [extractionMethods, narrative.resultsMarkdown].filter(Boolean).join("\n\n"); + const contents = [ + ["researcher-report", researcherReportMarkdown], + ["review-flow-table", prismaCountTableMarkdown], + ["review-flow-svg", prismaFlowSvg], + ["systematic-review", systematicReviewMarkdown] + ] as const; + return { + researcherReportMarkdown, + prismaCountTableMarkdown, + prismaFlowSvg, + systematicReviewMarkdown, + artifactProvenance: contents.map(([kind, content]) => provenance(input, kind, content)) + }; +} diff --git a/packages/longtable-scholar-research/src/retry.ts b/packages/longtable-scholar-research/src/retry.ts new file mode 100644 index 0000000..b6146c4 --- /dev/null +++ b/packages/longtable-scholar-research/src/retry.ts @@ -0,0 +1,82 @@ +import { + INSTITUTIONAL_RESEARCH_HARD_STOPS, + INSTITUTIONAL_RESEARCH_RETRYABLE_EVENTS, + type InstitutionalResearchRetryableEvent +} from "./workflow-types.js"; + +export const RESEARCH_FAILURE_INJECTION_MATRIX = [ + "network_timeout", + "delayed_export", + "partial_download", + "corrupt_pdf", + "empty_export", + "count_mismatch", + "placeholder_file", + "interrupted_run", + "missing_doi", + "session_expiry", + "result_cap", + "onedrive_hydration_delay" +] as const; + +export interface RetryPlanInput { + readonly code: string; + readonly maxAttempts: number; + readonly baseDelayMs: number; + readonly maxDelayMs: number; +} + +export interface RetryPlan { + readonly code: InstitutionalResearchRetryableEvent; + readonly maxAttempts: number; + readonly delaysMs: readonly number[]; + readonly requiresResearcherDecision: false; +} + +export interface RetryOutcome { + readonly code: InstitutionalResearchRetryableEvent; + readonly attempts: number; + readonly recovered: boolean; +} + +function positiveInteger(value: number, label: string): void { + if (!Number.isInteger(value) || value < 1) throw new Error(`${label} must be a positive integer.`); +} + +export function buildRetryPlan(input: RetryPlanInput): RetryPlan { + if ((INSTITUTIONAL_RESEARCH_HARD_STOPS as readonly string[]).includes(input.code)) { + throw new Error(`${input.code} is a hard stop and must never be retried automatically.`); + } + if (!(INSTITUTIONAL_RESEARCH_RETRYABLE_EVENTS as readonly string[]).includes(input.code)) { + throw new Error(`Unknown retry event: ${input.code}.`); + } + positiveInteger(input.maxAttempts, "Maximum attempts"); + positiveInteger(input.baseDelayMs, "Base delay"); + positiveInteger(input.maxDelayMs, "Maximum delay"); + if (input.maxDelayMs < input.baseDelayMs) throw new Error("Maximum delay must be at least the base delay."); + const delaysMs = Array.from({ length: Math.max(0, input.maxAttempts - 1) }, (_, index) => + Math.min(input.maxDelayMs, input.baseDelayMs * (2 ** index)) + ); + return { + code: input.code as InstitutionalResearchRetryableEvent, + maxAttempts: input.maxAttempts, + delaysMs, + requiresResearcherDecision: false + }; +} + +export function aggregateRetryAdvisory(outcomes: readonly RetryOutcome[]): { + readonly advisoryCount: 0 | 1; + readonly message: string; +} { + if (outcomes.length === 0) return { advisoryCount: 0, message: "No retryable events were recorded." }; + const counts = new Map(); + for (const outcome of outcomes) counts.set(outcome.code, (counts.get(outcome.code) ?? 0) + 1); + const summary = [...counts.entries()].sort(([left], [right]) => left.localeCompare(right)).map(([code, count]) => `${code} × ${count}`).join("; "); + const unrecovered = outcomes.filter((outcome) => !outcome.recovered).length; + return { + advisoryCount: 1, + message: `Retry summary: ${summary}. ${unrecovered} event(s) remained unresolved; see the retry ledger for artifact-level details.` + }; +} + diff --git a/packages/longtable-scholar-research/src/screening.ts b/packages/longtable-scholar-research/src/screening.ts new file mode 100644 index 0000000..d37b0f4 --- /dev/null +++ b/packages/longtable-scholar-research/src/screening.ts @@ -0,0 +1,323 @@ +import { createHash } from "node:crypto"; +import type { CapabilityResult, InstitutionalResearchHardStop } from "./workflow-types.js"; +import { buildScreeningConflictCheckpoint } from "./checkpoints.js"; + +export type ScreeningStage = "title_abstract" | "fulltext"; +export type ScreeningDisposition = "include" | "exclude" | "pending" | "full_text_needed_for_screening"; +export type ScreeningActor = "human" | "ai" | "deterministic_rule"; + +export interface ScreeningDecisionInput { + readonly paperId: string; + readonly stage: ScreeningStage; + readonly decision: ScreeningDisposition; + readonly reasonCode?: string; + readonly rationale: string; + readonly ruleVersion: string; + readonly codebookVersion: string; + readonly actor: ScreeningActor; + readonly actorId: string; + readonly modelVersion?: string; + readonly promptVersion?: string; + readonly evidenceArtifactIds: readonly string[]; + readonly decidedAt: string; + readonly adjudicatesDecisionIds?: readonly string[]; +} + +export interface ScreeningDecision extends ScreeningDecisionInput { + readonly id: string; +} + +export interface ScreeningCacheKeyInput { + readonly recordOrPdfHash: string; + readonly ruleVersion: string; + readonly codebookVersion: string; + readonly promptVersion: string; + readonly modelVersion: string; +} + +export interface TitleAbstractScreeningCounts { + readonly screened: number; + readonly fulltextCandidates: number; + readonly excluded: number; + readonly pending: number; +} + +export interface HumanAiScreeningConflict { + readonly id: string; + readonly paperId: string; + readonly stage: ScreeningStage; + readonly humanDecisionId: string; + readonly aiDecisionId: string; + readonly decisionIds: readonly string[]; + readonly humanDisposition: ScreeningDisposition; + readonly aiDisposition: ScreeningDisposition; +} + +export interface ScreeningConflictAuditRecord { + readonly id: string; + readonly createdAt: string; + readonly code: "HUMAN_AI_SCREENING_CONFLICT"; + readonly paperIds: readonly string[]; + readonly conflictIds: readonly string[]; + readonly decisionIds: readonly string[]; + readonly basis: string; +} + +export interface BundleScreeningConflictsInput { + readonly runId: string; + readonly protocolRevisionId: string; + readonly conflicts: readonly HumanAiScreeningConflict[]; + readonly createdAt?: string; +} + +export interface FulltextMissingThresholdInput { + readonly sought: number; + readonly retrieved: number; + readonly maximumMissingRate: number; +} + +function stableValue(value: unknown): unknown { + if (Array.isArray(value)) return value.map(stableValue); + if (value && typeof value === "object") { + return Object.fromEntries(Object.entries(value as Record) + .filter(([, nested]) => nested !== undefined) + .sort(([left], [right]) => left.localeCompare(right)) + .map(([key, nested]) => [key, stableValue(nested)])); + } + return value; +} + +function stableJson(value: unknown): string { + return JSON.stringify(stableValue(value)); +} + +function hash(value: unknown): string { + return createHash("sha256").update(stableJson(value)).digest("hex"); +} + +function requireText(value: string | undefined, label: string): string { + if (!value?.trim()) throw new Error(`${label} is required for a screening decision.`); + return value.trim(); +} + +function immutable(value: T): T { + if (value && typeof value === "object" && !Object.isFrozen(value)) { + for (const nested of Object.values(value as Record)) immutable(nested); + Object.freeze(value); + } + return value; +} + +export function classifyTitleAbstractAvailability( + record: { readonly abstract?: string }, + minimumUsableCharacters = 80 +): "screenable" | "full_text_needed_for_screening" { + const normalized = record.abstract?.replace(/\s+/g, " ").trim() ?? ""; + return normalized.length >= minimumUsableCharacters ? "screenable" : "full_text_needed_for_screening"; +} + +export function createScreeningDecision(input: ScreeningDecisionInput): ScreeningDecision { + requireText(input.paperId, "Paper ID"); + requireText(input.rationale, "Screening rationale"); + requireText(input.ruleVersion, "Screening rule version"); + requireText(input.codebookVersion, "Screening codebook version"); + requireText(input.actorId, "Screening actor ID"); + if (!Number.isFinite(Date.parse(input.decidedAt))) { + throw new Error("Screening decision timestamp must be an ISO-compatible date."); + } + if (input.decision === "exclude") requireText(input.reasonCode, "Exclusion reason code"); + if (input.stage === "fulltext" && input.decision === "full_text_needed_for_screening") { + throw new Error("full_text_needed_for_screening is valid only during title/abstract screening."); + } + if (input.actor === "ai" && (!input.modelVersion?.trim() || !input.promptVersion?.trim())) { + throw new Error("AI screening requires model and prompt version provenance."); + } + if (input.evidenceArtifactIds.length === 0) { + throw new Error("A screening decision requires at least one evidence artifact ID."); + } + const canonical = { + ...input, + paperId: input.paperId.trim(), + rationale: input.rationale.trim(), + ruleVersion: input.ruleVersion.trim(), + codebookVersion: input.codebookVersion.trim(), + actorId: input.actorId.trim(), + evidenceArtifactIds: [...new Set(input.evidenceArtifactIds.map((entry) => entry.trim()).filter(Boolean))].sort() + }; + return immutable({ ...canonical, id: `screening_${hash(canonical).slice(0, 24)}` }); +} + +export function appendScreeningDecision( + ledger: readonly ScreeningDecision[], + decision: ScreeningDecision +): ScreeningDecision[] { + const existing = ledger.find((entry) => entry.id === decision.id); + if (!existing) return [...ledger, decision]; + if (stableJson(existing) !== stableJson(decision)) { + throw new Error(`Screening ledger is append-only; decision ${decision.id} cannot be mutated in place.`); + } + return [...ledger]; +} + +function laterDecision(left: ScreeningDecision, right: ScreeningDecision): ScreeningDecision { + const timestampOrder = left.decidedAt.localeCompare(right.decidedAt); + if (timestampOrder !== 0) return timestampOrder > 0 ? left : right; + return left.id.localeCompare(right.id) > 0 ? left : right; +} + +export function reduceLatestScreeningDecisions( + ledger: readonly ScreeningDecision[], + stage: ScreeningStage +): Map { + const byPaper = new Map(); + for (const decision of ledger.filter((entry) => entry.stage === stage)) { + const existing = byPaper.get(decision.paperId); + byPaper.set(decision.paperId, existing ? laterDecision(existing, decision) : decision); + } + return new Map([...byPaper.entries()].sort(([left], [right]) => left.localeCompare(right))); +} + +export function selectFulltextCandidateIds( + paperIds: readonly string[], + ledger: readonly ScreeningDecision[] +): string[] { + const latest = reduceLatestScreeningDecisions(ledger, "title_abstract"); + return [...new Set(paperIds)] + .filter((paperId) => { + const decision = latest.get(paperId)?.decision; + return decision === "include" || decision === "full_text_needed_for_screening"; + }) + .sort(); +} + +export function calculateTitleAbstractScreeningCounts( + paperIds: readonly string[], + ledger: readonly ScreeningDecision[] +): TitleAbstractScreeningCounts { + const uniquePaperIds = [...new Set(paperIds)]; + const latest = reduceLatestScreeningDecisions(ledger, "title_abstract"); + let fulltextCandidates = 0; + let excluded = 0; + let pending = 0; + for (const paperId of uniquePaperIds) { + const decision = latest.get(paperId)?.decision; + if (decision === "include" || decision === "full_text_needed_for_screening") fulltextCandidates += 1; + else if (decision === "exclude") excluded += 1; + else pending += 1; + } + return { screened: uniquePaperIds.length, fulltextCandidates, excluded, pending }; +} + +export function buildScreeningCacheKey(input: ScreeningCacheKeyInput): string { + requireText(input.recordOrPdfHash, "Record or PDF hash"); + requireText(input.ruleVersion, "Screening rule version"); + requireText(input.codebookVersion, "Screening codebook version"); + requireText(input.promptVersion, "Screening prompt version"); + requireText(input.modelVersion, "Screening model version"); + return hash(input); +} + +function latestByActor( + ledger: readonly ScreeningDecision[], + actor: "human" | "ai" +): Map { + const latest = new Map(); + for (const decision of ledger.filter((entry) => entry.actor === actor)) { + const key = `${decision.paperId}:${decision.stage}`; + const existing = latest.get(key); + latest.set(key, existing ? laterDecision(existing, decision) : decision); + } + return latest; +} + +export function detectHumanAiScreeningConflicts( + ledger: readonly ScreeningDecision[] +): HumanAiScreeningConflict[] { + const humans = latestByActor(ledger, "human"); + const ais = latestByActor(ledger, "ai"); + const conflicts: HumanAiScreeningConflict[] = []; + for (const [key, human] of humans) { + const ai = ais.get(key); + if (!ai || ai.decision === human.decision) continue; + const decisionIds = [human.id, ai.id].sort(); + conflicts.push({ + id: `screening_conflict_${hash({ key, decisionIds }).slice(0, 20)}`, + paperId: human.paperId, + stage: human.stage, + humanDecisionId: human.id, + aiDecisionId: ai.id, + decisionIds, + humanDisposition: human.decision, + aiDisposition: ai.decision + }); + } + return conflicts.sort((left, right) => left.paperId.localeCompare(right.paperId) || left.stage.localeCompare(right.stage)); +} + +export function bundleScreeningConflicts(input: BundleScreeningConflictsInput): { + readonly auditRecord: ScreeningConflictAuditRecord; + readonly question: ReturnType; +} { + if (input.conflicts.length === 0) throw new Error("At least one human–AI screening conflict is required."); + const createdAt = input.createdAt ?? new Date().toISOString(); + const paperIds = [...new Set(input.conflicts.map((conflict) => conflict.paperId))].sort(); + const conflictIds = input.conflicts.map((conflict) => conflict.id).sort(); + const decisionIds = [...new Set(input.conflicts.flatMap((conflict) => conflict.decisionIds))].sort(); + const auditRecord: ScreeningConflictAuditRecord = { + id: `screening_conflict_bundle_${hash({ input: input.runId, paperIds, conflictIds }).slice(0, 20)}`, + createdAt, + code: "HUMAN_AI_SCREENING_CONFLICT", + paperIds, + conflictIds, + decisionIds, + basis: "Latest human and AI screening dispositions disagree for the affected paper/stage pairs." + }; + return { + auditRecord, + question: buildScreeningConflictCheckpoint({ + runId: input.runId, + protocolRevisionId: input.protocolRevisionId, + paperIds, + conflict: `${input.conflicts.length} human–AI screening conflict(s) require adjudication`, + createdAt + }) + }; +} + +export function applyScreeningAdjudication( + ledger: readonly ScreeningDecision[], + input: ScreeningDecisionInput & { readonly adjudicatesDecisionIds: readonly string[] } +): ScreeningDecision[] { + if (input.actor !== "human") throw new Error("Screening adjudication must be recorded by a human actor."); + if (input.adjudicatesDecisionIds.length < 2) throw new Error("Screening adjudication must reference the conflicting decision IDs."); + const existingIds = new Set(ledger.map((decision) => decision.id)); + const missingIds = input.adjudicatesDecisionIds.filter((id) => !existingIds.has(id)); + if (missingIds.length > 0) throw new Error(`Screening adjudication references unknown decisions: ${missingIds.join(", ")}.`); + return appendScreeningDecision(ledger, createScreeningDecision({ + ...input, + adjudicatesDecisionIds: [...new Set(input.adjudicatesDecisionIds)].sort() + })); +} + +function thresholdHardStop(code: InstitutionalResearchHardStop, reason: string): Extract, { status: "hard_stop" }> { + return { status: "hard_stop", code, reason }; +} + +export function evaluateFulltextMissingThreshold( + input: FulltextMissingThresholdInput +): CapabilityResult<{ readonly missing: number; readonly missingRate: number }> { + if (!Number.isInteger(input.sought) || !Number.isInteger(input.retrieved) || input.sought < 0 || input.retrieved < 0 || input.retrieved > input.sought) { + throw new Error("Full-text sought and retrieved counts must be valid non-negative integers."); + } + if (input.maximumMissingRate < 0 || input.maximumMissingRate > 1) { + throw new Error("Maximum full-text missing rate must be between 0 and 1."); + } + const missing = input.sought - input.retrieved; + const missingRate = input.sought === 0 ? 0 : missing / input.sought; + return missingRate <= input.maximumMissingRate + ? { status: "supported", value: { missing, missingRate } } + : thresholdHardStop( + "FULLTEXT_MISSING_THRESHOLD_EXCEEDED", + `Full-text missing rate ${(missingRate * 100).toFixed(1)}% exceeds the approved ${(input.maximumMissingRate * 100).toFixed(1)}% threshold.` + ); +} diff --git a/packages/longtable-scholar-research/src/word-renderer.ts b/packages/longtable-scholar-research/src/word-renderer.ts new file mode 100644 index 0000000..a10e37f --- /dev/null +++ b/packages/longtable-scholar-research/src/word-renderer.ts @@ -0,0 +1,47 @@ +import { createHash } from "node:crypto"; +import { existsSync, readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; +import { dirname, resolve } from "node:path"; +import { spawnSync } from "node:child_process"; +import type { ApprovedManuscript } from "./manuscript.js"; +import type { WordRenderProfile } from "./render-profiles.js"; + +export interface WordRenderInput { + readonly manuscript: ApprovedManuscript; + readonly profile: WordRenderProfile; + readonly outputPath: string; + readonly pythonExecutable?: string; +} + +export interface WordRenderResult { + readonly outputPath: string; + readonly contentSha256: string; + readonly profileId: string; +} + +function stable(value: unknown): unknown { + if (Array.isArray(value)) return value.map(stable); + if (value && typeof value === "object") { + return Object.fromEntries(Object.entries(value as Record).sort(([left], [right]) => left.localeCompare(right)).map(([key, entry]) => [key, stable(entry)])); + } + return value; +} + +export function renderWordManuscript(input: WordRenderInput): WordRenderResult { + const here = dirname(fileURLToPath(import.meta.url)); + const renderer = resolve(here, "..", "resources", "render_word.py"); + if (!existsSync(renderer)) throw new Error(`DOCX renderer resource is missing: ${renderer}`); + if (input.profile.templatePath) { + if (!existsSync(input.profile.templatePath)) throw new Error(`User-supplied DOCX template does not exist: ${input.profile.templatePath}`); + const actualTemplateSha256 = createHash("sha256").update(readFileSync(input.profile.templatePath)).digest("hex"); + if (actualTemplateSha256 !== input.profile.templateSha256) throw new Error("User-supplied DOCX template hash does not match the validated render profile."); + } + const payload = JSON.stringify(stable({ manuscript: input.manuscript, profile: input.profile })); + const result = spawnSync(input.pythonExecutable ?? "python3", [renderer, input.outputPath], { input: payload, encoding: "utf8", maxBuffer: 16 * 1024 * 1024 }); + if (result.status !== 0) throw new Error(`DOCX rendering failed: ${(result.stderr || result.stdout || "unknown renderer error").trim()}`); + return { + outputPath: input.outputPath, + contentSha256: createHash("sha256").update(payload).digest("hex"), + profileId: input.profile.id + }; +} diff --git a/packages/longtable-scholar-research/src/workflow-state.ts b/packages/longtable-scholar-research/src/workflow-state.ts new file mode 100644 index 0000000..109122f --- /dev/null +++ b/packages/longtable-scholar-research/src/workflow-state.ts @@ -0,0 +1,191 @@ +import { + INSTITUTIONAL_RESEARCH_HARD_STOPS, + nextInstitutionalResearchStage, + type InstitutionalResearchHardStop, + type InstitutionalResearchStage, + type ResearchRun, + type StageReceipt +} from "./workflow-types.js"; +import type { QuestionRecord } from "@longtable/core"; + +export interface CreateResearchRunInput { + readonly id: string; + readonly createdAt?: string; + readonly stage?: InstitutionalResearchStage; + readonly protocolRevisionId?: string; + readonly institutionProfileId?: string; +} + +export interface CreateStageReceiptInput extends StageReceipt {} + +export interface BlockResearchRunInput { + readonly code: InstitutionalResearchHardStop; + readonly questionRecordId: string; + readonly checkpointKey: string; + readonly safeCursor: string; + readonly blockedAt?: string; +} + +export interface ResearchDecisionRecordReference { + readonly id: string; + readonly checkpointKey: string; + readonly questionRecordId: string; +} + +function now(): string { + return new Date().toISOString(); +} + +function immutable(value: T): T { + if (value && typeof value === "object" && !Object.isFrozen(value)) { + for (const nested of Object.values(value as Record)) { + immutable(nested); + } + Object.freeze(value); + } + return value; +} + +function requireText(value: string | undefined, label: string): string { + if (!value?.trim()) { + throw new Error(`${label} is required.`); + } + return value.trim(); +} + +export function createResearchRun(input: CreateResearchRunInput): ResearchRun { + const createdAt = input.createdAt ?? now(); + return immutable({ + id: requireText(input.id, "Research run ID"), + createdAt, + updatedAt: createdAt, + status: "planned", + stage: input.stage ?? "SETUP", + ...(input.protocolRevisionId ? { protocolRevisionId: input.protocolRevisionId } : {}), + ...(input.institutionProfileId ? { institutionProfileId: input.institutionProfileId } : {}) + }); +} + +export function createStageReceipt(input: CreateStageReceiptInput): StageReceipt { + requireText(input.id, "StageReceipt ID"); + requireText(input.runId, "StageReceipt run ID"); + requireText(input.protocolRevisionId, "StageReceipt protocol revision ID"); + requireText(input.cursor, "StageReceipt cursor"); + return immutable({ + ...input, + inputArtifactIds: [...input.inputArtifactIds], + outputArtifactIds: [...input.outputArtifactIds] + }); +} + +function assertReceiptMatchesRun(run: ResearchRun, receipt: StageReceipt): void { + if (run.status === "blocked") { + throw new Error("A blocked research run requires a linked DecisionRecord before it can advance."); + } + if (run.status === "completed" || run.status === "failed") { + throw new Error(`A ${run.status} research run cannot advance.`); + } + if (receipt.runId !== run.id) { + throw new Error("StageReceipt belongs to a different research run."); + } + if (receipt.stage !== run.stage) { + throw new Error(`StageReceipt must complete the current stage ${run.stage}.`); + } + if (run.protocolRevisionId && receipt.protocolRevisionId !== run.protocolRevisionId) { + throw new Error("StageReceipt protocol revision does not match the research run."); + } +} + +export function advanceResearchRun( + run: ResearchRun, + receipt: StageReceipt, + advancedAt = now() +): ResearchRun { + assertReceiptMatchesRun(run, receipt); + const nextStage = nextInstitutionalResearchStage(run.stage); + return immutable({ + ...run, + updatedAt: advancedAt, + status: nextStage ? "running" : "completed", + stage: nextStage ?? run.stage, + latestSafeCursor: receipt.cursor + }); +} + +export function blockResearchRun(run: ResearchRun, input: BlockResearchRunInput): ResearchRun { + if (!INSTITUTIONAL_RESEARCH_HARD_STOPS.includes(input.code)) { + throw new Error(`Unknown institutional research hard stop: ${input.code}`); + } + requireText(input.questionRecordId, "Blocking QuestionRecord ID"); + requireText(input.checkpointKey, "Blocking checkpoint key"); + requireText(input.safeCursor, "Safe cursor"); + if (run.status === "completed" || run.status === "failed") { + throw new Error(`A ${run.status} research run cannot be blocked.`); + } + + const repeatedStop = run.status === "blocked" && + run.blockingCode === input.code && + run.blockingQuestionRecordId === input.questionRecordId && + run.blockingCheckpointKey === input.checkpointKey; + + return immutable({ + ...run, + updatedAt: input.blockedAt ?? now(), + status: "blocked", + blockingQuestionRecordId: input.questionRecordId, + blockingCode: input.code, + blockingCheckpointKey: input.checkpointKey, + latestSafeCursor: repeatedStop ? run.latestSafeCursor : input.safeCursor + }); +} + +export function resumeResearchRun( + run: ResearchRun, + decisions: readonly ResearchDecisionRecordReference[], + resumedAt = now() +): ResearchRun { + if (run.status !== "blocked") { + throw new Error("Only a blocked research run can resume."); + } + const questionRecordId = requireText(run.blockingQuestionRecordId, "Blocking QuestionRecord ID"); + const checkpointKey = requireText(run.blockingCheckpointKey, "Blocking checkpoint key"); + const decision = decisions.find((candidate) => + Boolean(candidate.id.trim()) && + candidate.questionRecordId === questionRecordId && + candidate.checkpointKey === checkpointKey + ); + if (!decision) { + throw new Error("A linked DecisionRecord is required before this research run can resume."); + } + + const { + blockingQuestionRecordId: _question, + blockingCode: _code, + blockingCheckpointKey: _checkpoint, + ...safeRun + } = run; + return immutable({ + ...safeRun, + updatedAt: resumedAt, + status: "running", + resumedByDecisionRecordId: decision.id + }); +} + +export function blockResearchRunForQuestion( + run: ResearchRun, + code: InstitutionalResearchHardStop, + question: QuestionRecord, + safeCursor: string +): ResearchRun { + if (question.status !== "pending" || question.hardStop !== true || !question.prompt.checkpointKey) { + throw new Error("A research run can be blocked only by a pending hard-stop QuestionRecord with a checkpoint key."); + } + return blockResearchRun(run, { + code, + questionRecordId: question.id, + checkpointKey: question.prompt.checkpointKey, + safeCursor, + blockedAt: question.createdAt + }); +} diff --git a/packages/longtable-scholar-research/src/workflow-types.ts b/packages/longtable-scholar-research/src/workflow-types.ts new file mode 100644 index 0000000..86e5a2d --- /dev/null +++ b/packages/longtable-scholar-research/src/workflow-types.ts @@ -0,0 +1,270 @@ +import { createHash } from "node:crypto"; + +export const INSTITUTIONAL_RESEARCH_STAGES = [ + "SETUP", + "PILOT", + "PROTOCOL_CHECKPOINT", + "PRODUCTION_SEARCH", + "EXPORT_AUDIT", + "TITLE_ABSTRACT_SCREENING", + "FULLTEXT_PLAN_CHECKPOINT", + "FULLTEXT_ACQUISITION", + "FULLTEXT_SCREENING", + "CORPUS_FREEZE_CHECKPOINT", + "DATA_EXTRACTION_PLAN_CHECKPOINT", + "DATA_EXTRACTION_PILOT", + "DATA_EXTRACTION", + "DATA_EXTRACTION_ADJUDICATION", + "EXTRACTED_DATA_FREEZE_CHECKPOINT", + "ANALYSIS", + "SYNTHESIS", + "RESEARCHER_REPORT", + "MANUSCRIPT_PACKAGE" +] as const; + +export type InstitutionalResearchStage = typeof INSTITUTIONAL_RESEARCH_STAGES[number]; + +export const INSTITUTIONAL_RESEARCH_HARD_STOPS = [ + "LOGIN_REQUIRED", + "MFA_OR_CAPTCHA_REQUIRED", + "SESSION_EXPIRED", + "TERMS_OR_ACCESS_UNCLEAR", + "QUERY_DRIFT_DETECTED", + "DATABASE_RESULT_CAP_REACHED", + "EXPORT_COUNT_MISMATCH", + "DOWNLOAD_PATTERN_CHANGED", + "UI_SIGNATURE_CHANGED", + "SCREENING_RULE_AMBIGUOUS", + "HUMAN_AI_SCREENING_CONFLICT", + "FULLTEXT_MISSING_THRESHOLD_EXCEEDED", + "ANALYSIS_METHOD_CHANGE", + "CORPUS_FREEZE_REQUIRED", + "EXTRACTION_SCHEMA_UNFROZEN", + "EXTRACTION_RULE_AMBIGUOUS", + "EXTRACTION_RELIABILITY_BELOW_THRESHOLD", + "DOUBLE_EXTRACTION_CONFLICT", + "EXTRACTED_DATA_FREEZE_REQUIRED" +] as const; + +export type InstitutionalResearchHardStop = typeof INSTITUTIONAL_RESEARCH_HARD_STOPS[number]; + +export const INSTITUTIONAL_RESEARCH_RETRYABLE_EVENTS = [ + "TRANSIENT_NETWORK_TIMEOUT", + "DELAYED_EXPORT_GENERATION", + "INCOMPLETE_DOWNLOAD", + "TEMPORARY_PAGE_RENDER_FAILURE", + "FILE_PROVIDER_HYDRATION_DELAY" +] as const; + +export type InstitutionalResearchRetryableEvent = typeof INSTITUTIONAL_RESEARCH_RETRYABLE_EVENTS[number]; + +export const INSTITUTIONAL_RESEARCH_CAPABILITIES = [ + "open", + "verifyAuthenticated", + "submitQuery", + "applyFilters", + "readResultCount", + "exportMetadata", + "verifyExport", + "resolveFulltext", + "downloadPermittedPdf", + "suspend", + "resume" +] as const; + +export type InstitutionalResearchCapability = typeof INSTITUTIONAL_RESEARCH_CAPABILITIES[number]; +export type ResearchRunStatus = "planned" | "running" | "blocked" | "completed" | "failed"; + +export interface SearchFilters { + readonly years?: readonly [number, number]; + readonly languages: readonly string[]; + readonly publicationTypes: readonly string[]; + readonly sourceTypes?: readonly string[]; +} + +export interface ProtocolRevisionInput { + readonly id: string; + readonly revision: number; + readonly frozenAt: string; + readonly decisionRecordId: string; + readonly databases: readonly string[]; + readonly queries: Readonly>; + readonly filters: SearchFilters; +} + +export interface ProtocolRevision extends ProtocolRevisionInput { + readonly protocolHash: string; +} + +export interface ResearchRun { + readonly id: string; + readonly createdAt: string; + readonly updatedAt: string; + readonly status: ResearchRunStatus; + readonly stage: InstitutionalResearchStage; + readonly protocolRevisionId?: string; + readonly institutionProfileId?: string; + readonly latestSafeCursor?: string; + readonly blockingQuestionRecordId?: string; + readonly blockingCode?: InstitutionalResearchHardStop; + readonly blockingCheckpointKey?: string; + readonly resumedByDecisionRecordId?: string; +} + +export interface StageReceipt { + readonly id: string; + readonly runId: string; + readonly stage: InstitutionalResearchStage; + readonly protocolRevisionId: string; + readonly inputArtifactIds: readonly string[]; + readonly outputArtifactIds: readonly string[]; + readonly cursor: string; + readonly createdAt: string; +} + +export type InstitutionAccessMode = "official_api" | "tdm" | "browser_sso" | "public_browser" | "manual_export"; + +export interface InstitutionDatabaseAccess { + readonly databaseId: string; + readonly accessMode: InstitutionAccessMode; + readonly exportFormats?: readonly string[]; + readonly resultCap?: number; + readonly recipeId?: string; +} + +export interface InstitutionStorageProfile { + readonly pdfVaultRoot: string; + readonly exportRoot?: string; +} + +export interface InstitutionProfileInput { + readonly id: string; + readonly institutionName: string; + readonly approvedDecisionRecordId: string; + readonly databases: readonly InstitutionDatabaseAccess[]; + readonly storage: InstitutionStorageProfile; + readonly checkpointPolicy: "mcp_first"; + readonly calibratedAt: string; + readonly lastVerifiedAt: string; + readonly resolverUrls?: readonly string[]; + readonly profileVersion?: number; + readonly calibrationStatus?: "draft_fixture" | "calibrating" | "approved"; + readonly permittedActions?: readonly string[]; + readonly prohibitedActions?: readonly string[]; +} + +export interface InstitutionProfile extends InstitutionProfileInput {} + +export interface ArtifactProvenance { + readonly id: string; + readonly sourceStage: InstitutionalResearchStage; + readonly inputArtifactIds: readonly string[]; + readonly protocolRevisionId: string; + readonly generator: string; + readonly generatorVersion: string; + readonly localPath?: string; + readonly paperId?: string; + readonly sha256: string; + readonly accessBasis?: string; + readonly createdAt: string; +} + +export type CapabilityResult = + | { readonly status: "supported"; readonly value: T } + | { readonly status: "unsupported"; readonly reason: string } + | { readonly status: "hard_stop"; readonly code: InstitutionalResearchHardStop; readonly reason: string } + | { readonly status: "retryable"; readonly code: InstitutionalResearchRetryableEvent; readonly reason: string }; + +export interface DatabaseAdapter { + readonly id: string; + readonly version: string; + readonly capabilities: readonly InstitutionalResearchCapability[]; +} + +const FORBIDDEN_PROFILE_KEY = /(?:password|passphrase|credential|cookie|session[_-]?token|access[_-]?token|refresh[_-]?token|auth(?:entication|orization)?[_-]?header|mfa|one[_-]?time[_-]?(?:code|password)|captcha)/i; + +function stableValue(value: unknown): unknown { + if (Array.isArray(value)) { + return value.map(stableValue); + } + if (value && typeof value === "object") { + return Object.fromEntries( + Object.entries(value as Record) + .filter(([, entry]) => entry !== undefined) + .sort(([left], [right]) => left.localeCompare(right)) + .map(([key, entry]) => [key, stableValue(entry)]) + ); + } + return value; +} + +function cloneJson(value: T): T { + return JSON.parse(JSON.stringify(value)) as T; +} + +function deepFreeze(value: T): Readonly { + if (value && typeof value === "object" && !Object.isFrozen(value)) { + for (const nested of Object.values(value as Record)) { + deepFreeze(nested); + } + Object.freeze(value); + } + return value; +} + +function assertNoAuthenticationMaterial(value: unknown, path = "profile"): void { + if (!value || typeof value !== "object") { + return; + } + for (const [key, nested] of Object.entries(value as Record)) { + const nestedPath = `${path}.${key}`; + if (FORBIDDEN_PROFILE_KEY.test(key)) { + throw new Error(`Institution profile contains forbidden authentication material at ${nestedPath}.`); + } + assertNoAuthenticationMaterial(nested, nestedPath); + } +} + +function protocolHash(input: ProtocolRevisionInput): string { + const canonicalProtocol = stableValue({ + databases: input.databases, + queries: input.queries, + filters: input.filters + }); + return createHash("sha256").update(JSON.stringify(canonicalProtocol)).digest("hex"); +} + +export function nextInstitutionalResearchStage( + stage: InstitutionalResearchStage +): InstitutionalResearchStage | undefined { + const index = INSTITUTIONAL_RESEARCH_STAGES.indexOf(stage); + return index >= 0 ? INSTITUTIONAL_RESEARCH_STAGES[index + 1] : undefined; +} + +export function createProtocolRevision(input: ProtocolRevisionInput): ProtocolRevision { + if (!input.id.trim() || !input.decisionRecordId.trim()) { + throw new Error("A protocol revision requires stable protocol and DecisionRecord IDs."); + } + if (!Number.isInteger(input.revision) || input.revision < 1) { + throw new Error("A protocol revision number must be a positive integer."); + } + if (input.databases.length === 0) { + throw new Error("A protocol revision requires at least one database."); + } + const copy = cloneJson(input); + return deepFreeze({ + ...copy, + protocolHash: protocolHash(copy) + }) as ProtocolRevision; +} + +export function createInstitutionProfile(input: InstitutionProfileInput): InstitutionProfile { + assertNoAuthenticationMaterial(input); + if (!input.id.trim() || !input.institutionName.trim() || !input.approvedDecisionRecordId.trim()) { + throw new Error("An institution profile requires stable institution and approval DecisionRecord IDs."); + } + if (!input.storage.pdfVaultRoot.trim()) { + throw new Error("An institution profile requires a Research PDF Vault root."); + } + return deepFreeze(cloneJson(input)) as InstitutionProfile; +} diff --git a/packages/longtable/README.md b/packages/longtable/README.md index 0cb1c95..4eb50a7 100644 --- a/packages/longtable/README.md +++ b/packages/longtable/README.md @@ -81,9 +81,17 @@ longtable question --prompt "" longtable decide --question --answer longtable spec read --cwd "" longtable search --query "" +longtable research doctor +longtable research init --cwd "" +longtable research status --cwd "" longtable panel --prompt "review this measurement plan" --json ``` +`longtable research` is the single public surface for institutional-database +search, protocol freeze, corpus, PDF Vault, screening, reporting, and manuscript +packaging. The older `longtable scholar-research` command remains a deprecated +compatibility alias for existing automation. + `longtable start` remains available for scripted workspace creation with `--no-interview --json`, but it is not the primary research-start surface. diff --git a/packages/longtable/src/cli.ts b/packages/longtable/src/cli.ts index 02bab30..50fb617 100644 --- a/packages/longtable/src/cli.ts +++ b/packages/longtable/src/cli.ts @@ -37,6 +37,11 @@ import { publisherConfigs, runResearchSearch, SEARCH_SOURCES, + buildResearchProjectLayout, + executeInstitutionalResearchCommand, + INSTITUTIONAL_RESEARCH_COMMANDS, + readLatestProtocolRevision, + writeResearchProjectScaffold, writeScholarResearchRunScaffold, summarizeConfiguredPublisherAccess, type EvidenceRun, @@ -412,9 +417,22 @@ function usage(): string { " longtable access doctor [--doi ] [--publisher auto|elsevier|springer_nature|wiley|taylor_francis|all] [--json]", " longtable access probe --doi [--publisher auto|elsevier|springer_nature|wiley|taylor_francis] [--json]", " longtable search --query [--intent literature|theory|measurement|citation|metadata|venue] [--field ] [--source all|crossref,arxiv,openalex,semantic_scholar,pubmed,eric,doaj] [--must ] [--exclude ] [--limit ] [--allow-partial] [--publisher-access] [--record] [--cwd ] [--json]", - " longtable scholar-research doctor [--json]", - " longtable scholar-research scaffold [--cwd ] [--run-id ] [--json]", - " longtable scholar-research smoke-fixture [--json]", + " longtable research doctor [--json]", + " longtable research init [--cwd ] [--json]", + " longtable research status [--cwd ] [--json]", + " longtable research pilot --run-id [--protocol-reference ] [--cwd ] [--json]", + " longtable research freeze --protocol-file [--cwd ] [--json]", + " longtable research ingest-export --file --database --format csv|ris|nbib|json|spreadsheet_xml [--cwd ] [--json]", + " longtable research screen --decisions-file [--cwd ] [--json]", + " longtable research acquire --candidate-path --vault-root --paper-id --acquisition-method --access-basis --version