Refactor corpus I/O, parsers, and pipeline orchestration - #6
Open
apodobe wants to merge 11 commits into
Open
Conversation
- Add atomic_write_text, load_json, write_json helpers in corpus_io - Use atomic writes in write_manifest and write_labs - Remove existing_keys skip in _append_lab_rows; always append rows - Route dedup_labs, apply_loinc_map, merge_labs through load_labs/write_labs Co-authored-by: apodobe <apodobe@users.noreply.github.com>
…cture_pdfs Remove module-level _skip_patient_dob and _configure_patient_dob. Thread patient_dob parameter through _first_iso_date, parse helpers, _parse_entry, and run(). run() loads DOB from corpus when not provided. Co-authored-by: apodobe <apodobe@users.noreply.github.com>
Introduce utc_now_iso() and corpus argparse helpers, then replace duplicated inline datetime formatting and --corpus parsing across scan/extract/pipeline modules. Co-authored-by: apodobe <apodobe@users.noreply.github.com>
Parse each lab document once and store the extracted dict in targets. Load patient_dob once via load_patient_dob and pass to _parse_entry. Reuse stored extraction in dry_run and write paths. Add test_dedup_vendor_priority for medsi-over-emias vendor priority. Add test_merge_parses_each_document_once to guard against re-parsing. Co-authored-by: apodobe <apodobe@users.noreply.github.com>
Refactor medbots/pipeline/run.py to import and invoke module functions directly instead of spawning python -m subprocesses for merge labs, LOINC mapping, dedup, goals/supplements/protocols extraction, discrepancies, LHM generation, goal reconciliation, validation, and corpus index writing. Keep subprocess only for optional legacy scripts (bridge_legacy_flat_pdfs, reconcile_weekly_pending). Update cli.py structure and validate commands to call local_structure_pdfs.run() and validate_corpus.main() directly. Co-authored-by: apodobe <apodobe@users.noreply.github.com>
Extract vendor detection, lab-source filtering, and structure-target selection into medbots/vendor_registry.py. Update merge_labs_corpus and local_structure_pdfs to use the shared helpers, and add unit tests. Co-authored-by: apodobe <apodobe@users.noreply.github.com>
Add _dedup_key() that builds grouping keys as (loinc:code|key:canonical_key, specimen_date). Rows with the same LOINC on the same date are deduplicated even when canonical_key differs. Vendor priority (medsi > gemotest > emias) still applies within each group. Co-authored-by: apodobe <apodobe@users.noreply.github.com>
Add medbots/log_config.py with setup_logging() and get_logger(). Use logger.info/error in pipeline/run.py for step progress and failures. Keep user-facing print in cli.py; route structure warnings to logger. Use logger in dedup_labs and merge_labs_corpus main() entry points. Co-authored-by: apodobe <apodobe@users.noreply.github.com>
Configure ruff (E/F/I/UP, line-length 100) and mypy (packages medbots, ignore_missing_imports) in pyproject.toml. Run both in GitHub Actions before pytest. Auto-fix import order and style issues across medbots; fix mypy narrowing in import_apple_health. Temporarily ignore_errors for local_structure_pdfs during incremental parser split. Co-authored-by: apodobe <apodobe@users.noreply.github.com>
Extract vendor-specific parse_* functions and shared utilities into medbots/parsers/ (common, emias, gemotest, medsi). Keep local_structure_pdfs.py as a thin orchestrator for manifest routing, run(), and backward-compatible re-exports for tests. Pure refactor with no behavior changes; all 100 tests pass. Co-authored-by: apodobe <apodobe@users.noreply.github.com>
Co-authored-by: apodobe <apodobe@users.noreply.github.com>
apodobe
marked this pull request as ready for review
August 11, 2026 14:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactoring from code-quality audit — data correctness, architecture cleanup, and CI hygiene.
Data correctness
atomic_write_text,write_jsonincorpus_io)_append_lab_rowsno longer drops duplicate keys; vendor priority indedup_labs(medsi > gemotest > emias)(loinc:{code}|key:{canonical_key}, specimen_date)merge_labs_corpusparses each document onceArchitecture
medbots/parsers/— split from 1265-line monolith:common,emias,gemotest,medsi;local_structure_pdfsis ~330-line orchestratorvendor_registry— unified vendor detection and entry targetingpipeline/run.py— direct function calls (subprocess only for optional scripts)time_util,cli_args,log_config— shared utilitiesCI / quality
log_config)Tests
Follow-up (not blocking)