Expertise modulates automation bias and sentinel behavior in human-AI collaborative diagnosis of neonatal pneumoperitoneum
Code for Lee et al., npj Digital Medicine (2026). doi:10.1038/s41746-026-03399-3 (open access).
Research and education use only. The models and tools in this repository are not intended for clinical use.
This repository contains the code behind a multi-reader, multi-case (MRMC) crossover reader study of AI-assisted diagnosis of neonatal pneumoperitoneum on radiographs. Fourteen readers (6 pediatric radiologists, 3 neonatologists, 5 radiology residents) interpreted 125 external radiographs from 11 tertiary hospitals, giving 1,750 interpretation events. Each case was read either unaided, with a reliable AI assistant, or with a deliberately error-injected AI assistant. The study examines how readers of different expertise respond when the AI is wrong.
Both assistants share one architecture: a RAD-DINO (ViT-B/14) backbone adapted with LoRA. The error-injected assistant was trained on systematically biased labels. Reader outcomes were analysed with generalized linear mixed models with crossed random effects for reader and case.
The study design, results, statistical methods and limitations are in the paper and its supplementary information. The neonatologist subgroup has only three readers, and the saliency-map analyses are exploratory; the paper discusses both.
neonatal-ai-sandbox.pages.dev is a web-based training tool built from the reader-study interface. It simulates reliable and error-prone AI assistance so that clinicians can practise checking AI output.
| Path | Contents |
|---|---|
image_preprocessing/ |
Preprocessing for the internal (internal_data_preprocessing.py) and external (external_data_preprocessing.py) cohorts |
model/rad_dino_finetune/ |
RAD-DINO + LoRA training (rdino_train.py), evaluation (rdino_test.py), error-injected model training on biased labels (bias_train.py), representation-focused batch sampler (rfbs.py), saliency maps (rdino_gradcam.py); options in rdino_config.py |
model/cnn_finetune/ |
CNN baselines (DenseNet201, EfficientNetB5, ResNet152, ResNeXt50) |
model/rad_dino_zeroshot/ |
Zero-shot RAD-DINO baseline |
quantitative_analysis/reader_study_full_analysis.py |
Primary reader-study analysis: GLMM, GEE sensitivity analysis, reading-time and saliency-map analyses, main-text tables and figures |
quantitative_analysis/revision_analyses/ |
Additional analyses from peer review (reading-time LMM full output, between-group differential, false-positive vs false-negative stratification); see its README |
quantitative_analysis/model-benchmarking.ipynb, metrics_p_value.ipynb |
Standalone model benchmarking and metric comparisons |
quantitative_analysis/standalone_model_performance/ |
Saved configs, ROC curves and confusion matrices for the RAD-DINO and CNN models, plus the two reader-study checkpoints |
figures/ |
Figures used in this README |
Dependencies are listed in requirements.txt. Comments at the top of that file explain how to install the PyTorch wheel that matches your hardware. A Dockerfile based on pytorch/pytorch:2.5.1-cuda11.8-cudnn9-devel is also provided. Model training was run on GPUs.
pip install -r requirements.txtThe mixed-effects analyses use pymer4, which calls R's lme4, lmerTest and emmeans. R must be discoverable through R_HOME.
Primary analysis. quantitative_analysis/reader_study_full_analysis.py reads the reader-study results from file paths set as constants near the top of the script (XLSX_PATH, EXTERNAL_CSV_PATH, OUTDIR; R_HOME is also set there). These inputs are not public, so edit the paths before running.
Revision analyses. These scripts read a reader-case CSV from the RS_DATA_CSV environment variable. The expected columns are documented in revision_analyses/README.md and at the top of each script.
export R_HOME="$(R RHOME)"
export RPY2_CFFI_MODE=ABI
export RS_DATA_CSV=/path/to/aided_set_long_format.csv
python quantitative_analysis/revision_analyses/r2_6_reading_time_lmm.py
python quantitative_analysis/revision_analyses/r2_7_between_group_differential.py
python quantitative_analysis/revision_analyses/r2_8_error_type_stratification.pyModel training and evaluation. Command-line options for the RAD-DINO scripts are defined in model/rad_dino_finetune/rdino_config.py. Running them requires the (non-public) image data.
- Checkpoints. The LoRA checkpoints used in the reader study are in
quantitative_analysis/standalone_model_performance/rad_dino/optimizedAI/weights/(reliable AI) and.../rad_dino/unreliableAI/weights/(error-injected AI). They are derived from microsoft/rad-dino (MIT License, research use only). - Images. Raw radiographs cannot be redistributed because of IRB and licensing restrictions. The external validation set can be requested through AI-Hub, subject to its access protocols.
- Derived data. De-identified reader-level metrics, AI prediction labels and reference standards are available from the corresponding authors on reasonable request.
Because the reader-study data are not public, the analysis scripts cannot be re-run from this repository alone.
Lee, J., Kim, Y., Kim, V., Park, C., Song, J. M., Kwon, J., Nam, Y., Lenehan, P., Kim, D. Y., Cho, Y. A., Kim, P. H., Hwang, J.-Y., Lee, J., Lee, B. S., Jung, E., Jung, A. Y., Choi, J., Kim, N. & Yoon, H. M. Expertise modulates automation bias and sentinel behavior in human–AI collaborative diagnosis of neonatal pneumoperitoneum. npj Digit. Med. (2026). https://doi.org/10.1038/s41746-026-03399-3
@article{lee2026expertise,
title = {Expertise modulates automation bias and sentinel behavior in human--{AI} collaborative diagnosis of neonatal pneumoperitoneum},
author = {Lee, Junseong and Kim, Yeonsu and Kim, Victoria and Park, Changhyun and Song, Jeong Min and Kwon, Jimin and Nam, Yoojin and Lenehan, Patrick and Kim, Dong Yeong and Cho, Young Ah and Kim, Pyeong Hwa and Hwang, Jae-Yeon and Lee, Jaeseong and Lee, Byong Sop and Jung, Euiseok and Jung, Ah Young and Choi, Jinwha and Kim, Namkug and Yoon, Hee Mang},
journal = {npj Digital Medicine},
year = {2026},
doi = {10.1038/s41746-026-03399-3},
url = {https://www.nature.com/articles/s41746-026-03399-3}
}The article is currently published as an Article in Press; the volume and article number have not yet been assigned. Machine-readable metadata is in CITATION.cff.
Correspondence: Namkug Kim (namkugkim@gmail.com) and Hee Mang Yoon (espoirhm@gmail.com), co-corresponding authors.
The code is released under the MIT License. The paper is open access under CC BY-NC-ND 4.0.