Prediction Models for In-Hospital Delirium Using Routinely Collected Electronic Health Record Data: Systematic Review.
JMIR medical informatics · 2026
Abstract originale
BACKGROUND: Delirium is a common and clinically important form of acute in-hospital mental status deterioration. Electronic health record (EHR)-based prediction models may support early identification and targeted prevention, but their methodological quality, validation rigor, and clinical readiness remain uncertain. OBJECTIVE: This systematic review aimed to synthesize and critically evaluate prediction models for in-hospital delirium developed using routinely collected EHR data, focusing on model characteristics, validation strategies, performance, risk of bias, and clinical applicability. METHODS: We searched PubMed, MEDLINE, Embase, PsycINFO, and Web of Science from inception to November 11, 2025. Eligible studies developed, validated, or evaluated multivariable prediction models using routinely collected EHR or administrative data to predict acute mental status deterioration during adult hospital admissions. Although eligibility criteria were broad, all included studies operationalized deterioration as delirium. Data extraction was informed by CHARMS (Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies) and TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) or TRIPOD-artificial intelligence guidance. Model performance, validation, calibration, and implementation features were synthesized narratively. Risk of bias and applicability were assessed using PROBAST (Prediction Model Risk of Bias Assessment Tool). RESULTS: Twenty-nine studies met the inclusion criteria. The evidence clustered into 4 overlapping prediction tasks: admission or early-stay risk stratification, perioperative or postoperative prediction, dynamic intensive care unit prediction, and external validation or workflow evaluation of existing tools. Most studies were retrospective cohorts (20/29, 69%) and were conducted in general ward, mixed ward-intensive care unit, intensive care unit, or emergency department settings. Machine learning or hybrid approaches were common (18/29, 62%), but more complex models did not consistently outperform statistical or rule-based approaches. Of 29 studies, internal discrimination was reported in 24 (83%; area under the receiver operating characteristic curve range 0.77-0.97) studies, whereas external discrimination was reported in 12 studies and calibration in 15 studies. Decision curve analysis was reported in 3 studies, and prospective evaluation or workflow integration remained limited. Overall risk of bias was low in 8 studies, unclear in 10 studies, and high in 11 studies, mainly because of analysis-domain limitations. CONCLUSIONS: Routinely collected EHR data can support delirium risk prediction across hospital settings, and many models show moderate to high discrimination. However, no single algorithm is ready for routine adoption. The field remains limited by heterogeneous prediction tasks, inconsistent outcome ascertainment, weak calibration and decision-analytic reporting, and insufficient external or prospective evaluation. Future studies should define the intended clinical use case before model development, evaluate calibration and clinical usefulness alongside discrimination, and test models across institutions, time periods, and workflows before deployment.
Parole chiave
Leggi il testo integrale originale
Testo estratto dalla pubblicazione: sono esclusi immagini, media, supplementi e parti soggette a diritti distinti.
Prediction Models for In-Hospital Delirium Using Routinely Collected Electronic Health Record Data: Systematic Review
Abstract
Background
Delirium is a common and clinically important form of acute in-hospital mental status deterioration. Electronic health record (EHR)–based prediction models may support early identification and targeted prevention, but their methodological quality, validation rigor, and clinical readiness remain uncertain.
Objective
This systematic review aimed to synthesize and critically evaluate prediction models for in-hospital delirium developed using routinely collected EHR data, focusing on model characteristics, validation strategies, performance, risk of bias, and clinical applicability.
Methods
We searched PubMed, MEDLINE, Embase, PsycINFO, and Web of Science from inception to November 11, 2025. Eligible studies developed, validated, or evaluated multivariable prediction models using routinely collected EHR or administrative data to predict acute mental status deterioration during adult hospital admissions. Although eligibility criteria were broad, all included studies operationalized deterioration as delirium. Data extraction was informed by CHARMS (Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies) and TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) or TRIPOD–artificial intelligence guidance. Model performance, validation, calibration, and implementation features were synthesized narratively. Risk of bias and applicability were assessed using PROBAST (Prediction Model Risk of Bias Assessment Tool).
Results
Twenty-nine studies met the inclusion criteria. The evidence clustered into 4 overlapping prediction tasks: admission or early-stay risk stratification, perioperative or postoperative prediction, dynamic intensive care unit prediction, and external validation or workflow evaluation of existing tools. Most studies were retrospective cohorts (20/29, 69%) and were conducted in general ward, mixed ward–intensive care unit, intensive care unit, or emergency department settings. Machine learning or hybrid approaches were common (18/29, 62%), but more complex models did not consistently outperform statistical or rule-based approaches. Of 29 studies, internal discrimination was reported in 24 (83%; area under the receiver operating characteristic curve range 0.77-0.97) studies, whereas external discrimination was reported in 12 studies and calibration in 15 studies. Decision curve analysis was reported in 3 studies, and prospective evaluation or workflow integration remained limited. Overall risk of bias was low in 8 studies, unclear in 10 studies, and high in 11 studies, mainly because of analysis-domain limitations.
Conclusions
Routinely collected EHR data can support delirium risk prediction across hospital settings, and many models show moderate to high discrimination. However, no single algorithm is ready for routine adoption. The field remains limited by heterogeneous prediction tasks, inconsistent outcome ascertainment, weak calibration and decision-analytic reporting, and insufficient external or prospective evaluation. Future studies should define the intended clinical use case before model development, evaluate calibration and clinical usefulness alongside discrimination, and test models across institutions, time periods, and workflows before deployment.
Introduction
Acute in-hospital mental status deterioration is a clinically important manifestation of acute brain dysfunction, encompassing disturbances such as confusion, inattention, agitation, and reduced consciousness. In current hospital-based prediction modeling research using routinely collected electronic health record (EHR) data, however, this construct has been operationalized almost exclusively as delirium. Delirium is common across hospital settings and is associated with increased morbidity, mortality, prolonged hospitalization, institutionalization, and persistent cognitive impairment following discharge [1,2].
Delirium is the most clinically and methodologically established target in this literature. Its prominence reflects both its prognostic significance and the availability of validated bedside assessment tools, most notably the Confusion Assessment Method (CAM) and its intensive care unit (ICU) adaptations [3].
Prediction modeling for in-hospital delirium is motivated by the need to support early identification and prevention in resource-constrained clinical environments. Universal application of intensive preventive strategies is rarely feasible, and risk stratification tools that identify patients at elevated risk early in the hospital course offer a pragmatic approach to targeting preventive interventions [4]. Advances in EHR-based modeling, including machine learning and natural language processing (NLP), have enabled scalable development of delirium prediction models using routinely collected clinical data [5].
However, recent systematic reviews have highlighted important limitations in the existing evidence base. Although many delirium prediction models report moderate to high discrimination, substantial heterogeneity exists in outcome definitions, predictor handling, validation strategies, and reporting quality. External validation, calibration assessment, and prospective evaluation remain inconsistently performed, limiting confidence in clinical generalizability and real-world usefulness [6-8].
Against this background, the present systematic review aims to synthesize and critically evaluate prediction models for in-hospital delirium using routinely collected EHR data. Although the search strategy was intentionally broad to capture a range of acute mental status outcomes, all eligible studies identified at full-text review operationalized deterioration as delirium. This finding underscores the central role of delirium as the dominant and currently most clinically actionable manifestation of acute in-hospital mental status change within the prediction modeling literature. Accordingly, this review focuses on delirium prediction models, with particular emphasis on methodological quality, validation practices, risk of bias, and implications for the development of clinically actionable decision support tools.
Methods
Study Design and Reporting Standards
This systematic review was conducted and reported in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [9]. Given the focus on prediction model development and validation, data extraction and synthesis were additionally informed by relevant items from the TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) and TRIPOD–Artificial Intelligence (TRIPOD-AI) reporting frameworks to support structured evaluation of model characteristics, validation strategies, and reporting quality [10,11]. This review was not prospectively registered in PROSPERO because protocol registration was not completed before screening had begun; this is acknowledged as a limitation.
Eligibility Criteria
Studies were eligible for inclusion if they met the following criteria: (1) involved adult patients (aged ≥18 years) admitted to hospital settings, including ICUs, general medical or surgical wards, or mixed ICU–ward cohorts; (2) developed, validated, or updated multivariable prediction models using routinely collected EHR or administrative data; (3) predicted delirium occurring during the index hospital admission; and (4) reported sufficient methodological or performance information to allow data extraction.
Eligible outcomes included delirium occurring during the index hospital admission, identified using validated clinical assessment tools (eg, CAM and CAM-ICU), diagnostic codes, or structured chart review. Although the search strategy was designed to capture a broader range of acute mental status deterioration outcomes, all studies meeting inclusion criteria at full-text review predicted delirium. Therefore, delirium was treated as the primary outcome for this review.
Studies using any statistical or machine learning–based modeling approach were eligible, including traditional regression-based models and advanced machine learning methods. Both retrospective and prospective study designs were included. Studies were excluded if they (1) focused exclusively on pediatric populations; (2) were conducted in outpatient, community, or psychiatric clinic settings without hospital admission; (3) relied solely on nonroutine data sources such as imaging, genomics, or specialized psychometric instruments not typically available in EHR systems; (4) predicted outcomes defined at the population level or occurring outside the index hospital admission (eg, long-term suicide risk or postdischarge psychiatric readmission); or (5) did not report delirium or an equivalent acute in-hospital mental status outcome.
Information Sources and Search Strategy
A comprehensive literature search was conducted to identify studies developing or validating prediction models for delirium and related acute mental status outcomes occurring during hospitalization. The search strategy was designed a priori to be intentionally broad to maximize sensitivity and capture models targeting a range of clinically relevant mental status changes. However, all studies meeting inclusion criteria at full-text review focused exclusively on delirium. This reflects the predominance of delirium as the most consistently defined and operationalized outcome in this field, and the present review therefore focuses specifically on delirium prediction models.
The following electronic databases were searched from inception to November 11, 2025: PubMed or MEDLINE, Embase, PsycINFO, and Web of Science. The search combined terms related to prediction modeling and machine learning (eg, prediction, risk model, machine learning, and AI), routinely collected health care data (eg, EHRs, administrative data, and claims data), acute mental status outcomes (eg, delirium, acute confusion, agitation, mental status change, and psychotropic medication use), and hospital settings (eg, inpatient, ward, and ICU).
Database-specific search strategies were developed using a combination of controlled vocabulary (eg, MeSH) and free-text terms. Full search strategies for each database are provided in Multimedia Appendix 1. No restrictions were placed on geographic location or health care setting. Only studies published in English were included. Reference lists of included studies and relevant review papers were manually screened to identify additional eligible studies. All retrieved records were imported into reference management software, and duplicate records were removed before screening.
Study Selection
After removal of duplicate records, titles and abstracts were screened to identify potentially eligible studies. Full-text papers were retrieved for all records deemed potentially relevant and assessed against the predefined eligibility criteria. Study selection was performed independently by 2 reviewers (CSL and GWC). Discrepancies at either the title or abstract or full-text screening stage were resolved through discussion and consensus, with consultation of a third reviewer (MHT) when necessary. The overall study selection process is summarized using a PRISMA flow diagram.
Data Extraction
Data were extracted independently by 2 reviewers (CSL and GWC) using a standardized data extraction form developed a priori. The extraction framework was informed by established guidance for prediction model studies, including the CHARMS (Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies), and reporting domains from the TRIPOD and TRIPOD-AI statements [10-12].
For each included study, extracted information included study metadata (first author, year, country, clinical setting, and study design), population characteristics (sample size and cohort definition), outcome definition and ascertainment method (eg, CAM-based assessment, ICD [International Classification of Diseases] codes, and chart review), prediction time horizon, modeling approach, predictor variables, feature selection methods, and model presentation.
Model development and validation characteristics were extracted in detail, including type of validation (internal, temporal, or external), validation datasets, and reported performance measures. Extracted performance metrics included measures of discrimination (eg, area under the receiver operating characteristic curve [AUROC]), sensitivity, specificity, and predictive values where available. Calibration measures and decision-analytic measures (eg, decision curve analysis) were recorded when reported. Information on missing data handling was extracted for each study, including use of complete-case analysis, single or multiple imputation, or algorithm-intrinsic handling of missingness. Where missing data handling was not explicitly described, this was recorded as not reported.
Risk of Bias and Quality Assessment
Risk of bias and applicability of included studies were assessed using the PROBAST (Prediction Model Risk of Bias Assessment Tool), which evaluates 4 domains: participants, predictors, outcomes, and analysis [13]. Two reviewers (CSL and GWC) independently completed PROBAST assessments using the tool’s signaling questions. Each domain was rated as low, high, or unclear risk of bias. Discrepancies were resolved through discussion and consensus, with involvement of a third reviewer (MHT) when necessary. PROBAST assessments were used to support qualitative interpretation of methodological strengths and limitations rather than to exclude studies or weight quantitative comparisons.
Data Synthesis and Analysis
Given substantial heterogeneity across studies with respect to clinical setting, population characteristics, outcome definitions, prediction horizons, modeling approaches, and validation strategies, a quantitative meta-analysis of model performance was not performed. Findings were therefore synthesized narratively using a structured framework focused on 4 questions: What clinical prediction task was being addressed? What routinely collected data were used? How well models performed under internal and external validation? How close the models were to clinically reliable implementation.
Prediction model characteristics and performance metrics were summarized descriptively. Performance differences were interpreted in relation to clinical setting, prediction horizon, outcome ascertainment, validation strategy, calibration reporting, and risk of bias rather than by algorithm type alone. This approach was chosen to distinguish technical feasibility from clinical readiness.
Results
Study Selection
The database search identified a total of 1086 records, including 274 from PubMed, 373 from Embase, 284 from Web of Science, and 155 from PsycINFO. After removal of duplicate and overlapping records, 673 unique records remained for title and abstract screening, of which 603 were excluded. Seventy reports were retrieved for full-text assessment. Following full-text review, 41 reports were excluded. The most common reasons for exclusion were the absence of a multivariable prediction model, outcomes not occurring during the index hospital admission, wrong publication type, wrong outcome definition, or an ineligible study population. A total of 29 studies met the inclusion criteria and were included in the final qualitative synthesis. The study selection process is summarized in Figure 1.
Study Characteristics
The 29 included studies did not represent a single uniform prediction problem. Instead, they formed a heterogeneous evidence base spanning early admission screening, perioperative risk prediction, short-term ICU warning systems, and external validation or prospective evaluation of existing tools. Country, center structure, and study design are summarized in Table 1. The narrative synthesis highlights the patterns most relevant to interpretation.
Table 1.Study characteristics (country, center, and study design).
Study ID Country Centers Study design
Ali et al (2023) [14] Netherlands Single-center Retrospective cohort
Bartolacci et al (2025) [15] United States Single-center Retrospective cohort
Bishara et al (2022) [16] United States Multicenter Retrospective cohort
Castro et al (2021) [17] United States Mixed (multicenter+external validation) Retrospective cohort
Ceppi et al (2023) [18] Switzerland Single-center Retrospective cohort
Contreras et al (2025) [19] United States Multicenter Retrospective cohort
Contreras et al (2023) [20] United States Single-center Retrospective cohort
Corradi et al (2018) [21] United States Single-center Retrospective cohort
Davoudi et al (2017) [22] United States Single-center Retrospective cohort
Heikal et al (2024) [23] Lebanon/United States Mixed (multicenter+external validation) Retrospective cohort
Holler et al (2025) [24] United States Multicenter Retrospective case–control
Hur et al (2021) [25] South Korea Mixed (multicenter+external validation) Retrospective cohort
Jauk et al (2020) [26] Austria Single-center Prospective cohort
Jauk et al (2022) [27] Austria Mixed (multicenter+external validation) Mixed retrospective–prospective
Jauk et al (2024) [28] Austria Single-center Prospective cohort
Jung et al (2022) [29] South Korea Multicenter Retrospective cohort
Li et al (2024) [30] China Single-center Retrospective cohort
Liu et al (2022) [31] United States Single-center Retrospective cohort
Lucini et al (2023) [32] Canada Multicenter Retrospective cohort
Matsumoto et al (2023) [33] Japan Single-center Retrospective cohort
Moon et al (2018) [34] South Korea Mixed (multicenter+external validation) Mixed retrospective–prospective
Mueller et al (2023) [35] United States Single-center Retrospective cohort
Pagali et al (2022) [36] United States Single-center Prospective cohort
Reeve et al (2025) [37] Switzerland Single-center Prospective cohort
Rudolph et al (2016) [38] United States Mixed (multicenter+external validation) Mixed retrospective–prospective
Sheikhalishahi et al (2023) [39] United States Mixed (multicenter+external validation) Retrospective cohort
Sun et al (2021) [40] Germany Multicenter Retrospective cohort
Sun et al (2022) [41] Germany Multicenter Mixed retrospective–prospective
Wong et al (2018) [42] United States Single-center Retrospective cohort
Most studies used retrospective cohort designs (20/29, 69%) [14-23,25,29-33,35,39,40,42]. Of 29 studies, prospective cohorts were less common (4/29, 14%) [26,28,36,37], 4 (14%) studies combined retrospective development with prospective validation or evaluation [27,34,38,41], and 1 (3%) study used a retrospective case-control design [24].
This design profile is important because much of the literature remains closer to model development than to implementation science. The strongest clinical evidence generally came from studies that moved beyond internal validation into external, temporal, or prospective evaluation.
Clinical setting, population focus, and sample size are summarized in Table 2. Clinical settings were unevenly represented. Of 29 studies, general ward populations accounted for 13 (45%) studies, mixed ward-ICU cohorts accounted for 8 (28%), ICU-only cohorts accounted for 7 (24%), and the emergency department accounted for 1 (3%). Fifteen (52%) studies were single-center, while 14 (48%) studies used multicenter data or combined multicenter development with external validation.
Table 2.Clinical setting, population focus, and sample size.
Study ID Clinical setting Population focus Total, N
Ali et al (2023) [14] General ward Older adult inpatients (≥60 years) 1168 patients
Bartolacci et al (2025) [15] Emergency department Older adult inpatients (≥65 years) 44,578 patients
Bishara et al (2022) [16] Mixed ICUa and ward Adult surgical inpatients 24,885 encounters
Castro et al (2021) [17] Mixed ICU and ward Adult medical inpatients (COVID-19) 2907 patients
Ceppi et al (2023) [18] General ward Adult rehabilitation inpatients 8774 stays
Contreras et al (2025) [19] ICU Adult ICU patients 104,303 patients
Contreras et al (2023) [20] ICU Adult ICU patients 13,395 patients
Corradi et al (2018) [21] Mixed ICU and ward Adult general inpatients 41,826 patients
Davoudi et al (2017) [22] General ward Adult surgical inpatients 51,457 patients
Heikal et al (2024) [23] Mixed ICU and ward Adult inpatients (ICU and ward) 40,208 admissions
Holler et al (2025) [24] General ward Older adult surgical inpatients (≥50 years) 14,334 encounters
Hur et al (2021) [25] ICU Adult ICU patients 12,409 patients
Jauk et al (2020) [26] General ward Adult general inpatients 4663 patients
Jauk et al (2022) [27] General ward Adult trauma surgery inpatients 93 patients
Jauk et al (2024) [28] General ward Adult surgical inpatients 738 patients
Jung et al (2022) [29] General ward Older adult orthopedic surgery inpatients 3980 patients
Li et al (2024) [30] ICU Adult cardiac surgery inpatients 507 patients
Liu et al (2022) [31] Mixed ICU and ward Adult general inpatients 34,035 patients
Lucini et al (2023) [32] ICU Adult ICU patients 38,426 patients
Matsumoto et al (2023) [33] General ward Adult surgical inpatients 11,863 patients
Moon et al (2018) [34] ICU Adult ICU patients 3284 patients
Mueller et al (2023) [35] Mixed ICU and ward Older adult inpatients (EDb admission) 28,531 patients
Pagali et al (2022) [36] Mixed ICU and ward Older adult inpatients (≥50 years) 8055 patients
Reeve et al (2025) [37] General ward Older adult surgical inpatients (≥60 years) 866 patients
Rudolph et al (2016) [38] General ward Older adult inpatients 27,871 patients
Sheikhalishahi et al (2023) [39] ICU Adult ICU patients 22,840 patients
Sun et al (2021) [40] General ward Adult general inpatients NRc
Sun et al (2022) [41] Mixed ICU and ward Adult general inpatients NR
Wong et al (2018) [42] General ward Adult general inpatients 18,223
aICU: intensive care unit.
bED: emergency department.
cNR: not reported.
Across all studies, adult inpatients constituted the primary population of interest, but the intended use cases differed. Some models were designed for broad hospital screening, some for older adult or surgical pathways, and others for high-frequency monitoring in intensive care. Sample sizes ranged from fewer than 100 patients in a small prospective deployment cohort [27] to more than 100,000 patients in large multidatabase studies [19], making direct comparison of performance estimates difficult.
Outcome definitions, assessment methods, and prevalence are summarized in Table 3. Delirium was the outcome across all included studies, but the clinical meaning of the outcome varied. Of the 29 studies, 27 (93%) modeled incident delirium, 1 (3%) modeled prevalent delirium in the emergency department [15], and 1 (3%) considered any delirium regardless of timing [38]. Outcome labels included in-hospital delirium, postoperative delirium, ICU delirium, and recurrent or short-term delirium risk.
Table 3.Outcome definition, assessment method, and prevalence.
Study ID Outcome label Assessment method Outcome prevalence
Ali et al (2023) [14] In-hospital delirium Chart review or adjudication 75/1345 (5.6%)
Bartolacci et al (2025) [15] In-hospital delirium CAMa-based screening 1701/44,578 (3.8%)
Bishara et al (2022) [16] Postoperative delirium Nurse routine screening (CAM-based) 1327/24,885 (5.3%)
Castro et al (2021) [17] Incident delirium EHRb-derived (codes or NLPc) 488/2907 (16.8%)
Ceppi et al (2023) [18] Incident delirium EHR-derived+chart review 125/8774 (1.4%)
Contreras et al (2025) [19] Incident delirium CAM-ICUd Reported without extractable event count
Contreras et al (2023) [20] ICU delirium CAM-ICU 12,871/56,297 windows (23%)
Corradi et al (2018) [21] Incident delirium Nurse routine screening (CAM-based) 3499/64,038 visits (5.5%)
Davoudi et al (2017) [22] Postoperative delirium ICDe codes 1608/51,457 (3.1%)
Heikal et al (2024) [23] ICU delirium or In-hospital delirium ICD codes+chart review NRf
Holler et al (2025) [24] Postoperative delirium CAM-based+ICD codes 7198/39,968 raw cohort (18.0%)
Hur et al (2021) [25] ICU delirium CAM-ICU Reported by dataset; event counts NR
Jauk et al (2020) [26] In-hospital delirium ICD codes+EHR text review 81/5530 (1.5%)
Jauk et al (2022) [27] In-hospital delirium Clinical judgment/chart review NR clinical cohort; 347/5347 external test set
Jauk et al (2024) [28] In-hospital delirium DOSg 103/738 (14%)
Jung et al (2022) [29] Postoperative delirium DSMh-based diagnosis+EHR-derived 196/3980 (4.9%)
Li et al (2024) [30] Postoperative delirium CAM-ICU 141/507 (28%)
Liu et al (2022) [31] Incident delirium CAM-ICU Positive CAM assessment rate reported; event count NR
Lucini et al (2023) [32] ICU delirium ICDSCi Episode prevalence reported; event count NR
Matsumoto et al (2023) [33] Postoperative delirium CAM-based screening 592/6497 derivation (9.1%); 427/5366 validation (8.0%)
Moon et al (2018) [34] ICU delirium CAM-ICU 688/3284 (21%)
Mueller et al (2023) [35] In-hospital delirium DOS/CAM-ICU 8057/28,351 (28.4%)
Pagali et al (2022) [36] In-hospital delirium CAM-based screening 1107/8055 (13.7%)
Reeve et al (2025) [37] Postoperative delirium DOS+ICD codes 100/866 (11.5%)
Rudolph et al (2016) [38] In-hospital delirium DSM-based diagnosis+EHR-derived 2343/27,625 retrospective (8%); 43/246 prospective incident (19%)
Sheikhalishahi et al (2023) [39] Incident delirium CAM-ICU NR
Sun et al (2021) [40] In-hospital delirium ICD codes NR
Sun et al (2022) [41] In-hospital delirium ICD codes NR
Wong et al (2018) [42] Incident delirium Nurse routine screening (CAM-based) 878/18,223 (4.8%)
aCAM: Confusion Assessment Method.
bEHR: electronic health record.
cNLP: natural language processing.
dICU: intensive care unit.
eICD: International Classification of Diseases.
fNR: not reported.
gDOS: Delirium Observation Screening Scale.
hDSM: Diagnostic and Statistical Manual of Mental Disorders.
iICDSC: Intensive Care Delirium Screening Checklist.
Outcome ascertainment was a major source of heterogeneity. Studies used structured tools such as CAM, CAM-ICU, the Delirium Observation Screening Scale, or the Intensive Care Delirium Screening Checklist, as well as ICD codes, EHR-derived definitions, NLP-enhanced ascertainment, and manual chart review or adjudication. These approaches identify overlapping but not identical clinical events, which limits the interpretability of pooled performance comparisons.
Outcome prevalence also varied substantially. Among the 20 studies with clearly extractable prevalence percentages, 6 (30%) reported prevalence below 5%, 10 (50%) reported prevalence between 5% and 20%, and 4 (20%) reported prevalence above 20%. Low prevalence was typical in general ward and broad inpatient cohorts, whereas higher prevalence was more common in ICU, surgical, and selected high-risk cohorts.
This variation has direct implications for clinical interpretation. In low-prevalence settings, even a model with good AUROC may have low positive predictive value (PPV) and may generate many false-positive alerts. In higher-prevalence settings, the same threshold can produce a very different balance between missed cases and unnecessary intervention.
Prediction horizon and TRIPOD classification are summarized in Table 4. Prediction horizons further separated the studies into different clinical tasks. Some models estimated risk at admission or during the first 24‐72 hours, some predicted postoperative delirium over a fixed perioperative period, and others updated risk dynamically using rolling ICU or hospital windows. These are not interchangeable tasks: short-term dynamic prediction benefits from more proximal clinical signals, whereas admission-time models must support earlier but less certain prevention decisions.
Table 4.Prediction horizon and TRIPODa classification.
Study ID Prediction horizon TRIPOD type
Ali et al (2023) [14] During hospitalization TRIPOD 4
Bartolacci et al (2025) [15] During EDb stay TRIPOD 4
Bishara et al (2022) [16] Postoperative period (≤7 days) TRIPOD 1b
Castro et al (2021) [17] During hospitalization TRIPOD 2b
Ceppi et al (2023) [18] During hospitalization TRIPOD 1a
Contreras et al (2025) [19] During ICUc stay TRIPOD 2b
Contreras et al (2023) [20] Dynamic short-term window (≤24 hours) TRIPOD 1b
Corradi et al (2018) [21] During hospitalization TRIPOD 1b
Davoudi et al (2017) [22] Postoperative period (≤7 days) TRIPOD 1b
Heikal et al (2024) [23] During ICU stay/During hospitalization TRIPOD 2a
Holler et al (2025) [24] Postoperative period (≤7 days) TRIPOD 2b
Hur et al (2021) [25] Early admission window (≤24‐72 hours) TRIPOD 2b
Jauk et al (2020) [26] During hospitalization TRIPOD 1b
Jauk et al (2022) [27] Early admission window (≤24‐72 hours) TRIPOD 3
Jauk et al (2024) [28] Postoperative period (≤7 days) TRIPOD 1b
Jung et al (2022) [29] Postoperative period (≤7 days) TRIPOD 2b
Li et al (2024) [30] Postoperative period (≤7 days) TRIPOD 1b
Liu et al (2022) [31] Dynamic short-term window (≤24 hours) TRIPOD 1b
Lucini et al (2023) [32] Dynamic short-term window (≤24 hours) TRIPOD 1b
Matsumoto et al (2023) [33] During hospitalization TRIPOD 2b
Moon et al (2018) [34] During ICU stay TRIPOD 2b
Mueller et al (2023) [35] Early admission window (≤24‐72 hours) TRIPOD 1b
Pagali et al (2022) [36] During hospitalization TRIPOD 4
Reeve et al (2025) [37] Postoperative period (≤7 days) TRIPOD 4
Rudolph et al (2016) [38] During hospitalization TRIPOD 2b
Sheikhalishahi et al (2023) [39] Dynamic short-term window (≤48 hours) TRIPOD 1b
Sun et al (2021) [40] During hospitalization TRIPOD 2b
Sun et al (2022) [41] During hospitalization TRIPOD 2b
Wong et al (2018) [42] During hospitalization TRIPOD 2a
aTRIPOD: Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis.
bED: emergency department.
cICU: intensive care unit.
Accordingly, apparent differences in discrimination should not be interpreted as simple evidence that one model family is superior. They may instead reflect differences in timing, population acuity, outcome prevalence, and availability of predictors close to delirium onset.
TRIPOD classifications also reflected a field still weighted toward development and validation rather than deployment. Type 1b and 2b studies predominated, while fewer studies focused on external validation, model updating, temporal validation, or prospective implementation.
Taken together, the study characteristics show that the literature is clinically broad but methodologically fragmented. The central synthesis question is therefore not only whether EHR-based delirium prediction is feasible but which models have been tested under conditions close enough to their intended clinical use.
Model Development and Predictors
Model categories are summarized in Table 5, with detailed model development and predictor characteristics provided in Multimedia Appendix 2 [14-42]. Machine learning models were the largest single group (10/29, 34%), followed by statistical or rule-based approaches (8/29, 28%), deep learning models (5/29, 17%), and hybrid approaches combining statistical, machine learning, or deep learning components (6/29, 21%). Tree-based ensemble methods such as random forests and gradient boosting were common, while deep learning was concentrated in ICU or large-scale EHR datasets.
Table 5.Model category across included studies.
Study ID Model category
Ali et al (2023) [14] Existing rule-based statistical model (external validation only)
Bartolacci et al (2025) [15] Not applicable (model validation study; no development)
Bishara et al (2022) [16] Classical machine learning and neural network models with clinical score comparator
Castro et al (2021) [17] Penalized logistic regression with simple clinical comparators
Ceppi et al (2023) [18] Multivariable logistic regression
Contreras et al (2025)[19] Deep learning models including transformers and large language models
Contreras et al (2023) [20] Tree-based machine learning and recurrent neural networks
Corradi et al (2018) [21] Tree-based machine learning (Random Forest)
Davoudi et al (2017) [22] Classical machine learning and statistical models
Heikal et al (2024) [23] Classical and ensemble machine learning models
Holler et al (2025) [24] Classical machine learning and neural networks
Hur et al (2021) [25] Classical machine learning and deep neural networks
Jauk et al (2020) [26] Tree-based machine learning with multiple outcome models
Jauk et al (2022) [27] Tree-based machine learning
Jauk et al (2024) [28] Tree-based machine learning
Jung et al (2022) [29] Gradient boosting models
Li et al (2024) [30] Classical machine learning models
Liu et al (2022) [31] Hybrid machine learning and deep learning models
Lucini et al (2023) [32] Recurrent neural networks
Matsumoto et al (2023) [33] Gradient boosting and penalized regression
Moon et al (2018) [34] Multivariable logistic regression
Mueller et al (2023) [35] Classical machine learning models
Pagali et al (2022) [36] Rule-based model with statistical recalibration
Reeve et al (2025) [37] External validation of existing model
Rudolph et al (2016) [38] Rule-based statistical model
Sheikhalishahi et al (2023) [39] Deep learning with attention mechanisms
Sun et al (2021) [40] Transformer-based deep learning
Sun et al (2022) [41] Transformer-based deep learning
Wong et al (2018) [42] Classical machine learning and statistical models
Several studies compared multiple modeling paradigms within the same dataset [16,17,20,22,24,25,31,33,42]. These comparisons did not show a consistent advantage for increasing model complexity. In some cohorts, tree-based or penalized regression models performed similarly to neural network approaches, and apparent internal advantages were not always preserved during external validation.
Predictor sets were built mainly from structured EHR data. Of 29 studies, demographics were included in 28 (97%), comorbidities or diagnostic history in 25 (86%) , laboratory values in 22 (76%), medications in 20 (69%), vital signs in 17 (59%), and nursing or care process variables in 7 (24%). This pattern indicates that most models relied on broadly available routine data rather than specialized research measurements.
Temporal handling of predictors was a key distinction between models (Table 6). Of 29 studies, 15 (52%) used static single-time snapshots, most often at admission, preoperatively, or within an early fixed window. Of 29 studies, 14 (48%) incorporated time-varying information through repeated recalculation, aggregated temporal summaries, rolling windows, or sequential time series modeling.
Table 6.Predictor window and temporal handling.
Study ID Predictor window Temporal handling
Ali et al (2023) [14] Admission-time or early admission window (≤24 hours) Static with repeated recalculation
Bartolacci et al (2025) [15] EDa presentation window Static (single-time snapshot)
Bishara et al (2022) [16] Preoperative window Static (single-time snapshot)
Castro et al (2021) [17] Preadmission+ early admission window Static (single-time snapshot)
Ceppi et al (2023) [18] Admission-time window Static (single-time snapshot)
Contreras et al (2025) [19] Early ICUb window (≤24 hours) Aggregated temporal summaries
Contreras et al (2023) [20] Mixed static+short-term dynamic ICU window Hybrid static+temporal modeling
Corradi et al (2018) [21] Dynamic ICU window (until outcome) Aggregated temporal summaries
Davoudi et al (2017) [22] Preoperative window Static (single-time snapshot)
Heikal et al (2024) [23] Dynamic ICU window or dynamic hospital window Aggregated temporal summaries
Holler et al (2025) [24] Preadmission window Static (single-time snapshot)
Hur et al (2021) [25] Early ICU window (≤24 hours) Aggregated temporal summaries
Jauk et al (2020) [26] Admission-time+early admission window Static with repeated recalculation
Jauk et al (2022) [27] Preadmission+early admission window Static (single-time snapshot)
Jauk et al (2024) [28] Early admission window (≤48 hours) Static with repeated recalculation
Jung et al (2022) [29] Preoperative window Static (single-time snapshot)
Li et al (2024) [30] Preoperative+perioperative+early ICU window Static (single-time snapshot)
Liu et al (2022) [31] Short-term dynamic window (≤24 hours) Hybrid static+temporal modeling
Lucini et al (2023) [32] Mixed static+dynamic ICU window Sliding or rolling time windows
Matsumoto et al (2023) [33] Admission-time+perioperative window Static (single-time snapshot)
Moon et al (2018) [34] Early ICU window (≤24 hours) Static (single-time snapshot)
Mueller et al (2023) [35] ED presentation+early admission window Static (single-time snapshot)
Pagali et al (2022) [36] Admission-time or early admission window Static (single-time snapshot)
Reeve et al (2025) [37] Preoperative window Static (single-time snapshot)
Rudolph et al (2016) [38] Early admission window (≤24 hours) Aggregated temporal summaries
Sheikhalishahi et al (2023) [39] Short-term dynamic window (≤24 hours) Sequential time series modeling
Sun et al (2021) [40] Admission-time+dynamic hospital window Aggregated temporal summaries
Sun et al (2022) [41] Dynamic hospital window (entire stay) Aggregated temporal summaries
Wong et al (2018) [42] Early admission window (≤24 hours) Static (single-time snapshot)
aED: emergency department.
bICU: intensive care unit.
Of 29 studies, only 8 (28%) used NLP or unstructured free-text data [17,18,23,26,29,38,40,41]. In several of these, NLP supported outcome ascertainment rather than contributing predictor features. Thus, despite the clinical importance of narrative documentation for cognitive and behavioral change, most models remained dependent on structured EHR fields.
Reporting of missing data handling, class imbalance, and interpretability was uneven (Table 7). Missing data handling was not reported in 24% (7/29) of the studies, and class imbalance handling was absent or not reported in 55% (16/29) of the studies. Interpretability was addressed most often through post hoc feature attribution (14/29, 48%) or intrinsically interpretable models (7/29, 24%), but these explanations were rarely linked to clinical workflow decisions.
Table 7.Missing data handling and model interpretability.
Study ID Missing data handling Interpretability approach
Ali et al (2023) [14] No imputation or model design–based None or not reported
Bartolacci et al (2025) [15] Simple imputation Intrinsic interpretability
Bishara et al (2022) [16] Simple imputation Hybrid (intrinsic+post hoc)
Castro et al (2021) [17] Simple imputation+complete-case Intrinsic interpretability
Ceppi et al (2023) [18] Exclusion-based (no imputation) Intrinsic interpretability
Contreras et al (2025) [19] Simple imputation+missingness indicators Post hoc feature attribution
Contreras et al (2023) [20] Time series imputation Post hoc feature attribution
Corradi et al (2018) [21] Model-intrinsic handling Post hoc feature attribution
Davoudi et al (2017) [22] Imputation (method not specified) Post hoc feature attribution
Heikal et al (2024) [23] Advanced imputation (DLa-based) Post hoc feature attribution
Holler et al (2025) [24] Exclusion-based (no imputation) Post hoc feature attribution
Hur et al (2021) [25] Simple imputation None or not reported
Jauk et al (2020) [26] Not reported Limited or unclear
Jauk et al (2022) [27] Not reported Post hoc feature attribution
Jauk et al (2024) [28] Not reported Post hoc feature attribution
Jung et al (2022) [29] Model-intrinsic handling Post hoc feature attribution
Li et al (2024) [30] Simple imputation+exclusions Post hoc feature attribution
Liu et al (2022) [31] Simple imputation Post hoc feature attribution
Lucini et al (2023) [32] Simple imputation + missingness indicators Post hoc feature attribution
Matsumoto et al (2023) [33] Advanced imputation (MLb-based) Hybrid (intrinsic+post hoc)
Moon et al (2018) [34] Not reported Intrinsic interpretability
Mueller et al (2023) [35] Advanced imputation (ML-based) Hybrid (intrinsic+post hoc)
Pagali et al (2022) [36] Simple imputation Intrinsic interpretability
Reeve et al (2025) [37] Simple imputation Intrinsic interpretability
Rudolph et al (2016) [38] Not reported Intrinsic interpretability
Sheikhalishahi et al (2023) [39] Time series imputation Attention-based or ante hoc
Sun et al (2021) [40] Not reported Post hoc feature attribution
Sun et al (2022) [41] Not reported Limited or unclear
Wong et al (2018) [42] Simple imputation+missingness indicators Post hoc feature attribution
aDL: deep learning.
bML: machine learning.
Overall, model development methods show that EHR-based delirium prediction is technically feasible, but the clinical value of additional data complexity remains uncertain without stronger validation, calibration, and implementation testing.
Model Performance
Model discrimination is summarized in Table 8, with additional model performance metrics, including sensitivity, specificity, PPV, negative predictive value (NPV), and threshold definitions provided in Multimedia Appendix 3 [14-42]. Reported discrimination suggested promising technical performance but should be interpreted cautiously. Of 29 studies, internal AUROC was reported in 24 (83%) studies, with values ranging from 0.77 [24] to 0.97 [23]. Confidence intervals were inconsistently reported, and the internal estimates came from heterogeneous designs, including split-sample validation, cross-validation, temporal evaluation, and prospective cohorts.
Table 8.Model discrimination (internal and external AUROCa).
Study ID Internal AUROC External AUROC
Ali et al (2023) [14] Not reported Not reported
Bartolacci et al (2025) [15] Not applicable Kennedy: 0.777; Zucchelli: 0.701; MDPb: 0.898; REDEEMc: 0.921
Bishara et al (2022) [16] XGBoost:d 0.851; Neural network: 0.841 Not performed
Castro et al (2021) [17] Not reported 0.75 (95% CI 0.71‐0.79)
Ceppi et al (2023) [18] 0.917 Not performed
Contreras et al (2025) [19] 0.848 (95% CI 0.818‐0.878) 0.824 (95% CI 0.818‐0.830)
Contreras et al (2023) [20] 0.87 (95% CI 0.86‐0.87; CatBoost on evaluation set) Not performed
Corradi et al (2018) [21] 0.909 (95% CI 0.898‐0.921) Not performed
Davoudi et al (2017) [22] −0.85 to 0.86 (best models: Random Forest and GAMe) Not performed
Heikal et al (2024) [23] ICUf CatBoost: 0.974; Ward CatBoost: 0.910 Not reported
Holler et al (2025) [24] 0.77‐0.79 0.64‐0.75
Hur et al (2021) [25] 0.919 (XGBoost, best-performing model) 0.721 (Random Forest, best-performing model)
Jauk et al (2020) [26] 0.855 (prospective evaluation) Not performed
Jauk et al (2022) [27] Not reported 0.924‐0.931 (retrained models on external cohort)
Jauk et al (2024) [28] 0.883 (95% CI 0.852‐0.915) Not performed
Jung et al (2022) [29] 0.80 (95% CI 0.77‐0.84) 0.82 (95% CI 0.80‐0.83)
Li et al (2024) [30] 0.92 (full feature set); 0.86 (selected feature set) Not performed
Liu et al (2022) [31] 0.952 (combined model, 6-hour prediction window) Not performed
Lucini et al (2023) [32] 0.909 (0‐12 hours); 0.895 (12‐24 hours) Not performed
Matsumoto et al (2023) [33] 0.85 (cross-validation, derivation cohort) 0.86‐0.90 (XGBoost); 0.86‐0.89 (LASSOg); 0.84‐0.88 (logistic regression)
Moon et al (2018) [34] 0.89 (training), 0.90 (test set) 0.72
Mueller et al (2023) [35] 0.839 (GBMh, best-performing model) Not performed
Pagali et al (2022) [36] 0.80 (modified MDP model) Not performed
Reeve et al (2025) [37] Not reported 0.77 (95% CI 0.72‐0.82)
Rudolph et al (2016) [38] 0.81 (retrospective C-statistic; 95% CI 0.80‐0.82) 0.69 (prospective C-statistic; 95% CI 0.61‐0.77)
Sheikhalishahi et al (2023) [39] 0.69‐0.81 (scenario-dependent) Not applicable
Sun et al (2021) [40] 0.82 (admission-time model) Up to 0.95 (discharge model; site-averaged)
Sun et al (2022) [41] 0.81‐0.85 (hospital-dependent; admission or discharge models) Approximately 8 percentage points decrease when applied cross-hospital
Wong et al (2018) [42] 0.855 (GBM, test set) Not performed
aAUROC: area under the receiver operating characteristic curve.
bMDP: Mayo Delirium Prediction.
cREDEEM: Risk Estimate of Delirium in Elderly Emergency Medicine.
dXGBoost: Extreme Gradient Boosting.
eGAM: Generalized Additive Model.
fICU: intensive care unit.
gLASSO: Least Absolute Shrinkage and Selection Operator.
hGBM: Gradient Boosting Machine.
Of 29 studies, external AUROC was reported in 12 (41%) studies [15,17,19,24,25,27,29,33,34,37,38,40]. Values ranged from 0.69 in prospective validation [38] to approximately 0.95 in large multisite evaluations [40]. However, external validation datasets differed substantially in clinical setting, prevalence, and outcome definition, limiting direct ranking of models.
Precision-recall performance and threshold reporting are summarized in Table 9. Precision-recall performance was much less frequently reported than AUROC. Of 29 studies, internal precision–recall area under the curve (PR-AUC) was reported in 6 (21%) studies [19-21,30,32,39], and external PR-AUC in 2 (7%) [19,33] studies. This is an important gap because many delirium prediction settings are low-prevalence tasks where AUROC can appear favorable despite limited PPV.
Table 9.Precision-recall performance and threshold definition.
Study ID Internal PR-AUCa External PR-AUC Threshold defined
Ali et al (2023) [14] NRb NR Yes (≥14.1% risk cutoff)
Bartolacci et al (2025) [15] NR NR Yes (predefined cutoffs per tool)
Bishara et al (2022) [16] NR NR Yes (model-specific cutoffs)
Castro et al (2021) [17] NR NR Yes (Youden-optimized cut points; eg, 0.12 and 0.15)
Ceppi et al (2023) [18] NR NR Not reported
Contreras et al (2025) [19] 0.192 0.118 Yes (example threshold 0.20; performance varies by hospital; additional thresholds in supplement)
Contreras et al (2023) [20] 0.62 (95% CI 0.59‐0.64) Not performed Yes (Youden index)
Corradi et al (2018) [21] 0.604 Not performed Yes (operating points chosen to maximize F_ and MCCc)
Davoudi et al (2017) [22] NR NR Yes (Youden’s J statistic)
Heikal et al (2024) [23] NR NR Yes (threshold lowered from 0.50 to 0.40)
Holler et al (2025) [24] NR NR Yes (fixed threshold=0.50)
Hur et al (2021) [25] NR NR No
Jauk et al (2020) [26] NR NR Yes (percentile-based thresholds: top 5% very high risk; next 10% high risk)
Jauk et al (2022) [27] NR NR Yes (top 15% risk classified as high or very high)
Jauk et al (2024) [28] NR NR Yes (85th or 95th percentile risk cutoffs)
Jung et al (2022) [29] NR NR Yes (Youden index; threshold=0.085)
Li et al (2024) [30] 0.80 (full); 0.73 (selected) Not performed Not reported
Liu et al (2022) [31] NR NR Yes (fixed-recall analysis; example recall=0.80)
Lucini et al (2023) [32] 0.786 (0‐12 hours); 0.745 (12‐24 hours) Not performed Yes (threshold=0.37)
Matsumoto et al (2023) [33] Reported (AUPRCd; value not specified) Reported (AUPRC; value not specified) Not explicitly fixed
Moon et al (2018) [34] NR NR Yes (Youden-based cutoffs C1/ C2)
Mueller et al (2023) [35] NR NR Yes (model-specific thresholds)
Pagali et al (2022) [36] NR NR Yes (≤5%, 6%‐29%, ≥30% risk strata)
Reeve et al (2025) [37] NR NR Yes (predefined PIPRAe risk categories)
Rudolph et al (2016) [38] NR NR Yes (risk-strata cut points)
Sheikhalishahi et al (2023) [39] 0.28‐0.45 Not applicable No
Sun et al (2021) [40] NR NR Yes (score-based alert categories)
Sun et al (2022) [41] NR NR Yes (hospital-specific alert thresholds)
Wong et al (2018) [42] NR NR Yes (thresholds set at 90% sensitivity and 90% specificity)
aPR-AUC: precision–recall area under the curve.
bNR: not reported.
cMCC: Matthews Correlation Coefficient.
dAUPRC: Area under the precision-recall curve.
ePIPRA: Pre-Interventional Preventive Risk Assessment.
Threshold-based metrics were difficult to compare. Although sensitivity, specificity, PPV, and NPV were often reported, thresholds were selected using different strategies, including data-driven optimization, fixed probability thresholds, and percentile-based risk strata. Few studies justified thresholds in relation to clinical resources, alert burden, or the intended intervention.
The performance synthesis therefore supports a cautious conclusion: routinely collected EHR data can discriminate delirium risk, but discrimination alone does not establish clinical usefulness. For deployment, external calibration, threshold consequences, and decision-analytic benefit are as important as AUROC.
Validation, Calibration, and Implementation Characteristics
Validation strategies are summarized in Table 10 and show a clear gap between model development and transportability testing. Of 29 studies, 9 (31%) reported internal validation only, 8 (28%) combined internal and external validation, 2 (7%) focused on external validation only, and 2 (7%) reported temporal validation only. Of 29 studies, 3 (10%) were prospective evaluations without a distinct internal or external validation phase; 2 (7%) combined external validation with prospective evaluation; 2 (7%) combined internal, external, and prospective evaluation; and 1 (3%) reported apparent performance only.
Table 10.Validation strategies.
Study ID Validation scope Internal validation approach External validation (type)
Ali et al (2023) [14] External only None or not applicable Geographic (different hospital)
Bartolacci et al (2025) [15] External only None or not applicable Geographic (independent EDa cohort)
Bishara et al (2022) [16] Internal only Combined internal validation None
Castro et al (2021) [17] Internal+external Split-sample Geographic (multiple hospitals)
Ceppi et al (2023) [18] Apparent only Apparent performance only None
Contreras et al (2025) [19] Internal+external Bootstrap Geographic
Contreras et al (2023) [20] Internal only Combined internal validation None
Corradi et al (2018) [21] Internal only Combined internal validation None
Davoudi et al (2017) [22] Internal only Cross-validation None
Heikal et al (2024) [23] Internal+external Combined internal validation Geographic (cross-setting)
Holler et al (2025) [24] Internal+external Combined internal validation Geographic (multihospital)
Hur et al (2021) [25] Internal+external Temporal internal validation Geographic
Jauk et al (2020) [26] Prospective only None or not applicable None
Jauk et al (2022) [27] Internal+external+prospective Cross-validation Geographic+temporal
Jauk et al (2024) [28] Prospective only None or not applicable Prospective clinical cohort
Jung et al (2022) [29] Internal+external Combined internal validation Geographic (independent hospital)
Li et al (2024) [30] Internal only Combined internal validation None
Liu et al(2022) [31] Internal only Combined internal validation None
Lucini et al (2023) [32] Internal only Split-sample None
Matsumoto et al (2023) [33] Temporal only Cross-validation Temporal
Moon et al (2018) [34] Internal+external Split-sample Geographic
Mueller et al (2023) [35] Internal only Cross-validation None
Pagali et al (2022) [36] Prospective only Prospective internal validation None
Reeve et al (2025) [37] External+prospective None or not applicable Prospective clinical cohort
Rudolph et al (2016) [38] External+prospective Split-sample Prospective clinical cohort
Sheikhalishahi et al (2023) [39] Internal only Cross-validation None
Sun et al (2021) [40] Internal+external Split-sample Geographic (multisite)
Sun et al (2022) [41] Internal+external+prospective Split-sample Geographic (live or workflow)
Wong et al (2018) [42] Temporal only Temporal internal validation None
aED: emergency department.
Overall, 16 studies reported some form of external or prospective validation [14,15,17,19,23-25,27-29,33,34,37,38,40,41]. Most external validation tested transfer across hospitals, health systems, time periods, or critical care databases. Cross-setting validation, such as applying an ICU-derived model to ward patients or vice versa, was uncommon.
Among the 7 studies with directly comparable internal and external AUROC values, 5 (71%) showed lower discrimination after external validation (Figure 2). The magnitude of decline varied, indicating that performance transportability was influenced by differences in population, data capture, outcome ascertainment, and workflow context.
In this subset, mean internal AUROC was 0.845 and mean external AUROC was 0.772, corresponding to a mean change of −0.073. This descriptive comparison should not be interpreted as a pooled effect estimate, but it illustrates the risk of relying on internal performance when judging deployment readiness.
The external validation evidence was therefore mixed: several models remained discriminative outside their development data, but validation was often conducted in settings similar to the development environment. Evidence for robust transport across substantially different institutions, care pathways, and outcome assessment practices remains limited.
Calibration methods and decision curve analysis are summarized in Table 11. Of 29 studies, calibration was assessed in 15 (52%) studies [15-17,21,24-28,30,32,33,36,37,41]. Methods included calibration plots, Brier scores, Hosmer-Lemeshow tests, calibration slope or calibration-in-the-large, Platt scaling, isotonic regression, and expected calibration error. The diversity of methods, combined with incomplete reporting, made calibration difficult to compare across studies.
Table 11.Calibration methods and decision curve analysis.
Study ID Calibration method Decision curve analysis
Ali et al (2023) [14] Not reported No
Bartolacci et al (2025) [15] Calibration plots, Brier score, Platt scaling, and Spiegelhalter z test No
Bishara et al (2022) [16] Calibration plots No
Castro et al (2021) [17] Hosmer-Lemeshow test; calibration plots Yes
Ceppi et al (2023) [18] Not applicable No
Contreras et al (2025) [19] Not reported No
Contreras et al (2023) [20] Not reported No
Corradi et al (2018) [21] Platt scaling; calibration plots No
Davoudi et al (2017) [22] Not reported No
Heikal et al (2024) [23] Not reported No
Holler et al (2025) [24] Calibration curves No
Hur et al (2021) [25] Brier score Yes
Jauk et al (2020) [26] Calibration plots (risk strata with confidence intervals) No
Jauk et al (2022) [27] Calibration plots No
Jauk et al (2024) [28] Calibration plots; Brier score (scaled) No
Jung et al (2022) [29] Not applicable No
Li et al (2024) [30] Expected calibration error No
Liu et al (2022) [31] Not reported No
Lucini et al (2023) [32] Isotonic regression; Brier score No
Matsumoto et al (2023) [33] Calibration slope, calibration intercept, and Brier score No
Moon et al (2018) [34] Not reported No
Mueller et al (2023) [35] Not reported No
Pagali et al (2022) [36] Calibration plots; Brier score No
Reeve et al (2025) [37] Calibration-in-the-large, calibration slope, and calibration plots No
Rudolph et al (2016) [38] Not reported No
Sheikhalishahi et al (2023) [39] Not reported No
Sun et al (2021) [40] Not reported No
Sun et al (2022) [41] Isotonic regression; calibration plots Yes
Wong et al (2018) [42] Platt scaling; calibration plots No
Calibration reporting was often less mature than discrimination reporting. Several studies relied mainly on visual assessment, and recalibration after external validation was rare. Of 29 studies, decision curve analysis was reported in only 3 (10%) [17,25,41], leaving limited evidence about whether model-guided decisions would improve net clinical benefit.
Prospective evaluation and implementation characteristics are summarized in Table 12. Of 29 studies, 8 (28%) included prospective evaluation [26-28,34,36-38,41], and 9 (31%) reported some form of workflow integration or implementation testing [26-28,34,36-38,40,41]. These ranged from silent prospective validation to live EHR alerts, but few assessed downstream effects on clinician behavior, prevention delivery, alert burden, or patient outcomes.
Table 12.Prospective evaluation and implementation.
Study ID Prospective evaluation Implementation tested
Ali et al (2023) [14] No No (compared against standard VMSa questions [nonintegrated comparison])
Bartolacci et al (2025) [15] No No
Bishara et al (2022) [16] No No
Castro et al (2021) [17] No No
Ceppi et al (2023) [18] No No
Contreras et al (2025) [19] No No
Contreras et al (2023) [20] No No
Corradi et al (2018) [21] No No
Davoudi et al (2017) [22] No No
Heikal et al (2024) [23] No No
Holler et al (2025) [24] No No
Hur et al (2021) [25] No No
Jauk et al (2020) [26] Yes Yes (fully embedded in hospital information system)
Jauk et al (2022) [27] Yes Yes (integrated into HISb)
Jauk et al (2024) [28] Yes Yes (real-time predictions integrated into HIS; blinded to staff during study)
Jung et al (2022) [29] No No (web-based tool only; no real-world impact evaluation)
Li et al (2024) [30] No No (future integration proposed only)
Liu et al (2022) [31] No No
Lucini et al (2023) [32] No No
Matsumoto et al (2023) [33] No No
Moon et al (2018) [34] Yes (after implementation) Yes (live EHRc Kardex alert)
Mueller et al (2023) [35] No No
Pagali et al (2022) [36] Yes Partial (EHR-integrated data capture; no automated alerts)
Reeve et al (2025) [37] Yes Yes (embedded in routine clinical workflow)
Rudolph et al (2016) [38] Yes Yes (EMRd-integrated; real-time execution approximately 8 seconds)
Sheikhalishahi et al (2023) [39] No No
Sun et al (2021) [40] No Yes (live EHR integration in 2 hospitals)
Sun et al (2022) [41] Yes Yes (production EHR integration)
Wong et al (2018) [42] No No
aVMS: Dutch safety management system (Veiligheidsmanagementsysteem).
bHIS: hospital information system.
cEHR: electronic health record.
dEMR: electronic medical record.
Risk of Bias and Applicability
Risk of bias was assessed using PROBAST, with domain-level judgments summarized in Multimedia Appendix 4 [14-42] and overall proportions illustrated in Figure 3. Of 29 studies, overall risk of bias was judged low in 8 (28%), unclear in 10 (34%), and high in 11 (38%). The main pattern was not a lack of clinical relevance but limited methodological assurance.
Low-risk studies generally had clearer participant selection, predictor timing, outcome ascertainment, and model evaluation [15,16,21,25,32,33,36,39]. These studies provide the most reliable evidence that routinely collected data can support delirium prediction.
Studies with unclear risk of bias were usually limited by incomplete reporting rather than obvious methodological failure [17,20,22,23,29-31,35,37,42]. Common sources of uncertainty included missing data handling, calibration assessment, feature selection, and validation procedures.
High risk of bias was identified in 11 studies, most often because of analysis-domain limitations [14,18,19,24,26-28,34,38,40,41]. Recurrent issues included apparent-only performance reporting, case-control designs, inadequate handling or reporting of missing data, limited calibration assessment, and incomplete reporting of feature selection.
Within the 11 high-risk studies, inadequate handling or reporting of missing data was identified in 6 studies [26-28,34,38,41], limited or absent calibration assessment in 4 studies [14,19,34,38], apparent-only performance without validation in 1 study [18], case-control design in 2 studies [24,40], and incomplete feature selection reporting in 2 studies [14,19]. Several studies had more than 1 limitation.
These risk-of-bias findings help explain why high AUROC values should not be interpreted as readiness for practice. Models can appear accurate in development datasets while still being vulnerable to overfitting, miscalibration, missing data artifacts, or poor transportability.
Applicability concerns were more limited than risk-of-bias concerns. Most studies evaluated adult inpatient populations and used routinely collected EHR predictors, supporting broad relevance to hospital practice. However, applicability was still context-dependent for ICU-only, perioperative, emergency department, rehabilitation, COVID-19, and single health system models.
The overall quality assessment therefore supports a nuanced interpretation: the field is clinically relevant and technically active, but the evidence base is not yet consistently strong enough to support unqualified clinical deployment. No study was excluded on the basis of PROBAST assessment. Instead, risk-of-bias judgments were used to interpret how much confidence should be placed in the reported performance and implementation claims.
Discussion
Principal Findings
This systematic review included 29 studies developing, validating, or evaluating prediction models for in-hospital delirium using routinely collected EHR data. The principal finding is that EHR-based delirium prediction is feasible across several hospital settings, but the current evidence is fragmented across different clinical prediction tasks. The literature supports the existence of measurable risk signals in routine data; it does not yet establish that any model class is consistently ready for routine clinical deployment.
Four findings are especially important for readers. First, most models were developed retrospectively, and only a minority underwent prospective evaluation or workflow testing. Second, model performance was commonly summarized by AUROC, while calibration, precision-recall metrics, and decision-analytic evaluation were less consistently reported. Third, increased algorithmic complexity did not consistently translate into better or more transportable performance. Fourth, risk of bias was mainly driven by analysis-domain limitations rather than by lack of clinical relevance.
These findings indicate that the next stage of the field should be less focused on producing additional internally validated models and more focused on defining clinical use cases, testing transportability, calibrating models for local populations, and evaluating whether model-guided care changes decisions or outcomes.
Interpretation in Context of Existing Literature
The heterogeneity observed in this review is consistent with previous systematic reviews of delirium prediction and broader clinical prediction modeling research [6-8]. Models differed not only in algorithm type but also in clinical setting, prediction timing, outcome ascertainment, and validation design. These differences mean that a single pooled estimate of performance would be difficult to interpret and could obscure clinically meaningful distinctions between prediction tasks.
The predominance of machine learning approaches also mirrors wider trends in hospital and critical care prediction research [43]. However, this review suggests that algorithmic sophistication is not the main bottleneck. In several studies, simpler statistical or tree-based approaches performed similarly to deep learning models, especially when evaluated under comparable conditions. Data quality, predictor timing, outcome definition, and validation context appeared at least as important as model family.
The risk-of-bias patterns also align with metaresearch showing that many published prediction models have limitations in the analysis domain, including missing data handling, overfitting, and incomplete calibration assessment [44]. Evidence that high-risk models often perform less well in external validation [45] is directly relevant here, because several delirium models showed lower discrimination when tested beyond their development data.
Clinical Implications
Clinically, delirium prediction models are attractive because they could help target prevention, screening, and staffing resources to patients most likely to benefit. The reviewed studies show that routine EHR data contain useful risk information in ward, ICU, perioperative, and emergency care contexts. This supports continued development of delirium prediction as a component of clinical decision support.
The implementation evidence, however, remains incomplete. Few studies assessed whether predictions changed clinician behavior, reduced delirium incidence, improved patient outcomes, or avoided alert fatigue. Without these evaluations, models with favorable discrimination may still have limited value in practice, particularly if thresholds are poorly calibrated to local prevalence and available resources.
Methodological Implications
Model Complexity and Performance
Across studies that directly compared modeling approaches, there was no consistent evidence that more complex models outperformed simpler alternatives. Tree-based machine learning, penalized regression, rule-based tools, and neural network approaches all achieved overlapping discrimination ranges. In several cases, models with favorable internal performance did not retain a clear advantage during external validation.
This finding does not imply that complex models are unnecessary, particularly for dynamic ICU prediction or high-dimensional time series data. Rather, it suggests that model choice should follow the clinical task, data structure, interpretability requirements, and implementation constraints. For many hospital use cases, a well-calibrated and externally validated simpler model may be more useful than a complex model with opaque behavior and limited transportability evidence. The relationship between model family and reported AUROC is shown descriptively for internal and external performance in Figures 4A and 4B, respectively.
Prediction Horizon and Task Comparability
Prediction horizon was one of the most important sources of heterogeneity. Admission-time models, perioperative models, rolling ICU models, and full-stay prediction models answer different clinical questions. They differ in how early an intervention can be triggered, how close predictors are to delirium onset, and how much uncertainty remains at the time of prediction.
Consequently, comparing AUROC values across prediction horizons can be misleading. A dynamic model predicting delirium in the next 12 hours may appear stronger partly because it uses proximal physiological information, whereas an admission-time model may be clinically valuable precisely because it operates before deterioration is obvious. Future studies should therefore define the intended prediction moment and intervention pathway before evaluating performance.
Validation and Generalizability
External validation remains the key step separating promising models from generalizable tools. Although some studies tested models outside the development dataset, validation often occurred in similar clinical settings or related health systems. Cross-setting validation and temporal validation were less common, despite being highly relevant for EHR models whose predictors and labels can change with local documentation practices.
The observed decline from internal to external AUROC in most directly comparable studies reinforces the need for conservative interpretation. Models intended for deployment should be tested across institutions, time periods, and patient groups that reflect their proposed use, and they should be recalibrated when transported to new settings.
Calibration and Reliability
Calibration is central to clinical reliability but was inconsistently assessed. Good discrimination indicates that a model can rank patients by risk; it does not show that predicted probabilities are accurate. For delirium prevention, inaccurate probabilities may lead to undertreatment of truly high-risk patients or excessive alerts for patients unlikely to develop delirium.
The limited use of recalibration and decision curve analysis is therefore a major evidence gap. Before deployment, models should report calibration-in-the-large, calibration slope, calibration plots, and clinically meaningful threshold analyses. Decision curve analysis or equivalent usefulness-based evaluation can help determine whether the model adds value beyond usual care or simpler screening rules.
Class Imbalance and Performance Metrics
Outcome prevalence varied widely, and several cohorts had low delirium prevalence. In such settings, AUROC can overstate practical usefulness because it is insensitive to the number of false positives generated at a chosen threshold. PPV and PR-AUC are especially important when the intended intervention is resource-intensive or when repeated alerts could reduce clinician trust.
The limited reporting of PR-AUC and threshold rationale therefore weakens the clinical interpretability of many studies. Future work should present threshold-specific consequences, including the number of patients flagged, false positives, false negatives, and expected resource implications at clinically plausible operating points.
Threshold Selection and Implementation
Threshold selection should be treated as a clinical design decision rather than a statistical afterthought. Data-driven thresholds such as Youden index may maximize a performance statistic in a development dataset, but they may not match local prevention capacity or acceptable alert burden. Fixed thresholds and percentile-based risk groups can be easier to implement, but they also require calibration to local prevalence and workflow.
For clinical deployment, threshold selection should be linked to the intended action: enhanced screening, multicomponent prevention, geriatric consultation, medication review, or ICU-specific intervention. The acceptable balance between sensitivity and specificity will differ across these use cases.
Use of Unstructured Data and NLP
The limited use of unstructured data is notable because delirium symptoms are often documented in narrative nursing, medical, and allied health notes. NLP may therefore improve both outcome ascertainment and predictor representation. However, free-text models raise additional challenges, including annotation burden, governance, changing documentation practices, and transportability across institutions. Future NLP-enhanced models should distinguish clearly between using language data to define the outcome and using language data as predictors. These uses have different risks for information leakage, temporal validity, and clinical implementation.
Implementation Considerations
Several findings have direct implications for deployment. A model should not be implemented solely because it has a high AUROC. It should have an explicitly defined clinical role, evidence of calibration in the target population, threshold analyses tied to available resources, and prospective evaluation showing that predictions can be acted on without excessive alert burden.
Implementation studies should therefore evaluate not only model performance but also workflow fit, clinician response, alert fatigue, equity, prevention delivery, and patient outcomes. This is particularly important for delirium, where prediction is useful only if it leads to timely and feasible prevention or treatment.
Limitations
Several limitations should be considered when interpreting the findings of this review. First, the quality of the included evidence base was variable, with a substantial proportion of studies judged to be at high risk of bias, primarily due to analytical limitations. Second, external validation and prospective evaluation were inconsistently performed, limiting confidence in the generalizability of reported performance.
This review also has inherent limitations. Only English-language studies were included, which may introduce language bias. The review was not prospectively registered before screening began, which may increase the risk of reporting bias. In addition, substantial heterogeneity in study design, outcome definitions, and reporting precluded formal meta-analysis. Finally, risk-of-bias assessment relied on the completeness of reporting in the original studies, which may have resulted in conservative or unclear judgments in some cases.
Future Research Directions
Future research should move from model development toward clinically anchored validation and evaluation. The immediate priority is external and temporal validation across heterogeneous populations, with calibration assessed in ways that can support local recalibration rather than only discrimination ranking.
Implementation research should then test whether predictions change care in practice. Studies should specify the intended intervention pathway, alert threshold, resource assumptions, and monitoring plan before deployment, and should measure clinician response, alert burden, prevention delivery, equity, resource use, and patient outcomes.
Reporting should make threshold consequences easy to interpret. Future studies should present the number of patients flagged, false positives, false negatives, and expected workload at clinically plausible operating points so that health systems can judge whether a model is compatible with local capacity.
Future models may benefit from combining structured EHR variables with unstructured clinical notes, because cognitive and behavioral changes are often documented narratively. NLP-enhanced models should distinguish outcome ascertainment from predictor use, and maintain temporal separation between predictors and outcomes to avoid information leakage.
Expansion beyond delirium to broader acute mental status deterioration may be valuable only when outcomes are clearly defined, temporally valid, and clinically actionable. Across all future work, transparent reporting and alignment with TRIPOD, TRIPOD-AI, CHARMS, and PROBAST will be essential to move from technically promising models toward reliable decision-support tools.
Conclusions
Routinely collected EHR data can support delirium prediction across hospital settings, but favorable discrimination alone does not establish clinical readiness. The evidence remains heterogeneous in outcome definition, prediction timing, prevalence, validation strategy, calibration reporting, and implementation maturity; risk of bias, particularly in the analysis domain, also remains common. More complex algorithms did not consistently improve performance. Future work should prioritize externally validated, well-calibrated, and clinically interpretable models evaluated prospectively within real workflows, with explicit attention to threshold consequences, alert burden, and patient benefit.
Supplementary material
Acknowledgments
This study was conducted as part of the authors’ academic research activities. The authors thank colleagues and peer reviewers who provided informal feedback during the development of the review protocol and data extraction framework. The authors used OpenAI ChatGPT to support language editing and formatting checks. All AI-assisted text and materials were reviewed, revised, and verified by the authors, who take full responsibility for the final content. No AI tool was used as an author.
Abbreviations
AUROCArea under the receiver operating characteristic curve
CAMConfusion Assessment Method
CHARMSChecklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies
EHRelectronic health record
ICDInternational Classification of Diseases
ICUintensive care unit
NLPnatural language processing
NPVnegative predictive value
PPVpositive predictive value
PR-AUCprecision–recall area under the curve
PRISMAPreferred Reporting Items for Systematic Reviews and Meta-Analyses
PROBASTPrediction Model Risk of Bias Assessment Tool
TRIPODTransparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis
TRIPOD-AITransparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis–Artificial Intelligence
Footnotes
Funding: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. Article-processing charges are covered under the UCL–JMIR institutional agreement via Jisc.
Data Availability: This study is a systematic review and did not generate new primary data. All data analyzed in this review were derived from published studies and are available within the paper and its multimedia appendices. Extracted data and risk-of-bias assessments are available from the corresponding author upon reasonable request. Multimedia Appendix 1 provides the full database search strategies. Multimedia Appendix 2 provides detailed model development and predictor characteristics across included studies. Multimedia Appendix 3 provides model performance metrics, including discrimination and classification measures. Multimedia Appendix 4 provides the PROBAST domain-level risk-of-bias and applicability assessments.
Conflicts of Interest: None declared.
References
1.LaHue SC, Douglas VC. Approach to altered mental status and inpatient delirium. Neurol Clin. 2022 Feb;40(1):45–57. doi: 10.1016/j.ncl.2021.08.004. doi. Medline.
2.Zoremba N, Coburn M. Acute confusional states in hospital. Dtsch Arztebl Int. 2019 Feb 15;116(7):101–106. doi: 10.3238/arztebl.2019.0101. doi. Medline.
3.Waszynski C. The confusion assessment method (CAM) Semantic Scholar. 2007. [13-01-2026]. https://www.semanticscholar.org/paper/The-Confusion-Assessment-Method-(CAM)-Waszynski/ad2f4dd6f43a70ddc4be594207b061cce10f6892 URL. Accessed.
4.Pagali SR, Miller DM, Manning DM. Predicting when a patient would be “out of the furrow”—a perspective on delirium prediction. Mayo Clin Proc. 2019 Oct;94(10):2145–2146. doi: 10.1016/j.mayocp.2019.08.001. doi. Medline.
5.Fu S, Lopes GS, Pagali SR, et al. Ascertainment of delirium status using natural language processing from electronic health records. J Gerontol A Biol Sci Med Sci. 2022 Mar 3;77(3):524–530. doi: 10.1093/gerona/glaa275. doi. Medline.
6.Lindroth H, Bratzke L, Purvis S, et al. Systematic review of prediction models for delirium in the older adult inpatient. BMJ Open. 2018 Apr 28;8(4):e019223. doi: 10.1136/bmjopen-2017-019223. doi. Medline.
7.Ruppert MM, Lipori J, Patel S, et al. ICU delirium-prediction models: a systematic review. Crit Care Explor. 2020 Dec;2(12):e0296. doi: 10.1097/CCE.0000000000000296. doi. Medline.
8.Xie Q, Wang X, Pei J, et al. Machine learning-based prediction models for delirium: a systematic review and meta-analysis. J Am Med Dir Assoc. 2022 Oct;23(10):1655–1668. doi: 10.1016/j.jamda.2022.06.020. doi. Medline.
9.Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021 Mar 29;372:n71. doi: 10.1136/bmj.n71. doi. Medline.
10.Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015 Jan 7;350:g7594. doi: 10.1136/bmj.g7594. doi. Medline.
11.TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024 Apr 18;385:q902. doi: 10.1136/bmj.q902. doi. Medline.
12.Moons KGM, de Groot JAH, Bouwmeester W, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med. 2014 Oct;11(10):e1001744. doi: 10.1371/journal.pmed.1001744. doi. Medline.
13.Wolff RF, Moons KGM, Riley RD, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med. 2019 Jan 1;170(1):51–58. doi: 10.7326/M18-1376. doi. Medline.
14.Ali MIM, Kalkman GA, Wijers CHW, Fleuren HWHA, Kramers C, de Wit HAJM. External validity of an automated delirium prediction model (DEMO) and comparison to the manual VMS-questions: a retrospective cohort study. Int J Clin Pharm. 2023 Oct;45(5):1128–1135. doi: 10.1007/s11096-023-01641-6. doi. Medline.
15.Bartolacci M, Carpenter KP, Jeffery MM, Mullan AF, Carpenter CR, Bellolio F. Validation of 4 risk stratification tools for delirium in the emergency department. JAMA Netw Open. 2025 Nov 3;8(11):e2540920. doi: 10.1001/jamanetworkopen.2025.40920. doi. Medline.
16.Bishara A, Chiu C, Whitlock EL, et al. Postoperative delirium prediction using machine learning models and preoperative electronic health record data. BMC Anesthesiol. 2022 Jan 3;22(1):8. doi: 10.1186/s12871-021-01543-y. doi. Medline.
17.Castro VM, Sacks CA, Perlis RH, McCoy TH. Development and external validation of a delirium prediction model for hospitalized patients with coronavirus disease 2019. J Acad Consult Liaison Psychiatry. 2021;62(3):298–308. doi: 10.1016/j.jaclp.2020.12.005. doi. Medline.
18.Ceppi MG, Rauch MS, Spöndlin J, Meier CR, Sándor PS. Assessing the risk of developing delirium on admission to inpatient rehabilitation: a clinical prediction model. J Am Med Dir Assoc. 2023 Dec;24(12):1931–1935. doi: 10.1016/j.jamda.2023.07.003. doi. Medline.
19.Contreras M, Kapoor S, Zhang J, et al. A large language model for delirium prediction in the intensive care unit using structured electronic health records. Sci Rep. 2025 Nov 6;15(1):38890. doi: 10.1038/s41598-025-22634-7. doi. Medline.
20.Contreras M, Silva B, Shickel B, et al. Dynamic delirium prediction in the intensive care unit using machine learning on electronic health records. IEEE EMBS Int Conf Biomed Health Inform. 2023 Oct;2023 doi: 10.1109/bhi58575.2023.10313445. doi. Medline.
21.Corradi JP, Thompson S, Mather JF, Waszynski CM, Dicks RS. Prediction of incident delirium using a Random Forest classifier. J Med Syst. 2018 Nov 14;42(12):261. doi: 10.1007/s10916-018-1109-0. doi. Medline.
22.Davoudi A, Ozrazgat-Baslanti T, Ebadi A, Bursian AC, Bihorac A, Rashidi P. Delirium prediction using machine learning models on predictive electronic health records data. 2017 IEEE 17th International Conference on Bioinformatics and Bioengineering (BIBE); Oct 23-25, 2017; Washington, DC, USA. 2017. pp. 568–573. Presented at. doi.
23.Heikal M, Saad H, Ghanime PM, et al. Using machine learning and electronic health records to identify neuropsychiatric risk scores for delirium in ICU and general hospital settings. Neuropsychiatr Dis Treat. 2024;20:1861–1876. doi: 10.2147/NDT.S479756. doi. Medline.
24.Holler E, Ludema C, Ben Miled Z, et al. Development and Validation of a routine electronic health record-based delirium prediction model for surgical patients without dementia: retrospective case-control study. JMIR Perioper Med. 2025 Jan 9;8:e59422. doi: 10.2196/59422. doi. Medline.
25.Hur S, Ko RE, Yoo J, Ha J, Cha WC, Chung CR. A machine learning-based algorithm for the Prediction of Intensive Care Unit Delirium (PRIDE): retrospective study. JMIR Med Inform. 2021 Jul 26;9(7):e23401. doi: 10.2196/23401. doi. Medline.
26.Jauk S, Kramer D, Großauer B, et al. Risk prediction of delirium in hospitalized patients using machine learning: an implementation and prospective evaluation study. J Am Med Inform Assoc. 2020 Jul 1;27(9):1383–1392. doi: 10.1093/jamia/ocaa113. doi. Medline.
27.Jauk S, Veeranki SPK, Kramer D, et al. External validation of a machine learning based delirium prediction software in clinical routine. Stud Health Technol Inform. 2022 May 16;293:93–100. doi: 10.3233/SHTI220353. doi. Medline.
28.Jauk S, Kramer D, Sumerauer S, Veeranki SPK, Schrempf M, Puchwein P. Machine learning-based delirium prediction in surgical in-patients: a prospective validation study. JAMIA Open. 2024 Oct;7(3):ooae091. doi: 10.1093/jamiaopen/ooae091. doi. Medline.
29.Jung JW, Hwang S, Ko S, et al. A machine-learning model to predict postoperative delirium following knee arthroplasty using electronic health records. BMC Psychiatry. 2022 Jun 27;22(1):436. doi: 10.1186/s12888-022-04067-y. doi. Medline.
30.Li Q, Li J, Chen J, et al. A machine learning-based prediction model for postoperative delirium in cardiac valve surgery using electronic health records. BMC Cardiovasc Disord. 2024 Jan 18;24(1):56. doi: 10.1186/s12872-024-03723-3. doi. Medline.
31.Liu S, Schlesinger JJ, McCoy AB, et al. New onset delirium prediction using machine learning and long short-term memory (LSTM) in electronic health record. J Am Med Inform Assoc. 2022 Dec 13;30(1):120–131. doi: 10.1093/jamia/ocac210. doi. Medline.
32.Lucini FR, Stelfox HT, Lee J. Deep learning-based recurrent delirium prediction in critically ill patients. Crit CARE Med. 2023 Apr 1;51(4):492–502. doi: 10.1097/CCM.0000000000005789. doi. Medline.
33.Matsumoto K, Nohara Y, Sakaguchi M, et al. Temporal generalizability of machine learning models for predicting postoperative delirium using electronic health record data: model development and validation study. JMIR Perioper Med. 2023 Oct 26;6:e50895. doi: 10.2196/50895. doi. Medline.
34.Moon KJ, Jin Y, Jin T, Lee SM. Development and validation of an automated delirium risk assessment system (Auto-DelRAS) implemented in the electronic health record system. Int J Nurs Stud. 2018 Jan;77:46–53. doi: 10.1016/j.ijnurstu.2017.09.014. doi. Medline.
35.Mueller B, Street WN, Carnahan RM, Lee S. Evaluating the performance of machine learning methods for risk estimation of delirium in patients hospitalized from the emergency department. Acta Psychiatr Scand. 2023 May;147(5):493–505. doi: 10.1111/acps.13551. doi. Medline.
36.Pagali SR, Fischer KM, Kashiwagi DT, et al. Validation and recalibration of modified Mayo delirium prediction tool in a hospitalized cohort. J Acad Consult Liaison Psychiatry. 2022;63(6):521–528. doi: 10.1016/j.jaclp.2022.05.006. doi. Medline.
37.Reeve KA, Schmutz Gelsomino N, Venturini M, et al. Prospective external validation of the automated PIPRA multivariable prediction model for postoperative delirium on real-world data from a consecutive cohort of non-cardiac surgery inpatients. BMJ Health Care Inform. 2025 Apr 10;32(1):e101291. doi: 10.1136/bmjhci-2024-101291. doi. Medline.
38.Rudolph JL, Doherty K, Kelly B, Driver JA, Archambault E. Validation of a delirium risk assessment using electronic medical record information. J Am Med Dir Assoc. 2016 Mar 1;17(3):244–248. doi: 10.1016/j.jamda.2015.10.020. doi. Medline.
39.Sheikhalishahi S, Bhattacharyya A, Celi LA, Osmani V. An interpretable deep learning model for time-series electronic health records: case study of delirium prediction in critical care. Artif Intell Med. 2023 Oct;144:102659. doi: 10.1016/j.artmed.2023.102659. doi. Medline.
40.Sun H, Depraetere K, Meesseman L, et al. A scalable approach for developing clinical risk prediction applications in different hospitals. J Biomed Inform. 2021 Jun;118:103783. doi: 10.1016/j.jbi.2021.103783. doi. Medline.
41.Sun H, Depraetere K, Meesseman L, et al. Machine learning-based prediction models for different clinical risks in different hospitals: evaluation of live performance. J Med Internet Res. 2022 Jun 7;24(6):e34295. doi: 10.2196/34295. doi. Medline.
42.Wong A, Young AT, Liang AS, Gonzales R, Douglas VC, Hadley D. Development and validation of an electronic health record-based machine learning model to estimate delirium risk in newly hospitalized patients without known cognitive impairment. JAMA Netw Open. 2018 Aug 3;1(4):e181018. doi: 10.1001/jamanetworkopen.2018.1018. doi. Medline.
43.Shillan D, Sterne JAC, Champneys A, Gibbison B. Use of machine learning to analyse routinely collected intensive care unit data: a systematic review. Crit Care. 2019 Aug 22;23(1):284. doi: 10.1186/s13054-019-2564-9. doi. Medline.
44.Andaur Navarro CL, Damen JAA, Takada T, et al. Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review. BMJ. 2021 Oct 20;375:n2281. doi: 10.1136/bmj.n2281. doi. Medline.
45.Venema E, Wessler BS, Paulus JK, et al. Large-scale validation of the prediction model risk of bias assessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination. J Clin Epidemiol. 2021 Oct;138:32–39. doi: 10.1016/j.jclinepi.2021.06.017. doi. Medline.
Associated Data
Supplementary Materials
Come citare questo studio
Hung-Min Huang, Chun-Shun Lu, Geng-Wei Chang, Ming-Hsu Tien, Yu-Kai Hsu. (2026). Prediction Models for In-Hospital Delirium Using Routinely Collected Electronic Health Record Data: Systematic Review.. JMIR medical informatics. DOI: 10.2196/91618
Consulta la pubblicazione originale ↗