TY - JOUR
T1 - Fusion of imaging and non-imaging data for disease trajectory prediction for coronavirus disease 2019 patients
AU - Tariq, Amara
AU - Tang, Siyi
AU - Sakhi, Hifza
AU - Celi, Leo Anthony
AU - Newsome, Janice M.
AU - Rubin, Daniel L.
AU - Trivedi, Hari
AU - Gichoya, Judy Wawira
AU - Banerjee, Imon
N1 - Publisher Copyright:
© 2023 Society of Photo-Optical Instrumentation Engineers (SPIE).
PY - 2023/5/1
Y1 - 2023/5/1
N2 - Purpose: Our study investigates whether graph-based fusion of imaging data with non-imaging electronic health records (EHR) data can improve the prediction of the disease trajectories for patients with coronavirus disease 2019 (COVID-19) beyond the prediction performance of only imaging or non-imaging EHR data. Approach: We present a fusion framework for fine-grained clinical outcome prediction [discharge, intensive care unit (ICU) admission, or death] that fuses imaging and non-imaging information using a similarity-based graph structure. Node features are represented by image embedding, and edges are encoded with clinical or demographic similarity. Results: Experiments on data collected from the Emory Healthcare Network indicate that our fusion modeling scheme performs consistently better than predictive models developed using only imaging or non-imaging features, with area under the receiver operating characteristics curve of 0.76, 0.90, and 0.75 for discharge from hospital, mortality, and ICU admission, respectively. External validation was performed on data collected from the Mayo Clinic. Our scheme highlights known biases in the model prediction, such as bias against patients with alcohol abuse history and bias based on insurance status. Conclusions: Our study signifies the importance of the fusion of multiple data modalities for the accurate prediction of clinical trajectories. The proposed graph structure can model relationships between patients based on non-imaging EHR data, and graph convolutional networks can fuse this relationship information with imaging data to effectively predict future disease trajectory more effectively than models employing only imaging or non-imaging data. Our graph-based fusion modeling frameworks can be easily extended to other prediction tasks to efficiently combine imaging data with non-imaging clinical data.
AB - Purpose: Our study investigates whether graph-based fusion of imaging data with non-imaging electronic health records (EHR) data can improve the prediction of the disease trajectories for patients with coronavirus disease 2019 (COVID-19) beyond the prediction performance of only imaging or non-imaging EHR data. Approach: We present a fusion framework for fine-grained clinical outcome prediction [discharge, intensive care unit (ICU) admission, or death] that fuses imaging and non-imaging information using a similarity-based graph structure. Node features are represented by image embedding, and edges are encoded with clinical or demographic similarity. Results: Experiments on data collected from the Emory Healthcare Network indicate that our fusion modeling scheme performs consistently better than predictive models developed using only imaging or non-imaging features, with area under the receiver operating characteristics curve of 0.76, 0.90, and 0.75 for discharge from hospital, mortality, and ICU admission, respectively. External validation was performed on data collected from the Mayo Clinic. Our scheme highlights known biases in the model prediction, such as bias against patients with alcohol abuse history and bias based on insurance status. Conclusions: Our study signifies the importance of the fusion of multiple data modalities for the accurate prediction of clinical trajectories. The proposed graph structure can model relationships between patients based on non-imaging EHR data, and graph convolutional networks can fuse this relationship information with imaging data to effectively predict future disease trajectory more effectively than models employing only imaging or non-imaging data. Our graph-based fusion modeling frameworks can be easily extended to other prediction tasks to efficiently combine imaging data with non-imaging clinical data.
KW - clinical event prediction
KW - fusion model
KW - graph neural network
UR - http://www.scopus.com/inward/record.url?scp=85164221491&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85164221491&partnerID=8YFLogxK
U2 - 10.1117/1.JMI.10.3.034004
DO - 10.1117/1.JMI.10.3.034004
M3 - Article
AN - SCOPUS:85164221491
SN - 2329-4302
VL - 10
JO - Journal of Medical Imaging
JF - Journal of Medical Imaging
IS - 3
M1 - 034004
ER -