MaxCorrMGNN: A Multi-Graph Neural Network Framework for Generalized Multimodal Fusion of Medical Data for Outcome Prediction

Autor:	D'Souza, Niharika S., Wang, Hongzhi, Giovannini, Andrea, Foncubierta-Rodriguez, Antonio, Beck, Kristen L., Boyko, Orest, Syeda-Mahmood, Tanveer
Rok vydání:	2023
Předmět:	Computer Science - Machine Learning Electrical Engineering and Systems Science - Signal Processing
Druh dokumentu:	Working Paper
Popis:	With the emergence of multimodal electronic health records, the evidence for an outcome may be captured across multiple modalities ranging from clinical to imaging and genomic data. Predicting outcomes effectively requires fusion frameworks capable of modeling fine-grained and multi-faceted complex interactions between modality features within and across patients. We develop an innovative fusion approach called MaxCorr MGNN that models non-linear modality correlations within and across patients through Hirschfeld-Gebelein-Renyi maximal correlation (MaxCorr) embeddings, resulting in a multi-layered graph that preserves the identities of the modalities and patients. We then design, for the first time, a generalized multi-layered graph neural network (MGNN) for task-informed reasoning in multi-layered graphs, that learns the parameters defining patient-modality graph connectivity and message passing in an end-to-end fashion. We evaluate our model an outcome prediction task on a Tuberculosis (TB) dataset consistently outperforming several state-of-the-art neural, graph-based and traditional fusion techniques. Comment: To appear in ML4MHD workshop at ICML 2023
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2307.07093 Zobrazit plný text záznamu View this record from Arxiv