The current cancer treatment practice collects multimodal data, such as radiology images, histopathology slides, genomics and clinical data. The importance of these data sources taken individually has fostered the recent raise of radiomics and pathomics, i.e. the extraction of quantitative features from radiology and histopathology images routinely collected to predict clinical outcomes or to guide clinical decisions using artificial intelligence algorithms. Nevertheless, how to combine them into a single multimodal framework is still an open issue. In this work we therefore develop a multimodal late fusion approach that combines hand-crafted features computed from radiomics, pathomics and clinical data to predict radiation therapy treatment outcomes for non-small-cell lung cancer patients. Within this context, we investigate eight different late fusion rules (i.e. product, maximum, minimum, mean, decision template, Dempster-Shafer, majority voting, and confidence rule) and two patient-wise aggregation rules leveraging the richness of information given by computer tomography images and whole-slide scans. The experiments in leave-one-patient-out cross-validation on an in-house cohort of 33 patients show that the proposed multimodal paradigm with an AUC equal to $90.9\%$ outperforms each unimodal approach, suggesting that data integration can advance precision medicine. As a further contribution, we also compare the hand-crafted representations with features automatically computed by deep networks, and the late fusion paradigm with early fusion, another popular multimodal approach. In both cases, the experiments show that the proposed multimodal approach provides the best results.
翻译:目前的癌症治疗实践收集了多式数据,如放射图象、病理学幻灯片、基因组学和临床数据等。这些数据来源的重要性个别地促进了最近放射学和病理学的上升,即从放射学和病理学图像中提取常规收集的定量特征,以预测临床结果,或用人工智能算法指导临床决定;然而,如何将这些特征纳入单一多式联运框架仍然是一个未决问题。因此,在这项工作中,我们开发了一种多式迟期混合模式,将放射学、病理学和临床数据计算出的手制特征结合起来,以预测非小细胞肺癌病人的辐射治疗结果。在此背景下,我们调查了八种不同的晚期融合规则(即产品、最低、平均值、决定模板、Dempster-Shafer、多数投票和信任规则),以及两个以耐心为基础的汇总规则,利用计算机通货法图像和整面扫描法所提供的信息的丰富性。在离心机的病人一室、病理学和临床数据交叉分析结果中进行实验,在每组内提出一个最慢的计算模型中,我们提出一个最接近的模型展示了最精确的模型。