A Multi-input Deep Learning Model to Classify COVID-19 Pneumonia Severity from Imaging and Clinical Data

The study of multi-input models that can process heterogeneous data is a challenge at the frontier of machine learning. We implemented a multi-modal approach aiming at exploiting both imaging and clinical information of patients to predict the severity of their outcome. As a specific use case, we developed a fully automated algorithm to predict the outcome of COVID-19 patients based on chest X-Ray images and clinical data, provided by the AI4COVID Challenge (1589 subjects). The system can distinguish between severe cases, those who needed intensive care or died, and mild ones. The system is composed of three Convolutional Neural Networks (CNN) for pre-processing, lung segmentation (U-Net architecture), and outcome classification. The first CNN is devoted to recognize the gray-level encoding needed to standardize the images. The U-Net for lung segmentation has been trained using two datasets collected for Tuberculosis screening. We achieved a Dice Similarity Coefficient (DSC) equal to 0.96 ± 0.03. This was needed to focus the final classifier on evaluating features within the lung. Without the careful selection of the lung, in fact, the prediction strongly depended on features outside the lung district (e.g. ECG cables, respiratory masks). The outcome classifier is a multi-input CNN made of two branches joined at the bottom. The first branch is a ResNet that takes the segmented images as input, while the second branch is a Multi-Layer Perceptron (MLP) that takes in the preprocessed clinical parameters. We obtained an AUC equal to 0.84 and an accuracy equal to 76%. We also computed the saliency maps with the gradCAM and the feature importance to obtain a reasonable explanation of the classifier. This method based on data aggregation and on merging clinical and imaging information can be applied also to domains different from COVID-19 patients.