Artificial Intelligence (AI) embraces techniques and algorithms that, over the years, have been applied to diferent fields and domains. The complexity and rise of data in healthcare have prompted research into exploring the application of AI techniques in the medical field, resulting in viable and promising approaches to sustain innovation, early diagnosis and support cures. In this work, we briefly report some of the projects we carried out in this domain at the University of Naples Federico II node of the CINI-AIIS Lab, focusing on the main aims and contributions.
most common application of traditional ML is precision medicine- predicting what treatment protocols are likely Artificial intelligence (AI) is the simulation of human in- to succeed on a patient considering his/her attributes telligence in machines to perform tasks like abstraction and the treatment context. The great majority of ML and problem-solving. Healthcare is one of the domains and precision medicine applications require a training that has been impacted by AI, and it is considered one dataset for which the outcome variable (i.e. onset of disof the most promising applications of AI. Indeed, AI in ease) is known (this is called supervised learning.). Deep healthcare can be used to support patients and physicians, Learning (DL) is a family of ML algorithms that use Artias well as to transform patient care and administrative ifcial Neural Networks (ANNs) to simulate the structure processes. Also, AI-powered systems can analyze patient of the human brain. DL approaches have gained popudata to identify potential health risks and help physicians larity in pattern recognition tasks, particularly in image make more accurate diagnoses, which is useful in situ- processing, thanks to Convolutional Neural Networks ations where patients have complex medical conditions (CNNs), a type of DL architecture that can learn the best or multiple medical conditions. set of features for a given task. As a consequence, DL
AI applications are not limited to a specific technology, is increasingly being used in healthcare, particularly in but rather a collection of them. Machine Learning (ML) medical image analysis and radiomics to extract clinically is a subset of AI that includes the set of algorithms that relevant features from images beyond what can be perprovides the systems with the ability to automatically ceived by the human eye. However, medical imaging is learn and improve from experience. In healthcare, the not the only domain in which DL is being proven useful. Making sense of human language has indeed been one of Intiazle-dIAby20C2I3N:I3,rMdaNyat2i9o–n3a1l,C2o0n2f3e,rPeinscae, IotnalAyrtificial Intelligence, orga- the main goals of AI researchers. Natural Language Pro* Corresponding author. cessing (NLP), strongly based on ML and DL techniques, $ carlo.sansone@unina.it (C. Sansone) includes applications such as speech recognition and text 0000-0002-1266-2154 (A. Bosco); 0000-0002-2578-9349 analysis. In healthcare, the dominant applications of (S. Capuozzo); 0000-0001-5033-9617 (M. Gravina); NLP involve the creation, understanding and classifica(0M00.0P-.0M00a1u-r6e8l5li2)-;00307070-(0S0.0M2-a0r7ro54n-e7)6;90600(0V-.0M00o2s-c7a5t6o4)-;3356 tion of clinical documentation and published research. 0000-0001-5425-1890 (G. Pontillo); 0000-0003-1470-8053 Moreover, NLP systems can analyse unstructured clini(M. Postiglione); 0000-0001-7003-4781 (A. M. Rinaldi); cal notes on patients, prepare reports, transcribe patient 0000-0002-0975-2052 (L. Rinaldi); 0000-0002-8732-1733 (C. Russo); interactions and conduct conversational AI. 0000-0003-4033-3777 (G. Sperlì); 0000-0001-9763-8745 It should be clear that thanks to its multitude of appli(0C0.00T-o0m00m2-a8s1in7o6)-;60905000(-C0.00S1a-n9s8o2n8e-)0079 (G. Cringoli); cations, AI plays a key role in healthcare. In this paper, © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License we will illustrate some of the projects exploiting AI techCPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) niques in the medical field carried out at the University of S i n g l-t e a s k iitr a n n g
...
identifying mentions of biomedical entities (e.g. disorddeartas, adnrudgiss, gaenfuentidcaimnfeonrmtaaltsiotenp) firnommaunnystrduocwtunrsetdretaemxt Merge d1atasets Get pre2dictions Train s4tudent tasks, such as the development of chatbots, Q/A systems or knowledge graphs. BioNER systems have evolved Figure 1: Overview of the training framework for TaughtNet. over time through deep learning techniques based on I) multiple datasets are combined to create a larger one called Bidirectional Long-Short Term Memory networks [1] , to serve as a reference during the Student’s training; II), and character-level features of words [2, 3]. Large-scale the Teacher models make predictions for each sample in , language models pre-trained on biomedical corpora [4, 5] and their output distributions are combined to create a new have recently shown potential in enhancing the state-of- corpus; III), the Student is trained using two loss components, the-art. However, developing a BioNER system is still tornuethbadsaetdaofnrotmheTe a.chers and the other based on the ground dificult due to the high frequency of synonyms, alternate spellings and polysemous words. Furthermore, due to the privacy concerns related to the healthcare domain, there is a lack of annotated data, and inference time 3. Distributions aggregation. A single output disand memory constraints arise when embedding mod- tribution is generated by integrating the output els in clinical workflows. When building BioNER models, distributions from each teacher. researchers are often forced to integrate datasets from 4. Student Training. A Student model with paramediferent sources that include annotations for diferent ters is trained by taking both the ground truth entity types. and the knowledge of Teachers into considera
To handle these issues without having to design, train tion. Specifically, we define the loss function as and deploy a single Transformer-based BioNER model for follows: each available dataset — which is extremely impractical due to their constraining memory requirements and to the problems which would arise due to overlapping pre- ℒ( ; , 1 , . . . , ) = dictions, e.g. two models assigning two diferent entity · ℒ ( ; , 1 , . . . , )+ types to the same mention —, we propose TaughtNet [6], a multi-task framework based on knowledge distillation + (1 − ) · ℒ ( ; ), (1) that fine-tunes a single transformer architecture to recog- where ℒ and ℒ are the knowledge distillanize multiple entity types. TaughtNet produces a single, tion and ground-truth loss, respectively, while independent Student Transformer model that is capable is a hyperparameter controlling their weight on of recognizing a variety of entity types. The proposed the overall loss ℒ. method is compared to other state-of-the-art approaches, and the results show that TaughtNet achieves higher In our experiments, we show that not only does Taughperformance across multiple benchmark datasets. tNet allow to recognize multiple biomedical entity types
We show an overview of our framework in Figure 1. while ensuring state-of-the-art performance, but it also The methodological workflow assumes that a set of can be applied to smaller and lighter students which can single-task Teachers represented by their parameters be more easily deployed on hardware with limited re 1 , 2 , . . . , ( being the number of teachers) has al- sources w.r.t. heavy transformer models. ready been trained, and can be summarized with the steps listed below: 3. The KFM 1. Datasets aggregation. The available single-entity datasets are merged together to build an aggre- The scanning process of parasite eggs contained in faecal gated multi-entity dataset . samples of farm animals is a common task in the veteri2. Retrieval of Teacher predictions. Each sample in nary medicine domain which can be automated thanks is provided to each teacher as input and the to Convolutional Neural Networks (CNNs) for Object resulting output distributions are stored. Detection. The Kubic FLOTAC Microscope (KFM) is a compact, low-cost, versatile, and portable digital microscope designed to autonomously analyze faecal specimens pre- agents [8]. Consequently, the presence of the methylation pared with FLOTAC or Mini-FLOTAC, in both field and process in GBM patients’ brains can be associated with a laboratory settings, for diferent parasites and hosts [ 7]. predictive biomarker of response to therapy and a progHaving been proven to acquire images comparable to nosis factor [9]. Unfortunately, identifying methylation the view provided by traditional optical microscopes, the signs is not trivial, and often requires time-consuming, KFM can autonomously scan and acquire images in a few expensive and invasive procedures. In a recent work minutes, allowing the operator to focus on a diferent task. [10], we propose to face MGMT promoter methylation The KFM is composed of electro-mechanical components identification by analyzing Magnetic Resonance Imaging which enable both manual and automatic FLOTAC/Mini- data with a Deep Learning based approach, consisting FLOTAC reading discs 3D scans, a firmware that allows of a CNN operating on suspicious regions on the FLAIR 3D movements of the camera, remote interactions, and series, which are pre-selected through an unsupervised scans retrieval, and external agents which enable users to Knowledge-Based filter leveraging both FLAIR and T1connect with the KFM hardware (KFM web interface and weighted series.
KFM app) and to process scans for parasite eggs detection Given the absence of segmentation masks, determin(KFM AI server). The device can be remotely controlled ing somehow the ROI representing the tumour region (the optical part with LED and digital camera and tray) by is a crucial and critical step. We proposed to select the any user with external devices, like smartphones, tablets area of interest in an unsupervised manner, leveraging or PCs, through a dedicated web interface and app, mak- past medical experience [11] for tumour recognition. In ing it possible to easily manage multiple KFMs with a particular, we took advantage of simple characteristics single device, start several scans in parallel and switch of lesioned tissues for each considered series: in T1-W from one KFM to another efortlessly. Images collected slices, tumour areas have pixels whose intensity is higher with the KFM can be stored in the KFM AI server and/or than cerebrospinal fluids (CSF) but lower than any other transmitted to diagnostic hubs in order to have a quick kind of tissue, while in FLAIR slices, pixels with the highdiagnosis or a parasitological consultation. The whole est intensity belong to the tumour region. Leveraging system flow is described in Figure 2. these characteristics, it is possible to pre-select potentially lesioned tissues from volumes of these series by applying a threshold on the histogram of the signal intensities occurrences. The output of this process is a mask consisting of huge clusters, in case of tumour presence, corresponding to the ROIs, representing the potentially lesioned areas, with little outliers, or of sparse outliers in the opposite case, as shown in Figure 3. The resulting ROI is used to select from the FLAIR slices the portion of the image to be considered by the actual methylation Figure 2: A schematic picture of the working flow of the detection module. We chose the FLAIR sequence for the KFM system, composed of the hardware, the AI server and good performance shown in the literature for related the client app interface. tasks [12]. The proposed module aims to compensate for the lack of segmentation masks with an early fusion technique, where information coming from multiple sources is merged to highlight diferent characteristics [13]. 4. MGMT Promoter Methylation We adopted a CNN to face the task of MGMT promoter methylation identification. In particular, we built a seIdentification quential network from the ground called MGMTClassifier , which is composed of seven Convolutional Blocks and One of the most aggressive malignant tumours is the two fully connected layers separated by Rectified Linear Glioblastoma Multiforme (GBM), which is known for Unit (ReLU) as an activation function. These Convoluits extremely low survival rate. Even if alkylating tional Blocks consist of a convolutional layer, followed chemotherapy is typically adopted to fight this tu- by batch normalization and ReLU function. To reduce mour, sometimes it may result ineficient since the the number of training parameters and avoid overfitting O(6)-methylguanine-DNA methyltransferase (MGMT) at the same time, we adopt depth-wise separable convoenzyme repair abilities counter the cytotoxic efects of lution [14]. This simple but efective DL-based approach alkylating agents, preventing the tumours cells destruc- is able to outperform state-of-the-art solutions while contion. However, MGMT promoter regions may be subject sisting of less than 0.29% of their parameters (about 10 to a phenomenon called methylation, a biological process million of typical CNNs versus 40561 of the proposed preventing MGMT enzymes from destroying the alkyl approach). by exploiting CNNs in the solution. On the basis of the results reported in [15, 16], we aim to include multiple and complementary sequences acquired during the MRI exam, namely DCE, T2 and DWI, proposing an approach based on multimodal learning, where heterogeneous sources of data are fused to provide a shared representation. Moreover, we will focus on a methodology based on intermediate fusion, leveraging the ability of deep neural networks to provide an efective high-level representation of the input and taking into account both MR images and clinical information.
brain health and pathology, the brain-age paradigm has emerged as a promising approach. Briefly, machine learning methods are used to model chronological age as a 5. Axillary Lymph Node Status function of structural brain MRI scans in healthy peoAssessment in Breast Cancer ple, and the resulting model of ‘normal’ brain ageing is used for neuroimaging-based age prediction in unseen Among women, breast cancer (BC) is the most frequent subjects [17]. The extent to which each subject deviform of tumour, and the axillary lymph node status ates from healthy brain-ageing trajectories, expressed (ALNS) is considered a crucial indicator, representing as the diference between predicted and chronological one of the most influencing and independent prognostic age (brain-predicted age diference, brain-PAD), has been factors. Magnetic Resonance Imaging (MRI) is always proposed as an index of structural brain health, sensitive performed for BC stage definition and plays a key role in to brain pathology in a wide spectrum of neurological primary tumour examination. The most important MRI and psychiatric disorders [18]. We applied the brainsequence is the dynamic-contrast enhanced (DCE) that, age paradigm to a target clinical population of patients thanks to the high contrast resolution, provides informa- with Fabry Disease (FD), a rare genetic multisystemic tion about the tumour morphology, size, and perfusional disorder that also involves the brain but lacks quantitabehaviour allowing the distinction between benign and tive neuroimaging biomarkers. We selected MRI scans malignant lesions, the prediction of biological aggressive- of FD patients and healthy controls from the same Inness and the prognostic evaluation. The T2-weighted stitution. The Fabry stabilization index (FASTEX) was (T2) imaging is a standard component of breast MRI ex- recorded as a measure of multi-organ FD severity. We ams, most prominently utilized for the identification of trained and evaluated a model of healthy brain ageing on cysts, allowing a better depiction of lesion morphology a large dataset (total N = 2160; male/female = 1293/867; and perifocal or prepectoral edema within the breast. mean age = 33 years, age range = 4-86) comprising 3D Difusion-Weighted Imaging (DWI) is another sequence T1-weighted brain scans of healthy subjects from 8 pubacquired during the MRI exam. It reflects the mobility licly available sources. Our brain-age model was based of water molecules difusing in tissues, revealing tissue on the DenseNet264 architecture [19] adapted from the organization at the microscopic level and providing com- implementation available at Project MONAI 1 by adding plementary information for lesion assessment in compar- a linear regression layer for the prediction of a continison with the DCE scan. uous variable and a 0.2 dropout rate after each dense
Assessment of ALNS indicates inherent primary tu- layer to reduce the risk of overfitting. Mean absolute mour properties, whose examination enables the discov- error (MAE) and coeficient of determination (R2) were ery of minimally invasive solutions for the sentinel node used to quantify model performance. The final model biopsy currently being utilized. In [15], we only focused was applied to the internal cohort of FD patients and on the DCE sequence to analyze diferent tumour bound- healthy controls to generate brain-predicted ages and coring options for the prediction of ALN status. In particular, responding brain-PAD values (Figure 4). Lastly, within a the work described in [15] evaluates how the amount of linear modelling framework, brain-PAD was tested for the included non-tumour tissue impacts the assessment 1https://docs.monai.io/en/stable/_modules/monai/networks/nets/ densenet.html perform a query only employing metadata. The design and implementation of a robust and accurate methodology for querying them in the pathology domain using a novel approach are mandatory. In particular, the Content-Based Image Retrieval (CBIR) methodology can be involved in the PACSs using a query-by-example task.
In this context, one of many crucial points of CBIR concerns the representation of images as feature vectors, and the accuracy of retrieval mainly depends on feature aFnigdupreerf4o:rTmhaenmceodisemlweiatshutrhede loonwtehset vparleivdiaotuiosnlylousnsseisecnhcoasseens, extraction. In order to allow intelligent multimodality of the test set (left). The final model is also applied to the query posing [20], Content-Based Image Retrieval (CBIR) target clinical population (right), composed of the internal techniques can be used [21]. The feature extraction methcohort of FD patients and healthy controls, to generate brain- ods and similarity functions are crucial aspects of a CBIR predicted ages and corresponding brain-PAD values. [22]. Over the years, diferent approaches for image retrieval have been proposed, also in the computational pathology field using diferent techniques such as deep learning [23]. Thus, our study explored diferent repredemographics-adjusted associations with a diagnostic sentations of WSI patches by features extracted from pregroup (FD vs healthy controls) and FASTEX score. trained Convolution Neural Networks (CNNs). In order
We selected 52 FD patients (40.6 ± 12.6 years; 28 fe- to perform a helpful comparison, we evaluated features males) and 58 healthy controls (38.4 ± 13.4 years; 28 extracted from diferent layers of state-of-the-art CNNs females). The brain-age model achieved accurate out- using diferent dimensionality reduction techniques. Furof-sample performance (MAE = 4.01y, R2 = 0.90). FD thermore, we provided a qualitative analysis of obtained patients had significantly higher brain-PAD than healthy results. To accomplish a more precise analysis, we anacontrols (estimated marginal means: 3.1 vs -0.1, p=0.01), lyzed in depth all the best CNN configurations using a and brain-PAD was also associated with multi-organ clin- confusion matrix. We intend to understand which dataset ical severity as assessed with the FASTEX score (B = 0.10, category is not correctly retrieved. According to Figure 5, p = 0.02). In conclusion, using deep learning and the it is clear that the queries using benign examples are not brain-age paradigm, we found that patients with FD have correctly retrieved. They are often misunderstood with older-appearing brains compared to healthy controls. in situ images and normal ones. Furthermore, normal The gap between brain-predicted and chronological age and invasive queries are usually recognized with good correlates with multi-organ disease severity, potentially precision value. representing a novel quantitative imaging biomarker of brain involvement in FD. 0.75
processes mainly due to the extensive digitization of his- 0.55 tology slides that resulted in the availability of many Whole Slide Images (WSIs). Their use is essential, espe- 0.50 cially in cancer diagnosis and research, and raises the pressing need for increasingly influential information archiving and retrieval systems. Picture Archiving and 0.45 Communication Systems (PACSs) represent an actual possibility to archive and organize this growing amount 0.40 of data. The storage, retrieval, and analysis of biomedical P@5 P@10 P@50 P@100 images are essential tools of Picture Archiving and Communication Systems (PACSs). The same ones are also fFriogmurreet5r:iePvraelcuissiionngcComNNpalraisyoenrsaatn5d,1g0l,o5b0alaanvde1ra0g0eopbotaoilninedg useful in related contexts such as computational pathol- to reduce dimensionality. ogy (CPATH), where their difusion is still limited. A weakness of traditional PACSs concerns the capability to
0.70 0.65 0.60 VGG16 - block5_pool ResNet152V2 - conv4_block36_out DenseNet201 - conv5_block32_concat InceptionV3 - mixed6 Xception - add_10 InceptionResNetV2 - mixed_7a MobileNetV2 - block_14_add NASNetLarge - normal_concat_12 EfficientNetV2L - block6y_add
A. Sadeghian, M. Afzali, Contrast-enhanced [1] G. Lample, M. Ballesteros, S. Subramanian, weighted-t1 and flair sequences in mri of meningeal K. Kawakami, C. Dyer, Neural architectures for lesions, American Journal of Nuclear Medicine and named entity recognition, in: HLT-NAACL, 2016. Molecular Imaging 12 (2022) 63. [2] X. Ma, E. Hovy, End-to-end sequence labeling [13] D. Ramachandram, G. W. Taylor, Deep multimodal via bi-directional LSTM-CNNs-CRF, in: Proceed- learning: A survey on recent advances and trends, ings of the 54th Annual Meeting of the Associa- IEEE signal processing magazine 34 (2017) 96–108. tion for Computational Linguistics (Volume 1: Long [14] F. Chollet, Xception: Deep learning with depthPapers), Association for Computational Linguis- wise separable convolutions, in: Proceedings of the tics, Berlin, Germany, 2016, pp. 1064–1074. URL: IEEE conference on computer vision and pattern https://aclanthology.org/P16-1101. doi:10.18653/ recognition, 2017, pp. 1251–1258. v1/P16-1101. [15] D. Santucci, E. Faiella, M. Gravina, E. Cordelli, [3] M. Habibi, L. Weber, M. L. Neves, D. L. Wiegandt, C. de Felice, B. Beomonte Zobel, G. Iannello, C. SanU. Leser, Deep learning with word embeddings im- sone, P. Soda, Cnn-based approaches with diferent proves biomedical named entity recognition, Bioin- tumor bounding options for lymph node status preformatics 33 (2017) i37 – i48. diction in breast dce-mri, Cancers 14 (2022) 4574. [4] I. Beltagy, K. Lo, A. Cohan, Scibert: A pretrained [16] M. Gravina, E. Cordelli, D. Santucci, P. Soda, C. Sanlanguage model for scientific text, in: EMNLP/IJC- sone, Evaluating tumour bounding options for deep NLP, 2019. learning-based axillary lymph node metastasis pre[5] S. Gururangan, A. Marasović, S. Swayamdipta, diction in breast cancer, in: 2022 26th International K. Lo, I. Beltagy, D. Downey, N. A. Smith, Don’t stop Conference on Pattern Recognition (ICPR), IEEE, pretraining: Adapt language models to domains and 2022, pp. 4335–4342.
tasks, in: ACL, 2020. [17] J. H. Cole, K. Franke, Predicting age using neu[6] V. Moscato, M. Postiglione, C. Sansone, G. Sperli, roimaging: innovative brain ageing biomarkers, Taughtnet: Learning multi-task biomedical named Trends in neurosciences 40 (2017) 681–690. entity recognition from single-task teachers, IEEE [18] T. Kaufmann, D. van der Meer, N. T. Doan, Journal of Biomedical and Health Informatics (2023) E. Schwarz, M. J. Lund, I. Agartz, D. Alnaes, D. M. 1–12. doi:10.1109/JBHI.2023.3244044. Barch, R. Baur-Streubel, A. Bertolino, et al., Com[7] G. Cringoli, A. Amadesi, M. P. Maurelli, B. Celano, mon brain disorders are associated with heritable G. Piantadosi, A. Bosco, L. Ciuca, M. Cesarelli, P. Bi- patterns of apparent aging of the brain, Nature fulco, A. Montresor, et al., The kubic flotac mi- neuroscience 22 (2019) 1617–1623. croscope (kfm): a new compact digital microscope [19] G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Weinfor helminth egg counts, Parasitology 148 (2021) berger, Densely connected convolutional networks, 427–434. in: Proceedings of the IEEE conference on computer [8] M. Christmann, B. Verbeek, W. P. Roos, B. Kaina, vision and pattern recognition, 2017, pp. 4700–4708.
O6-methylguanine-dna methyltransferase (mgmt) [20] A. M. Rinaldi, C. Russo, C. Tommasino, Visual query in normal tissues and tumors: enzyme activity, posing in multimedia web document retrieval, in: promoter methylation and immunohistochemistry, 2021 IEEE 15th International Conference on SemanBiochimica et Biophysica Acta (BBA)-Reviews on tic Computing (ICSC), IEEE, 2021, pp. 415–420.
Cancer 1816 (2011) 179–190. [21] V. Jeyakumar, B. Kanagaraj, A medical image re[9] W. Haque, E. Thong, S. Andrabi, V. Verma, E. B. trieval system in pacs environment for clinical deButler, B. S. Teh, Prognostic and predictive impact cision making, in: Intelligent Data Analysis for of mgmt promoter methylation in grade 3 gliomas, Biomedical Applications, Elsevier, 2019, pp. 121– Journal of Clinical Neuroscience 85 (2021) 115–121. 146. [10] S. Capuozzo, M. Gravina, G. Gatta, S. Marrone, [22] A. M. Rinaldi, C. Russo, A content based image C. Sansone, A multimodal knowledge-based deep retrieval approach based on multiple multimedia learning approach for mgmt promoter methylation features descriptors in e-health environment, in: identification, Journal of Imaging 8 (2022) 321. 2020 IEEE International Symposium on Medical [11] M. C. Clark, L. O. Hall, D. B. Goldgof, R. Velthuizen, Measurements and Applications (MeMeA), IEEE, R. Murtagh, M. S. Silbiger, Unsupervised brain 2020, pp. 1–6. tumor segmentation using knowledge-based fuzzy [23] J. Ma, X. Jiang, A. Fan, J. Jiang, J. Yan, Image matchtechniques, in: Fuzzy and neuro-fuzzy systems in ing from handcrafted to deep features: A survey, medicine, CRC Press, 2017, pp. 137–170. International Journal of Computer Vision 129 (2021) [12] S. Roozpeykar, M. Azizian, Z. Zamani, M. R. 23–79.