Cancer still represents one of the leading causes of death worldwide, resulting in a considerable healthcare impact. Recent research eforts from the clinical and molecular oncology scientific communities were able to increase considerably life expectancy of patients for some cancer types. Most of the current cancer diagnoses are primarily determined by pathology laboratories, providing an essential source for information to guide the treatment of patients with cancer. Pathology observations essentially characterize the results of microscopic or macroscopic studies of cells or tissues following a biopsy or surgery. Clinicians and researchers alike, require systems that automatically detect, read and generate structured data representations from pathology examinations. The resulting structured or coded clinical information, normalized using controlled vocabularies like the ICD-O or SNOMED-CT is critical for large-scale analysis of specific tumor types or to determine response to specific treatments or prognosis. Text mining and NLP approaches are showing promising results to transform medical text into useful clinical information, bridging the gap between free-text and structured representation of clinical information. Nonetheless, in the case of cancer text mining approaches, most eforts were exclusively focused on medical records in English. Moreover, due to the lack of high quality manually labeled clinical texts annotated by oncology experts most previous eforts, even for English relied mainly on customized dictionaries of names or rules to recognize clinical concept mentions despite the promising results of advanced deep learning technologies. To address these issues we have organized the Cantemist (CANcer TExt Mining Shared Task) track at IberLEF 2020. It represents the first community efort to evaluate and promote the development of resources for named entity recognition, concept normalization and clinical coding specifically focusing on cancer data in Spanish. Evaluation of participating systems was done using the Cantemist corpus, a publicly accessible dataset (together with annotation consistency analysis and guidelines) of manually annotated mentions of tumor morphology entities and their mappings to the Spanish version of ICD-O. We received a total of 121 systems or runs from 25 teams for one of the three Cantemist sub-tasks, obtaining very competitive results. Most participants implemented sophisticated AI approaches; mainly deep learning algorithms based on LongShort Term Memory Units and language models (BERT, BETO, RoBERTa, etc) with a classifier layer such as a Conditional Random Field. In addition to using pre-trained language models, word and character embeddings were also explored. Cantemist corpus: https://doi.org/10.5281/zenodo.3773228 Cancer is one of the leading causes of mortality worldwide, producing around one in six deaths in 2018. Lung, breast and colorectal cancers are amongst the most common types of cancer of the more
than 200 varieties [
Due to the highly specialized clinical languages and the need to standardize medical vocabularies,
a range of diferent terminology systems has been constructed for oncology data. The availability
of knowledge resources and medical terminologies is key for semantic interoperability and practical
clinical coding. Schulz et al. provides an overview of controlled vocabularies relevant to oncology [
To this end, we have created the first Gold Standard text corpus of tumor morphology mentions,
manually mapped to the latest Spanish version of ICD-O [
To increase the exploitation and impact of the CANTEMIST corpus, it was used for a shared task in the context of the IberLEF 2020 evaluation initiative. This paper provided an overview the results, data, methods, outcome and future outlook of the CANTEMIST shared task.
Cantemist explores the automatic detection of tumor morphology mentions in medical documents in
the Spanish language, as well as the assignment of eCIE-O codes (Morfología neoplasia, in Spanish) to
each mention. To the authors’ knowledge, Cantemist is the first shared task specifically focusing on
Named Entity Recognition of a critical type of concept related to cancer, namely tumor morphology, in
Spanish. Previous community evaluation eforts within the cancer domain include the cancer genetics
shared task using data in English [
The Cantemist task is structured into three independent sub-tasks, each taking into account a particularly important use case scenario: • Cantemist-NER track. It requires finding tumor morphology mentions automatically in text. All tumor morphology mentions are defined by their corresponding character ofsets (start character and end character) in UTF-8 plain text medical documents. • Cantemist-NORM track. Clinical concept normalization task that requires to return all tumor morphology entity mentions together with their corresponding eCIE-O codes, i.e. finding and normalizing tumor morphology mentions. • Cantemist-CODING track. It requires returning for each of the documents a ranked list of correct eCIE-O code assignments. This is essentially an indexing or multi-label classification task (oncology clinical coding).
The Cantemist track was organized in three participation periods or phases: 1. Training phase. During the first participation period, the training subset of the complete corpus was released, containing plain text documents and their annotations in the proper format (see section 3 for more details on corpus format). During this period, participants start building their systems. 2. Development phase. Then the development set was released (plain text documents and their annotations). This set was used to further fine-tune and evaluate the systems. 3. Test phase. Finally, the test set was released. In this case, only the plain text documents were provided to the participants. They had to use their systems to predict the correct annotations for these documents. After the submission deadline, the organizers evaluated the participants’ predictions against the manual annotations done by clinical experts. Each team was allowed to submit up to 5 runs.
In the first two subtasks, Cantemist-NER and Cantemist-Norm, the main evaluation metric has been micro average f1-score. In addition, precision and recall have been computed.
true positives Precision (P) = true positives + false positives
true positives
Recall (R) = true positives + false negatives F1 score (F1) = 2 ∗ ( ∗ )
( + )
For the Cantemist-Coding sub-track, the same metrics were computed. However, based on our
experience with past clinical coding eforts, the primary evaluation metric used was Mean Average
Precision (MAP). It is a ranking metric and therefore, participants has to submit their predictions
ranked by confidence. Mean Average Precision (MAP) is a score extensively used in ranking problems:
∑( ( ) ∗ ( ))
= number of relevant documents
where P(k) is the precision at the position k, and rel(k) is an indicator function equaling 1 if the item
at rank k is a relevant document, zero otherwise. MAP has shown good discrimination and stability
[
Additionally, as annotations with the code 8000/6 represent nearly 30% of all annotations, we also computed all metrics without taking into account this code for the Cantemist-Norm and CantemistCoding sub-tasks. The code 8000/6 corresponds to mentions of metastasis.
We have compared every system to a baseline prediction, in this case a Levenshtein lexical lookup approach using a sliding window of varying length. The baseline systems essentially scans a novel input text looking for mentions previously found in the training or development annotations. For every test set document the baseline system performed the following steps: 1. Select one annotation from the training and development sets to generate a dictionary entry. 2. Scan the test set document with a sliding window the same size of the dictionary entry, plus 2 characters. 3. If the Levenshtein distance from the current text inside the window to the annotation is smaller than 1, annotate it and add to it the code that the original annotation had. 4. Select another annotation and repeat steps 2 and 3 with it.
Its results are found in Table 1.
1https://github.com/TeMU-BSC/cantemist-evaluation-library
The Cantemist corpus is a collection of 1301 oncological clinical case reports written in Spanish. In ddition to this ofical corpus post-workshop document collection will be released to sum a total of 1900 clinical cases. The training subset contains 501 documents, the development subsets 500, and the test subset 300. All documents of the corpus have been manually annotated by clinical experts with mentions of tumor morphology (in Spanish, “morfología de neoplasia”). Every tumor morphology mention is linked to an eCIE-O code (the Spanish equivalent of ICD-O). Figure 1 shows an example of an annotated document fragment. The Cantemist corpus is publicly available at Zenodo 2.
To the best of our knowledge, there are no state-of-the-art, quality-controlled, publicly available NLP resources for oncologic histology in Spanish. The Cantemist corpus fills in this gap. It is a corpus of oncology clinical case reports written in Spanish, with tumor morphology mentions annotated and mapped to a controlled terminology, eCIE-O. It is publicly available, follows strict guidelines, and was annotated performing regular quality control analysis. It may be employed to develop new tools, to test the existing ones, and even to complete future oncology histology corpora, since the annotation guidelines are also published 3.
Spanish documents were chosen because it is the second world language with more native speakers,
after Mandarin Chinese. In addition, it follows past eforts of the shared task organizers [
To increase the usefulness and practical relevance of the Cantemist corpus, we selected clinical cases afecting all genders and that comprised most ages (from children to the elderly) and of various complexity levels (solid tumors, hemato-oncological malignancies, neuroendocrine cancer, etc.). We should emphasize that this corpus can be used toward generating further Gold Standard annotations, including temporal events and negations, or that focus on entities extracted from specific sections (for instance, imaging), of the individual clinical case records. The Cantemist cases include clinical signs and symptoms, personal and family history, current illness, physical examination, complementary tests (blood tests, imaging, pathology), diagnosis, treatment (including adverse efects of chemotherapy), evolution and outcome.
The lack of resources for oncology in Spanish was reflected in the training data employed by
Cantemist participants. While in the past shared tasks we have organized participants make use of other
public or private resources, this is rarely the case for Cantemist participants. See Figure 2 which
compares the participants’ answers to the question “which datasets or corpus did you use?” in Cantemist
and CodiEsp [
Corpus annotation. The manual annotation of the Cantemist corpus was performed by clinical experts following the Cantemist guidelines. These guidelines contain rules for annotating morphology neoplasms in Spanish oncology clinical cases; as well as for mapping these annotations to eCIE-O.
The annotation guidelines were created by clinical experts in three phases:
1. Firstly, a preliminary initial version of the guidelines was created after the clinical experts
revised neoplasm morphology annotations in the SPACCC corpus [
Similarly to what is required in the case of laboratory experiments, we should emphasize the need for scientific rigor in the creation of NLP tools that assist in medical research. In our case, a medical doctor was regularly consulted by annotators (themselves scientists with PhDs on cancer-related subjects) for the most dificult pathology expressions. This same doctor periodically checked a random selection of annotated clinical records and these annotations were compared and discussed with the annotators. To normalize a selection of very complex cases, MD specialists in pathology from one of the largest university hospitals in Spain (Hospital Clínic, Barcelona) were consulted.
Corpus format. The Gold Standard Cantemist corpus is distributed in Brat standof format [
For Cantemist-NER, every line of the ANN file contains the mention string of the annotation, its start character ofset, and its end character ofset, which uniquely locate the mention in the text document. For Cantemist-Norm, the eCIE-O codes are included as comments in the annotations. See Figure 3 and 4 for examples of Cantemist-NER and Cantemist-Norm ANN files, respectively.
Finally, the Gold Standard Cantemist corpus was also distributed in the same format as previously
used for the CodiEsp dataset [
Corpus statistics. The Cantemist corpus contains 1,301 documents, with a total of 63,016 sentences and 1093,501 tokens. All the 1,301 documents are annotated, i.e. tumor morphology mentions are found in them. There are 16,030 of such mentions, and each of them was manually mapped to an eCIE-O code. There are 850 unique codes. See Table 2 for a complete corpus overview.
The Cantemist test set was released together with an additional collection of 4,932 clinical case
documents that belong to diverse medical disciplines. We call these 4,932 documents as the background set.
Participants have generated automatic mention predictions for the test and also the background set,
although they were only evaluated on the test set predictions. In that way, we examined whether
systems were able to scale to larger data collections and prevented from manual annotation correction.
Additionally, the code predictions for this background set constitute the Cantemist Silver Standard
corpus, similar to the CALBC initiative [
Cantemist contained three independent subtasks: Cantemist-NER, Cantemist-Norm, and CantemistCoding. Participants could choose whether to submit results for one, two, or all subtasks. Participants could submit up to 5 runs for each subtask.
Cantemist has received considerable attention from the community. Indeed, 66 teams registered for this task, and 25 of them submitted predictions (one did not qualify due to a format error). From the 25 teams, 23 participated in the Cantemist-NER subtask, 10 in the Cantemist-Norm subtask, and 9 in the Cantemist-Coding subtask. Since five runs were allowed per subtask, the total number of systems participating in the shared task is considerably higher. Indeed, we received 62 prediction runs for Cantemist-NER, 30 for Cantemist-Norm, and 29 for Cantemist-Coding. In total, Cantemist shared task lead to the creation of 131 systems.
We should emphasize that 20% of the teams come from industry rather than academia. Additionally, as Table 4 shows, participants belonged to institutions (industry or academia) from a number of diferent countries including Spain, China, India, USA or Argentina.
• Cantemist-NER. HITSZ-ICRC team has obtained the highest f1-score, 0.87. Their system is highly balanced: the precision has been 0.871 and the recall 0.868. It is almost equivalent to the f1-score obtained by the Vicomtech team, 0.869. • Cantemist-Norm. Again, the HITSZ-ICRC team has obtained the highest f1-score, 0.825. Their precision has been 0.824 and their recall 0.826. And again, Vicomtech has obtained a really close f1-score, 0.821. • Cantemist-Coding. In this subtask, both Vicomtech and ICB-UMA have obtained equivalent scores for the main metric (MAP), 0.847. The Vicomtech team has developed a system with balanced precision (0.875), recall (0.836), and f1-score (0.855). Diferently, the ICB-UMA team has maximized the MAP metric, since their system has a really low precision (0.007) and really high recall (0.928).
Missed annotations are more complex. The annotations that automatic systems fail to predict seem to be the more complex. We have extracted the annotations systematically missed by the top 5 participants, according to the f1-score: HITSZ-ICRC, Vicomtech, SINAI, NLNDE, and NCU-IISR. We have compared these annotations with the complete set of test annotations.
The missed annotations are longer. The median number of characters is 26 for the dificult
annotations, while the median number of characters for all test set annotations is 14. In addition, there
is a higher percentage of abbreviations from the Spanish Medical Abbreviation DataBase [
Missed codes are more specific and less frequent. The codes that automatic systems do not assign properly seem to be more specific. In the subset of missed annotations, 8% of the codes contain an “H”. This percentage is as low as 2% in the entire test set. Additionally, 13.2% of the missed annotations include the sixth diferentiation digit in their code (the sixth digit in eCIE-O indicates the tumor diferentiation). In contrast, this percentage is 5.6% in the entire test set. Besides, missed test codes are less frequent in the training and development sets. The median of appearances of the missed codes in the training and development set is 1, whereas for the test set codes is 3. Finally, 20.8% of the missed annotations have the metastasis code (8000/6), while this code accounts for 34.6% of the complete test set.
Most successful teams employ deep learning for all subtasks (for NER, normalization, and clinical
coding). Indeed, there are two types of architectures that most participant repeat:
• A transformer-based language model and a classification layer. For example, the DTIMAI [
The preeminence of deep learning against other approaches is particularly seen in the performance drop of teams that employed it in Cantemist-NER but chose other approaches in Cantemist-Norm (Figure 6, Table 5).
From the teams using deep learning, Recurrent Neural Networks, transformers (which are the core
of the latest language models) and CRFs are the most utilized technologies. And within the language
models, participants have chosen to use, mostly, BERT [
P
To the authors’ knowledge, Cantemist is the first NER shared task in Spanish focusing on cancer, specifically on tumor morphology. Notably, we use oncology clinical case reports, since their language and format are closer to clinical narrative texts from EHRs, but with no issues due to patient data privacy. Finally, in addition to NER, we introduce the normalization of tumor morphology concept mentions to a standard normative terminology, eCIE-O (the Spanish equivalent of the International Classification of Diseases for Oncology), which is globally used by pathologists for classification and statistical reporting purposes.
The Cantemist corpus is the first collection of clinical case reports in Spanish with annotated tumor morphology mapped to eCIE-O. We believe that the Cantemist Silver Standard may serve to further extend this corpus. Both corpora are publicly available at the Zenodo Medical NLP community 4. We plan to also release a Cantemist post-workshop Gold Strandard collection to further promote research and development of new tools beyond the shared task period.
Medical language is complex and varies considerably among diferent medical disciplines. Available corpora specific to a particular medical speciality or domain are essential for fine-tuning novel language models. We strongly encourage the community to replicate the Cantemist initiative for other medical specialties, such as cardiology or radiology. Notably, the current Cantemist corpus is limited to 1,301 clinical case reports with 16,030 annotations. As discussed in the Results section, codes that automatic systems fail to predict are usually the less frequent. Larger corpora will result in better NLP tools, and we are currently adding 600 case reports to our corpus. Finally, published clinical cases share many similarities with hospital records. However, we expect to be able to access real health records in the immediate future, since they would produce much better suited tools for application in real-world oncology reports.
Cantemist, a shared task on Spanish NLP, is included in the conference of the Spanish Society of Natural Language Processing (SEPLN). The clinical texts and the terminology are written in Spanish. Interestingly, teams from all around the world registered for this task, with 22 teams from Spain (22), followed in number of participant teams by China (9), India (8), and the USA (5). Other represented countries were Russia, Germany, Belarus and Taiwan.
Of note, participants originate from various locations and heterogeneous backgrounds (Figure 8). When asked about their team members’ expertise, a high number of participants worked in artificial intelligence, machine learning and data mining. Others self-identified with NLP, information retrieval, and text mining fields of knowledge. Also, a small proportion of participants selected medicine or biomedicine as their background. There are also answers pointing to medical informatics or bioinformatics. Finally, 21.2% of registered teams work in commercial organizations, while the remaining teams work in academic institutions.
Named Entity Recognition is an established NLP task. Indeed, 90.9% of the participant teams that answered the question reported having worked on another NER task before (Figure 9). Moreover, the same percentage reported needing less than four weeks to complete the task (Figure 9). Technologies such as Recurrent Neural Networks and, more recently, transformer-based language models have taken over NER tasks and have become state of the art, which might explain the remarkable equality among top-scoring teams of Cantemist-NER: there are five teams with f1-scores between 0.85 and 0.871.
In a situation such as the one described in the previous task (a mature NLP task with an established state-of-the-art technology), we expect the emergence of stakeholders interested in transferring the knowledge from academia to industry. Indeed, more than 20% of Cantemist registered participants are from the industry (Figure 10). Additionally, most respondents to the participants’ survey think that commercial firms and healthcare professionals could benefit from these tasks (Figure 11), and only 27% of them answered “no” to the question, “Would you be interested in support for turning your system into a software product/startup?” (Figure 10). Named Entity Recognition is a building block for downstream NLP tasks, and the resources originating from the Cantemist shared task have the potential to impact text mining applications for oncology.
The Cantemist shared task has been a participation success, considering the amount of
participants and the interest of international teams in a Spanish text mining task for a Spanish conference.
In addition, the Cantemist corpus is a pioneer work on the distribution of domain-specific medical
NLP corpus, and in languages other than English. We would like to encourage similar initiatives in
other medical specialties and other languages, including Basque, Catalan and Galician, which are of
interest for the Spanish National Plan for the Advancement of Language Technology [
We propose that future evaluation eforts for oncology text mining in Spanish should also take into
account the annotation of clinically important information such as negation or temporal expressions,
as well as examining scalability and robustness related aspects [
We acknowledge the Encargo of Plan TL (SEAD) to BSC for funding, and the scientific committee for their valuable comments, their guidance, and their help with the review of the proceedings. Besides, we would also like to thank the organization of IberLEF. Finally, we would particularly like to thank the team of Bitac, Gloria González, and Toni Mas, who worked with us to create the dataset and are still working to make it grow. We do acknowledge positive support from Jose Antonio Lopez-Martin (Hospital 12 de Octubre) and the Sociedad Española de Oncología Médica (SEOM).
IBS_Software IBS_Software IBS_Software IBS_Software lasigeBioTM Tong Wang Tong Wang DTIMAI DTIMAI episource XIntao UAB UAB UAB BigByte PaccanaroLab fernandrez fernandrez fernandrez fernandrez Run-2 Run-4 Run-5 Run-5-unoficial 1-run 1-systemDL 2-systemDL 1-systemBERT1 2-systemBERT2_Casing process_1 1-run 1-systemCasedBETO 2-systemRegex 3-systemMBERT 1-run 1-nn_merge 1-bilstm_merged 2-bilstm_0 3-bilstm_dropout 4-bilstm_0_b P
R
F1 HITSZ-ICRC HITSZ-ICRC HITSZ-ICRC HITSZ-ICRC HITSZ-ICRC Vicomtech Vicomtech Vicomtech Vicomtech Vicomtech SINAI SINAI SINAI SINAI SINAI NLNDE NLNDE NLNDE NLNDE NLNDE mhjabreel
MAP F1 Vicomtech Vicomtech Vicomtech Vicomtech Vicomtech NLNDE NLNDE NLNDE NLNDE NLNDE mhjabreel Fadi Fadi Fadi Fadi lasigeBioTM lasigeBioTM lasigeBioTM lasigeBioTM lasigeBioTM episource Bigbyte ICB-UMA ICB-UMA ICB-UMA ICB-UMA ICB-UMA Kathrync Kathrync 1-BETO 2-SciBERT 3-TwoExperts 4-OneRoundEns. 5-TwoRoundEns. 1-systemBiLSTM 2-systemMeta 3-systemBiafine 4-systemBiafineD. 5-systemEnsemble coding_processed 1-BILSTM 2-BILSTM CRF CRF_BILSTM 1-X-Transformer. 2-X-Transformer. 3-X-Transformer. 4-X-Transformer. 5-X-Transformer. coding coding 1-Multi 2-Multi-Onco 3-Multi-Onco-M. 4-Scielo 5-Scielo-Onco 1-Cantemist-coding 2-Cantemist-coding