In a context of digitalization and modernization of healthcare, automatic analysis of clinical data plays a leading role in improving the quality of care. Since much of the information lies in an unstructured form within clinical notes, it is necessary to make use of modern Natural Language Processing techniques to extract and build structured knowledge from the data. However, clinical texts pose unique challenges due to the extensive usage of i) acronyms, ii) non-standard medical jargons and iii) typos over technical terms. In this paper, we present a prototype spell-checker specifically designed for medical texts written in Italian.
traces regarding activities, patients, as well as medical notes. In particular, it is relevant to consider both structured and unstructured data, i.e., clinical and textual.
Since a portion of the healthcare data is in textual form, it is increasingly of interest to provide a Natural Language Processing (NLP) pipeline to extract and analyse useful information. Unstructured texts are often noisy, with typing errors and extensive use of non-standard acronyms and medical jargon, which are usually accompanied by a less rigorous structure of the document itself. To overcome these problems, researchers must begin to address issues of spelling correction, acronyms disambiguation and entity normalization. However, in languages other than English it is very dificult to find advanced models, data or other resources.
In this paper we deal with the spelling correction task (i.e. the correction of typos) of notes
written by physicians, so as to provide the most correct text for the sophisticated Information
Extraction (IE) techniques that generally follow the initial data cleaning phase. Indeed, this work
is part of a larger project [
Specifically, with this work we introduce a prototype of a spell-checker designed to work on Italian clinical texts. Although it is still a work-in-progress, to the best of our knowledge it represents currently the first and unique study specifically designed to correct medical texts in Italian.
The automatic spelling correction task is the first step to be taken in order to analyse clinical
texts, representing one of the most important open problems in Natural Language Processing.
The correction process can be divided, according to [
According to the Claude Shannon’s noisy-channel framework, the problem is broken down
into error modelling (aka channel model) and language modelling. The first one measure “fitness”
of the correction candidate with respect to the corrected string, meanwhile the second one
expresses the probability of correct word occurrence, considering - possibly - also the context.
Most of the works, such as [
simplest n-grams [
Recently, it has also been shown how good results can be obtained through purely neural
approaches. For example, a state-of-the-art corrector for Italian ([
or, by applying Bayes’ rule: The proposed spell checker prototype is based on Shannon’s noisy channel framework described by the equation: ˆ = arg max (|)
∈ ˆ = arg max (|) ()
∈ where ˆ indicates the best correction for the misspelled word ; word is selected from a given and finite vocabulary .
Since the prior probability () carries too little information, we replace it with a Language Model (LM) that involves context (|− 1). Finally, we weight the LM with a parameter and, for numerically stability reasons, we move on the logarithmic space. The equation is therefore: ˆ = arg max (|) + (|− 1)
∈
3.1. Data
Unlike English, languages such as Italian are characterized by limited publicly available resources.
Furthermore, considering the specificity of clinical language, the availability of medical texts
is even more dificult. Medical terms such as surgical procedures, drugs, anatomical parts etc.
constitute a very specific vocabulary that difers from what we can normally find in Italian public
corpora. It is therefore necessary to find a collection of suitable medical/scientific documents
and build new Language Models on them. Following the suggestions of [
For computational reasons, and to avoid too rare expressions (a symptom of a possible error) we only consider the elements that occur more than 8 times for the terms and 48 for the n-grams. The resulting vocabulary has a total of 787,940 unique words.
Source Wikipedia News Ministry of Health Dica33 Notes TOTAL
Sentences 1,096,672 247,872 39,838 1,059,063 58,160 2,501,605
Words 25,605,524 5,878,905 1,151,371 37,333,844 962,408 70,932,052
As regards clinical documents, we relied on a sample of 200 anamnesis notes that were provided by the hospital. The texts, once anonymized, were manually corrected by physicians, thus constituting the gold-standard. Acronyms and abbreviations are excluded from the correction process. Out of the total 9374 words, 269 (2.87%) constitute typos, of which 28 (0.30%) are real-word errors. Errors attributable to purely medical context are 107 (40%), of these 25 are names of drugs/active substances. 88% of typos have Damerau–Levenshtein distance of 1 from the correction (e.g. mammamria → mammaria); only 10% have DLD 2 (e.g. ematochici → ematochimici) and a less than 1% for higher distances (e.g. idrixixizima → idroxizina). 3.2. Model For simplicity, and considering the rarity with which they occur, we have chosen to discard realword errors, thus focusing on the remaining 88% of typos. These errors are easily identifiable by searching for terms that do not match the vocabulary. Potential acronyms and abbreviations are excluded; in this regard we have built, in collaboration with domain names, a blacklist of terms not to be corrected.
Once a potential error has been found, a list of candidate corrections is generated, considering the words “similar" to the original one as candidates. Also in this case, the Damerau–Levenshtein distance is used as similarity metric between strings. Since the generation of candidates is a very demanding task in computational terms, we rely on the optimized tools SymSpell10, which 9https://webhose.io/free-datasets/italian-news-articles/ – Crawled Oct 2015 10https://github.com/wolfgarbe/SymSpell can operate under the “CLOSEST" regime (i.e. finding the first word at the shortest distance) or “ALL" (all words with maximum distance ).
Always with reference to Equation 3, we list below the solutions that have been implemented and tested.
The simplest way to implement the channel model is to assign a probability ( ) for the event = (i.e. the word found is not an error even if it does not appear in the vocabulary) and use the Damerau–Levenshtein distance to evaluate other cases.
(|) = {︃
if = − ((, )) otherwise
In 1991, [
⎪⎨ 1− ⎪ ⎩ |()| if = if ∈ () otherwise
Finally, we tested a slightly more sophisticated variant, proposed by [
⎪ ⎩ (|) = Π (,) (,) if = if ∈ () otherwise
We have chosen to approach the language model in two ways: through n-grams, either
monodirectional (− 2, − 1, ) or bidirectional (− 1, , +1), or through contextualized word
embeddings (Masked Language Model). The n-grams are calculated according to the “stupid
backof” scheme [
11Transformations allowed by Damerau–Levenshtein are: deletion, insertion, substitution and transposition Model
We tested the diferent models on the gold standard described above, comparing the results with the state of the art as shown in Table 2. Unfortunately the absence of clinical corpus, as well as publicly available models, makes it dificult, if not impossible, the comparison between spell-checkers in the hospital setting. For this reason we have relied on commonly used general purpose tools such as: Hunspell15, LanguageTool16, Google Docs17 and Microsoft Ofice 18. To standardize the results we have chosen to correct all the possible typos suggested by the software, replacing them with the first suggestion. It is also necessary to consider the specificity of the medical vocabulary, which is usually absent in generic tools. For this reason we have excluded all the terms that belong to the vocabulary in our possession and the blacklist defined with the experts. In the Table 2 it is possible to distinguish the two cases (with or without the vocabulary extension) by means of the label “+voc".
As regards the models developed by us, the optimal configuration of the parameters is shown in Table 3. Any change to them does not bring any benefit, as described below. Among the proposed solutions, the best model is the combination of uniform distribution, as error model, and n-grams for the language model.
The advantage of n-grams, over word embeddings, may also be due to the generic nature of the embeddings used; unfortunately, however, the few data available did not allow us to train a 15http://hunspell.github.io/ 16https://languagetool.org/it 17https://docs.google.com 18https://www.ofice.com Parameter alpha lambda epsilon n-grams size
Value new model from scratch. By excluding Wikipedia and newspapers from the initial corpus, thus focusing on purely medical texts, we can obtain 1 157 061 sentences, useful – after subdivision into training set (96%) and validation set (4%) – to continue the training of the ELECTRA model. However, the efort is vain, failing to difer much from the performance of ELECTRA without ifne-tuning; for this reason the results are omitted from the table. The ELECTRA model is however better than other architectures such as RoBERTa and its multilingual variant. A possible advantage in the use of the neural model, in addition to a slight improvement in Precision, consists in the significant reduction of processing times (15 min for the n-grams case vs 25 sec for ELECTRA).
Focusing instead on n-grams, a reduction in their size (passing from 5 to 3) slightly decrease
the performances (F1 from 66.31 to 66.19). Similarly, the monodirecional/bidirectional choice
is almost irrelevant in terms of score. On the contrary, the application of standardization
techniques (stemming and numbers masking) considerably worsens the scores, obtaining F1
62.97. The result is not surprising, as a similar phenomenon has already been observed by [
With regard to the error model, on the other hand, the use of the confusion matrix is counterproductive, lowering the F1 score by 8 points on average. While using distance alone as a probability measure has no benefit over the uniform distribution.
Most of the gaps highlighted in the previous section are probably attributable to the scarcity of
data. The examples collected by [
The texts that characterize the medical domain are also particularly interesting. Just think, for example, that the addition of medical notes (which weighs just over 1%) made it possible to improve performance by 4 points, passing from F1 64.37 to F1 66.31, although with a diferent nature of texts. Indeed, all the texts used in our corpus present linguistic characteristics that are very diferent from those that appear in clinical documents. The Wikipedia entries, for example, provide encyclopedic information, as well as the pages of medical disclosure and personal essays. The search for texts closer to clinical reality will be a fundamental objective that we will pursue in future works.
We also think that - in addition to a mere increase in the size of the datasets for statistical learning purposes - also the integration of syntactic parsing and POS tagging can improve the results. This is especially true in a low resources languages, like Italian, where machine learning is severely limited. For this reason we are conducting a study aimed at evaluating the reliability of these techniques on medical texts.
Finally, the abbreviations (standard and non-standard) usually used in texts remain to be addressed. In this regard it is dificult to think of a pipeline that orders the 3 tasks to be performed: POS tagging, acronyms disambiguation and spelling correction. More likely, the 3 tasks will be carried out in parallel, as one can help the other. We will also evaluate this possibility in future works.
Not having reached the state of the art, represented by Google’s spell checker, we believe there is still room for improvement. Moreover, the task is of fundamental importance in order to continue with the analysis of the texts and, ultimately, for clinical decision support.
This research has been partially carried out within the “Circular Health for Industry” project funded by “Compagnia di San Paolo” under the call “IA, uomo e società”.