This paper tackles the epidemiological event extraction task applied to digitized documents. Event extraction is an information extraction task that focuses on identifying event mentions from textual data. In the context of event-based health surveillance from digitized documents, several key issues remain challenging in spite of great efforts. First, image documents are indexed through their digitized version and thus, they may contain numerous errors, e.g. misspellings. Second, it is important to address international news, which would imply the inclusion of multilingual data. To clarify these important aspects of how to extract epidemic-related events, it remains necessary to maximize the use of digitized data. In this paper, we investigate the impact of working with digitized multilingual documents with dierent levels of synthetic noise over the performance of an event extraction system. This type of analysis, to our knowledge, has not been alleviated in previous research.
The surveillance of epidemic outbreaks has been an ongoing challenge globally and it has been a key component of public health strategy to contain diseases spreading. While digital documents have been the standard format in the modern days, many archives and libraries still keep printed historical documents and records. Historians and geographers have a growing interest in these documents as they still hold many crucial information and events in the past to analyze, noticeably in health and related to epidemics events in an international context.
Copyright ' 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). ? This work has been supported by the European Union’s Horizon 2020 research and innovation program under grants 770299 (NewsEye) and 825153 (Embeddia).
Event extraction (EE) is an important information extraction (IE) task that focuses on identifying event mentions from text and extracting information relevant to them. Typically, this entails predicting event triggers, the occurrence of events with specic types, and extracting arguments associated with an event. In the context of event-based health surveillance from digitized documents, for extracting relevant events, even though the historical documents are in physical form, few of them have been converted into digital form for further storage as records in a database. However, due to the digitization process, several issues can arise, most commonly in the case when the original document is distorted, whether through deterioration due to aging or was damaged in the storing process, which will aect the converted content. Moreover, errors from the digitization process could also be a factor that causes adulteration of the converted documents e.g. word variations or mispellings.
In this article, we propose to experiment with an approach to event extraction
with the ability of handling not only multilingual data, but also large amounts of
data without relying on any additional natural language processing (NLP) tools.
The architecture is based on the DAnIEL system [
The paper is organized as follows: Section 2 briey overviews the related works on epidemiological event extraction. Section 3 introduces the DAnIEL system and its characteristics and in Section 4 the dataset built specically for the DAnIEL system is presented in detail. The Section 5 describes the experiments and an extrinsic evaluation of the results. We conclude and propose possible suggestions for future research in Section 6. 2
Specic to epidemiological event extraction, there exist a few of empirical works
targeted to extract events related to disease outbreaks. For instance, similar to
the chosen system for this paper, DAnIEL, there are two other systems,
BIOCASTER [
To our knowledge, there are no works related to the analysis of the impact
of documents digitization for event extraction in the epidemiological domain.
In return, few studies have been devoted to other information extraction tasks
i.e. the extraction of named entities from digitized historical data [
DAnIEL [
By avoiding grammar analysis and the usage of other NLP toolkits (e.g
partof-speech tagger, dependency parser) and by focusing on the general structure of
journalistic writing style [
DAnIEL uses a minimal knowledge base for matching between the extracted possible disease names or locations and the knowledge base entries. Its central processing chain includes four phases. In the Article segmentation phase, the system rst divides the document into salient positions: title, header, body and footer. In Pattern extraction, for detecting events, the system looks for repeated substrings at the salient zones aforementioned. In Pattern ltering , the substrings that satisfy this condition will be matched to a list of disease/location names that was constructed by crawling from Wikipedia.
For the string matching between the extracted character sequences and knowledge base entries, the system is parameterized with a ratio. For instance, a small ratio value could oer a perfect recall but with high noise (many irrelevant entries are selected). For a maximum value ( 1:0), the system will match the exact extracted substrings which could be detrimental to the morphologically rich languages (e.g. Greek, Russian). There are cases where the canonical disease name cannot be found in the text, as in the case of aforementioned languages, but grammatical cases of nouns. For example, in Russian, Prostuda (prostuda) means cold, and since this disease name cannot be found in the text article, we used the instrumental case in Russian that can generally be distinguished by the - om (-om) sux for most masculine and neuter nouns, the - o /- o (-oju/-oj) sux for most feminine nouns. A ratio of less than 1:0 will consider the instrumental case for singular prostudo as a true positive.
Finally, the Detection of disease location pairs (in some cases, the number of victims also) produces the end result with one or more events that are described by pairs of disease-location. 4
In this section, we present the dataset that was created for the DAnIEL
system [
A DAnIEL event is dened at document-level, meaning that an article is considered as relevant if it is annotated with a disease location pairs (and rarely, the number of victims). An example is presented in Figure 1, where the event is a listeria outbreak in USA and the number of victims is unknown.
Thus, in this dataset the event extraction task is dened as identifying articles that contain an event and the extraction of the disease name and location, i.e. the words or compound words that evoke the event. Since the events are epidemic outbreaks, there is no pre-set list of types and subtypes of events, and thus the task of event extraction is simplied to detecting whether an article contains an epidemiological event or not.
Common to event extraction, the dataset is characterized by imbalance. In this case, only around 10% of these documents are relevant to epidemic events, which is very sparse. The number of documents in each language is rather bal] , "comment " : " " , " date " : "2012 01 12" , " l a n g u a g e " : " en " , " document_path " : "doc_en /20120112_www. cnn .
com_48eddc7c17447b70075c26a1a3b168243edcbfb28f0185 " , " u r l " : " http : / /www. cnn . com/2012/01/11/ h e a l t h / l i s t e r i a o u t b r e a k / i n d e x . html " anced, except for French, having about ve times more documents compared to the rest of the languages. More statistics on the corpus can be found in Table 1.
The DAnIEL dataset is annotated at document-level, which dierentiates itself from other datasets used in research for the event extraction task. A document is either reporting an event (disease-place pair, and sometimes the number of victims) or not. We will elaborate the evaluation framework in Section 5. The focal point of this set of experiments is to observe how the level of noise stemming from the digitization process impacts the performance of the models. However, there is no adequate historical document dataset provided with manually curated event annotation that could directly be used to measure the performance of the models over deteriorated historical documents. Thus, the noise and degradation levels have to be articially generated into clean documents, so as to measure the impact of ATR over event detection using DAnIEL. We shall thus use readily available data sets over contemporary and digitally-born datasets, which are free of any ATR-induced noise.
In order to create such an appropriate dataset, the raw text from the DAnIEL dataset was extracted and converted into clean images 3. The rationale is to simulate what can be found in deteriorated documents due to time eect, poor printing materials or inaccurate scanning processes, which are common conditions in historical newspapers. We used four types of noise: Character Degradation adds small ink dots on characters to emulate the age eect on articles, Phantom Character appears when characters erode due to excessive use of documents, Bleed Through appears in double-paged document image scans where the content of the back side appears in the front side as interference, and Blur is a common degradation eect encountered during a typical digitization process. After contaminating the corpus, all the text was extracted from noisy images 4, for initial clean images (without any adulteration) and the noisy synthetic ones. An example with the degradation levels is illustrated in Figure 2. The noise levels were empirically chosen with a considerable level of diculty 5.
The experiments were conducted in the following manner: for each noise type, the dierent intensity is generated to see its relation to the performance of the model. Character error rate (CER) and word error rate (WER) were calculated for each noise level, that can align long noisy text even with additional or missing text with the ground truth, thus enables it to calculate the error rate of OCR process. The experiments are performed under conditions of varying word error rate (WER) and character error rate (CER): original text, OCR from highquality text images, and OCR on synthetically degraded text images. 5.1
For the evaluation of the performance of the event detection task, we use the standard metrics: Precision (P), Recall (R), and F-measure (F1). For measuring the document distortion due to the OCR process, we also report the standard metrics: character error rate (CER) and word error rate (WER). We perform two types of evaluations, both at the document level (included in the DAnIEL system):
Event identication: a document represents an event if both triggers were found, regardless of their types; Event classication: a document represents an event if the triggers are correctly found and match exactly with the groundtruth ones.
4 The Tesseract optical character recognition (OCR) Engine v4.0 https://github.
com/tesseract-ocr/tesseract [
For event identication on clean textual data, one can notice from the Table 2, that usually DAnIEL favors recall instead of precision and tends to suer from an imbalance between precision and recall, which may be due to the high imbalance of the data. It is aso not surprising that the DAnIEL system the highest performance values for event identication for Chinese and Greek, since for Chinese, there are few relevant documents comparing with the other languages (16 documents that report an event), and for Greek, there are 26 of them.
We also can note the large dierence between the two chosen ratios. More exactly, an increase in this value comes in the detriment of the languages that are not only morphologically rich, but also in the case where the exact name of the disease is not located in the text.
In the case of event classication, we observe from Table 3, that DAnIEL is balanced regarding the precision and recall metrics, being able to have higher F1 on the under-represented languages (Chinese, Russian, and Greek). We also notice that, in all the cases, DAnIEL does not detect the number of victims. We assume that this is due to the fact that many of the annotated numbers cannot be found in the text, e.g. 10000 cannot be detected since the original text has the 10; 000 form, or it is spelled ten thousands. Generally, for the detection of locations, we recall that DAnIEL is capable to detect locations due to the usage of external resources and article metadata.
For the experiments on noisy data, we will use a ratio value of 0:8, since the maximum value for the ratio creates results prone to suer from word variations or misspelings of words (which is a direct consequence of the digitization process). 5.3
based languages (e.g. Chinese), CER is commonly used instead of WER as the
measure for OCR, and, thus, we report only the CER [
We note also that, regarding the Chinese documents, the high values for CER, for every type of noise, might be caused by the existence of the enormous number of characters in the alphabet that, by adding such an eect as Character Degradation can change drastically the recognition of a character (and in Chinese, one single character can often be a word). Otherwise, while Character Degradation noise and Blur eect have more impact on the performance of DAnIEL than Phantom Character type since it did not generate enough distortion to the images. A similar case applies for the Bleed Through noise.
Regarding the experiments presented in Tables 5 and 6, we notice, rst of all, that the Character Degradation eect, Blur, and most of all, all the eects mixed together, have indeed an impact or eect over the performance of DAnIEL, but with little variability. Meanwhile, Phantom Degradation and Bleed through had very little to no impact on the quality of detection with DAnIEL.
The cause of the decrease in performance of DAnIEL is that, in order to detect events, the system looks for repeated substrings at salient zones. In the case of many incorrectly recognised words during the OCR process, there may be no repetition anymore, implying that the event will not be detected. However, since DAnIEL only needs two occurrences of its clues (substring of a disease name and substring of a location), it is assumed to be robust to the loss of many repetitions, as long as two repetitions remain in salient zones.
Regarding all the aforementioned results for the DAnIEL system, computing the number of aected event words (disease, location, number of cases), we also notice that a very small number of them have been modied by the OCR process, only 1:98% for all the languages together, for all the eects mixed together, close to the 1:63% that were aected by the OCR on clean data. This is due to the imbalance in the DAnIEL dataset: only 10:14% of a total of 4; 822 documents contain events. It brings us to the conclusion that the event extraction task is not considerably impacted by the degradation of the image documents.
One interesting observation is that the precision or the recall can increase,
resulting in a higher F1, despite the higher noise eect applied. One possible
explanation for this phenomenon is that with a greater level of noise, some false
positives disappear. Documents, which were previously classied wrongly due to
being too ambiguous to the system (for instance documents relating vaccination
campaigns are usually tagged as non-relevant in the ground truth dataset), were
given much more distinction thanks to the noise, thus making them look less like
relevant samples to the system. More formally: let document X be a false positive
in its raw format (Xraw). Let XNoisy be its noisy version. If the paragraph that
triggered both system’s misclassications disappeared in Xnoisy , there are good
chances that it will be classied as non-relevant. In that case, Xraw is a false
positive but Xnoisy is a true negative. That may seem counter-intuitive but noise
can improve classication results, see for instance [