This article describes the participation of ERTIM (INALCO) to the French NER task of CLEF HIPE 2020 lab with the mXS system, a combination of pattern mining and machine learning, implemented in 2010-2013. Due to multiple reasons, almost no upgrades or improvements were achieved since then, only a minimal linking module and some lexical entries were added. No training and almost no adaptation were implemented for this lab. Results on historical data show severe degradations, in particular concerning the recognition of organisations.
Named Entity Recognition for French language was implemented by several
resource building, evaluation campaigns or shared tasks. Many resources have
already been described[
Our system, mXS3, was developed during the QUAERO[
Since its implementation, mXS was minimally updated. Recently, the
TALAD4 research project (NLP and Discourse Analysis) provided the opportunity
to use mXS on political data (radio interviews). It is in the context of this project
that we decided to participate in the CLEF HIPE 2020 lab [
Our system has already been exhaustively described [
The same preprocessing are used during training and inference (prediction) steps. For French, the tokenisation and POS tagging are done using TreeTagger [9], some postprocessing are implemented (e.g. proper nouns, numbers, verbs) to provide relevant features for named entity classi cation. The second step consists in looking up each token in a gazetteer which uses several large lexicons (in total more than 1M entries) containing names for French (mainly common and proper nouns, including multiword expressions).
Both preprocessing are considered as an enrichment process: raw textual data is transformed into sequences of disjunctive itemsets. 2.2
One of the main di erences of mXS compared to other existing software is that named entity recognition is considered as a segmentation task rather than a classi cation task. In other words, the system tries to nd positions where entities begin and end, while most systems consider this task as assigning a class (named entity category) to each element in the sequence.
Based on this assumption, during the training step the software runs a
pattern mining algorithm [
Once patterns are extracted, they are considered as features for a machine
learning model implemented using the SciKit [
The recognition is a decoding step where individual segmentation probabilities are constrained (started entities must end within the considered sentence, nesting can be allowed and follow guidelines, etc.). The resulting annotation corresponds to the segmentation which respects the constraints and maximises the joint probabilities over the sequence. 2.5
We did implement a simple entity linker, executed after recognition, and based on Wikipedia. As a rst version, the system only links PERS entities. The main steps are: 1. The name is checked against a local database (very limited for now, it does only contain names and Wikipedia links of a small set of entities), 2. If no match, the wikiapi5 python module, connected to the French Wikipedia, is searched for entries based on the entity label (as far as we understood, this API executes the search both in Wikipedia titles and pages), 3. Retrieved results are ltered with an approximate string matching based on the title of the Wikipedia page, 4. A similarity (cosine) between the BoW of processed document and the
Wikipedia page content is computed, 5. The page which has highest similarity is selected. 3
As previously mentioned, due to multiple reasons including the COVID19, we did not have much time to prepare the system for the CLEF HIPE 2020 lab. Actually we did not use the development and training data at all. Fortunately, this shared task was conducted using annotation guidelines mainly inspired by the QUAERO campaign, which facilitated our work.
Input data is simply converted into raw text (with some minor French tokenisation adaptations) and passed to mXS. Output of our system as tagged texts are converted to the expected column format. A simple script checks and lters out unwanted named entities. For linking, another process runs our linker as previously described, and we used the wikimapper6 python module to obtain the corresponding WikiData key. 4
We report our system o cial results [
Clearly, our main problem is recall, as expected. Table 2 reports detailed results per type. We note that PERS, TIME and LOC types have a limited degradation, while ORG su er from a very severe loss. Given the fact that the lab focused on historical data, this reveals that locations and persons are quite robust over time periods, as depicted below in Figure 1, what is not the case for organisations.
We conducted some additional experiments reported in Table 3, with the coarse strict metric (in Table 1, COARSE-LIT-micro-strict) computed using the provided scorer7. First, we corrected some scripts which led to minimal improvements (up). In a second step, we added to mXS's lexicon dev and train data entries with their types (up,lex), but this did almost not change scores. Finally, we re-trained the machine learning component of the system (up,lex,train) with only CLEF HIPE 2020 lab data or merged with the data the system was initially trained on (QUAERO). Those last experiments more signi cantly improved the system performance, especially for precision, but recall still remains an issue.
From those results, we rst notice to what extent a system's performance is degraded when changing the dataset, both for the domain (historical) and annotation scheme (there are notable di erences between QUAERO and HIPE). Despite our e orts, the system does not reach the organizers baseline (CRF). Secondly, even with a modular system where lexicon is an external resource, it is not su cient to update only this component, the machine learning model parameters have to be recomputed. Finally, from the fact that scores are better
1 F
0:5 P 0:5 0 1800 1 0 1 R 0:5 0 1800 when training with CLEF HIPE 2020 lab data only we make the assumption that QUAERO and HIPE datasets are somehow heterogeneous.
Figure 1 depicts F1, precision and recall of entity types over time periods (20 years, each point is plotted at the beginning of the period). For person and location types, results are quite stable. This is not the case for other types, especially for the organisation type which uctuates a lot. The global score (ALL) is quite stable and increases smoothly, at the exception of the 1940-1960 period where there is an important loss. 5
This paper describes the results obtained by our old-fashioned mXS system for the CLEF HIPE 2020 lab (named entity classi cation and linking). Diverse reasons prevented us from updating and training the system for this shared task. Performance is severely degraded compared to what was obtained during the QUAERO evaluation campaign (2012). Our analysis showed that organisation entities have higher performance losses performance decrease compared to person and location entities. Our participation provides and estimation of how much a system trained on recent radio transcripts decreases its quality when faced with historical data. 8. Rosset, Sophie, Grouin, Cyril, Zweigenbaum, Pierre: Entites nommees structurees : guide d'annotation Quaero. Tech. Rep. 2011-04, LIMSI-CNRS (2011) 9. Schmid, H.: Probabilistic Part-Of-Speech Tagging using Decision Trees. In: New methods in language processing. p. 154 (2013)