This note describes IRISA's system for the task of named entity processing on historical newspapers in French. Following a standard entity detection and linking pipeline, our system implements three steps to solve the named entity linking task. Named Entity Recognition (NER) is rst performed to identify the entity mentions in a document based on a Conditional Random Fields classi er. Candidate entities from Wikidata are then generated for each mention found, using simple search. Finally, every mention is linked to one of its candidate entities in a so-called linking step leveraging various string metrics and the semantic structure of Wikidata to improve on the linking decisions.
Entity linking is a core task in textual document processing, which consists in identifying the entities of a knowledge base (KB) that are mentioned in a text. For instance, approaches from the literature implement three stages to solve mention ambiguity in texts. The rst stage consists in the detection of named entities within the text and is known as named entities recognition (NER). To further link the mention found in the text, candidate entities are generated for each mention detected in the rst stage. Finally, every mention is linked to one of its candidate entities in a so-called linking step. This last step can be performed independently for each individual mention, or collectively for all mentions at once. In the rst case, every mention in a text is assumed to be independent from other mentions and is linked to a candidate entity on sole basis of some similarity between the mention and the candidate entities, so-called local scores. By contrast, for collective entity linking, entity mentions and the corresponding entities are not assumed independent one from another but somehow semantically related within a (coherent) document, i.e., mention-to-entity linking decisions are interdependent. In this case, the local mention-entity scores are complemented with global scores re ecting to which extent the candidate entities chosen for the mentions under consideration are related in the KB, according to a so-called entity relatedness measure. The last two stages of the pipeline are also known as named entity linking (NEL).
In the context of the shared task CLEF HIPE 2020 |Identifying Historical
People, Places and other Entities|which is a named entity processing on historical
newspapers in French, German and English [
Our system for CLEF HIPE 2020 follows a standard pipeline for entity linking
and implements three separate stages:
1. We devise a NER stage on top of the baseline provided by CLEF HIPE 2020
organizers. This system used Conditional Random Fields (CRFs) to detect and
classify named entities. We added several features that we found e ective for
the task of NER.
2. The generation step consists in looking to Wikidata directly when searching
entities similar to a given mention. As a lookup in the heavy database (CLEF
HIPE 2020 Wikidata dump) is costly in time, we performed automatic searches
for the entity mentions using online Wikidata. Note that this search is based
on Wikidata indexing algorithm.
3. The linking step is to decide which candidate should be retained for each
mention within a document. We tried to link the mentions separately or collectively,
training a classi er to predict if a mention is related to one of its candidate
entities. The former is based solely on the similarity between a given mention and
its candidate entities. The latter which performs the linking collectively for all
the mentions at once, beside the previous similarity metrics, makes use of the
entity relatedness measure WSRM that we have proposed in [
Our source code, datasets and experimental results are made available online for reproducibility purposes5.
The note is organized as follows. We give the description of our method in Sec. 2. Then we group the experimental results in Sec. 3 before discussing the perspectives and conclude in Sec. 4. 2
This section gives the description of our system. We distinguish two independent
tasks for named entity processing, namely the NER and the NEL. Our solutions
for the NER and the NEL are described respectively in Sec. 2.1 and Sec. 2.2.
5 https://gitlab.inria.fr/celvaigh/hipe2020
2.1
The NER task aims at detecting the surface forms in a text that correspond to
named entities and at classifying those forms as a type (PER, LOC, ORG, TIME,
PROD). The NER system that we developed originally came from the NER
baseline provided by the organizing team of the evaluation campaign CLEF HIPE
2020. This system used Conditional Random Fields (CRFs) based on a Python
implementation [
NEL
In the NEL stage, we assume that the annotations are known for the mentions
(person, organization, location, etc.) for each document. Those annotations are
provided by the NER system described in Sec. 2.1 or by an oracle NER. For the
candidate generation stage we rely on a simple Wikidata web search. The
candidate selection stage accounts for the WSRM entity relatedness measure between
candidate entities within the document in an e cient manner, relying here on
Wikidata, the KB provided by CLEF HIPE 2020 for named entity processing
(see [
Local Scores The local scores depict the similarity between a mention and its candidate entities. If we assume the mentions to be independent in text, the linking problem can be formalized as e^= argmax (m;ei) (1)
ei
where ei is a candidate entity, m is an entity mention, and is the local score
function. We tried several metrics for . Beside the longest contiguous matching
sub-sequence, we tried a Levenshtein distance to handle the OCR noise, Wikipedia
popularity [
Collective Entity Linking In a collective NEL setup, the local score is complemented with a global score accounting for the intricate interrelationships that candidate entities of the di erent mentions may share. The latter is known as an entity relatedness measure and used to assess entity relationships in the KB, which will allow to estimate the interdependence of the mentions in the text. The CEL problem can be thus formalized as
0 n n
(e^1;:::;e^n) = argmax@X (mi;ei)+X
e1;:::;n
(2)
i=1 i=1j=1;j6=i
where n is the total number of mentions in a text and (ei;ej ) donates the
entity relatedness measure. In the collective linking version of our system, we used
the semantic entity relatedness measure WSRM [
e02E where E denotes the set of entities in the KB and jSj the cardinality of the set S.
Because the directions of the relations are somewhat arbitrary in KBs, depending on how the relation vocabulary was designed (think about the publishes and publishedBy symmetric RDF properties), we use a symmetric version of WSRM de ned as
1
(ei;ej ) = (WSRM(ei;ej )+WSRM(ej ;ei)) :
2
(4)
Using the NEL Output to Correct the NER Predictions We also exploited
the output of the NEL in order to enhance the NER results. First, we used the type
(obtained from Wikidata) of the entities retrieved by the NEL and forwarded it to
the NER stage, which can be updated accordingly. Then we leveraged WSRM [
Experimental validation was conducted on the CLEF HIPE 2020 French corpus to assess the quality of our system. The dataset is described in Sec. 3.1. Results for the NER are provided in Sec. 3.2, and in Sec. 3.3 for the NEL.
Due to the high number of results given by the CLEF HIPE 2020 scorer, we decided to focus only on a couple of them, that were given in the produced json le: NE-COARSE-LIT - ALL - strict - F1 micro and NE-COARSE-LIT - ALL ent type - F1 micro. 3.1
The evaluation corpus is composed of newspaper articles sampled among several Swiss, Luxembourgish and American historical newspapers on a diachronic basis. This corpus is digitised based on an OCR algorithm which hightails the historical context of the evaluation campaign. The time-span of the whole corpus goes from 1798 until 2018. We used only the French version of the corpus composed of a train, a validation and a test sets 6. 3.2
The NER classi er described in Sec. 2.1 is trained on the CLEF HIPE 2020 dataset. We added several features to the ones of the baseline. We performed a random search to select the best features while controlling the over tting. We provide the list of the features used in Tab. 1 and the list of the best hyper-parameters in Tab. 2. The system was trained on the train le and then tested on the dev and test les provided by the organizers.
We compared our NER system with the baseline provided by the CLEF HIPE 2020 organizers on the validation set (dev le). The results are gathered in Tab. 3. We can see that our NER system outperforms the baseline. We believe that its good performance is due to the choice of the selected features, e.g., the use of the tokens present in the text as features for the classi er. The ne tuning of the hyper-parameters of our CRF also partly explains the results better than those of the baseline. The results of our system on the test le are gathered in Tab. 3 6 Details statistics about the data can be found at https://impresso.github.io/
CLEF-HIPE-2020/datasets.html Parameter c1, the coe cient for L1 regularization, between 0 and 1 c2, the coe cient for L2 regularization, between 0 and 1 min freq, cut-o threshold for occurrence frequency of a feature, between 0 and 1 max iterations, the maximum number of iterations for optimization algorithm, between 100 and 1000 all possible transitions num memories, the number of limited memories for approximating the inverse Hessian matrix, between 4 and 8
Table 2. The best parameters for the CRF.
Best value found 0.1798 0.0551 192 false 0 4 Results of the entity linking process evaluated in terms of micro-averaged F1 classi cation scores are reported in Tab. 4. The three systems that we submitted to CLEF HIPE 2020 were ranked second (team7 results). We rst evaluated the entity linking based on the sole use of the local scores donated by team7 bundle2 fr 1. Second, we added the global score devising a collective entity linking which we named team7 bundle2 fr 2. And nally, we changed the collective linking system to lter the non-linkable mentions (NIL) based on a threshold, meaning we only link a mention to a candidate entity if the prior probability is below a xed threshold (here 0.5). We can see that the collective linking gave the best results, while the collective linking with a xed threshold is worse than the non-collective one. These results show the bene t of the collective linking.
A few experiments have been carried out to exploit the outputs of the NEL in order to enhance the NER results. The rst one consisted in using the types of the entities found by the NEL to change the NER labels; e.g., if the NER detects the entity 'Europe' and classi es it as 'PERS', the NEL links it to 'Q46' and gives the information that the type of 'Q46' is 'LOC'. The second consisted in generating closely related entities to the ones found by the NEL. We found that the output of the NEL stage can correct the NER, but can also introduce too much noise. Despite not being able to directly incorporate the output of the NEL with the existing features, we believe that applying a major vote between the di erent versions of the NER|with and without the NEL output|can lead to an increase of the accuracy of the NER. Nonetheless, our system opened the door to incorporate the semantics of the KB into the NER task.
We built an entity processing system based on a CRF classi er for the NER task,
and a collective entity linking system for the NEL one, exploiting the WSRM
entity relatedness measure that we have proposed in [