With the ever-growing popularity of the eld of NLP, the demand for datasets in low resourced-languages follows suit. Following a previously established framework, in this paper1, we present the UNER dataset, a multilingual and hierarchical parallel corpus annotated for named-entities. We describe in detail the developed procedure necessary to create this type of dataset in any language available on Wikipedia with DBpedia information. The three-step procedure extracts entities from Wikipedia articles, links them to DBpedia, and maps the DBpedia sets of classes to the UNER labels. This is followed by a post-processing procedure that signi cantly increases the number of identi ed entities in the nal results. The paper concludes with a statistical and qualitative analysis of the resulting dataset.
Named entity recognition and classi cation (NERC) is an essential Natural
Language Processing (NLP) task involved in many applications like interactive
question answering, summarizing, relation extraction, and text mining. It was
originally de ned in the 6th Message Understanding Conference (MUC-6) as the
identi cation of \Person", \Location" and \Organization" types [
NERC systems that depend on machine learning models need to be trained with a large amount of language-speci c annotated data to attain high accuracy.
Commons License Attribution 4.0 International (CC BY 4.0).
Nevertheless, annotating this type of data is expensive, time-consuming, and
requires speci c annotators, training to guarantee good inter and intra-annotator
agreements [
The complexity of generating NERC annotated data increases when
multilingualism and a complex hierarchy of types are involved [
Our goal is to parse data from Wikipedia3 corpora in multiple languages,
extract named entities through hyperlinks, align them with entity classes from
DBpedia4 [
This paper presents the data extraction and annotation work ow and its application on English5 and Croatian6 languages under the licence CC BY-NCSA 4.0. It is organized as follows: In Section 2, we present the state-of-theart concerning NERC automatic data generation; in Section 3, we describe the details of the data extraction and annotation work ow, and in Section 4, we provide statistical analysis and qualitative evaluation of the annotation. Section 5 is dedicated to the discussion of the results, and in Section 6, we present our conclusions and possible future directions for research, including applications of the UNER corpora. 2
Due to the importance of the NERC task, the challenge of creating a quality annotated dataset has been the object of many studies.
Yadav and Bethard [
To overcome the problem of cost, Lawson et al. [
A generic method for extracting MWEs from Wikipedias was proposed in
Bekavac and Tadic [
using local regular grammars. An automatic multilingual solution is presented
by Ni et al. [
Kim et al. [
A complete automatic work ow is proposed by Weber & Vieira [
The work ow we have developed allows the extraction of texts and metadata from Wikipedia (for any language present in this database), followed by the identi cation of the DBpedia classes via the hyperlinks associated with certain tokens (entities) and the translation to UNER types and sub-types (these last two steps being language independent).
Once the main process of data extraction and annotation is over, the
workow proposes post-processing steps to improve the tokenization and implement
the IOB format [
The whole work ow is presented in detail in the project GitHub web-page7 together with all scripts that have been used, and that can be applied to any other Wikipedia language. 3.1
The following items are important components that were used for the dataset creation. These are mappers that map entities from the source to a target class or hierarchy.
1. UNER/DBpedia Mapping: This is a mapper that connects each pertinent DBpedia class with a single UNER tag. A single extracted named entity might have more than one DBpedia class. For example, entity 2015 European Games have the following DBpedia classes with the respective UNER equivalences: { dbo:Event { Name-Event-Historical-Event { dbo:SoccerTournament { Name-Event-Occasion-Game { dbo:SocietalEvent { Name-Event-Historical-Event { dbo:SportsEvent { Name-Event-Occasion-Game { owl:Thing { NULL The value on the left represents a DBpedia class and its UNER equivalent is on the right side of the class. It maps all the DBpedia classes to UNER equivalent classes. 2. DBpedia Hierarchy: This mapper assigns priorities to each DBpedia class.
This is used to select a single DBpedia class from the collection of classes that are associated with an entity. Following are classes are their priorities. { dbo:Event { 2 { dbo:SoccerTournament { 4 { dbo:SocietalEvent { 2 { dbo:SportsEvent { 4 { owl:Thing { 1 For entity 2015 European Games , the DBpedia class SoccerTournament presides over the other classes as it has a higher priority value. If the extracted entity has two assigned classes with the same hierarchy value the rst from the list is chosen as the nal one. All the DBpedia classes were assigned with a hierarchy value according to DBpedia Ontology8. 3.2
The main process is schematized in the gure below and is divided into three sub-processes. 1. Extraction from Wikipedia dumps: For a given language, we obtain its latest dump from the Wikimedia website9. Next, we perform text extraction preserving the hyperlinks in the article using WikiExtractor10. These are hyperlinks to other Wikipedia pages as well as unique identi ers to those named-entities. We extract all the unique hyperlinks and sort them alphabetically. These hyperlinks will be referred to as named-entities henceforth. 2. Wikipedia-DBpedia entity linking: For all the unique named-entities from the dumps, we query the DBpedia endpoint using a SPARQL query with SPARQLWrapper11 to identify the various classes associated with the entity. This step produces, for each named-entity from step 1, a set of DBpedia classes it belongs to.
3. Wikipedia-DBpedia-UNER back-mapping: For every extracted namedentity obtained in step 1, we use the set of classes produced in step 2, along with an UNER/DBpedia mapping schema, to assign UNER classes to each named-entity. For an entity, all the classes obtained from the DBpedia response are mapped to a hierarchy value, highest valued class is resolved and chosen and then it is mapped to UNER class. For constructing the nal annotation dataset, we only select those sentences that have at least one single named entity. This reduces the sparsity of annotations and thus reduces the false negatives rate in our test models. This step produces an initial tagged corpus from the whole Wikipedia dump for a speci c language. 3.3
The post-processing steps correspond to four di erent Python scripts that provide: 1. The improvement of the tokenization (using regular expressions) by isolating punctuation characters that were connected with words. In addition, it applies the IOB format to the UNER annotations inside the text. 2. The calculation of the following statistic information concerning the generated corpus: Total number of tokens, Number of Non-entity Tokens (tag \O"), Number of Entity Tokens (tags \B" or \I") and Number of Entities (tag \B"). The script also provides a list of all UNER tags with the number of occurrences of each tag inside the corpus. 3. Listing the entities inside the corpus (tokens and the corresponding UNER tag). Each identi ed entity appears once in this list, even if it has multiple occurrences in the corpus. 4. The increment of the annotations by an automatic process using the list of entities (output of item 3 above). The script checks if, inside the corpus, entity tokens from the mentioned list were not tagged as a UNER entity. If so, the corresponding UNER tag is associated to them. Entities composed of one single character, two lower-case characters, and digits were discarded. Also, multi-tokens entities were prioritized using the longest match approach.
Post-processing steps from 1 to 4 were applied to the UNER Croatian corpus, while for the English one, only steps 1 to 3 due to its size. Results and qualitative analysis are presented in the following section. 4
We have decided to test the proposed work ow with two di erent languages:
English and Croatian. The Croatian Wikipedia [
After applying the main process of the proposed work ow, we obtain, for each language, annotated text les divided into folders. The sizes of the English and Croatian UNER corpora are presented in the following table.
Corpus Total Size Number of folders Number of les English UNER 3.3 GB 172 17,150
Croatian UNER 108 MB 5 411
After applying post-processing steps 1-3 in both Croatian and English UNER datasets, we obtain the following statistics concerning both corpora.
Therefore, concerning the English Corpus, 8.9% of tokens are entities, 7.1% in the Croatian one.
As explained previously, the UNER hierarchy is composed of categories, types, and subtypes. UNER includes the most common classes used in NERC (\Person", \Location", Organization", being more detailed (subtypes): { \Person": correspond to UNER \Name-Person-Name" { \Location": correspond to all subtypes inside the UNER types \Name-Location". { \Organization": correspond to all subtypes inside the UNER type \NameOrganization"
English UNER Corpus Croatian UNER Corpus Total Number of Tokens 325,395,838 9,388,224 Number of Non-Entity Tokens 320,719,350 8,436,254 Number of Entity Tokens 31,676,488 951,970 Number of Entities 15,101,318 668,231 Number of Di erent Entities 630,519 157,418
Therefore, it is possible to analyse the generated corpora in terms of these more generic classes.
These main classes correspond to 68.2% of NEs in the English corpus and 75,5% in the Croatian one. As explained in section 3.1, the annotation of a certain named-entity depends on the existence of hyperlinks. However, these links are not always associated with the tokens if the entity is mentioned repeatedly in the article. Therefore, to improve the corpus generated through the proposed work ow, we have established another post-processing step, which we applied to the Croatian corpus. By using the list of entities present in the corpus, we identify in the text the tokens that should have been tagged as entities, but that did not have the corresponding hyperlink.
In the following table, we present the statistics of the nal post-processed Croatian corpus compared to the one obtained before this last step.
The percentage of entity tokens is increased from 7.1% to 16.2%. In the next table, we focus the statistical analysis on the UNER types that can be associated with the classic NERC classes: Person, Location and Organisation. The automatic annotation of the text extracted from Wikipedia requires the identi cation of the DBpedia classes associated with the respective tokens (via hyperlink) and the translation to UNER using (UNER/DBpedia equivalences). In order to evaluate this step, we have performed an analysis of 943 entities randomly selected concerning the English UNER Corpus. For each one, we have checked the DBpedia associated classes and the nal UNER chosen tag. The next table presents the results of this evaluation.
The analysis of this sample shows that 91% of the entities are correctly tagged with UNER tags. However, 6% are associated to the correct UNER type but to a generic sub-type. For example, Bengkulu should be tagged as Name-LocationGPE-City but received the tag Name-Location-GPE-GPE Other.
The incorrect tags are due to errors in the DBpedia classes associated with the tokens or due to the equivalence between DBpedia and UNER rules: { Buddhism is associated only to the DBpedia class EthnicGroup and, therefore, is wrongly tagged as Name-Organization-Ethnic Group other while it should be associated to the UNER tag Name-Product-Doctrine Method-Religion. { Brit Awards , due to the prioritization of DBpedia class hierarchy in the choice of UNER tags, is wrongly tagged as Name-Organization-CorporationCompany while it should receive the tag Name-Product-Award.
It is possible to observe a considerable number of False Negative instances inside the corpora not being processed by the last post-processing step. This is due to the fact that not all entities in Wikipedia extracted articles have hyperlinks.
A qualitative analysis of the nal Croatian UNER corpus shows that the nal post-processing step considerably reduces the number of False Negative instances. Nevertheless, with the actual set of rules for this nal automatic annotation step, we can observe some problems concerning mostly mono-tokens entities, for example, all jezik instances are associated with UNER tag Name-ProductPrinting-Magazine while in the text it may not correspond to a magazine but to the common noun jezik (language in English ). Also, it is important to mention that while the UNER hierarchy proposes a very complex hierarchy in terms of time and numerical expressions, this work ow covers basically UNER types and sub-types of the Name category. 5
The implemented work ow and the generated corpora show the potential of
the method presented by Weber & Vieira [
In this paper, we describe an automatic process for generating multilingual Named-Entity recognition corpora by using Wikipedia and DBpedia data. We also present the UNER corpus, a hierarchical named-entity corpus in Croatian and English, developed with the proposed method. We show some statistics of the corpus and detail the procedure used to create it, nishing with a qualitative evaluation. In our future work, we plan to extend our corpus to other under-resourced languages while evaluating our work ow's performance across the languages. Also analysing the di erent limitations due to the uneven content of Wikipedia across languages. Following this intrinsic evaluation of our dataset, we will train models on the obtained data to extrinsically evaluate the corpus. 7
The work presented in this paper has received funding from the European Union's Horizon 2020 research and innovation program under the Marie SklodowskaCurie grant agreement no. 812997 and under the name CLEOPATRA (Crosslingual Event-centric Open Analytics Research Academy).