This paper explores automatic methods to identify relevant biography candidates in large databases, and extract biographical information from encyclopedia entries and databases. In this work, relevant candidates are defined as people who have made an impact in a certain country or region within a pre-defined time frame. We investigate the case of people who had an impact in the Republic of Austria and died between 1951 and 2019. We use Wikipedia and Wikidata as data sources and compare the performance of our information extraction methods on these two databases. We demonstrate the usefulness of a natural language processing pipeline to identify suitable biography candidates and, in a second stage, extract relevant information about them. Even though they are considered by many as an identical resource, our results show that the data from Wikipedia and Wikidata differs in some cases and they can be used in a complementary way providing more data for the compilation of biographies.
In the last decade, large biographical databases have
become available in a number of languages
Main_Page provides structured information taken from Wikipedia (and other sources)
The aforementioned APIS project aims to develop new
methods for re-using qualitative (biographical) research
products (encyclopedias) for quantitative research and, in
doing so, facilitate a digital transformation process against
the background of Humanities
mation about these candidates. The work presented here is within the scope of the aforementioned APIS project and aimed at enriching the O¨ BL, by means of automatically suggesting candidates for inclusion in the newer additions of the O¨ BL. We investigate the case of biography candidates that died between 1951 and 2019 and who had an impact in the present Republic of Austria. The current work focuses on processing the English texts, which can easily be adapted to process other languages as well, and selecting relevant candidates for inclusion in the O¨ BL. However, since previous work on adding biographies to the O¨BL was carried out by a team of historians, with decisions sometimes being based on factors which are difficult to model, evaluation of our pipeline is not straight-forward. We discuss this aspect in Section 4.
2.
The field of information extraction (IE) has long been a
thriving area of research within natural language
processing (NLP) and has steadily maintained a close relationship
with gathering information from the web. A large amount
of the early work on IE was done during the Message
Understanding Conferences (MUC) which took place from
1987 until 1997, and was later continued at the Automatic
Content Extraction (ACE) program
Other areas related to IE are named entity recognition
(NER), named entity linking (NEL) and wikification. NER
refers to the NLP task of detecting entities in text,
including (proper) names, locations, institutions, dates and so on.
On the other hand, NEL refers to the process of linking
named entities to entries in large databases or knowledge
bases, essentially linking information together. Both
Petram et al. (2015) and Brouwer and Nijboer (2017) have
explored entity linking in the context of biographical data,
in order to gain further information on persons. Hachey
et al. (2013) implement and evaluate three existing NEL
approaches, while making use of Wikipedia to augment
their approach. However, this should not be confused with
wikification, which commonly refers to the task of linking
wikipedia pages to concepts, persons and so on mentioned
in texts, i.e. not linking named entities and databases in a
strict sense
Extracting biographical information from the web is an area that has been gaining more attention, especially for the (automatic) creation of biographies and biographical databases. Increasingly, this area uses IE and closely related methods. Garcia and Gamallo (2015) have explored different machine learning methods to extract biographical relations in Portuguese. Furthermore, Wikipedia has increasingly become a common source of information for various applied methods. Approaches have either been used to extract information from Wikipedia as a source of information, such as Gotti and Langlais (2017). Russo et al. (2015) explores methods to extract biographical information from Wikipedia and DBpedia. Relevant information that was extracted includes the name, birthplace, birth date, and so on, of a person.
Although NLP methods are being used in the context of creating new biographies and biographical databases, there are many issues that need to be addressed. More specifically, Fokkens et al. (2014) point out these issues that were encountered during the course of the BiographyNet project. The authors point out that historic methods can often be hard to transfer to computational or automatic methods, since they rely on facts that may not be extracted directly, in addition to being based on interpretations, logic, analysis and so on. However, they also raise awareness of the potential biases that historians could face when using NLP methods. In particular, the authors argue that when using rulebased methods, the rules and heuristics need to be clearly indicated, and when machine learning approaches are used, the training data and features used should be described. Fokkens et al. (2014) use the example of provenance modelling to demonstrate where these biases could occur and how obvious they would be. More obvious cases could be ambiguous geographical locations, which would factor heavily in a rule-based approach. Less obvious would be unbalanced datasets that may be used in machine learning, leading to persons to be associated mistakenly with certain topics. In their final conclusion, the authors emphasise that the awareness on both the historian and NLP sides needs to be raised to the problems explained in the paper. Work on extracting biographical information from Wikipedia using Wikidata on a larger-scale has been carried out by Plum (2018). Research carried out for that project can be seen as preliminary work for the methods we describe here. It features a dataset of around 130, 000 lexicon pages about individual persons, which was selected by using Wikidata with similar parameters set out here. A short analysis of common structures containing information was carried out. Plum (2018) not only demonstrates how these common structures can be exploited and simple rules applied in order to extract information about the date of death and occupation of a person. The author also points out some of the pit-falls when working with such a large data-set, including the amount of processing time and choosing an appropriate data-structure. We make use of the analysis put forth and apply similar rules to extract basic information. In addition, we make sure to take into account the possible obstacles to overcome.
3.
In this section we present the processing pipeline used
to carry out the experiments described in Section 4. The
pipeline consists of three main steps, and is depicted in
Figure 1. First, we pre-select a large amount of entities
using Wikidata and in accordance to parameters set out by
the project scope (Section 3.1.). We make the gathered
meta information easily accessible via a local MongoDB6
database. Next, we preprocess the data, using Stanford
CoreNLP
According to the parameters of the joint project, we select entities that are listed as human and that have died between 1951 and 2019. The query returns the date and place of death and birth, a short description, as well as the Wikidata link and other identification numbers. Figure 2 shows the query that was used and Figure 3 shows an example of the returned results.
A problem that occurs with the links to the individual
articles is that Wikipedia uses two types of links, one using
the name and one using an id number. Therefore, we use
a second Wikidata query to retrieve the ids for each page
in addition to the previously returned links (which use the
name). The id number cannot be returned using a standard
Wikidata query, i.e. there is no relation that can be
specified in order to gather this for all entities at the time of the
query. Instead, we use a separate query which takes each
Wikidata id individually, and generates the corresponding
Wikipedia id by extracting it from the Wikipedia article
itself. The choice was made to use these ids, as it easier to
retrieve the article for each entity from the Wikidump, which
we describe in due course. Using the name could lead to
some errors, due to differences in spelling across different
languages. This problem was also pointed out by
The Wikidata query returned 401, 695 entities (equivalent to lexicon pages), and of these 172, 131 had corresponding articles in English. From the total number returned, we hope to use cross-lingual methods in future to extract from articles in languages other than English (see Section 5.). It is also worth mentioning at this point that some discrepancies exist between Wikidata and Wikipedia information, leading to some entities of the overall retrieved not being used. A account of this will be presented in Section 3.2. Once the selection of the entities and retrieval of basic meta information was carried out, the next task involved extracting the corresponding articles from the Wikidump. A Wikidump is a snapshot of the whole Wikipedia encyclopedia in XML format. We use a Python script called WikiExtractor7 to convert the Wikidump from XML format to plain text. The script processes the whole Wikidump, returning each article as plain text, as well as minimal meta information including the Wikipedia id and name of the article. It does not retain any structure of the XML file or provide any further markup or information. This was the main reason for using the script, as processing a Wikidump is time-consuming in itself and can involve many complications. Therefore, we opted to use this already available script in order to be able to process the vast XML files which have a complex structure. From the converted plain text, we extract all relevant articles using the previously obtained Wikipedia ids. Articles are grouped together, indicated by a begin and end tag and stored across plain text files with sizes between 1Mb - 2Mb. We run a Python script to extract each article by identifying the beginning and end tags, and store it in individual JSON files, which makes the following step of pre-processing easier.
In order to facilitate the information extraction task, we
automatically annotate each article with linguistic
information. Due to the large amount of data we are processing, this
task could not be carried out during extraction. Therefore,
we process each article using Stanford CoreNLP
continuous processing time. Using a more multi-threaded workload could optimise this task, however, this will be addressed in future work.
Once processed, we were left with 170, 517 articles, down 1, 614 from the previous number. Taking the timeconsuming nature of this processing into account, we had to set up a time-out for each request to the CoreNLP annotation pipeline, meaning that extremely long articles were not processed. Upon further inspection we also discovered that a small number of texts had not been converted properly, and contained either no text or corrupted text. 3.3.
For the information extraction step, we follow a shallow rule-based approach in order to take advantage of the basic information from Wikidata. Exploiting the structure of Wikipedia articles, we extract the name and date of death of an entity. Not only do we test if using simple rule-based methods are viable for this kind of data, we also test a method to determine candidates of relevance to the project. As described in the introduction, candidates should have had an influence in the Republic of Austria. Therefore, we also extract locations mentioned in the articles and determine whether they are Austrian, hence possibly hinting at the fact that the entity of the article has some kind of connection to Austria.
By extracting the tokens of the heading of each article, we
extract the name. We remove any information that is
contained in brackets, which is sometimes the case in order to
disambiguate certain persons. Furthermore, we use a
simple rule based on observations and preliminary work carried
out by
As the articles have been annotated in terms of named entities, we are able to extract locations by simply searching for any LOCATION or CITY tags. We employ this approach for each text, making a list of each location for each article. Next, we determine the country of each location. Our first approach was to use the GeoCoder api8 connected to GeoNames to retrieve the country of each location. Unfortunately, GeoNames is restricted to 1, 000 requests for locations per hour. With such a large dataset, this would not be a viable approach. Other than buying requests as part of a premium service, we opt to download the full GeoNames list of locations, which is freely available. This list contains locations, as well as their country. Using MongoDB, we create a database and index the location names to ensure fast searching. This way we are able to query a local database to determine the country for each location. Using a custom Python script we query whether a location among those found in each article belongs to the Republic of Austria. If this is the case, we include the article or entity as a candidate. In the first iteration, we found that locations were being found in the documents, such as City. These location names are always part of a longer name, but are picked up as they are tagged individually. These locations returned matches in the database, although these were mostly mistakenly added. Therefore, we added the criterion that any location has to have a population of more than zero. We found that locations that have been mistakenly added or are meant for some other purpose usually have a population of zero.
In order to test out how a method of ranking relevant candidates could work, we count all locations in one article that are in Austria. In addition, we try to count the false matches as well. As there are a large amount of false matches we put these in contrast to the main location count. The idea is that these count could put the ranking into better perspective, if for instance the counts are equal, this candidate could possibly be excluded.
4.
In this section we present the results of the information extraction task, as well as the selection of possible candidates. The evaluation of our results presents a challenge, due to problems selecting a gold standard. We compare the extracted names and dates of birth to those returned by Wikidata, but this assumes Wikidata as a gold standard. Concerning the location extraction, we rely on the performance of Stanford CoreNLP and the rules we employ as to extracting the country. For reasons that will be explained later on, we do not have a gold standard for evaluation purposes here. Therefore, it should be clear that this is not an evaluation of the extraction itself, but rather the process of selecting candidates.
4.1.
As described in the previous section, we extracted the name and date of death from the Wikipedia articles. We compared each result with the information obtained from Wikidata. Of the 170, 517 articles, the name did not match in 18, 267 cases. Upon closer inspection, we found that this is largely due to differences in spelling, and slight differences in the name. Table 1 shows a selection of the most common errors: The first two rows are examples of differences in shorter names. Rows 3 and 4 show different levels of preciseness in naming, i.e. abbreviations. The last row shows an example where Wikidata returned the name in German, whereas we extracted the name in English.
Robert Joshua Francisco Javier Vidarte Joe C. Davis, Jr.
Vincent Graber Karl Aloys von und zu Liechtenstein
Bob Joshua Paco Vidarte Joe C. Davis Jr.
Vincent J. Graber Sr.
Prince Karl Aloys of Liechtenstein In terms of the date of death, we had 30, 153 cases where the date of death did not match the Wikidata records. Using a Python script we counted the different errors that occurred, and found that we could classify three main errors: “no date” errors, “minor difference” errors and “birthday” errors. A breakdown of how many times each error occurred is shown in Table 2. In the first case, our extraction rule returned 0000-00-00, indicating no date was extracted. This error was caused by the Stanford NER algorithm not detecting a date, or it being missing in the article itself. The second most common error was the “minor difference error”. In this case, the difference between the Wikidata date and our extracted date was minor, i.e. only between one to five days difference. We suggest that this shows that Wikidata also gathers information from other sources, or that it could be caused by timezone differences. The last error to occur was the “birthday” error. Here, the date we extracted did not match the date of death extracted from Wikidata, but rather the corresponding date of birth. This is caused by the fact that we extract a date in sentences, even if only one date is found by the NER algorithm.
Taking these results into account, it is interesting to see where differences in data lie. Using Wikidata and Wikipedia as complementary components has many benefits. On the one hand, Wikidata serves excellently as a tool to pre-select data according to some criteria. As processing all Wikipedia, or complete Wikidumps is extremely time-consuming, this reduces the time dramatically. On the other hand, Wikidata can serve to some extent as a kind of gold standard against which to compare the results of any extraction carried out on Wikipedia articles. Of course, this is only to a limited extent, as not all relations are available in Wikidata. This is the case with our extraction of locations in order to determine candidates.
Going beyond the use as a gold standard, the two data sources can also be used to extract information in a more complementary way, i.e. using Wikidata for basic information, and building on these known relations to extract further information from Wikipedia. It may also be of interest to compare contradicting information, as seen here with the differences in date of death. 4.2.
Using our location matching script, we were able to obtain 13, 521 possible candidates. A short investigation of candidates picked at random shows that our technique is probably not precise enough. For each candidate, we list the location that caused its inclusion in our candidates list. The ten most common locations are listed in Table 3. While this list includes many valid locations, it is clear that many articles are chosen as candidates due to matches caused by Hall and Point. Further examples include Sand, Fall and Gray and so on. These match proper locations in Austria, however, they also match English nouns and adjectives, and are most probably part of longer location names. Another common problem in this regard was the matching of names which are ambiguous, as they also match locations in Austria, and therefore contributed to being considered as candidates.
Hall Vienna Point Salzburg Bergen Nassau Innsbruck Graz Ko¨ nigsberg Inn
Count In terms of the ranking of the candidates, this is just as hard to evaluate as to measure. A brief manual analysis indicates that the ranking mechanism at this point is too crude. Quite often candidates rank very highly, even though there is no relevance to Austria whatsoever. This is mainly due to the false matches, described previously. An extract of some of the candidates below shows how highly some irrelevant candidates rank. At the other end of the scale, the reverse applies. Candidates that should probably be considered with priority are ranked very low, due to only a few ore one locations being matched. However, this is mostly due to extremely short Wikipedia articles, which do not hold much information.
Further evaluation of the locations extracted against a gold standard is not possible. Wikidata queries rely on a relation, such as born in, to be present in order to extract the corresponding location. In our case, we want to go beyond these relations and find any kind of mention of locations that are relevant. Ultimately, the candidates we are able to derive are to be evaluated in an iterative process by historians from the APIS/OEBL team in order to say how well our method performs. Other forms of automatic evaluation do not exist at this point in time, especially considering there is no gold standard for this work, as it is mainly aimed at finding completely new candidates.
5.
In this paper we presented an NLP pipeline to identify biography candidates and to extract information about them from Wikipedia and Wikidata. We show that shallow extraction methods work well for obtaining basic information about biography candidates. However, for determining possible relevant candidates there is still work to be done. While our simple method of matching locations works as a wide net, there are many irrelevant inclusions. We acknowledge that this metric by itself is too simple, however, we feel that it could become an aspect of a future metric. As the goal of this project is detecting relevant candidates, we are currently working on improving our basic method, hopefully making use of statistical or machine learning based approaches in order to determine whether a person has had some kind of relevance in a certain area. This could also allow us to rank candidates according to their relevance for that particular area and time period.
In collaboration with the AAS we are working on a ranking system which at present is based on implicit expert knowledge. We hope to convert this knowledge to the machine, and as a part of this ongoing effort we are working on an annotating the dataset of the previous O¨ BL biographies. By extracting sentences that show some kind of surface relevance to Austria, we are researching the possibility of training a machine learning classifier on the word and/or context embeddings of these sentences, in order to automatically detect them in text. This would also eliminate the need of the large-scale pre-processing beforehand.
In the future, we also aim to refine our information extraction methods and to test them on different encyclopedic repositories. In addition, we plan to explore cross-lingual methods for extracting information from data sources in other languages such as Czech, German, Hungarian, and Slovak as the core languages of the former AustrianHungarian monarchy.
This research is a pilot endeavour to detect relevant candidates for a biographical dictionary in online sources. It aims to contribute to three goals, 1) the further compilation of a digital, semi-automatic biographical dictionary on the case study of the O¨ BL, 2) the further development of an editing system for biographical dictionaries, which might be used as a research infrastructure, on the use case of O¨ BL, and 3) triggering interdisciplinary collaboration and further pilot studies on methods and tools to detect people of ”relevance”.
The APIS project is funded by a research grant (project number O¨ AW0405) of the Austrian Nationalstiftung fu¨ r Forschung, Technologie und Entwicklung (Programm “Digital Humanities - Langzeitprojekte zum kulturellen Erbe”). We are grateful for technical support to access the APIS and O¨ BL data sets by Katalin Lejtovicz and Matthias Schlo¨ gl and are thankful for the feedback and evaluation of the (art)historians A´goston Bernad and Maximilian Kaiser. 6.