This paper presents a simple method to rank georeference candidates to optimally support the workflow of a citizen science web application for toponym annotation in historical texts. We implement the general idea of efficient crowdsourcing based on human and artificial intelligence working hand in hand. For named entity recognition, we apply recent neural pretraining-based NER tagger methods. For named entity linking to geographical knowledge bases, we report on georeference ranking experiments testing the hypothesis that textual proximity indicates geographic proximity. Simulation results with online reranking that immediately integrates user verification show further improvements.
Named entity recognition (NER) in texts
Named entity linking (NEL) of toponyms is
normally cast as a consecutive task to NER
and consists in annotating each toponym
mention with a unique identifier from a
domainspecific knowledge base. This linking of
toponyms, also known as toponym resolution (TR)
The main problems of toponym resolution are the ambiguity of toponym names (e.g. in Switzerland alone there are 12 mountains called Schwarzhorn), and especially in the case of historical texts, the renaming of geographical entities over time and changes in the spelling of names, which leads to insufficient coverage of name variants even in large contemporary geographical ontologies. For our corpus, additional problems arise from multilinguality, (a) because many geographical entities genuinely have more than one name due to the multilingual cultural background of Switzerland, and (b) because we are dealing with a multilingual text corpus.
The historical corpus of Alpine texts for which
we aim at a complete, fine-grained and precise
toponym annotation consists of the early
yearbooks of the Swiss Alpine Club (SAC) published
since 1864
Given the difficulties of toponym annotation
in domain-specific historical texts, automatic
toponym resolution methods are not able to achieve
the desired performance. For NER using a
domain-specific rule-based system, Kew et al.
(2019) report a recall of 63% and a precision of
88% when evaluating on 1300 sentences sampled
from the full corpus (1864-2015). For a modern
neural NER approach
However, the task of toponym annotation is
more complex and knowledge-intensive than OCR
correction. Our goal is to provide an
efficient workflow that ensures that automatic
preannotation and human correction from citizen
scientists profit from each other as early as possible.
For NER, this means to retrain the neural NER
models regularly and to update the pre-annotations
without interfering with already curated material.
Recent neural NER taggers
For NEL, it means to minimize the user’s efforts to identify the correct georeference of a toponym mention. Ideally, the NEL component should (a) precompute all possible georeference candidates for a mention (taking into account typical spelling variations) in order to free the user from performing time-consuming knowledge base queries on his own, and (b) rank these candidates such that the true reference appears first on the list. Figure 1 illustrates the intended setup for linking the mention “Matterhorn” to its intended georeference.2 Verifying a suggested georeference candidate by a single click is far less time consuming than searching through a long unordered list (sometimes up to 70 candidates).
The remainder of this paper reports on simple
and efficient methods to optimally rank
georeference candidates for toponym annotation based on
the principle of textual and geographical
proximity
(Re)Ranking Georeference Candidates We investigate two scenarios: (a) ranking candidates using as only evidence automatically computed georeference candidates, and (b) dynamically reranking candidates simulating a human validation process where the automatic rankings and the human corrections serve as iteratively improving evidence. The original ranking happens offline during automatic NEL. In contrast, the reranking happens online during the annotation process and can be done in the client’s web browser.
The ranking algorithms for both scenarios rely
on the hypothesis that textual proximity indicates
geographic proximity
science web application currently features the offline “. . . Mo¨nchjoch, erste Ueberschreitung. Alphubel und Alphubelpass, Feegletscher, Ru¨mpfishorn, sa¨mmtlich erste
Besteigungen.” ambiguous toponym Alphubel is rewarded twice due to its smallest distance to all other candidates in a context window of n 1 toponyms. In other words, each context toponym “votes” for the target toponym candidate with the smallest distance.
Ranking. More formally (see Algorithm 1), given a target toponym tti at position i we determine the set of context toponyms tcti n; :::; cti 1; cti 1; :::; cti nu. Then, target candidate tcj indexes all admissible georeference candidates of the target toponym tti, and for each context toponym ctk the context candidate cckl indexes all its admissible georeferences. The score of every target candidate tcj is initialized with 0, and for each context position k, the score of tc^j with the smallest distance of all target/context pair ptcj ; ccklq is incremented by 1. Thus, for a given context size n containing 2n toponyms, 2n is the maximum candidate score in a window.
Aggregating all window scores over a yearbook results in a single global score for each georeference. Our final scoring normalizes the yearbook scores of each georeference into the range r0; 1s and multiplies it with the score of each candidate georeference from the local window. In this way, the overall prominence of a georeference in a yearbook (in early years, each SAC yearbooks had one mountain region as a main topic) is combined with the proximity in a “local story” told within the context window. Candidates are then sorted in descending order according to the final total score in order to produce the candidate ranking. “Wir fliegen mit dem Blick u¨ber den [Tschingelgletscher] hin und einen Moment verweilen wir bei der ja¨hen Gneistafel des Lauterbrunner Breithorns, welches, scharf in seinen breiten Gra¨ten, uns nur kahle Platten zeigt und von dessen Fuss einige sekunda¨re Gletscher in’s Lo¨tschenthal herabha¨ngen. Ueber [Ebene Fluh], Grosshorn und Gletscherhorn fliegen wir neuerdings hinweg, senken wieder den Blick in den grossen Ocean des [Aletschfirns] und stehen gebannt vor den scharfen Formen der Jungfrau [. . . ]”.
Dynamic Reranking As soon as humans correct a georeference, new information is available that can be used to update and improve existing candidate rankings on the fly and to further minimize the effort of a crowd corrector. In order to assess the expected benefit, we define the following correction simulation strategy that assumes the user to correct all toponyms in reading order of the text. Each time a user verifies a reference candidate, we update the candidate ranking of the following toponym.
The dynamic reranking is also based on window and yearbook scores and only differs in the following aspects: (a) The yearbook scores are not updated by the window scores. (b) The window score rewards by 10 points instead of 1 point if a verified candidate is involved. (c) The window score rewards by 3 points if only one candiAlgorithm 1 Candidate Ranking
Input: pages P , window size n Output: pages P initialize yearbook scores ys for each page P P do for each target toponym tti P page do initialize window score map ws tc Ð get candidatespttiq ct Ð get context toponymsptti; nq for each context toponym ctk P ct do cck Ð get candidatespctkq for each context candidate cckl P cck do for each target candidate tcj P tc do
set tc^jto tcjif closest to cckl end for end for ws tc^j 1 end for update page with ws end for increment ys by ws end for update P with ys sort candidates of P by ys and ws return P date is involved. Figure 3 shows how candidates for Alphubel are ranked and reranked. Both candidates are assigned the same window score but the southern central candidate has a higher yearbook score and is thus ranked first by the ranking. When a user reads the sentence and adds and verifies Tschingelgletscher, which is close the correct candidate, the dynamic reranking updates the candidate positions and ranks the correct candidate on the first place. 3
Ranking and Reranking Experiments We test the quality of our ranking method on German pages of the yearbook 1864 and 1874. Our NEL uses two different geographical ontologies, SwissNames3D3 for toponyms within Switzerland and GeoNames4 for all others. We made this choice in order to achieve maximal coverage in Switzerland and to avoid linking ambiguity due to multiple knowledge bases. For linking with SwissNames3D, only 23 relevant entity types out of 103 are used5, for GeoNames 127 out of 676 entity types (feature codes) are used. Table 1 reports the number of toponym candidates and their ambiguity. Note that 30% of the toponyms cannot be resolved by the NEL, and therefore, they do not contribute to the ranking.
3https://shop.swisstopo.admin.ch/en/products/landscape/ names3D 4https://www.geonames.org 5We exclude field names (traditional “Flurnamen” in German) due to their extensive ambiguity.
For our experiments, we randomly sampled 20 pages from the yearbook 1864 and 1874 that contain at least 4 ambiguous toponyms and manually resolved all ambiguous cases. Additionally, we invested roughly one hour per page to verify or add other toponyms on the page.
Evaluation Systematically evaluating NEL
systems is still a challenging task
In Table 2, we report results for 3 different ranking conditions: Randomized (rand.) is a baseline that shuffles the candidates arbitrarily. Ranking (rank.) reports the outcome of the proximity ranking algorithm. Reranking (rerank.) shows the results of our dynamic reranking derived from the correction simulation. Our evaluation measure reflects the overall frequency of a correct georeference being ranked first (labeled as rank1), second (rank2), third (rank3) or below rank three (rank4+).
Additionally, we report the relative improvement of ranking in comparison to random shuffling, the improvement of reranking in comparison to ranking and relative error reductions. Further, for comparability, we report the mean reciprocal rank (MRR) of the true references. For a given set of ranks of true references R, the MRR is computed as
MRRpRq 1 |R| 1
¸ |R| i 0 Ri (1) with Ri P r1; 4s because we map all rank positions ¡ 4 to 4 for consistency with the absolute ranks reported (rank1 to rank4+).
An important hyperparameter of our approach is the sliding window size n. We evaluated our ranking system with values between 1 and 10 and decided to use a size of 4, which is efficient to compute and performs as well as larger windows. Table 3 shows the rank1 and MRR results for varying window sizes.
Discussion We see that the simple ranking algorithm works pretty well in general. Especially for the yearbook sample 1864, there is a stark relative improvement over random shuffling of 267%. # topo topo w/o georef topo w/ georef -ambig +ambig +ambig (in %)
Reranking then cannot improve much more on top of that. For 1874, ranking works decently, but leaves many true georeferences on second position. Reranking almost doubles the number of rank1 rankings. Reranking also reduces the number of rank3 and rank4+ cases in comparison to ranking. The poor ranking performance in 1874 is probably due to several ambiguous toponym occurrences where a lot of the surrounding named entities were not found by the NER component initially. The reranking based on incremental user corrections alleviates this problem.
It is interesting to note that by qualitatively looking at reranking errors we could detect several errors in the initial ground truth. A next step for improving the ranking is probably the inclusion of external prominence features (population size, existence of a Wikipedia page, etc.) directly available from some of our geographical knowledge bases. 4
We have shown that a simple ranking approach using a sliding window of 4 is an effective way to profile the intended georeferences on top positions in our two test sets. The quality of our automatic preannotation in historical texts is low enough to profit from a dynamic reranking that integrates human verification as early as possible into the georeference suggestions prominently presented to the user. In crowdsourcing, human and artificial intelligence should work hand in hand in order to efficiently produce high-quality annotations. The disambiguation of rank1 cases using a two-click verification speeds up the process and leaves more time for citizen scientists to address the difficult toponym resolution problems that need real detective work.