This paper introduces the Perdido Python library for geoparsing and geocoding French texts. The architecture of the Perdido Geoparser, which includes three layers: back-ofice, API, and Python library, is outlined. We also provide details on the methods used in the development of the processing chain and the various tasks covered, such as named entity recognition and classification (NERC), and toponym resolution. Lastly, we showcase the diferent features of the Python library and explain how to use it. The library is built as an overlay using API services, enabling users to manipulate, visualize, and export the results of geoparsing and geocoding. A Jupyter notebook1 is also provided to demonstrate all the functionalities implemented in the library.
https://ludovicmoncla.github.io (L. Moncla)
Geocoding methods in the literature are divided into two categories: those that rely on
external resources such as knowledge bases and gazetteers, and those that rely on trained
models [
The architecture presented in this article has been developed and enriched during diferent
projects such as itinerary reconstruction from hike descriptions [
Perdido Geoparser is implemented in three layers: the back-ofice part hosted on a server, a REST API that exposes the back-ofice functionalities in the form of web services and the Python library that ofers an extra layer to query the services and manipulate, visualize and export the results.
Back-ofice implements a processing chain for geoparsing: pre-processing (tokenization,
lemmatization, morpho-syntactic annotation), named entity recognition and classification and toponym
resolution. The pre-processing steps are performed using Treetagger1. Named entity
recognition and spatial information annotation rely on a dual cascade of transducers that use lexical
resources and pattern descriptions (local context-free grammars, morpho-syntactic patterns, …).
The transducers are implemented within the Unitex2 platform and act by insertion to tag named
entities and spatial information in the text. The processing chain produces two output formats,
an XML-TEI3 format [
The Perdido6 Python library is available as an open-source on GitHub and can also be easily installed through the PIP package management system7. This makes it convenient to integrate into a Python environment and use with minimal coding required.
The library provides three main classes: G e o p a r s e r and G e o c o d e r which allow to call the corresponding web services of the API and P e r d i d o which allows to manipulate, visualize and export the results. Other classes are also available, such as the P e r d i d o C o l l e c t i o n class, which extends the role of the P e r d i d o class for a set of documents processed by Perdido, or the T o k e n , E n t i t y , and T o p o n y m classes, which provide various attributes and methods for retrieving and viewing the objects manipulated by the P e r d i d o class.
The constructor of the G e o p a r s e r class takes several optional arguments in parameter: for
both the geotagging and geocoding stages (these last parameters correspond to those of the
constructor of the G e o c o d e r class). Concerning the geotagging, the v e r s i o n parameter allows to
select which version of the annotation cascades will be used among the two currently existing
versions: Standard (default) and Encyclopedie. The Standard version has been developed for
geotagging texts with a very important spatial dimension, such as descriptions of routes or
hikes [
The methods p a r s e ( ) and g e o c o d e ( ) of the G e o p a r s e r and G e o c o d e r classes, respectively, call
the geoparsing and geocoding web services of the API and return a P e r d i d o object. These are
6https://github.com/ludovicmoncla/perdido
7https://pypi.org/project/perdido/
the methods that are executed when an instance of the classes G e o p a r s e r or G e o c o d e r is used
as a function. The method p a r s e ( ) takes as parameter the text that we want to geoparser and
the method g e o c o d e takes as parameter a place name (or a list of place names) to geocode.
For disambiguation, the method c l u s t e r _ d i s a m b i g u a t i o n ( ) of the class P e r d i d o implements
a spatial density clustering (DBSCAN) [
The P e r d i d o class provides diferent attributes and methods to access the output formats and propose diferent ways of visualizing the geoparsed results. For example, the attribute t e i allows to retrieve directly the XML-TEI format returned by the geoparsing web service (see Figure 1). The method t s v _ f o r m a t ( ) of the class T o k e n allows to retrieve tokens in TSV format according to the IOB (short for inside, outside, beginning) annotation scheme8. The TSV format allows to store one token per line and for each token: its index, its form, its lemma, its part of speech and its semantic category(ies). For display purpose, the t o _ s p a c y _ d o c ( ) method is provided by the P e r d i d o class. This method transforms a P e r d i d o object into a SpaCy D o c 9 object, allowing to use the displaCy10 library for NER visualization. Two modes are possible, the first one displays only named entities (i.e. proper names) (Fig. 2a), the second one displays nested named entities (Fig. 2b). Perdido provides also the g e t _ f o l i u m _ m a p ( ) method for visualizing results on a map (Fig 3).
(a) named entities (b) nested named entities
Finally, Perdido proposes several methods to export the results of geoparsing, such as the method t o _ x m l ( ) , which saves the content of the attribute t e i in an XML file, the method t o _ g e o j s o n ( ) , which saves the content of the attribute g e o j s o n in a json file, or the method t o _ i o b ( ) , which saves the results of the annotation of named entities in TSV format according 8IOB/BIO is a common tagging format for tagging tokens in a chunking task in computational linguistics, a token is annotated B-<tag> if it is the beginning of a chunk, I-<tag> indicates that the tag is inside a chunk. An O-<tag> indicates that a token belongs to no entity/chunk. 9https://spacy.io/api/doc 10https://spacy.io/universe/project/displacy to the IOB annotation scheme. These methods take as parameter the path to which the user wants to save the files.
Two datasets are currently available in the library. The first contains 3,385 encyclopedic articles (corresponding to volume 7 of Diderot and d’Alembert’s Encyclopedia (1751-1772)), provided by ARTFL11 within the framework of the GEODE12 project. The second one contains 30 descriptions of hikes collected in the framework of the ANR CHOUCAS13 project, where each description is associated with its GPS track.
This article describes the overall architecture of the Perdido geoparsing tool and the recent development of its Python library. The library ofers two main functions: geoparsing and geocoding of French texts. However, it is still a work in progress, and several improvements are planned. One proposed improvement is the implementation of a trained model for automatic annotation of nominal entities (or unnamed entities) upstream of the existing annotation cascade. 11https://artfl-project.uchicago.edu 12https://geode-project.github.io 13http://choucas.ign.fr Another improvement being considered is the use of machine learning to train models, which will be integrated with the current approach to make it more versatile for analyzing diverse texts. Besides technical improvements, we also plan to conduct an evaluation campaign using benchmarks and our own corpora for the comparison of our approach with baselines.
Additionally, several other options are being explored for the geocoding step, such as using centroids, distances, or interpreting the spatial context extracted from the text to improve toponym disambiguation.
The authors are grateful to the ASLAN project (ANR-10-LABX-0081) of the Université de Lyon, for its financial support within the French program ”Investments for the Future” operated by the National Research Agency (ANR).