The paper outlines the motivation and build-up of the DFG-funded GEOcite project at University of Passau. The project works on a domain-specific approach to automatically extract, segment, match and visualize literature references in the German speaking geography domain with the objective to provide a novel basis for a scientometric monitoring instrument for the community. In this paper, we describe the GEOcite corpus, its construction and elaborate on a preliminary evaluation of diferent approaches to extract and segment references from the digitized part of the corpus. We further evaluate the EXCITE segmentation model [1] on diferent datasets of German research papers. The results of our evaluation show small improvements with domain-specific and increased training data.
(M. Steinbrink)
citation patterns, the disciplinary structure of German speaking geography is investigated. In
the first phase of the project (2014-2018), the citation relationships of all geographers who held
a professorship at a German, Austrian or Swiss university in 2012 were collected. The citation
data was retrieved from journal papers published by these actors in the decade from 2003 to
2012. The data collection was carried out partly automated using Scopus. Reference data from
geographical journals not listed in Scopus were manually extracted. The results of the first
project phase show that the discipline has clearly split into diferent subdisciplinary clusters
[
This paper matches with a couple of focus topics at the ULITE workshop1: GEOcite completely
builds on open source software and has the objective to produce ”Open infrastructures and
services for reference mining”; in addition, GEOcite is an application of an established software
framework for reference extraction and matching EXCITE [
As a software solution, GEO cite aims to locate, collect and process extensive historical and recent citation data (from 1949 to the present). Digital and analogue archives are used for this. Extensive digitization work is being carried out, which forms the basis for an automated citation data extraction. For this task the EXCITE tools are used (see Figure 1). Our aim is to create a database for the analysis of current structures of disciplinary knowledge networks and their historical genesis and development. GEOcite enables us to create the conditions for bibliometric network analysis to better understand the disciplinary dynamics. The data obtained are made available to the scientific community and is thus permanently available for empirical research and historical discipline observation.
Figure 1 gives an overview of the structure and the data flows of the GEO cite tool. At the center is the GEOcite database [M1]. This links three datasets [M1a, b, c] necessary for the planned bibliometric network analyses.
In GEOcite, as in the first phase of the project, the actors considered are also the geographic
professors in German speaking countries. The GEOprof -Database [actor data, M1a] contains a
list of all geography professors since 1949 as well as additional biographical attribute data [
The second dataset [M1b] is a comprehensive compilation of the bibliographic information of journal papers published by those geography professors listed in the GEOprof -Database (citing articles). To cover both historical and current publication activity, the database feeds from two diferent sources: In addition to texts listed in Scopus, the data is taken from analog journals that were digitized by the Göttingen Digitization Center (GDZ). Due to copyright legislation only the title pages and the bibliographies of a paper were scanned. The copies were provided as single files in TIFF format and needed to be processed in several steps: First, automated text recognition (OCR) was performed. Errors detected during text recognition were corrected manually. The image files were then converted into PDF format. After that the documents were merged to combine the title pages of an article and the associated bibliography into one file. The open source programs Cermine [9] and Grobid [10] were used to extract bibliographic data from the title pages, specifically authors and title of an article.
The third component depicted [M1c], represents a list of all works cited in the articles. In
addition to the complete bibliographic information of the references, the dataset also contains
the link to the actor data (GEOprof database) [M1a] and the source texts [M1b] in which they
were cited. While this bibliographic information can be queried directly in Scopus, extracting
the information of the cited works from the digitized corpus is more challenging. This is
precisely the application field of the EXCITE project, which has been running since 2016 and
is funded by the DFG [
GEOcite reuses the following EXCITE tools2 [
2. Exparser [
In the following Table 1, we outline the GEOcite corpus consisting of active male and female professors of Geography in Germany and other German speaking countries. In addition, we list the amount of considered relevant papers in Scopus and our digitized article corpus. We divide our data into bins of 20 years (1949–1968; 1969–1988; 1989–2008; 2009-2022). 1 Included are those professors who were actively holding a professorship in
the respective time interval. 2 The Scopus corpus includes only scientific papers written by relevant actors. 3 23 articles from Scopus do not include a publication date. 4 The digitalized corpus also contains articles from authors, which are not part of our research group. total 1,180 29,4843 16,052
As the first output of the GEO cite project, a comprehensive list of the geographic professors in
Germany, Austria and Switzerland since 1949 is available. The GEOprof dataset is available to
the research community as a static download (CSV format) at the geoscience data publisher [
We further created a geography-specific dataset of annotated and segmented references, extracted from scientific articles. At the end of the project, a web-based platform for bibliometric network analysis of the collected geogrphic citation data is planned. All datasets and tools are or will be available for reuse5.
2https://github.com/exciteproject/ 3https://github.com/exciteproject/EXannotator 4https://geographische-netzwerkstatt.uni-passau.de/de/geoprof/ 5https://github.com/GeoCite
In the following section, we describe a preliminary evaluation and discussion of an analysis of reference segmentation in our Geocite corpus, as well as the next steps to be taken in the project in order to optimize the segmentation process.
In order to create a comprehensive citation network between our actors, it is essential to extract
important bibliographic data with preferably low error rates. While the default Exparser models
provide suficient results, the used toolchain allows its user to train a model on a custom dataset.
To maximize the results of the segmentation of our extracted references, we trained three
models on diferent datasets to examine the efects of more domain specific articles and more
articles in general. To increase our training data for the machine learning (ML) models used by
Exparser, we extracted references from 170 German geography research papers. These articles
were randomly chosen from our digital corpus and were published between 1952 and 2019. The
EXCITE dataset7 contains 125 German articles. We further annotated and segmented these
references according to the specified EXCITE requirements [ 11]. As test set we combined 10%
of our dataset and 10% of the EXCITE German Goldstandard. Training parameters were set
identical as reported by Hosseini et al. [
Label publisher last page surname article-title
url volume source given-names
editor first page
year identifier issue other training sets combined (Combined model).
F1 GEOcite model
F1 EXCITE model
F1 Combined model
Our evaluation shows that a domain-specific dataset does not necessarily improve the output of the Exparser segmentation model. In particular we can observe that important tags for our work, like surname, given-names, and article-title achieve lower F1-scores that the EXCITE model. For several tags, we notice slightly improved results with the combined model. Our results indicate that more general data and training data from our target domain can improve the Exparser segmentation model. Considering the predominating use of English language in the scientific community, it might be no surprise that the majority of datasets in this domain were collected from English publications. This unfortunately limits the usage of large scale datasets like PMC Open Access [12] or DocBank [13].
One subject of our future research is the utilization of more sophisticated ML models. In recent years a paradigm shift for ML can be observed [ 14]. Models like BERT [15] or GPT-3 [16] trained on broad data at scale and used as foundation models show exceeding results in NLP or Computer Vision tasks. We experiment with multilingual models (e.g. XLM-R [17]) for text features and instance segmentation models (e.g. Mask R-CNN [18]) for structural features. The underlying idea is to use these available large scale datasets for language independent models and circumvent the sparsity of data in diferent languages.
With the completion of the project, we will release a dataset of our citation network, as well as all extracted citations from our corpus. Additionally, we will provide a REST API for all members of the scientific community to query data from diferent actors, corresponding citations, and various other attributes.
Similar to our GEOprof dataset, we will provide an interactive website where our citation network and research results are visualized. Furthermore our software platform GEO cite will be entirely Open Source.
Initial empirical analyses based on the GEOcite corpus are also already planned. For example, there will be further investigations on the question of the unity of geography (see above). In addition, work is planned on paradigm genesis and evolution in German speaking geography as well as specific bibliometric studies on the disadvantage of female geographers in the sense of the so called Matilda efect [ 19, 20] in the course of the discipline’s history. As another example, self-citation behavior [21] of this special community covered in the Geocite corpus can be analysed over the covered period.
This work was funded by DFG under grant 249237273, Die Säulen der Einheit und die Brücken im Fach: Geographische Forschung zwischen Rhetorik und Praxis (GEOcite) project, https://geographische-netzwerkstatt.uni-passau.de/geocite/. der geographischen ProfessorInnenschaft im deutschsprachigen Raum ab 1949, 2021. doi:1 0 . 5 8 8 0 / F I D G E O . 2 0 2 1 . 0 1 8 . [9] D. Tkaczyk, P. Szostek, M. Fedoryszak, P. J. Dendek, Ł. Bolikowski, CERMINE: automatic extraction of structured metadata from scientific literature, Int. J. Doc. Anal. Recognit. 18 (2015) 317–335. [10] Grobid, https : / / github.com / kermitt2 / grobid, 2008–2022.
a r X i v : 1 : d i r : d a b 8 6 b 2 9 6 e 3 c 3 2 1 6 e 2 2 4 1 9 6 8 f 0 d 6 3 b 6 8 e 8 2 0 9 d 3 c . [11] Excite documentation, https://exparser.readthedocs.io/en/latest/ReferenceParsing/, 2019.
[Online; accessed 1-May-2022]. [12] Pmc open access subset [internet]. bethesda (md): National library of medicine, https: //www.ncbi.nlm.nih.gov/pmc/tools/openftlist/, 2003. [Online; accessed 1-May-2022]. [13] M. Li, Y. Xu, L. Cui, S. Huang, F. Wei, Z. Li, M. Zhou, Docbank: A benchmark dataset for document layout analysis, 2020. a r X i v : 2 0 0 6 . 0 1 0 3 8 . [14] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al., On the opportunities and risks of foundation models, arXiv preprint arXiv:2108.07258 (2021). [15] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [16] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, arXiv preprint arXiv:2005.14165 (2020). [17] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale (2019). a r X i v : 1 9 1 1 . 0 2 1 1 6 . [18] K. He, G. Gkioxari, P. Dollár, R. Girshick, Mask r-cnn, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961–2969. [19] M. W. Rossiter, The matthew matilda efect in science, Social Studies of Science 23 (1993) 325 – 341. doi:1 0 . 1 1 7 7 / 0 3 0 6 3 1 2 9 3 0 2 3 0 0 2 0 0 4 . [20] P. Aufenvenne, C. Haase, F. Meixner, M. Steinbrink, Participation and communication behaviour at academic conferences – an empirical gender study at the german congress of geography 2019, Geoforum 126 (2021) 192–204. URL: https://www.sciencedirect.com/science/ article/pii/S0016718521001986. doi:h t t p s : / / d o i . o r g / 1 0 . 1 0 1 6 / j . g e o f o r u m . 2 0 2 1 . 0 7 . 0 0 2 . [21] A. Kacem, J. W. Flatt, P. Mayr, Tracking self-citations in academic publishing, Scientometrics 123 (2020) 1157–1165. doi:1 0 . 1 0 0 7 / s 1 1 1 9 2 - 0 2 0 - 0 3 4 1 3 - 9 .
• GitHub.