Toponym resolution plays a crucial role in geoparsing. While recent approaches leveraging lightweight LLMs, such as Mistral 7B, have shown promise, they still sufer from ineficiency and suboptimal accuracy. In this work, we propose an improved method that enhances both inference speed and resolution accuracy. Instead of processing toponyms individually, our approach resolves multiple toponyms simultaneously within the same text, leveraging contextual relationships among toponyms to refine predictions while reducing inference time. Furthermore, we integrate Retrieval-Augmented Generation (RAG) to incorporate candidate locations retrieved from GeoNames during inference, providing additional geographic context and improving disambiguation. To further accelerate processing, we adopt vLLM as an optimized inference engine. Experimental results on seven public datasets with 83,365 toponyms demonstrate that our solution increases accuracy from 0.90 to 0.93 and is seven times faster than previous LLM-based methods using the same base model.
Italy ∗Corresponding author.
CEUR
ceur-ws.org
However, the LLM-based approach still has two key limitations. First, it processes each toponym individually, even when multiple toponyms appear in the same text, which may cause the LLM to overlook important contextual relationships between the toponyms. Additionally, geographic knowledge is utilized only after inference, leaving the LLM without crucial disambiguation cues in inference. Second, resolving toponyms individually requires multiple inference passes, which increases computational cost.
To address these issues, we propose a parallel inference approach that resolves multiple toponyms simultaneously, enhancing both accuracy and computational eficiency. We further integrate RAG to incorporate candidate locations for toponyms from GeoNames during inference, and utilize vLLM [21] to accelerate processing.
Our approach, illustrated in Figure 1, consists of two phases: training (fine-tuning) and inference. For training, we use the LGL dataset [22], the same as in our previous work [20], which is the LocalGlobal Lexicon (LGL) corpus. This corpus contains 588 human-annotated news articles with 5,088 toponyms from 78 local newspapers. We employ Mistral-7B-v0.2 as the base model and apply Low-Rank Adaptation (LoRA) [23] for model fine-tuning.
Figure 3 in the Appendix presents a training example. In Instruction, we specify the toponyms to be resolved. The char_index in «START:char_index» indicates the starting character index of the toponym in the text, which helps distinguish between multiple occurrences of the same name. Candidate locations for each toponym are retrieved from GeoNames and ranked in two steps: first grouped into exact (matching the primary or alternative names) and non-exact matches, then ranked by population within each group. The top candidates from the exact group are selected first; if fewer than 17 are found, the highest-ranked non-exact matches are added to complete the Top-17. They are then included in Instruction. In Input, the target toponyms are marked within the original text. The Output consists of unambiguous references for all the toponyms.
During inference, candidate locations are queried and ranked using the same strategy, and the Top-17 candidates are retained. Given the context that includes the target toponyms and their candidate locations, the model generates an unambiguous reference for each toponym. This reference consists of the toponym’s full name along with its higher-level administrative divisions (e.g., country, state, or province) necessary to uniquely identify the location. Each reference is sequentially queried in GeoNames and, if necessary, in Nominatim to obtain geographic coordinates. GeoNames is deployed locally, while Nominatim is queried online as a fallback, mainly for fine-grained locations such as points of interest. Most references are resolved by GeoNames, with only a small fraction requiring Nominatim. A caching mechanism is also employed to ensure fast query performance.
Training dataset
Fine-tuning
Toponym & context I am in <START:8> Glasgow <END>, a city in <START:27>Mont<END>.
GMloanstgow ;
Estimate unambiguous reference unraemfebreignucoeus GMloanstgaonwa,, MUSontana, US; 1. Montana, US; 2. Oblast Montana, Bulgaria; … 1. Glasgow, Scotland, UK; 2. Glasgow, Montana, US; ...
Query GeoNames
For LoRA, we set the attention dimension, scaling parameter, and dropout rate to 16, 16, and 0.1, respectively. We used the AdamW optimizer for fine-tuning with a learning rate of 0.003, over 300 epochs, and a batch size of 16. This fine-tuning process was performed on an NVIDIA Tesla V100 GPU, utilizing approximately 14 GB of GPU memory.
For testing, we employed the seven public datasets: TR-News [24], GeoWebNews [
We compared our approach with 10 representative methods, including transformer-based entity
linkers (BLINK [31], GENRE [
Figure 2 demonstrates that the new solution based on the Mistral-7B-v0.2 model improves the accuracy of the previous solution [20] using the same base model from 0.9 to 0.93. This performance matches that of the previous solution using the Llama2-70B model. Compared with non–LLM-based approaches, the new method improves the voting ensemble by 11% and GENRE, the best individual approach, by 15%. Moreover, the new solution is seven times faster than our previous implementation using the same base model, achieving performance that is comparable to—or even exceeds—that of traditional deep learning– and rule-based methods such as CHF and CamCoder.
BLINK GENRE CLAVIN
CHF CamCoder
Voting FT−Baichuan2 (7B) FT−Mistral (7B)−v0.2
FT−Mistral (7B) FT−Llama2 (70B)
Ours 0.00 0.25 0.50 Accuracy@161km 0.75 0.4
In this work, we presented an improved approach for toponym resolution based on a light-weight LLM, leveraging parallel inference, RAG, and faster inference engines to enhance both accuracy and eficiency. Our experiments on seven public datasets demonstrate that the proposed solution outperforms previous 1https://github.com/Novetta/CLAVIN LLM-based methods, achieving higher accuracy and significantly faster processing times. One limitation of the approach lies in the candidate ranking algorithm, which currently considers only string similarity and population. This may result in the correct candidate being excluded from the Top-17 list. In future work, we will propose a more robust ranking strategy that considers the spatial relationships among toponyms within the same text.
The authors employed ChatGPT and Mistral to polish the text. Following this, the manuscript underwent
a thorough review and necessary modifications by the authors, who assume complete responsibility for
the final content.
[15] Z. Yan, C. Yang, L. Hu, J. Zhao, L. Jiang, J. Gong, The integration of linguistic and geospatial
features using global context embedding for automated text geocoding, ISPRS International Journal
of Geo-Information 10 (2021) 572.
[16] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers
for language understanding, arXiv preprint arXiv:1810.04805 (2018).
[
G. Lengyel, G. Lample, L. Saulnier, et al., Mistral 7b, arXiv preprint arXiv:2310.06825 (2023). [20] X. Hu, J. Kersten, F. Klan, S. M. Farzana, Toponym resolution leveraging lightweight and opensource large language models and geo-knowledge, International Journal of Geographical Information Science (2024) 1–28. [21] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, I. Stoica, Eficient memory management for large language model serving with pagedattention, in: Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. [22] M. D. Lieberman, H. Samet, J. Sankaranarayanan, Geotagging with local lexicons to build indexes for textually-specified spatial data, in: 2010 IEEE 26th international conference on data engineering (ICDE 2010), IEEE, 2010, pp. 201–212. [23] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, Lora: Low-rank adaptation of large language models, arXiv preprint arXiv:2106.09685 (2021). [24] E. Kamalloo, D. Rafiei, A coherent unsupervised model for toponym resolution, in: Proceedings of the 2018 World Wide Web Conference, 2018, pp. 1287–1296. [25] J. O. Wallgrün, M. Karimzadeh, A. M. MacEachren, S. Pezanowski, Geocorpora: building a corpus to test and train microblog geoparsers, International Journal of Geographical Information Science 32 (2018) 1–29. [26] M. Gritta, M. T. Pilehvar, N. Limsopatham, N. Collier, What’s missing in geographical parsing?,
Language Resources and Evaluation 52 (2018) 603–623. [27] G. DeLozier, B. Wing, J. Baldridge, S. Nesbit, Creating a novel geolocation corpus from historical texts, in: Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016), 2016, pp. 188–198. [28] P. Rayson, A. Reinhold, J. Butler, C. Donaldson, I. Gregory, J. Taylor, A deeply annotated testbed for geographical text analysis: The corpus of lake district writing, in: Proceedings of the 1st ACM SIGSPATIAL Workshop on Geospatial Humanities, 2017, pp. 9–15. [29] M. C. Ardanuy, D. Beavan, K. Beelen, K. Hosseini, J. Lawrence, K. McDonough, F. Nanni, D. van Strien, D. C. Wilson, A dataset for toponym resolution in nineteenth-century english newspapers, Journal of Open Humanities Data 8 (2022). [30] M. Gritta, M. T. Pilehvar, N. Collier, A pragmatic guide to geoparsing evaluation, Language resources and evaluation 54 (2020) 683–712. [31] L. Wu, F. Petroni, M. Josifoski, S. Riedel, L. Zettlemoyer, Zero-shot entity linking with dense entity retrieval, in: EMNLP, 2020.
Instruction: Given the following toponyms, marked with unique identifiers in the text, estimate their full addresses or unambiguous references. The full address should include the toponym’s formal name followed by its higher-level administrative units, such as city, state/province, and country. Input Toponyms: ‘Hurricane’ marked with «START:23»«END»; ‘Hurricane’ marked with «START:372»«END»; ‘Putnam County’ marked with «START:398»«END» Reference Information: Below are candidate addresses for each toponym. While these are valid options, the correct address might not be included among them. Use your geographic knowledge, along with these candidates, to infer the most accurate address.
Partial candidate locations of ‘Hurricane’: 1. Hurricane, Washington County, Utah, United States; 2. Hurricane, Putnam County, West Virginia, United States; ... 17. Lac Hurricane, Mauricie, Quebec, Canada; Input: Sherif’s deputies and «START:23» Hurricane «END» police found the two.‘The guys were down in the area for about 10 minutes when they found them near the tracks,’ the dispatcher said. Neither of the teenagers were hurt and both are safe, the dispatcher said. Keya Phillips and Seth Pettry, both 14, were found Monday after a tip came in that the two were hiding under a railroad bridge in «START:372» Hurricane «END» , according to a «START:398» Putnam County «END» dispatcher.
Output: («START:23»Hurricane, Putnam County, West Virginia, United States «END») («START:372»Hurricane, Putnam County, West Virginia, United States «END») («START:398»Putnam County, West Virginia, United States «END»)