In this study, we investigate the use of mid-sized and open-source large language models to enhance the extraction of geographic information from texts, focusing on toponym resolution. Our approach involves ifne-tuning Llama2 (7B) to accurately derive the unambiguous references of toponyms within textual contexts and subsequently assign geo-coordinates using geocoders. The method is evaluated on two challenging datasets featuring 28,342 global toponyms. The findings demonstrate notable performance improvements compared to existing state-of-the-art methods while maintaining computational eficiency.
CEUR ceur-ws.org
Specifically, we fine-tuned Llama2 (7B) [
10], an open-source and powerful model in language comprehension and inference, to estimate toponyms’ unambiguous references, followed by their conversion to geographical coordinates using free geocoders. Our approach demonstrates greater eficacy than seveal leading methods across two challenging datasets. Besides, the approach is computationally eficient, requiring about 14 GB of memory for operation on a standard GPU. CEUR Workshop Proceedings
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Our approach, depicted in Figure 1, involves two phases: training (fine-tuning) and geocoding. Initially, we fine-tune Llama2 using Low-Rank Adaptation (LoRA) [ 11], a technique that optimizes GPU resource usage, to predict the unambiguous references (e.g., city, state, county) of toponyms based on their context. Our training dataset is the LGL1 (Local-Global Lexicon) corpus, developed by Lieberman et al. [12], comprising 588 human-annotated news articles with 5088 toponyms from 78 local newspapers. In the geocoding phase, the fine-tuned model ifrst deduces the unambiguous reference of toponyms from their contextual cues. It is then fed into a sequence of free geocoders—primarily GeoNames2, followed by Nominatim3 and ArcGIS4. This sequential querying strategy is designed to consult the next geocoder if one fails, enhancing the reliability and precision of the geocoding process.
Training dataset
LGL
Fine-tuning Toponym & context I am in <START> Glasgow <END>, a city in Mont.
unrEaemsfetbirmeignauctoeeus uGnlaasmgboigwu,oMuosnrteafneare,nUcSe geQoucoedryers 48.1C9o6o9rd6i,n -a1t0e6s.63671
For LoRA, the attention dimension, the scaling parameter (ℎ ), and the dropout rate are set to 8, 16, and 0.1, respectively. We employed the AdamW optimizer for fine-tuning with a learning rate of 0.003, over 300 epochs, and a batch size of 128. This fine-tuning process was executed on an NVIDIA Tesla V100 GPU, utilizing about 14 GB of GPU memory.
For testing, we used two public datasets, detailed in Table 1. The geographical distribution of the toponyms in the test dataset is shown in Figure 2. Our evaluation employed two metrics [13]: Accuracy@161km for geocoding precision within 161 km (100 miles), and Mean Error (ME) for average distance error. 1https://github.com/milangritta/Pragmatic-Guide-to-Geoparsing-Evaluation/blob/master/data/Corpora/lgl.xml 2https://www.geonames.org/ 3https://nominatim.org/ 4https://developers.arcgis.com/documentation/mapping-apis-and-services/geocoding/
We compared our approach with 10 representative methods. These include a Voting system [7], CamCoder [14], CHF [15], Clavin5, Blink [16], GENRE [17], Bootleg [18], and the three standard geocoders: Nominatim, GeoNames, and ArcGIS. Among these, CamCoder is a deep learning-based geoparser; CHF and Clavin are rule-based; and Blink, GENRE, and Bootleg are deep learning-based entity linkers. The Voting system integrates seven methods, such as GENRE, Blink, and CamCoder.
The outcomes of our evaluation are presented in Table 2. The results show that our approach outperforms others. On average, it exceeds the performance of the previously best method, the voting system, by 7% in Accuracy@161km and 61% in ME. Compared to the top individual method, GENRE, our approach demonstrates more substantial improvements of 13% in Accuracy@161km and 83% in ME. These findings underscore the efectiveness of our proposed approach.
5https://github.com/Novetta/CLAVIN This research presents an innovative method for toponym resolution utilizing mid-sized, opensource large language models, specifically Llama2 (7B). Its eficiency is validated through testing on two public datasets, establishing a new standard in the field. Furthermore, it maintains significant computational eficiency with a reasonable GPU memory requirement of 14 GB. Future research will aim to investigate a broader range of open-source LLMs for this task and conduct extensive comparative analyses with existing methods across a more diverse array of test datasets. Furthermore, eforts will be directed towards augmenting the models’ geographical knowledge during the inference process by incorporating a toponym’ candidates retrieved from gazetteers, aiming to enhance accuracy and performance further.
The authors employed ChatGPT to polish the language. Following this, the manuscript underwent a thorough review and necessary modifications by the authors, who assume complete responsibility for the final content. synthetic data by rules, International Journal of Geographical Information Science (2021) 1–28. doi:10.1080/13658816.2021.1947507. [5] X. Hu, Z. Zhou, Y. Sun, J. Kersten, F. Klan, H. Fan, M. Wiegmann, Gazpne2: A general place name extractor for microblogs fusing gazetteers and pretrained transformer models, IEEE Internet of Things Journal (2022) 1–1. doi:10.1109/JIOT.2022.3150967. [6] X. Hu, Z. Zhou, H. Li, Y. Hu, F. Gu, J. Kersten, H. Fan, F. Klan, Location reference recognition from texts: A survey and comparison, ACM Computing Surveys 56 (2023) 1–37. [7] X. Hu, Y. Sun, J. Kersten, Z. Zhou, F. Klan, H. Fan, How can voting mechanisms improve the robustness and generalizability of toponym disambiguation?, International Journal of Applied Earth Observation and Geoinformation 117 (2023) 103191. [8] G. Mai, C. Cundy, K. Choi, Y. Hu, N. Lao, S. Ermon, Towards a foundation model for geospatial artificial intelligence (vision paper), in: Proceedings of the 30th International Conference on Advances in Geographic Information Systems, 2022, pp. 1–4. [9] Y. Hu, G. Mai, C. Cundy, K. Choi, N. Lao, W. Liu, G. Lakhanpal, R. Z. Zhou, K. Joseph, Geo-knowledge-guided gpt models improve the extraction of location descriptions from disaster-related social media messages, International Journal of Geographical Information Science 37 (2023) 2289–2318. [10] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al., Llama 2: Open foundation and fine-tuned chat models, arXiv preprint arXiv:2307.09288 (2023). [11] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, Lora: Low-rank adaptation of large language models, arXiv preprint arXiv:2106.09685 (2021). [12] M. D. Lieberman, H. Samet, J. Sankaranarayanan, Geotagging with local lexicons to build indexes for textually-specified spatial data, in: 2010 IEEE 26th international conference on data engineering (ICDE 2010), IEEE, 2010, pp. 201–212. [13] M. Gritta, M. T. Pilehvar, N. Collier, A pragmatic guide to geoparsing evaluation, Language resources and evaluation 54 (2020) 683–712. [14] M. Gritta, M. Pilehvar, N. Collier, Which melbourne? augmenting geocoding with maps (2018). [15] E. Kamalloo, D. Rafiei, A coherent unsupervised model for toponym resolution, in:
Proceedings of the 2018 World Wide Web Conference, 2018, pp. 1287–1296. [16] L. Wu, F. Petroni, M. Josifoski, S. Riedel, L. Zettlemoyer, Zero-shot entity linking with dense entity retrieval, in: EMNLP, 2020. [17] N. De Cao, G. Izacard, S. Riedel, F. Petroni, Autoregressive entity retrieval, in: International Conference on Learning Representations, 2021. URL: https://openreview.net/forum?id= 5k8F6UU39V. [18] L. Orr, M. Leszczynski, S. Arora, S. Wu, N. Guha, X. Ling, C. Re, Bootleg: Chasing the tail with self-supervised named entity disambiguation, arXiv preprint arXiv:2010.10363 (2020). [19] J. O. Wallgrün, M. Karimzadeh, A. M. MacEachren, S. Pezanowski, Geocorpora: building a corpus to test and train microblog geoparsers, International Journal of Geographical Information Science 32 (2018) 1–29. [20] M. Gritta, M. T. Pilehvar, N. Limsopatham, N. Collier, What’s missing in geographical parsing?, Language Resources and Evaluation 52 (2018) 603–623.