This paper provides an initial exploration of the capabilities of large language models (LLMs) to extract spatial relations from unstructured social media text. The approach examines the performance of GPT-4o and Gemini 1.5-Pro using a diverse set of spatial relation terms, and seeks to determine whether certain spatial relation types (topological, distance and direction) are more challenging to extract. To evaluate, GPT-4o and Gemini 1.5-Pro output is compared to manually labeled spatial relation triplets from Reddit place descriptions. The findings demonstrate challenges in extracting spatial relations for LLMs, with the highest model only achieving 0.48 precision. However, performance varied across spatial relation types, as direction relations were extracted with higher precision (0.75) compared to distance relations (0.62) and topological relations (0.35).
[
Extracting spatial relations from unstructured text, like Reddit, remains a challenge for established
methods, for several reasons. Spatial relation triplets may not follow a simple syntactic structure
of <subject, relation, object> (e.g.“check out the nice parts of Bradford, such as Ilkley and Saltaire.”),
terms used can be complex and contextual (e.g.‘about 10 minute drive’), and applied presuming spatial
knowledge (e.g. “Camden is just past King’s Cross.”) [
CEUR
ceur-ws.org
This study is an initial exploration into the capabilities of using LLMs to extract explicit spatial relation triplets from a manually annotated Reddit text corpus. Preliminary experiments test LLM’s ability to identify spatial relation terms across general types and sub-types. By focusing on explicit spatial relations, LLMs semantic interpretation of text is explored, not its ability to infer implicit relations through reasoning.
LLMs have already shown impressive capabilities in information extraction tasks, including NER10[]
and relation extraction 1[
There are several directions this preliminary work can be extended. So far, there have been limited attempts to extract spatial relations from unstructured tex1t8[], and from social media data. Furthermore, no study has compared LLM’s performance in extracting spatial relation terms across spatial relation types (topological, distance and direction). Finally, the use of prompts has been relatively unexplored, as previous work has only used zero-shot prompting.
Reddit data collected by Berragan et al[.19] was used in this study. The dataset consists of Reddit posts and comments from 186 UK place-based subreddits, posted between 2011 to 2022. To evaluate LLMs performance, text from the subreddit r/Leeds were manually annotated to create the test set. Triplets were labelled if they contain a subject (a reference to a place to be located), a spatial relation term and an object (a reference to a place already located), in the form <subject, relation, object>. A place was defined as any named location (e.g., ‘Leeds’, ‘Headingley’), or generic place descriptor (e.g., ‘home’, ‘city center’).
Each triplet within the test set was assigned to spatial relation general-type (topological, distance or direction) and six sub-types (see Figure1), which have been defined in [ 20]. Only containment and connectivity relations were chosen for topological spatial relations, as terms relating to other topological relations were not frequently present in the Reddit text corpus. Natural language terms were assigned to a general type and sub-type, aided by previous work21[, 22].
In total, 300 spatial relation triplets were labeled from the Reddit text corpus. A comment or post may contain multiple spatial relation triplets. It was ensured that equal amounts of triplets for each spatial relation general type and sub-type were present within the test set. An additional 600 comments or posts that did not contain spatial relation triplets (non-spatial) were included in the test set to better reflect the sparsity of such relations in the Reddit text corpus. This also enabled the evaluation of false positives, allowing assessment of whether LLMs incorrectly extracted spatial relationships that were not present in the text.
Prompts were designed to include definitions for spatial relation sub-types, and to constrain the LLM to
only extract explicit spatial relations (see AppendixA). Additionally, place names were not provided
as an input, therefore the model is required to extract them. The base prompt is used for the
zeroshot approach, which does not provide any training data. It is then adapted for two diferent prompt
approaches (AppendixB). Few-shot prompting uses examples, and chain-of-thought provides reasoning
steps to improve model performance2[
To extract the spatial relation triplets, GPT-4o and Gemini 1.5-Pro was used via OpenA2I5[] and Google AI [26] API services. The outputs were compared to the test set, and performance is evaluated using precision, recall and F1-score. A correct spatial relation triplet includes exact place names and spatial relation term found in the text, in the correct order and with the correct general type and sub-type labels. Performance scores for each spatial relation general-type and sub-type were calculated as an average across the three prompt approaches.
The results of the experiment are presented in Tabl1e. Overall, GPT-4o and Gemini 1.5-Pro exhibited moderate performance scores, with lower than expected precision. Few-shot and chain-of-thought
Containment Connectivity
Relative Direction Cardinal Direction
Quantitative Distance Qualitative Distance prompting appeared to improve performance across the models with higher recall scores. Precision and recall scores varied by spatial relation type considerably, as certain spatial relations were more challenging to extract (Figure2).
Direction relations were less challenging compared to topological and distance relations. In particular, cardinal relations were extracted with high precision (exampale). Likewise, topological connectivity relations were easily identified and captured by the models (exampleb).
Most spatial relation sub-types had lower recall compared to precision, therefore the models was confident in predictions but missed relevant cases. In particular, quantitative distance relations proved challenging, as both models frequently missed numerical spatial relation terms (exampcle). In contrast, relative direction and qualitative distance had lower precision scores, as the terms often rely on spatial references and interpretation, making them more prone to inaccuracies (examdp)l.e
Finally, containment relation proved the most challenging for GPT-4o and Gemini 1.5-Pro. Contrasting to other spatial relations, it produced very low precision scores. Examining the output, there was a high frequency of implied ‘in’ relations. Some of these were geographically correct however did not explicitly appear in the text, whilst others were presumed due to the sentence structure (examep)l.eSimilar ifndings have been reported in previous studies [18], and have been attributed to LLM’s tendency to hallucinate information that appears plausible but is not grounded in the input text.
This paper presents initial experiments using LLMs to extract explicit spatial relations from Reddit. Both GPT-4o and Gemini 1.5-Pro encountered significant challenges in this task, exhibiting numerous inaccuracies and missed cases. However, ability to extract spatial relations varied across diferent types, warranting further investigation. One possible explanation is the presence of biases in training datasets. For instance, topological and directional relations, such a“snext to” or“north of”, may be overrepresented, leading to better generalization for these types of relations. Additionally, understanding the extent to which LLMs rely on memorization versus reasoning remains an open questio2n7][. This is particularly relevant for Reddit-based data, given its reported inclusion in LLM training corpor2a8][.
Future research should explore techniques to enhance LLM performance, including the integration of geographic knowledge [17]. For topological relations, resources such as gazetteers and spatial ontologies could validate containment relationships. Retrieval-augmented generation (RAG) approaches may help mitigate hallucinations by incorporating external knowledge source2s9[].
Although the findings are specific to the manually annotated dataset, further investigation using benchmark datasets is needed. Nevertheless, this experiment establishes an initial foundation for future assessments of LLM’s capabilities in spatial relation extraction.
This research has been funded by Economic and Social Research Council (ESRC) via the White Rose Doctoral Training Partnership (WRDTP).
The authors used ChatGPT to rephrase sentences to improve clarity and conciseness. Any suggestions were reviewed and edited as needed, and the authors take full responsibility for the publication’s content. [13] K. Kopanov, Comparative Performance of Advanced NLP Models and LLMs in Multilingual GeoEntity Detection, in: Proceedings of the Cognitive Models and Artificial Intelligence Conference, ACM, İstanbul Turkiye, 2024, pp. 106–110. doi1:0.1145/3660853.3660878. [14] A. G. Cohn, R. E. Blackwell, Evaluating the Ability of Large Language Models to Reason about
Cardinal Directions, 2024. doi1:0.48550/ARXIV.2406.16528. [15] N. Fulman, A. Memduhoğlu, A. Zipf, Distortions in Judged Spatial Relations in Large Language
Models, The Professional Geographer (2024) 1–9. do1i0: .1080/00330124.2024.2372792. [16] N. Ramrakhiyani, V. Vasudeva, K. Girish, Extracting Orientation Relations between Geo-Political
Entities from their Wikipedia Text, GeoExT@ ECI (2023) 44–50. [17] L. Hu, W. Li, J. Xu, Y. Zhu, GeoEntity-type constrained knowledge graph embedding for predicting natural-language spatial relations, International Journal of Geographical Information Science (2024) 1–24. doi:10.1080/13658816.2024.2412731. [18] E. Haris, A. G. Cohn, J. G. Stell, Exploring Spatial Representations in the Historical Lake District
Texts with LLM-based Relation Extraction, 2024. doi1:0.48550/ARXIV.2406.14336. [19] C. Berragan, A. Singleton, A. Calafiore, J. Morley, Mapping cognitive place associations within the United Kingdom through online discussion on Reddit, Transactions of the Institute of British Geographers (2024) tran.12669. do1i:0.1111/tran.12669. [20] P. Kordjamshidi, M. Van Otterlo, M.-F. Moens, Spatial Role Labeling Annotation Scheme, in: N. Ide, J. Pustejovsky (Eds.), Handbook of Linguistic Annotation, Springer Netherlands, Dordrecht, 2017, pp. 1025–1052. doi:10.1007/978-94-024-0881-2_38. [21] J. Chen, A. G. Cohn, D. Liu, S. Wang, J. Ouyang, Q. Yu, A survey of qualitative spatial representations, The Knowledge Engineering Review 30 (2015) 106–136. do1i:0.1017/S0269888913000350. [22] M. Egenhofer, D. M. Mark, R. Sharif, Natural-Language Spatial Relations Between Linear and Areal Objects: The Topology and Metric of English-Language Terms, International Journal of Geographical Information Science 12 (1998) 215–246. [23] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, L. Zettlemoyer, Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?, 2022. do1i0:.48550/ARXIV. 2202.12837. [24] A. Gao, Prompt Engineering for Large Language Models, SSRN Electronic Journal (2023). d1o0.i: 2139/ssrn.4504303. [25] OpenAI, Models: Gpt-4o (128k context window), 2023. URL:https://platform.openai.com/docs/ models#gpt-4o, large language model. [26] Google, Gemini 1.5 pro: A multimodal mixture-of-experts model with extended context window, 2024. URL: https://ai.google.dev/gemini-api/docs/models#gemini-1.5-pr,olarge language model. [27] J. Roberts, T. Lüddecke, S. Das, K. Han, S. Albanie, GPT4GEO: How a Language Model Sees the World’s Geography, 2023. URL:https://arxiv.org/abs/2306.00020. doi:10.48550/ARXIV.2306. 00020. [28] H. Field, Reddit soars after announcing OpenAI deal that allows use of its data for training AI models (2023). URL: https://www.cnbc.com/2024/05/16/ reddit-soars-after-announcing-openai-deal-on-ai-training-models.html#:~:text=Reddit% 20soars%20after%20announcing%20OpenAI%20deal%20that,previously%20announced%20a% 20similar%20deal%20with%20Google. [29] W. Fan, Y. Ding, L. Ning, S. Wang, H. Li, D. Yin, T.-S. Chua, Q. Li, A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models, in: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ACM, 2024, pp. 6491–6501. URL: https://dl.acm.org/doi/10.1145/3637528.3671470. doi:10.1145/3637528.3671470.