We report our initial findings from an examination of potential systematic bias in Large Language Models' (LLM) spatial reasoning capabilities. We devised a series of questions to probe the spatial reasoning abilities of four LLMs: GPT-3.5, GPT-4, Gemini, and Llama-2, targeting four specific biases rooted in human spatial perception: hierarchical, proximity and directional biases. The questions encompassed scenarios challenging the models' spatial reasoning, and each question was posed 10 times independently to gauge the consistency of the LLMs' responses. The models demonstrated a strong understanding of straightforward geographical relationships, achieving 87% accuracy in questions that did not challenge biases in spatial reasoning. However, when faced with questions highlighting these biases, the models' accuracy dropped to 24%. We discuss the design of a large-scale experiment aimed at examining spatial cognition biases in large language models and identifying potential mitigation strategies.
CEUR ceur-ws.org
Recent studies primarily view Large Language Models (LLMs) in geography as tools linking
natural language to geographic information systems [
We investigate the possibility that biases in human spatial reasoning may manifest in LLMs,
focusing on four well-studied ones: (a) Hierarchical bias refers to the cognitive tendency to
infer the direction between two points based on the dominant geographical orientation of their
∗Corresponding author.
†These authors contributed equally.
larger categorical groups (such as states or regions), leading to inaccuracies when exceptions
to these general orientations exist [
LLMs rely on associative learning and contextual data processing to understand and generate
human-like text [
To investigate hierarchical bias in large language models (LLMs), we initially conducted a
study with ten questions, five challenging questions where bias is likely to be exhibited based
on scenarios where humans typically struggle, and five control questions to serve as a baseline
for comparison. This study included four models: GPT-3.5 [
The outcomes for hierarchical bias (a) are illustrated in Table 1. The models were instructed to determine intercardinal directions between cities, using the prompt: ’What is the intercardinal direction from [City A] to [City B]?’ For example, all models consistently (0/10) provided inaccurate directions between Portland and Toronto. We attribute the error to the general northward alignment of Canada relative to the United States (Figure 1a). This observation is
Type l a c i h c r a r e i H
San Antonio TX to Houston TX
in line with the findings of Stevens and Coupe [
GPT-4 demonstrates the highest accuracy in this assessment, achieving a 75% success rate, followed by Gemini with 55%, GPT-3.5 with 53%, and LLaMA-2 at 47%. In scenarios designed to highlight hierarchical bias, GPT-4 distinctly outperforms its counterparts, registering a 50% accuracy rate. In comparison, Gemini scores 34%, GPT-3.5 26%, and LLaMA-2 only 10%. However, when evaluating tasks absent of suspected hierarchical bias, all models exhibit improved performance, with accuracy rates exceeding 75%. The remainder of this study will focus on GPT-4 to explore further biases.
The outcomes for biases (b) through (d) are presented in Table 2. To demonstrate proximity bias (b), GPT-4 was tasked with evaluating the relative distances between cities, employing the query: ’Which is closer to [City X]: [City A] or [City B]?’ For instance, despite New Haven, Connecticut being closer to Philadelphia, Pennsylvania by both road distance (∼250km) and great circle measurements, the model consistently determined that Pittsburgh, Pennsylvania is the closer city (∼450km) (0/10) (Figure 1b). However, when New Haven is replaced with Johnstown, which is ∼390km from Philadelphia in Pennsylvania, the model consistently gives the correct answer (10/10). This possibly reflects a bias of perceiving distances within the state as shorter than across states.
In examining the rotation bias (c), the model was asked: ’Which city is further west, [City A] or [City B]?’ For instance, when inquiring which city is further west between Wilmington, North Carolina and Jacksonville, North Carolina, the model mistakenly (2/10) pointed to the latter, possibly reflecting a simplification of the US east coast curvature (Figure 1c). However, it correctly identified the relative westward position when comparing Wilmington to Morehead City, North Carolina, both being coastal cities, possibly suggesting that the presence of a common geographical feature, forces more precise comparisons (10/10).
We report our initial findings from an examination of potential systematic bias in the spatial reasoning capabilities of GPT-3.5, GPT-4, Gemini, and Llama-2. The models show distinct patterns in their performance: They achieve 87% accuracy in questions that do not challenge biases in spatial reasoning, indicating a strong understanding of straightforward geographical relationships. On the other hand, they only achieved a 24% accuracy rate in the questions which highlight these biases.
Our current study draws from the psychology experiments that served as our inspiration; however, it does not ofer a statistically valid assessment of how biases in spatial perception impact LLMs. To address this limitation, our future exploration will focus on the proximity bias, denoted as (b), which relates to the tendency to underestimate distances within categories while overestimating distances between them. This analysis will involve querying the models with hundreds of relevant cities and examining their interrelationships.
While our approach may allow us to verify the existence of biases in LLM’s spatial reasoning skills, we may not be able to pinpoint the source of these biases – whether they originate from learned human errors, generalized geographical input data, or the models’ inherent tendencies towards conceptual associations. Nevertheless, we may be able to mitigate these issues. One potential strategy involves training LLMs with datasets explicitly detailing spatial relationships between various locations. Utilizing Natural Language Geographic Data for this purpose could usher in a deliberate development of spatial reasoning skills within these models, enabling them to more accurately comprehend and process geographic relationships. In the next phase of our research, we plan to explore methods for fine-tuning an open-source LLM using such spatially explicit datasets, evaluating its ability to discern intercardinal directions and ultimately enhance its spatial reasoning capabilities.
N. Fulman was supported by the Health + Life Science Alliance Heidelberg Mannheim and received state funds approved by the State Parliament of Baden-Württemberg. A. Memduhoğlu was supported by the Scientific and Technological Research Council of Türkiye (TUBITAK) under the program 2219 (1059B192202917).