=Paper=
{{Paper
|id=Vol-3385/paper1
|storemode=property
|title=Evaluating the Similarity of Location-based Corpora Identified in Reddit Comments
|pdfUrl=https://ceur-ws.org/Vol-3385/paper1.pdf
|volume=Vol-3385
|authors=Cillian Berragan,Alex Singleton,Alessia Calafiore,Jeremy Morley
|dblpUrl=https://dblp.org/rec/conf/ecir/BerraganSCM23
}}
==Evaluating the Similarity of Location-based Corpora Identified in Reddit Comments==
https://ceur-ws.org/Vol-3385/paper1.pdf
Evaluating the Similarity of Location-based Corpora
Identified in Reddit Comments
Cillian Berragan1 , Alex Singleton1 , Alessia Calafiore2 and Jeremy Morley3
1
University of Liverpool, Liverpool, L69 3BX, United Kingdom
2
University of Edinburgh, Edinburgh, EH8 9YL, United Kingdom
3
Ordnance Survey, Southampton, SO16 0AS, United Kingdom
Abstract
Social interaction is typically studied from the context of physical movement, where geographic dis-
tance and ease of connectivity influence the strength of interaction between regions. From the point
of view of social media networks however, these limitations appear to still persist, despite interactions
not being reliant on physical movement, suggesting non-physical geographic characteristics influence
interaction between social communities. Unlike geotags, which provide explicit geographic information
about social media users as coordinates, unstructured text presents an alternative perspective for the
study of social interaction between regions, instead allowing for the comparison between the language
used when mentioning locations in context. Our paper analyses the corpora associated with major cities
across the UK, first vectorising Reddit comments through transformer-based embeddings, which cap-
ture semantic information, then using these to establish unsupervised clusters and similarity between
them. We find that distinct groups emerge which broadly conform with established regional identities
of locations across the UK, but with interesting deviations.
Keywords
Social media, Natural Language Processing, Social Interaction
1. Introduction
Social interaction is typically studied in the context of mobility, using data sources like Census
or transport records, where physical movement is restricted by distance and ease of connectiv-
ity between two locations [1, 2]. In contrast to this, social interaction has also been studied us-
ing phone call data [3], and social media networks [4], where the spatial and temporal bounds
of connectivity between two locations does not restrict interactions. Despite this however,
many studies have found that geographic identities within communities still persist in these
networks, with interaction strength influenced by the geographic distance between them [5, 6].
Social media also presents rich semantic information regarding locations through text as-
sociated with geotagged social media posts. Comparative analysis of corpora associated with
GeoExt 2023: First International Workshop on Geographic Information Extraction from Texts at ECIR 2023, April 2,
2023, Dublin, Ireland
£ C.Berragan@liverpool.ac.uk (C. Berragan); ucfnale@liverpool.ac.uk (A. Singleton); calafio@ed.ac.uk
(A. Calafiore); jeremy.morley@os.uk (J. Morley)
Ȉ 0000-0003-2198-2245 (C. Berragan); 0000-0002-2338-2334 (A. Singleton); 0000-0002-5953-2891 (A. Calafiore);
0000-0002-3658-8796 (J. Morley)
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons Licence Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
geotagged locations similarly exhibit regionality; for example, tweets from the North East of
England are statistically different compared with the South [5].
Our paper explores the similarity of corpora with respect to locational mentions from data
taken directly from text, without relying on geotagged metadata. This approach offers an alter-
native perspective for the analysis of social interaction, built directly from the semantic infor-
mation associated with locations, rather than the location associated with social media users
themselves. Collective semantic information from social media embeds the regional identity
of locations across a continuous spectrum, allowing for the direct comparison between these
identities and their relationships.
2. Methodology
The following section gives an overview of our data source and the data processing methodol-
ogy used in our paper. All code, analysis and data are available on our DagsHub repository.
Reddit is a public discussion, news aggregation social network, among the top 20 most vis-
ited websites in the United Kingdom. As of 2020, Reddit had around 430 million active monthly
users, comparable to the number of Twitter users [7, 8]. Reddit is divided into separate inde-
pendent subreddits each with specific topics of discussion, where users may submit posts which
each have dedicated nested conversation threads that users can add comments to. Subreddits
cover a wide range of topics, and in the interest of geography, they also act as forums for the
discussion of local places. The United Kingdom subreddit acts as a general hub for related
topics, notably including a list of smaller and more specific related subreddits. This list pro-
vides a ‘Places’ section, a collection of local British subreddits, ranging in scale from country
level (/r/England), regional (/r/thenorth, /r/Teeside), to cities (/r/Manchester) and small
towns (/r/Alnwick). In total there are 213 subreddits that relate to ‘places’ within the United
Kingdom1 . For each subreddit, every single historic comment was retrieved using the Pushshift
Reddit archive [9]. In total 8,282,331 comments were extracted, submitted by 490,535 unique
users, between 2011-01-01 and 2022-04-17.
We extracted and geolocated all place names in this collection of comments using a cus-
tom built geoparsing pipeline. To identify place names, we used a BERT transformer-based
NER model trained on the WNUT 2017 dataset [10], available on the HuggingFace Model Hub.
We then implemented a disambiguation methodology using contextual place names and two
gazetteers to geolocate place names; OS Open Names and ‘natural’ location types from the
Gazetteer of British Place Names. Processed comments consist of a collection of geolocated
place names, alongside their natural language context sentence.
2.1. Similarity of Place Corpora
Comparing the similarity between two or more distinct texts first relies on an appropriate
method for processing the text into a numerical format. For each location we obtained a corpus
of comments, consisting of sentences where each location is mentioned. These were then
processed into a single vector, reflecting the semantic information attributed with locations.
1
https://www.reddit.com/r/unitedkingdom/wiki/british_subreddits
Typically, a TF-IDF approach is used to generate document embeddings [11], however we found
comparative analysis between embeddings did not always provide insightful information. Each
vector shared similar properties, giving cosine similarities which did not result in any distinct
variation between locations. This is likely a problem with the language between locations
sharing similar properties, meaning the more nuanced semantic information is not captured
through a TF-IDF method.
We therefore extracted embeddings from a deep neural network called a transformer. Unlike
TF-IDF or simpler neural network models, transformers are able to use contextual information
to generate word embeddings, meaning the same word in two different contexts will not share
the exact same vector, capturing different embedded semantic information [12]. Additionally,
transformers are pre-trained on a large corpus of text, meaning general information regarding
the English language is already embedded within the model, allowing for improved under-
standing of semantic information. These core features mean that embeddings generated from
transformers are likely to capture information that allows for more the accurate comparative
analysis. We generated embeddings using the all-mpnet-base-v2 model from the sentence-
transformers library in Python [13]. Unlike a standard ‘BERT’-like transformer, this library
implements modifications to base models that more appropriately captures semantic informa-
tion in their output embeddings.
Before calculating embeddings we first masked every mention of a location with a generic
token ‘PLACE’, this ensured that when analysing embeddings, no explicit geographic informa-
tion was captured accidentally. For example, Manchester and Liverpool may mention matching
locations frequently in each of their comments because they are geographically close. To both
remove noise and reduce the computational requirements for this work, only locations with
over 10,000 unique mentions were kept, from these a random sample of 1,000 comments were
selected for each. Once embeddings were generated for every comment in each city corpus,
the mean for each corpus was generated, giving a vector 768 decimal values for each city.
With a single vector for each selected location, we first calculated K-Means clusters to de-
termine whether distinct groupings of locations could be identified across the UK. To visualise
these clusters we used a PCA decomposition to reduce the dimensionality from 768 down to
2 dimensions. Finally, we calculated the cosine similarity between each and every location
vector.
3. Results & Discussion
Figure 1 gives K Means clusters for transformer embeddings decomposed into two dimensions
with 𝑘 = 5. These Clusters show corpora that share similar semantic properties, however, it
is worth noting that while points that are closer together likely indicate increased similarity,
the position of these points reflect PCA decomposed values, which capture less information
compared with the clusters calculated on non-decomposed vectors. Notably London appears
as a single value in a cluster, suggesting the corpus associated with the capital of the UK is
semantically distinct from the rest of the country. There is also a single cluster associated with
the four Scottish cities considered in our study (Cluster 1), as well as a cluster for Cambridge
and Oxford (Cluster 5). Figure 1 (B) reveals that clusters do broadly appear to reflect distance-
(A) (B)
London
Manchester
Bristol
Birmingham
Brighton
Edinburgh
Liverpool
Cambridge
Glasgow
Nottingham
Leeds
Oxford Aberdeen
Sheffield
Newcastle
Dundee
1 2 3 4 5
Figure 1: Average transformer vector associated with each location corpus coloured by K Means clus-
ters where 𝐾 = 5. (A) PCA decomposed into 2 dimensions. (B) Visualised with their easting and
northing coordinates.
restricted geographic properties, while also capturing some divergences from this, with loca-
tions like London, Newcastle, Bristol and Brighton geographically distant from locations they
share clusters with.
With our high dimensional transformer embeddings we compare the cosine similarity be-
tween them on Figure 2. The highest and lowest similarity score for each location is high-
lighted in red and green respectively. As with Figure 1, corpora in Scottish cities appear to
largely share similarities, with Glasgow and Edinburgh sharing their highest similarity values.
The city with the lowest similarity to the most other locations is Oxford, which shares low val-
ues with cities in Scotland, as well as Liverpool and Manchester. London again stands out, with
overall very low similarities with all other cities, but the highest similarity with Manchester.
4. Conclusion
Our paper demonstrates the ability to compare Reddit comments relating to cities across the
UK, using document embeddings generated from a transformer neural network. Instead of
focussing on physical interactions between people or social media interactions, our work iden-
tifies relationships between cities through their semantic footprint, and analysing each corpus
computationally allows for direct comparisons between cities through clustering and cosine
similarity.
Our analysis reveals distinct clusters which largely reflect geographic proximity of locations,
however, interesting deviations from proximity do emerge. Oxford and Cambridge are both
clustered and share a high cosine similarity, but generate the lowest similarity with many other
locations in the UK, including London. London in particular appears distinct from the rest of
Aberdeen
Birmingham
Brighton
Bristol
Cambridge
Dundee
Edinburgh
Glasgow
Leeds
Liverpool
London
Manchester
Newcastle
Nottingham
Oxford
Sheffield
in n
B am
B n
br l
D ge
nb e
G gh
ow
ve s
Lo ol
an on
r
ng e
m
d
d
am sto
N este
Li ed
e
l
ee
to
or
el
ot ast
o
ha
E nd
id
ur
M nd
sg
effi
gh
rp
Le
gh
rd
xf
r i
u
c
ch
la
O
be
ew
ri
Sh
ti
di
m
A
C
ir
N
B
Figure 2: Cosine similarity between each and every location related transformer vector embedding.
Values scaled between 0 and 1. Green highlights indicate the highest value in each row, while red
indicates the lowest value in each row.
the UK, while cities that are not geographically close exhibit clustering and high similarity,
such as Liverpool and Newcastle.
The information generated through our work presents an alternative view of relationships
between cities that are not captured by existing data sources, all of which rely on explicit ge-
ographic coordinate information. Instead, we build similarities and clusters directly from the
semantic information that exists within their respective corpora. Unlike traditional data, which
captures objective social interactions between regions, the deviations from the restriction of
geographic distance between several cities in our work appears to reflect the more subjective
language that shapes the cultural and perceived identity of regions, and the relationships be-
tween them.
While our work enables the direct numerical comparison between city-based corpora, it
cannot explain the similarities and dissimilarities between them. Additional work may explore
the use of topic-modelling to identify shared topics between locations, and differences in the
sentiment towards these topics may explain dissimilarity.
References
[1] A. Rae, From spatial interaction data to spatial interaction information? Geovisualisa-
tion and spatial structures of migration from the 2001 UK census, Computers, Environ-
ment and Urban Systems 33 (2009) 161–178. doi:10.1016/j.compenvurbsys.2009.
01.007 .
[2] H. Titheridge, K. Achuthan, R. L. Mackett, J. Solomon, Assessing the extent of transport
social exclusion among the elderly, Journal of Transport and Land Use 2 (2009). doi:10.
5198/jtlu.v2i2.44 .
[3] S. Sobolevsky, M. Szell, R. Campari, T. Couronné, Z. Smoreda, C. Ratti, Delineating Geo-
graphical Regions with Networks of Human Interactions in an Extensive Set of Countries,
PLoS ONE 8 (2013) e81707. doi:10.1371/journal.pone.0081707 .
[4] B. Lengyel, A. Varga, B. Ságvári, Á. Jakobi, J. Kertész, Geographies of an Online Social
Network, PLOS ONE 10 (2015) e0137248. doi:10.1371/journal.pone.0137248 .
[5] R. Arthur, H. T. P. Williams, The human geography of Twitter: Quantifying regional
identity and inter-region communication in England and Wales, PLOS ONE 14 (2019)
e0214466. doi:10.1371/journal.pone.0214466 .
[6] C. Ratti, S. Sobolevsky, F. Calabrese, C. Andris, J. Reades, M. Martino, R. Claxton, S. H.
Strogatz, Redrawing the Map of Great Britain from a Network of Human Interactions,
PLoS ONE 5 (2010) e14248. doi:10.1371/journal.pone.0014248 .
[7] N. Murphy, Reddit’s 2019 Year in Review - Upvoted,
https://www.redditinc.com/blog/reddits-2019-year-in-review/#content, 2019.
[8] Statista, Most popular social networks worldwide as of January 2022, ranked by number of
monthly active users, https://www.statista.com/statistics/272014/global-social-networks-
ranked-by-number-of-users/, 2022.
[9] J. Baumgartner, S. Zannettou, B. Keegan, M. Squire, J. Blackburn, The Pushshift Reddit
Dataset, 2020. arXiv:arXiv:2001.08435.
[10] L. Derczynski, E. Nichols, M. van Erp, N. Limsopatham, Results of the WNUT2017 Shared
Task on Novel and Emerging Entity Recognition, in: Proceedings of the 3rd Workshop
on Noisy User-generated Text, Association for Computational Linguistics, Copenhagen,
Denmark, 2017, pp. 140–147. doi:10.18653/v1/W17- 4418 .
[11] J. Daniel, M. James H, Speech and Language Processing: An Introduction to Natural
Language Processing, Computational Linguistics, and Speech Recognition, prentice hall,
2007.
[12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polo-
sukhin, Attention Is All You Need, arXiv:1706.03762 [cs] (2017). arXiv:1706.03762.
[13] N. Reimers, I. Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-
Networks, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Lan-
guage Processing and the 9th International Joint Conference on Natural Language Pro-
cessing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China,
2019, pp. 3980–3990. doi:10.18653/v1/D19- 1410 .