Reddit has been an important source of news and information exchange in the past two decades. This social media platform is composed of communities forming spaces where users with common interests can share their experiences and opinions. Many of these online communities, called 'subreddits', are highly active, with some subreddits having tens of millions of followers. The extraction of geographic information from social media text data has been an increasing topic of research in the last years. While most of the work has been done on Twitter data, recent restrictions to its Application Programming Interface (API) have limited the access for academic research. Reddit.com could be a good alternative source due to its depth of discussions and communal interactions. Additionally, it allows more anonymity to users, thereby rendering it more ethically acceptable as it reduces users' traceability. However, this characteristic complicates the process of objective verification and evaluation. Nevertheless, we believe that Reddit is a rich source of information that could be used for training models for geolocation extraction. This paper presents a few first such experiments of extracting geographic data from location-based communities on Reddit and classifying posts on a city-level using BERT for text classification.
CEUR ceur-ws.org
around two billion monthly visits to Reddit, and more than 130,000 active communities, or
subreddits [
The topic of geographic information extraction from text is challenging due to considerable
linguistic variability inherent in such data and its complex relations within a geospatial
framework. A large part of the research was focused on Named Entity Recognition (NER), which is
the categorization of important entities within unstructured text, specifically location related
entities [
The authors in [
Since 2023, Twitter API is not afordable for academic research. In addition, Twitter reduced Tweets with precise geolocation to less than 0.2% in 2021 [27] and many of the user-set place names and geolocations on Twitter were proven unreliable [28]. Therefore, research needs to be extended beyond those labels and platform. Reddit is one of the strong alternative candidates with a freely available API. To our knowledge, research on geoparsing using Reddit data was limited due to the lack of ground-truth data. Harrigian [29] attempted to classify Reddit user geolocation using subreddit metadata and extracting localized data via queries for questions such as ’where do you live?’. The results showed that models trained on Reddit data significantly outperformed transferred models trained on Tweets.
We used a subset of the Reddit Data Dump dataset from Academic Torrents1, which was collected using the Pushshift API [ 30]. Namely we used posts from the last quarter of 2022 for the training and validation, and tested the model on data from February 2023 to avoid immediate temporal influences. Subreddits of large cities by population in the USA and in Germany were chosen manually for the classification task (the full list is available under https://github.com/Milast/GeoExt_Reddit). Overall there were 181 subreddits from 126 cities that included the names of cities such as ’/rBerlin’2, as well as other location-related subreddits such as ’/rChicagoFoods’, ’/rportlandmusic’, etc., which were aggregated under the relevant city name. To train the model, only cities with over 100 posts during the mentioned time frame were included in the experiments, resulting in 57 cities being selected.
A pre-trained BERT model from Hugging Face [31] was used for the text classification. We chose the multilingual bert-base-multilingual-cased model, due to the ability of the case sensitive model to deal with the capitalization of nouns in the German language. The title and content of Reddit posts were combined and used for the classification. The code from the tutorial of Winastwan [32] was modified to fit the task. The model comprises a BERT tokenizer with a maximal text length of 512, which generated the input into the BERT transformer model with 12 layers of transformer encoder and hidden size of 768. A classification layer was used to discriminate between the cities. The fine-tuned model was then evaluated on the test data.
First, a country-level classification was performed and achieved 98.2% accuracy (see Appendix C). This result was somewhat expected due to the nature of BERT embeddings, providing higher similarity between languages than semantic similarity [33]. Since the focus was on two countries with a large language diference (US and Germany), we then conducted a non-hierarchical city-level classification, assuming that country level diferences would be captured automatically. After the city-level classification of the original text, we used the SpaCy library in Python [ 34] for the NER to keep posts that included Named Entities (NEs) that could imply, or infer, a 1www.academictorrents.com 2The notation ’/r’ indicates a subreddit on the platform. location, such as organizations, events, etc. The NER tags used and their meanings can be found in the appendix A. After seeing an improvement of 0.08 in the overall F1-score, we pursued to further filtering the posts to keep those with NEs related specifically to a geographic location, namely ’GPE’ (Countries, cities, states) and ’LOC’ (Non-GPE locations, mountain ranges, bodies of water). The results are summarized in table 1. There was a significant improvement in the classification after filtering the posts for NEs that could potentially infer a location, and another improvement after filtering for location specific NEs. It is important to mention that the size of the dataset changed after filtering, which might have had an impact on the results. The size of the training and testing data, and the full results per city for the three experiments were added in appendixB and appendixC respectively. Some communities had more accurate classification results in all of three experiments, namely large German cities such as Munich, Berlin and Hamburg, as well as Las Vegas and Kanzas City achieving up to 70-80% accuracy while other cities only around 50%. Interestingly, both Las Vegas and Kanzas City included an aggregated subreddit ’<city_name>_r4r’, which stands for Redditors for Redditors and facilitates connections between users. These subreddits improved the classification results to a larger extent than other topics such as food or music, and could be studied further in the future.
In this paper, we presented the initial results for geoparsing experiments on recent Reddit data. The analysis was done in a statistical manner without attempting to locate specific users, adhering to ethical recommendations. We performed these experiments using BERT classification on country and city levels. For countries, the classification worked near-perfectly due to the influence of language. For cities, we also included experiments where NER was used to detect posts that contain either location-inferring NEs, or location-specific NEs, showing an improving classification accuracy. There are many influences that should be analyzed next such as linguistic ambiguities, influences of specific NER tags and aggregated subreddits, and extension to other countries. Geoparsing using Large Language Models (LLMs) could also be compared to supervised classification, as well as utilizing KGs constructed from Gazeteers and other sources, with or without the assistance of LLMs. Other future work could be on hierarchical classification, using additional training data and diferent BERT models, and examining user networks in the localized communities. Although working with Reddit data is challenging due to the anonymous nature of the platform and the lack of explicit geolocation information, our experiments demonstrate an interesting basis for further research on geoparsing. SBP-BRiMS 2017, Washington, DC, USA, July 5-8, 2017, Proceedings 10, Springer, 2017, pp. 281–291. [16] T. Miyazaki, A. Rahimi, T. Cohn, T. Baldwin, Twitter geolocation using knowledge-based methods, in: Proceedings of the 2018 EMNLP Workshop W-NUT: The 4th Workshop on Noisy User-generated Text, 2018, pp. 7–16. [17] F. Lovera, Y. Cardinale, D. Buscaldi, T. Charnois, A knowledge graph-based method for the geolocation of tweets, in: Workshop Proceedings of the 19th International Conference on Intelligent Environments (IE2023), IOS Press, 2023, pp. 53–62. [18] B. Wing, J. Baldridge, Hierarchical discriminative classification for text-based geolocation, in: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 336–348. [19] B. Huang, K. M. Carley, A hierarchical location prediction neural network for twitter user geolocation, arXiv preprint arXiv:1910.12941 (2019). [20] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [21] L. F. Simanjuntak, R. Mahendra, E. Yulianti, We know you are living in bali: Location prediction of twitter users using bert language model, Big Data and Cognitive Computing 6 (2022) 77. [22] I. Salogni, Salogni at geolingit: Geolocalization by fine-tuning bert, in: Proceedings of the Eighth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian.
Final Workshop (EVALITA 2023), CEUR. org, Parma, Italy, 2023. [23] K. Lutsai, C. H. Lampert, Geolocation predicting of tweets using bert-based models, arXiv preprint arXiv:2303.07865 (2023). [24] Y. Scherrer, N. Ljubešić, Helju@ vardial 2020: Social media variety geolocation with bert models, in: Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects, 2020, pp. 202–211. [25] X. Hu, Z. Zhou, J. Kersten, M. Wiegmann, F. Klan, Gazpne2: A general and annotation-free place name extractor for microblogs fusing gazetteers and transformer models (2021). [26] OpenStreetMap contributors, Planet dump retrieved from https://planet.osm.org , https: //www.openstreetmap.org, 2017. [27] A. Kruspe, M. Häberle, E. J. Hofmann, S. Rode-Hasinger, K. Abdulahhad, X. X. Zhu, Changes in twitter geolocations: Insights and suggestions for future usage, arXiv preprint arXiv:2108.12251 (2021). [28] A. Kumar, J. P. Singh, N. P. Rana, Authenticity of geo-location and place name in tweets (2017). [29] K. Harrigian, Geocoding without geotags: A text-based approach for reddit. arxiv, 2018. [30] J. Baumgartner, S. Zannettou, B. Keegan, M. Squire, J. Blackburn, The pushshift reddit dataset, in: Proceedings of the international AAAI conference on web and social media, volume 14, 2020, pp. 830–839. [31] H. Face, huggingface.co, 2024. URL: https://huggingface.co/bert-base-multilingual-cased. [32] R. Winastwan, Text classification with bert in pytorch, 2021. URL: https:// towardsdatascience.com/text-classification-with-bert-in-pytorch-887965e5820f. [33] J. Libovickỳ, R. Rosa, A. Fraser, How language-neutral is multilingual bert?, arXiv preprint arXiv:1911.03310 (2019). [34] M. Honnibal, I. Montani, spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing, 2017. To appear.
Original text (and Country-level) Text filtered for location inferable NER Text filtered for location specific NER