In this work we present a solution to the Thematic Unsupervised Classification , a track presented in this year Rest-Mex competition, in which four topic-based groups are to be found from a unlabeled text item news related to tourism in Mexico. Our approach includes LDA topic modeling on a term-document representation, as well as BERT embeddings. Our BERT-based approach achieved a second place among all participating teams in the competition, demonstrating the efectiveness of mixing pre-trained models with traditional machine learning techniques.
A very common method of unsupervised learning is clustering, which aims to identify distinct
groups in data, that is, we seek to learn something about the structure and patterns inherited by
the data. On other hand, unsupervised text classification aims to perform categorization without
using annotated data during training and therefore ofer the potential to reduce annotation
costs. In this direction little research has been conducted on unsupervised text classification, see
for instance the survey [
For this task, 50,000 news items were collected on 4 diferent topics related to tourism. The challenge is to identify the groups in an automatic way. We call our approach a segmentationbased one, as detailed below.
All data was obtained from google news over the last two years regarding 4 unrevealed touristic topics, downloaded and tagged. Figure 1 shows a news example of the corpus.
As illustrated in Figure 2 we decided to homogenize the text in the following way: • Removing hyperlinks. • Removing numerical and special characters due to the large amount of numeric data that does not provide relevance to the topics, such as dates or monetary amounts, for example. • Converting the entire text to lowercase.
The first proposal is to use of Latent Dirichlet Allocation (LDA), a widely used statistical model
in NLP for unsupervised text clustering. LDA aims to discover the underlying topics in a set
of text documents and group them into coherent categories [
Topic 1 Topic 2 Topic 3 Topic 4
autoridades, fiscalia, noticias, leon, personas, elementos, seguridad, policia, dos, mas parte, asi, ciudad, ser, si, notificaciones, tambien, hace, mexico, mas lectura, anuncio, min, articulo, seguridad, foto, tiempo, mas, mexico, nacional personas, millones, tambien, gobierno, mexico, guanajuato, yucatan, pesos, mil, mas
Note that in Table 1 frequent words overlap among diferent topics identified by LDA, which can make it challenging to diferentiate them clearly. This overlap of words can be attributed to various factors, such as thematic similarity between topics or the presence of shared vocabulary. Hence, accurately discerning the boundaries and distinctive characteristics of each topic can be dificult. In our particular case, we have observed that the LDA model faces dificulties in accurately and eficiently diferentiating more specific topics due to the intersections of words among the identified topics, see Figure 3 where the overlap between clusters is shown.
For the second approach, we considered classical clustering methods: k-means and hierarchical clustering, using the same word frequency representation of Section 2.3. The resulting topic groups of k-means and hierarchical clustering can be seen in Figure 4a and Figure 4b respectively. The figures show the two-dimensional SVD representation of the text and the corresponding topics to which each news belongs. We can observe that the observations of the obtained topics, for the most part, do not overlap with each other. However this traditional approach shows a more, at least visually speaking, clear segmentation of our text data.
Contextualized word and sentence embeddings produced by pre-trained models such as BERT have demonstrated the state-of-the-art results in NLP tasks, recently Selva et al. [15] showed that using BERT versus BOW as document representation surpass in topic coherence, moreover, the use of term frequency to select topic words fails to capture the semantics of clusters precisely because of words with high frequency may be common across diferent clusters, as seen in (a) Visualization of corpus representation and clustering results obtained with K-means algorithm. (b) Visualization of corpus representation and clustering results obtained with Hierarchical Clustering
To evaluate each system in the unsupervised classification task, an alignment must first be done.
Given the Gold Standard, the output of each k system must be renumbered so that the themes
correspond. This is because the only restriction that the participating teams have is that they
must identify 4 groups with the news shared in the competition [
3rd JCMQ-Team_run_5 0.2182 HM MCE-Team_2ndIterKmeans 0.2031
As showed in Figures 4 and 4a, a term-frequency representation plus traditional clustering methods achieves a good segmentation, at least visually speaking. However as noted in Table 2, pre-trained contextualized embeddings surpass the simple term-frequency representation nonetheless in Figure 5 this segmentation is not clear enough, at least visually. We hypothesize that this might be related to the overlap between news items in terms of tourism-related keywords, such as COVID, since this created a big change in the market. We believe that more features could have been included in this approach, such is the case of better handling the scope of bert-embeddings, straightforward we passed news items through the pre-trained model all-MiniLM-L6-v2 sentence transformer [16] cutting of up to 128 items, resulting in a 384 dimensional dense vector space. Because of the little known results on this topic and being this the first time of this track in the Rest-Mex 2023 competition we wanted to try something simpler.
The authors thank Dr. Johan Johan VanHorebeek from Mathematics Research Center (CIMAT) for his support and guidance in creating this project. Delgado, R. Guerrero-Rodríguez, L. Bustio-Martínez, Overview of rest-mex at iberlef 2022: Recommendation system, sentiment analysis and covid semaphore prediction for mexican tourist texts, Procesamiento del Lenguaje Natural 69 (2022). [10] E. Olmos-Martínez, M. Á. Álvarez-Carmona, R. Aranda, A. Díaz-Pacheco, What does the media tell us about a destination? the cancun case, seen from the usa, canada, and mexico, International Journal of Tourism Cities (2023) [11] M. A. Alvarez-Carmona, R. Aranda, A. Rodriguez-Gonzalez, D. Fajardo-Delgado, M. G.
A. Sanchez, H. Perez-Espinosa, J. Martinez-Miranda, R. Guerrero-Rodriguez, L. BustioMartinez, A. D. Pacheco, Natural language processing applied to tourism research: A systematic review and future research directions, Journal of King Saud University-Computer and Information Sciences (2022). [12] Author, A.-B.: Contribution title. In: 9th International Proceedings on Proceedings, pp.
1–2. Publisher, Location (2010) [13] Haj-Yahia, Z., Sieg, A., & Deleris, L. A. (2019, July). Towards unsupervised text classification leveraging experts and word embeddings. In Proceedings of the 57th annual meeting of the Association for Computational Linguistics (pp. 371-379). [14] LNCS Homepage, http://www.springer.com/lncs. Last accessed 4 Oct 2017 [15] Selva Birunda, S., & Kanniga Devi, R. (2021). A review on word embedding techniques for text classification. Innovative Data Communication Technologies and Application: Proceedings of ICIDCA 2020, 267-281. [16] Reimers, N., & Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.