This paper presents our participation in the CLEF2023 CheckThat! Lab [1], focusing on Task 5, which addresses Authority Finding in Twitter [2]. This study addresses the gap in rumor verification by exploring the use of Bio and Lists information from trusted authorities' bios as evidence sources in social media. A Twitter Bio is a brief description or introduction about oneself or an organization displayed on a user's profile, providing a snapshot of their identity, interests, or purpose. Twitter Lists are curated groups of Twitter accounts created by users to organize and categorize specific individuals or topics for easy viewing and engagement. Previous research has primarily focused on propagation networks and user-generated content. The findings highlight the potential of incorporating Bio and Lists information to enhance existing rumor verification systems. Additionally, the study evaluates diferent retrieval approaches, showing that combining Bio and Lists information leads to efective lexical retrieval. A hybrid approach combining lexical and semantic retrieval further improves performance. These findings contribute to advancing rumor verification methods in social media.
Previous research on rumor verification in social media has primarily focused on using
propagation networks as a source of evidence, examining reply stances [
The task of expert finding, which aims to identify individuals with relevant expertise in a
particular domain, has gained significant attention in recent years [
Previous research in this area has often focused on incorporating spatial information to
address the expert-finding problem. For example, McDonald et al.[
Pal and Counts [13] proposed a probabilistic clustering framework that utilized nodal and topical features to identify authoritative experts in a given topic. Ghosh et al.[14] developed the Cognos system, which utilized Twitter Lists to determine user expertise and demonstrated comparable performance to Twitter’s oficial system in identifying top users for specific topics. Additionally, Wei et al.[15] investigated the use of multiple types of relationships on Twitter to identify experts associated with a particular topic. Studies, such as the one referenced in [16], integrate user-related Twitter data to improve expert finding outcomes. They employ a semi-supervised graph ranking technique to rank expertise levels.
The subsequent sections of this paper are structured as follows: Section 2 provides an overview of the data employed in this study, Section 3 elucidates the ranking metrics to enhance the understanding of the results, Section 4 presents the proposed methodology, Section 5 ofers the experimental results and in-depth discussions, and finally, Section 6 concludes the study.
In this section, we present the Authority Finder in Twitter dataset provided as part of the
CheckThat! 2023 competition [
Given a tweet stating a rumor, we retrieve a ranked list of authorities that may help verify the rumor. The dataset is available in Arabic only, and it’s composed of the following: • Rumors: The data is in a JSON format. It contains JSON objects representing rumors.
Figure 1 illustrates the diferent entries for each rumor. • Relevance Judgments (Qrels): The file is a TAB-separated file in TREC format. Each rumor ID is associated with user IDs of the authorities. The file is in the following format: – rumor_id: Unique ID for the given rumor – 0: Literally 0 (this column is needed to comply with the TREC format).
– user_id: Unique ID for the given authority. – relevance: 2 if the authority is highly relevant to the rumor (has higher priority to be contacted); 1 if it is relevant; 0 is assumed for all pairs not appearing in the qrels ifle. • Users Metadata: It has a collection of 1000 JSON files. Figure 2 represents the format of each file. • Twitter Lists Metadata: a collection of 1000 JSON files with Twitter Lists information (see
Figure 3 for the format)
P@1[17], P@5[18] and nDCG@5[19] are evaluation metrics commonly used in information retrieval and recommendation systems to measure the quality and relevance of ranked Lists. The oficial evaluation measure is P@5 to evaluate how systems are able to retrieve authorities at the top of a short retrieved list.
• Precision at K (P@K): Precision at K measures the proportion of relevant items among the top K items in a ranked list.
– TP (True Positives): The number of relevant items in the top K items.
– K: The value of K for which you want to calculate precision. • Normalized Discounted Cumulative Gain at K (nDCG@K): nDCG is a measure that considers both the relevance and the position of items in a ranked list. It gives higher scores to relevant items that appear at the top of the list.
Where: Where: (1) (2) (3) = ∑︁
=1 log2( + 1) – DCG (Discounted Cumulative Gain): The sum of the relevance scores of the top K items, with decreasing weights, calculated as follows. – IDCG (Ideal Discounted Cumulative Gain): The maximum achievable DCG for the given list.
– K: The value of K for which you want to calculate nDCG.
Our methodology consists of several steps inspired by [20], including preprocessing, creating user documents, indexing and retrieval, and incorporating lexical and semantic scoring for ranking users. Below is a description of each step in the process:
We used Pyserini1, a Python toolkit for information retrieval, to index our user collection for lexical retrieval. We experimented using diferent values of BM25 [21] parameters. We set the language to Arabic and performed normalization on both documents and queries. The Lucene Arabic analyzer was employed for stemming and removing stop words. As our queries (rumors) are expressed in tweets, we preprocessed the data by discarding URLs, emojis, and non-Arabic characters. We considered three types of user representation indexes: • Bio_index: each user is represented by their translated Twitter profile name and description. • Lists_index: each user is represented by concatenating their translated Twitter list names and descriptions. • Bio_lists_index: each user is represented by concatenating their translated Twitter profile name and description with their translated Twitter Lists.
We employed the BM25 [21] retrieval model to compute the lexical score for a candidate user given a rumor , considering only the textual representation of users. The BM25 algorithm calculates a term-weighted score for a given document-query pair as follows: 25(, ) = ∑︁ () ·
(1 + 1) · (, ) 1 · ((1 − ) + · || ) + (, ) ∈∩ where and are the textual representations of the user and rumor , respectively, (, ) is the frequency of term in the user document , || is the length of the user document, is the average document length in the collection, and () is the inverse document frequency of term . The parameters 1 and control the term frequency saturation and field-length normalization, respectively.
Assuming authoritative users have more followers and are involved in more Twitter Lists than regular users. We computed the initial score (, ) by incorporating the number of Twitter Lists, followers, and users as follows: (, ) = (, ) × 2[( + 2)( + 2)] 1https://github.com/castorini/pyserini (4) (5) where is the number of Twitter Lists the candidate is a member of, is the number of followers the candidate has, and is the number of users the candidate is following. We added 2 when computing the logarithm of both factors for smoothing in case or is zero. In any of those cases, the initial score will fall back to the lexical score.
Contextualized transformer-based models such as BERT that are pre-trained on large corpora have shown superiority in document reranking. Given that, we also employed a semantic reranking approach based on BERT to re-rank the initial retrieved users, using the same technique as [20] for the generation of input data. Practically we used three pretrained Arabic models: • AraBERT [22], a BERT-based language model, is optimized for Arabic NLP tasks, demonstrating efectiveness in sentiment analysis, and text classification. • ArBERT [23] is a large-scale pre-trained masked language model for Modern Standard
Arabic, based on BERT-base architecture, with 163 million parameters. • MARBERT [23] is an Arabic language model pre-trained on a diverse Twitter dataset specifically designed to handle Arabic dialect variations.
As the initial score incorporates user profile features like the count of Twitter Lists, followers,
and followees, which are crucial in measuring user popularity, we adopted an existing hybrid
approach [20], that combines both the initial and semantic scores to compute a final score of
candidate users as follows:
ℎ(, ) = × ˆ(, ) + (1 − ) × (, ),
(6)
where ∈ [
The results presented in this study were obtained using the Development Dataset prior to the release of the Test dataset, with the objective of identifying the most appropriate configuration to be employed during the evaluation phase. This pre-release assessment aimed to discern the optimal setup for subsequent evaluations and ensure the reliability and validity of the findings.
The following tables (2 3 4 5) show the results of the diferent previously mentioned steps of ranking.
Table 2 (Lexical Retrieval): The table compares the performance of three diferent index
approaches (Bio Index, Lists Index, and Bio Lists Index) using various combinations of the
parameters k1 and b. We used Gready-Search to evaluate the optimal configuration of BM25,
where k1 ∈ [
In Table 3, we present the results of the initial retrieval using the Lists and Bio Lists indexes, where we adopted the optimal BM25 parameter settings (k1=3 and b=0.1). The Bio Lists Index yielded better results, with P@1=0.3500, P@5=0.1667, and nDCG@5=0.2345, further demonstrating the benefit of combining Bio and Lists information and it shows the importance of using user network features.
In Table 4, we present the semantic reranking results using three Arabic-language BERT models (AraBERT, ArBERT, and MARBERT) with the initial retrieval results from Table 2. The initial retrieval outperformed all the semantic models in terms of P@1, P@5, and nDCG@5, suggesting that the lexical approach is more efective than the semantic approach in this case.
The hybrid retrieval results of reranking the top 100 candidate users from the initial retrieval (Bio+Lists) by interpolating initial and semantic scores are presented in Table 5. MARBERT performed the best among the semantic models, with P@1=0.3445, P@5=0.1835, and nDCG@5=0.2427, demonstrating that the hybrid approach combining lexical and semantic retrieval can lead to better performance than using either method alone.
During the evaluation cycle, an attempt was made to employ Hybrid reranking with MARBERT. However, the limited availability of computational resources hindered the production of the necessary results within the specified evaluation cycle deadline. As a result, the submission solely consisted of the results obtained using Initial Retrieval with the optimal configuration of BM25.
This study evaluates diferent retrieval approaches for authority finding, where each user is represented by their translated Twitter profile name and description (Bio index) or by the translated Twitter list names and descriptions (Lists index) or combined (Bio-Lists index). The findings indicate combining Bio and Lists information leads to the most efective lexical retrieval strategy. The initial retrieval process demonstrates the superiority of the Bio Lists Index over the Lists Index, emphasizing the importance of incorporating Bio and Lists information. Moreover, the results showed that both lexical and initial retrieval models outperformed all the semantic retrieval models. However, a hybrid approach combining lexical and semantic retrieval, particularly using the MARBERT model, shows improved performance. These results highlight the benefits of combining diferent retrieval methods to achieve optimal information retrieval outcomes. Proceedings of the 1998 ACM conference on Computer supported cooperative work, 1998, pp. 315–324. [10] B. Dom, I. Eiron, A. Cozzi, Y. Zhang, Graph-based ranking algorithms for e-mail expertise analysis, in: Proceedings of the 8th ACM SIGMOD workshop on Research issues in data mining and knowledge discovery, 2003, pp. 42–48. [11] C. S. Campbell, P. P. Maglio, A. Cozzi, B. Dom, Expertise identification using email communications, in: Proceedings of the twelfth international conference on Information and knowledge management, 2003, pp. 528–531. [12] N. Craswell, A. P. De Vries, I. Soborof, Overview of the trec 2005 enterprise track., in:
Trec, volume 5, 2005, pp. 1–7. [13] A. Pal, S. Counts, Identifying topical authorities in microblogs, in: Proceedings of the fourth ACM international conference on Web search and data mining, 2011, pp. 45–54. [14] S. Ghosh, N. Sharma, F. Benevenuto, N. Ganguly, K. Gummadi, Cognos: crowdsourcing search for topic experts in microblogs, in: Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval, 2012, pp. 575–590. [15] W. Wei, G. Cong, C. Miao, F. Zhu, G. Li, Learning to find topic experts in twitter via diferent relations, IEEE Transactions on Knowledge and Data Engineering 28 (2016) 1764–1778. [16] Z. Cheng, J. Caverlee, H. Barthwal, V. Bachani, Finding local experts on twitter, in:
Proceedings of the 23rd International Conference on World Wide Web, 2014, pp. 241–242. [17] E. Yilmaz, J. A. Aslam, Estimating average precision with incomplete and imperfect judgments, in: Proceedings of the 15th ACM international conference on Information and knowledge management, 2006, pp. 102–111. [18] E. M. Voorhees, The trec question answering track, Natural Language Engineering 7 (2001) 361–378. [19] K. Järvelin, J. Kekäläinen, Ir evaluation methods for retrieving highly relevant documents, in: ACM SIGIR Forum, volume 51, ACM New York, NY, USA, 2017, pp. 243–250. [20] F. Haouari, T. Elsayed, W. Mansour, Who can verify this? finding authorities for rumor verification in twitter, Information Processing & Management 60 (2023) 103366. [21] S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, M. Gatford, et al., Okapi at trec-3, Nist Special Publication Sp 109 (1995) 109. [22] W. Antoun, F. Baly, H. Hajj, AraBERT: Transformer-based model for Arabic language understanding, in: Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Ofensive Language Detection, European Language Resource Association, Marseille, France, 2020, pp. 9–15. URL: https://aclanthology.org/ 2020.osact-1.2. [23] M. Abdul-Mageed, A. Elmadany, E. M. B. Nagoudi, ARBERT & MARBERT: Deep bidirectional transformers for Arabic, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Association for Computational Linguistics, Online, 2021, pp. 7088–7105. URL: https://aclanthology.org/2021.acl-long.551. doi:10.18653/v1/2021.acl-long.551.