Personalization in Information Retrieval (IR) is a topic studied by the research community since a long time. Nevertheless, the availability of high-quality, real-world datasets for large-scale experiments and model evaluation remains limited. This paper helps to fill this gap by introducing SE-PQA (StackExchange - Personalized Question Answering), a new curated dataset designed for the development and evaluation of personalized models in the domain of community Question Answering (cQA). SE-PQA encompasses over one million queries and two million answers, annotated with a rich set of features that capture the social interactions among users on a cQA platform. We provide reproducible baseline methods for the cQA task based on the resource, including deep learning and personalized approaches. The results of the preliminary experiments conducted show the appropriateness of SE-PQA to train efective cQA models; they also show that personalization remarkably improves the efectiveness of all the methods tested.
The problem of Personalization in Information Retrieval (IR) has been explored by the research
community since a long time [
Data from StackExchange has been utilized in the training of language models for sentence similarity [
We infer the relevance of an answer to a question based on the number of up-votes it receives from community members. For experiments involving personalized cQA models, we consider only the single answer explicitly labelled as the best answer by the user who submitted the question as relevant. Using this information, we define two versions of the dataset: the base version, where we consider all answers with a positive score as relevant for a question, and the personalized (pers) version, which considers only the single answer selected by the user as the best answer as relevant for both the user and the question. A variety of user-generated information from the training set can be utilized in the personalization phase. For each question, we include all prior user posts to prevent data leakage during the training. However, user data is not limited to these documents: it also encompasses social interactions between users, tags assigned by users to previous questions along with their meanings, and badges earned by users. Additionally, the dataset contains biographic text (about me) introducing each user, a comprehensive set of numeric features (e.g., user reputation score, number of up-votes and down-votes for each post, number of views), and temporal information (e.g., user creation date, last access date, post creation timestamp).
Comparison with available datasets. In Table 1 we summarize the basic statistics of the main datasets used in the literature for personalized IR tasks. The AOL query log, released in 2006, remains a widely used resource despite privacy concerns, containing around 20 million web queries from over 657,000 users. However, the dataset’s limitation lies in representing web pages solely by their URLs without text content. To address this, researchers have used a 2017 version that includes scraped text content, though this approach introduces the issue of web page content changes over time. The more recent AOLIA dataset resolves this by retrieving 2006 web page versions from the Internet Archive, providing a cleaner and higher-quality query set. Additionally, synthetic datasets such as PERSON, Amazon product search, and MAG have been developed to circumvent real-world dataset limitations. PERSON and MAG use citation networks to create personalized evaluation frameworks, while the Amazon product search dataset employs item categories and properties to generate user queries. As depicted in Table 1, the proposed dataset exhibits similarities with other datasets concerning corpus volume and other statistical measures. Notably, it stands out as the largest dataset in terms of the number of queries provided. Unique to this dataset is the explicit annotation by users for relevance assessment. A single user labels the best answer, while various community members contribute by either up-voting or down-voting an answer based on its perceived relevance.
In this section, we briefly describe the experimental setup and discuss the results of the preliminary experiments conducted. We adopt a two-stage ranking architecture: the first stage uses BM25 as a fast ranker; for the second stage, we rely on a linear combination of the scores computed by BM25, a neural re-ranker based on a pre-trained language model, and, when used, a personalization model exploiting user history, represented by the tags used by the users. In the second stage, three neural models are employed: MiniLM, which was trained and tuned using billions of training pairs, including StackExchange data; DistilBERT; MonoT5 small and base. For the DistilBERT and MonoT5 models, ifne-tuning is performed using all the training queries of SE-PQA. To fine-tune MonoT5 model we rely on Adapter modules [14, 15, 16]. The intermediate dimension of the Adapter is set to 48. The personalization score is computed for an answer a generated in response to a query q written by user u, wherein the interests of u are captured through the set of tags assigned to all her/his previous questions posted before time t. The personalized score, called the TAG model, assesses the relevance of a to q by computing the intersection of tags associated with a and u’s previous questions, normalized by the total number of tags associated with u’s previous questions plus one, to account for cases where the set of tags is empty. We use P@1, NDCG@3, NDCG@10 and MAP@100 as our evaluation metrics. The experimental results are presented in Tables 2 and 3 for the base and pers datasets, respectively. Statistically significant improvements, indicated by *, are determined using a Bonferroni-corrected twosided paired Student’s t-test at a 99% confidence level. Neural re-rankers based on MiniLM outperform DistilBERT and T5-small, due to MiniLM’s extensive training set. DistilBERT and T5-small, nfie-tuned for 10 epochs, show MAP@100 improvements of approximately 22% and 33% over BM25, respectively. T5-base, on the other hand, outperforms all the baselines, obtaining relative improvements in terms of MAP@100 of 6% and 46% over MiniLM and BM25, respectively. The most notable result is that TAG improves, by a statistically significant margin, any cQA method it is combined with and for all the metrics considered on the pers version of the dataset, thus showing the advantages of personalization. The improvement due to the addition of this simple personalized model reaches up to 8% in terms of MAP@100 compared to their non-personalized baseline. We claim that personalization is particularly useful for multi-domain collections, where we can exploit information about users’ interests in multiple topics of diferent domains. To validate our hypothesis, we perform a series of experiments considering single-domain data extracted from SE-PQA. Specifically, we consider 50 partitions of SE-PQA (base version) built by isolating the data from the 50 communities. We apply to each one of these subsets the non-personalized and personalized combinations of models using the bi-encoder model MiniLM, and measure the performance according to the same metrics used for the previous cQA tests. For Academia Apple Bicycles Christianity Cooking DIY Hermeneutics Law Money Music Rpg Scifi
MiniLM MiniLM MiniLM MiniLM MiniLM MiniLM MiniLM MiniLM MiniLM MiniLM MiniLM MiniLM + TAG 0.453* MiniLM + TAG 0.335* MiniLM + TAG 0.436* MiniLM + TAG 0.549* MiniLM + TAG 0.619* MiniLM + TAG 0.335* MiniLM + TAG 0.632* MiniLM + TAG 0.677* a fair comparison, we performed for each community the optimization of the weights on singledomain validation data. We notice that the contribution of the TAG model is lower in this setting, and in some cases missing. Specifically, for 25 out of 50 communities, personalization does not lead to any improvement, i.e., = 0. On the other 13 communities, we do not observe statistically significant improvements for P@1 over the non-personalized methods. As expected, the absolute metrics are slightly higher for single-domain tests due to the higher recall in the first-stage retrieval, since we drastically reduce the size of the collection indexed, allowing the first-stage ranker to perform better. However, in terms of the absolute performance boost due to the TAG model, we achieve a 2% improvement on P@1 when using all communities together, while the average boost decreases to 1.1% when considering the communities separately. In Table 4, we report the results for the 12 communities for which personalization achieves statistically significant improvements.
Despite significant eforts by the IR community in studying personalization, a comprehensive dataset for evaluating and comparing diferent approaches has been lacking. This work addresses this gap by introducing a large-scale dataset encompassing 14 years of StackExchange user activity. Detailed information about the dataset is provided, along with its potential for training and evaluating both classical and personalized models for the community question-answering (cQA) task. Preliminary experiments demonstrate that personalization significantly enhances state-of-the-art methods based on pre-trained large language models. The analysis and unique features of the SE-PQA dataset suggest numerous future research directions, including the development of more complex personalized models utilizing additional user features not employed in the current models.
We acknowledge the CINECA award under the ISCRA initiative, for the availability of high-performance computing resources and support. on Research and Development in Information Retrieval, SIGIR ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 645–654. URL: https://doi-org.unimib.idm.oclc.org/10. 1145/3077136.3080813. doi:10.1145/3077136.3080813. [14] C. Poth, H. Sterz, I. Paul, S. Purkayastha, L. Engländer, T. Imhof, I. Vulić, S. Ruder, I. Gurevych, J. Pfeifer, Adapters: A unified library for parameter-eficient and modular transfer learning, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Singapore, 2023, pp. 149–160.
URL: https://aclanthology.org/2023.emnlp-demo.13. [15] J. Pfeifer, A. Kamath, A. Rücklé, K. Cho, I. Gurevych, AdapterFusion: Non-destructive task composition for transfer learning, in: P. Merlo, J. Tiedemann, R. Tsarfaty (Eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Association for Computational Linguistics, Online, 2021, pp. 487–503. URL: https: //aclanthology.org/2021.eacl-main.39. doi:10.18653/v1/2021.eacl-main.39. [16] M. Braga, A. Raganato, G. Pasi, AdaKron: An adapter-based parameter eficient model tuning with kronecker product, in: N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, N. Xue (Eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), ELRA and ICCL, Torino, Italia, 2024, pp. 350–357. URL: https://aclanthology.org/2024.lrec-main.32.