Searchers who cannot resolve a known-item information need using a search engine (e.g., as the searcher only remembers vague details about a movie from some years ago) might post a respective question on a question answering platform, hoping that the discussion with other people can help to identify the item. To foster research on applying query performance prediction to known-item information needs, especially in the light of the upcoming tip-of-my-tongue known-item retrieval track at TREC 2023, we build a large-scale dataset of 1.28 million known-item questions (47 % have an identified answer) from the r/tipofmytongue subreddit. As the “performance” of a known-item question, we use the time it took the community to solve the question (or the absence of a solution) and evaluate the efectiveness of seven standard pre-retrieval query performance predictors in a pilot study. Not surprisingly, none of the tested predictors can really assess the performance of known-item questions.
Retrieving answers to known-item questions will be the focus of the upcoming ToT track at TREC 2023.1 The task can be very challenging since known-item questions often do not mention a suited identifier for the item or only a loosely similar one [ 8, 9], and since information in the question might even be wrong [10]. As training data, the organizers of the ToT track announced a dataset by Bhargav et al. [8]: 15,000 questions with linked known items (focus: movies and books) extracted from the subreddit r/tipofmytongue/.2 Having run an adapted version of Bhargav et al.’s approach, our resulting TOMT-KIS dataset (tip-of-my-tongue known-item search) contains 1.28 million known-item questions (47 % with an identified answer) and is freely available under a permissive license.3
As an indicator for the “performance” of a known-item question, we simply use the time elapsed to answer the question. For example, the left question in Figure 1 was answered within a minute and thus performed better than the middle question (answered after three days) that again performed better than the rightmost question (not answered after a week). In a pilot study with seven standard pre-retrieval query performance predictors, we find that none of the predictors can really assess the performance of known-item questions.
To create our TOMT-KIS dataset, we have crawled all questions and discussions from the tip-ofmy-tongue subreddit. Note that answers are not explicitly tagged in the subreddit discussions, but the guidelines state that the asker should reply with “Solved!” to a post with the correct answer. In creating the dataset that the TREC 2023 ToT track organizers suggest as training data, Bhargav et al. [8] focused on questions for which the asker replied exactly with “Solved!” to some post and from these they only kept the questions where the answer contains exactly one link to a Wikipedia, IMDb, or GoodReads page. Using this rather restrictive approach, only 15,000 questions with a specified known-item answer were identified.
Title Content UTC
I remember a movie where a [Movie] From the 1990s? young woman having an 1576446820
afair with a married man [. . . ] [late-2000s] Video clip of Around the time “In Rainbows” Thom Yorke talking about came out, I remember seeing a 1598985699 Radiohead album video clip of Thom Yorke [. . . ] [AVniadteoom] yRescurregaetriyonscoefnGereys sTcheensee ipneGopreleysreAcrneaattoemdya[. . . ] 1479737186 Answer Identification Analyzing diferent questions from the subreddit, we observed that askers often do not reply to the correct answer with “Solved!” but with other posts (e.g., “This was it!”), and we found a further metadata field (maintained by moderators but not used by Bhargav et al.) that indicates whether a question is solved. We thus adapted the answer identification method of Bhargav et al. to recall more questions with answers. For questions that the moderator metadata indicates as solved, we use four manually created lexical patterns as a rule-based answer identification method: if a post from the asker in reply to a post contains ‘solved’, ‘thank’, ‘yes’, or ‘amazing’, we view the post as the answer. On 50 random questions and 50 questions for which this heuristic identified an answer (none of them used to develop the rules), the achieved precision is 92 % at a recall of 78 %.
Dataset Description Our TOMT-KIS dataset is available in JSONL format. For each question, we include all the attributes available in the crawled data and add the chosen answer when our heuristic could extract it. Table 1 shows an excerpt for the questions from Figure 1.
Overall, TOMT-KIS includes 127 attributes for each question, such as title, content,
created_utc (indicating the posted question’s timestamp), and link_flair_text (indicates
whether the question is solved; set by moderators). The complete tree of the discussion on each
question is stored in the comments field. To simplify subsequent processing steps, we run our
precision-oriented answer identification heuristic on questions tagged as solved by a moderator
and add four “new” attributes when the heuristic could identify an answer: (
For further analyses, we manually merged the original tags to form larger categories (e.g., combining [song], [music], etc. to a “Music” category). Table 3 shows general statistics for all questions from our new TOMT-KIS dataset and for the four most popular merged categories. As a proxy for the “performance” of a known-item question, we use the time elapsed until the solution was posted (columns Δ ; lower is better). Obviously, the performance varies between years and between categories. For instance, relatively more known-item questions in the movies category are solved than in the music category (52 % for movies vs. 46% for music) and the average time until an answer is posted is lower (9 hours for movies vs. 14 hours for music). Since query performance prediction is evaluated via measuring the correlation of a predictor’s query ranking (by predicted performance) to the ground truth ranking [5, 7, 11], we build training, validation, and tests sets for the set of all questions and for the shown four most popular categories by each time sampling 100 questions for validation and 100 questions for test while keeping all other questions as training data.
Table 4 shows length statistics of the titles and contents of the questions, and of the identified answers in our TOMT-KIS dataset as the average number of characters and words (whitespace tokenization). Overall, the titles of questions are much shorter (14 words on average across all categories) than the explanations in the content field (81 words on average), while the identified answers again are rather short (12 words on average). There are some notable diferences between categories like the titles for movie questions being longer (15 words on average) or the answers in the book category being longer (16 words on average).
We conduct a pilot study on TOMT-KIS with the seven pre-retrieval query performance predictors implemented in the qpptk toolkit4—including the state-of-the-art max-var approach [6, 7]. In the experiments, we use the Robust04 CIFF index5 from the Open-Source IR Replicability Challenge [12] to compute corpus-relative values that the predictors need.
Following previous practice on evaluating query performance prediction [6], Table 5 shows the rank correlations of the predictors to the ground truth in form of Kendall’s , Spearman’s , and Pearson’s (a score of 1 indicates perfect correlation, 0 indicates random correlation, and -1 indicates perfect inverse correlation). In all scenarios, the existing predictors only achieve correlation scores close to 0 which indicates that they are not suited for known-item question performance prediction. For the development of more efective known-item question performance prediction, our new TOMT-KIS dataset can form an ideal starting point.
We have constructed the new large-scale TOMT-KIS dataset6 for known-item question performance prediction by crawling the complete tip-of-my-tongue subreddit (1.28 million questions; 47 % with heuristically identified answers). As a proxy for the performance of a question, we use the time elapsed until the solving answer was posted. In a pilot study, none of the existing pre-retrieval query performance predictors implemented in the qpptk toolkit could really predict a known-item question’s performance. Known-item question performance prediction thus is still not “solved” and forms an interesting subject for future research—with our dataset as a possible starting point. Other interesting directions could be to use TOMT-KIS as an enrichment of the training data provided by the organizers of the upcoming TREC 2023 ToT track. 6Data and code freely available under a permissive license: https://github.com/webis-de/QPP-23 [6] D. Carmel, E. Yom-Tov, Estimating the Query Dificulty for Information Retrieval, Morgan & Claypool Publishers, 2010. URL: https://doi.org/10.2200/S00235ED1V01Y201004ICR015. [7] C. Hauf, D. Hiemstra, F. de Jong, A survey of pre-retrieval query performance predictors, in: Proceedings of CIKM 2008, 2008, pp. 1419–1420. URL: https://doi.org/10.1145/1458082. 1458311. [8] S. Bhargav, G. Sidiropoulos, E. Kanoulas, ‘It’s on the tip of my tongue’: A new dataset for known-item retrieval, in: Proceedings of WSDM 2022, 2022, pp. 48–56. URL: https: //doi.org/10.1145/3488560.3498421. [9] I. K. H. Jørgensen, T. Bogers, “Kinda like The Sims . . . But with ghosts?”: A qualitative analysis of video game re-finding requests on Reddit, in: Proceedings of FDG 2020, 2020, pp. 40:1–40:4. URL: https://doi.org/10.1145/3402942.3402971. [10] C. Hauf, M. Hagen, A. Beyer, B. Stein, Towards realistic known-item topics for the ClueWeb, in: Proceedings of IIiX 2012, 2012, pp. 274–277. URL: https://doi.org/10.1145/ 2362724.2362773. [11] G. Faggioli, O. Zendel, J. S. Culpepper, N. Ferro, F. Scholer, An enhanced evaluation framework for query performance prediction, in: Proceedings of ECIR 2021, 2021, pp. 115–129. URL: https://doi.org/10.1007/978-3-030-72113-8_8. [12] R. Clancy, N. Ferro, C. Hauf, J. Lin, T. Sakai, Z. Z. Wu (Eds.), Proceedings of OSIRRC@SIGIR 2019, 2019. URL: http://ceur-ws.org/Vol-2409.