We describe our contribution of query entity linking and query interpretation software as components of the Information Retrieval Experiment Platform (TIREx) to the 1st International Workshop on Open Web Search (WOWS'24). Query interpretation is the task of determining all plausible user intents behind a search query and can be used to diversify the search results of a retrieval system. As TIREx components, the query entity linking method has identified a total of 89,289 entity candidates in 2,544 queries from 31 standard information retrieval datasets. In addition, a total of 2,304 interpretations of 1,225 search queries from 18 keyword query datasets have been found, which means that on average there is more than one plausible interpretation per search query. This is an indication that most search queries are ambiguous.
The Information Retrieval Experiment Platform (TIREx) [
To streamline further research towards query understanding, we contribute query entity linking
and query interpretation components from Kasturia et al. [
As demonstrated by the example above, search engine queries can be ambiguous. Automated
approaches to interpreting search queries can help a search engine understand a user’s intent or
diversify search results based on all possible interpretations. Moreover, prior studies have shown that
extending queries with (linked) named entities can increase the efectiveness of sparse retrieval [
In this paper, we describe the query entity linking and query interpretation components and report on entity and interpretation statistics of queries from standard information retrieval datasets. As part of these exemplary analytics, we find that on average queries from almost all common information retrieval datasets have more than one plausible interpretation. We also observe that the number of interpretations of a query does not correlate with the number of relevant documents. Consequently, this opens a direction for future work to determine whether query interpretations as an intermediate step in retrieval would increase the number of relevant search results when a query is ambiguous (i.e., the number of plausible interpretations is greater than one).
The query interpretation approach of Kasturia et al. [
The entity linking approach tries to find entities for all the (2) potential segments of an -term query. The entity linking module is based on titles of Wikipedia articles, redirects, and disambiguation pages. Each Wikipedia article represent an entity and its title, all redirect candidates, and disambiguation names from Wikipedia serve as plausible query segments that refer to this entity. The about 13 million distinct key–value pairs (keys are potential query segments and values are lists of entities that can be referred to by this segment) are stored in a RocksDB table1 for fast exact-match access and in a Lucene index2 to quickly find imperfect matches.
Entities from perfect and imperfect matches are then ranked by commonness scores (i.e., the likelihood
of an entity-mention link). To compute the commonness of a mention-entity pair, a Wikipedia dump in
combination with the computation methodology from Ferragina and Scaiella [
Table 1 displays the identified and linked entities in two queries from TREC Web Track 2009 [
Query segmentation methods aim to rank the possible 2 valid segmentations of an -term query
according to retrieval efectiveness when the segments are treated as phrases to be matched in the
search results. The query interpretation approach employs query segmentation approaches from Hagen
et al. [
Most of the 2 segmentations of an -term query do not yield plausible interpretations. Kasturia et al.
[
highest weighting segment is contained in a higher-ranked segmentation. A second filter removes segmentations for which the score ratio to the lowest kept higher-ranked segmentation falls below a threshold of 0.66.
To build query interpretations for a query, the approach combines segmentations with linked entities. To “fill” the segmentation skeletons forwarded by the segmentation phase, the approach collects the entities ranked by commonness for each segment (discarding entities with a commonness of 0) and adds the option of not linking a segment to an entity, but keeping it as a phrase as a fallback. The potential interpretations can then be derived by a Cartesian product of the not-0-common entities and the unlinked respective segments.
In the interpretation ranking, three weights from the entity linking literature are combined: (1) the
above described commonness CMN, (2) the likelihood of two entities to occur together
(relatedness REL), and (3) the likelihood of an entity to occur with the unlinked segments (context CXT). The
relatedness and context weights computations are calculated by using Wikipedia-based joint word–
entity embeddings4 provided by Yamada et al. [
An interpretation ’s score is the averaged weighted sum of the commonness, relatedness, and context
scores of the entities ∈ with the weights = = = 1 suggested by Kasturia et al. [
∑︁ ( · CMN() + · REL() + · CXT()) , |{ ∈ }| · ∈
Table 2 presents the found query interpretations for the example queries from Table 1 and an additional query from TREC Web Track 2012. For the query “obama family tree” the most plausible interpretations have been identified, which is that a user is looking up the family tree of Barack Obama. These interpretations are on rank 1 and rank 2 and can be seen as equivalent because the concept of a family tree is semantically equivalent to the linked entity Family_Tree. The interpretation on rank 3 assumes that the segment “obama” is a concept and not an entity which is wrong, but the assigned relevance is desirably low. For the query “pork tenderloin” the only found interpretation is the dish pork tenderloin. Desirabel (but less likely) would also be the interpretation ⟨pork | Tenderloin,_San_Francisco⟩ as
someone might look for meat vendors in Tenderloin in San Francisco. However, the system did not provide this plausible interpretation. Another interesting example is the query “last supper painting”. The system identifies three plausible interpretations which translate to a user who is interested in (1) the painting process of the painting “Last Supper” by Leonardo da Vinci, (2) any painting that depicts the last supper, and (3) information about “Last Supper” by Leonardo da Vinci. All three interpretations are highly likely which is reflected in the assigned relevance scores.
Both, the query entity linking and the query interpretation software, are Java-based implementations that require external data (e.g., RocksDB and Lucene Index for entity linking) to function as intended. To integrate these components into TIREx and ensure reproducible execution across all systems, we dockerize the query interpretation and entity linking software and bundle the image with all necessary external data. Both Docker images are available in our public container registry5 to use outside TIREx. The use in combination with TIREx is documented in examplary Jupyter notebooks in the code repositories of query entity linking6 and query interpretation7 components.
To gain insights into the identified entities and interpretations, we perform a statistical analysis of the entity and interpretation frequencies of queries from common information retrieval datasets available in TIREx.
TIREx accesses datasets through the Python package of ir_datasets [
Table 3 presents query and entity statistics from 31 diferent information retrieval datasets. All datasets contain between 30 (TREC Precision Medicine Track 2017 dataset) and 325 (NFCorpus test collection) queries of which almost all queries contain at least one entity according to the query entity linking approach. Across all datasets, almost 98% of the queries contain at least one entity. The highest ratio of queries without any entity has the dataset of TREC Web Track 2010. The seven queries from that dataset for which no entity have been identified are “horse hooves”, “iron”, “vldl levels”, “kiwi”, “tornadoes”, “raised gardens”, and “ocd”. Except for Obessive-compulsive disorder (OCD), no other likely entity would be expected in these queries. Although there are songs called “kiwi” or “iron”, neither seems to be common enough to be included as a plausible entity in an interpretation of these queries.
A fairly obvious observation is that natural language queries are on average much longer than keyword queries. While keyword queries consist of approximately three query terms, the average length of natural language queries vary between 5 and 18 terms depending on the dataset. Intuitively, the number of mentions of entities correlates with the length of the query. We calculated a Spearman’s = 0.63 between these two values over all 2,544 queries, which means that the number of plausible entities increases monotonically as the length of a query increases. The query with the most entity mentions is “is it possible to determine rates of forced convective heat transfer from heated cylinders of non-circular cross-section, (the fluid flow being along the generators)” from the Cranfield dataset. The query entity linking approach identified 22 diferent segments that might refer to entities in that query.
The query interpretation approach depends on query segmentation, which requires keyword queries and does not scale well for long natural language queries. We therefore only report on query interpretation statistics for keyword query datasets. Table 4 presents query interpretation statistics for 18 keyword query datasets in TIREx. As most statistics are influenced by them, the number of queries and the query length are copied from Table 3 to facilitate comparisons.
Overall, the query interpretation approach has identified 2,304 plausible interpretations from 1,225 analyzed queries. Therefore, queries across all datasets have an average of about 1.8 plausible interpretations using the query interpretation approach, with queries from the TREC Terabyte Track 2006 dataset having the most interpretations of 2.3. One of two queries which have the most interpretations is “how has african american music influence history” from the TREC Web Track 2014 dataset with 20 diferent interpretations. The highest ranked intepretations with a score close to one are ⟨how | has | African-American | Music | influence | History⟩ and equivalent variations of that in which the entities African-American, Music, and History have not been linked, and thus have been considered concepts. Since the concepts music and history are equivalent to their linked entities, the interpretation variations can be considered semantically equivalent. Another interesting interpretation of that query is ⟨how | has | African-American_Music | influence | History⟩ which is probably a more relevant interpretation because it correctly identifies the relation between “african-american” and “music”. Unfortunately, this interpretation has been ranked lower.
Ideally, the more ambiguous a query is (i.e., the more plausible interpretations it has), the more documents that fulfill all the diferent information needs become relevant. To analyze if this is given in the datasets, we compute correlation coeficients as Spearman’s between the number of automatically identified interpretations and the number of relevant documents (relevance > 0) in the datasets. We have found no correlation ( ≈ 0) for most datasets or in the worst case a negative correlation ( = − 0.35). This points into an interesting future direction to use query interpretation as an intermeditate step to diversify search result to increase this correlation. A related research question would be to analyze whether each diferent interpretation results in a similar number of relevant documents, which would lead to a linear growth of relevant documents as the number of interpretations increases.
To summarize, we contributed query entity linking and query interpretation components to TIREx. A total of 89,289 Wikipedia-linked entities and 2,304 segmentation-based interpretations were automatically identified, which can be reproducibly used for future research with the help of TIREx. As part of a preliminary analysis of the query interpretations, we found that the number of relevant documents for a query does not correlate with the number of plausible interpretations. This fact points in the direction of future research in which query interpretation can be used to diversify search results and what other efects query interpretation has as an intermediate step on an information retrieval pipeline.
Although working with a static entity collection and precomputed commonness scores brings the advantage of reproducibility, entities that are relevant now may not be as relevant in the future. Therefore, the identified linked entities included in our query interpretations may become outdated, or their relevance score may become inadequate. A method to automatically update the entity index and the associated commonness scores from an up-to-date knowledge base can compensate for this limitation and will be implemented in the future.
Wikipedia is a well maintained knowledge base for general knowledge. However, entities from specialized areas may be inadequately represented or simply not exist on Wikipedia, and consequently the entity linking method will fail to identify these entities. A mechanism for exchanging (or extending) the knowledge base that is used for query entity linking can help to find more relevant entities. For example, the addition of a knowledge base such as PubMed9 could increase the discoverability of medical-related entities. We aim to implement an easy method to modify and extend the knowledge base for entity linking and the interpretation of queries. H. Wachsmuth, M. Potthast, M. Hagen, Overview of touché 2020: Argument retrieval, in: A. Arampatzis, E. Kanoulas, T. Tsikrika, S. Vrochidis, H. Joho, C. Lioma, C. Eickhof, A. Névéol, L. Cappellato, N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction. 11th International Conference of the CLEF Association (CLEF 2020), volume 12260 of Lecture Notes in Computer Science, Springer, Berlin Heidelberg New York, 2020, pp. 384–395. [41] H. Wachsmuth, N. Naderi, Y. Hou, Y. Bilu, V. Prabhakaran, T. A. Thijm, G. Hirst, B. Stein, Computational argumentation quality assessment in natural language, in: P. Blunsom, A. Koller, M. Lapata (Eds.), 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2017), 2017, pp. 176–187. [42] L. Braunstain, O. Kurland, D. Carmel, I. Szpektor, A. Shtok, Supporting human answers for advice-seeking questions in CQA sites, in: N. Ferro, F. Crestani, M.-F. Moens, J. Mothe, F. Silvestri, G. M. D. Nunzio, C. Hauf, G. Silvello (Eds.), Advances in Information Retrieval - 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20-23, 2016. Proceedings, volume 9626 of Lecture Notes in Computer Science, Springer, 2016, pp. 129–141. [43] M. Rafalak, K. Abramczuk, A. Wierzbicki, Incredible: Is (almost) all web content trustworthy? analysis of psychological factors related to website credibility evaluation, in: C.-W. Chung, A. Z. Broder, K. Shim, T. Suel (Eds.), 23rd International World Wide Web Conference, WWW ’14, Seoul, Republic of Korea, April 7-11, 2014, Companion Volume, ACM, 2014, pp. 1117–1122. [44] W. R. Hersh, R. T. Bhuptiraju, L. Ross, P. Johnson, A. M. Cohen, D. F. Kraemer, TREC 2004 genomics track overview, in: TREC, 2004. [45] N. Craswell, B. Mitra, E. Yilmaz, D. Campos, E. Voorhees, Overview of the TREC 2019 deep learning track, in: TREC 2019, 2019. [46] P. Bajaj, D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, M. Rosenberg, X. Song, A. Stoica, S. Tiwary, T. Wang, MS MARCO: A Human Generated MAchine Reading COmprehension Dataset, 2018. arXiv:1611.09268. [47] N. Craswell, B. Mitra, E. Yilmaz, D. Campos, Overview of the TREC 2020 deep learning track, in: TREC, 2020.