In this manuscript, we review the work we undertake to build a large-scale benchmark dataset for an understudied Information Retrieval task called Semantic Query Labeling. This task is particularly relevant for search tasks that involve structured documents, such as Vertical Search, and consists of automatically recognizing the parts that compose a query and unfolding the relations between the query terms and the documents' fields. We first motivate the importance of building novel evaluation datasets for less popular Information Retrieval tasks. Then, we give an in-depth description of the procedure we followed to build our dataset.
Community Question Answering platforms (e.g., Quora1, Yahoo! Answer2, Stack Overflow 3, etc.).
Community Question Answering research datasets include Quora Dataset4, Yahoo! Answers
Dataset [
One of the IR sub-fields that received limited attention from academicians for the study of the application of Deep Learning techniques is Vertical Search. However, nowadays, many diferent kinds of vertical online platforms, such as e-commerce websites (e.g., Amazon 6), media streaming services (e.g., Netflix 7, Spotify8), job-seeking platforms (e.g., LinkedIn9), digital libraries (e.g., DBLP10), and several others, provide access to domain-specific information through a search engine to millions of users every day. What makes Vertical Search interesting from a research perspective and, potentially, for the application of sophisticated Machine Learning-based approaches is that vertical platforms usually organize their information in structured documents, which require to be treated appropriately during search to leverage the additional information encoded in their structure. However, search functionalities on vertical platforms are usually delivered as standard keyword-based search, or trough uncomfortable faceted search interfaces, which require additional efort form the user. Unlike in Web Search, user queries in vertical systems often contain references to specific structured information contained in the documents. Nevertheless, Vertical Search is often managed as a traditional retrieval task, treating documents as unstructured texts and taking no advantage of the latent structure carried by the queries. Exploiting this latent information could unfold the relations between the query terms and the documents’ structure, thus enabling the search engine to leverage the latter during retrieval.
Semantic Query Labeling [
Semantic Query Labelling is a challenging task that can add context and structure to
keywordbased queries, usually composed of a few terms that may be ambiguous. The main challenges
of this task are related to the vocabulary overlap among diferent semantic classes, which
could require the use of contextual information and disambiguation techniques, and vocabulary
mismatch [
Despite semantic query labelling could play an important role in Vertical Search, very little
work has been done in this regard. The majority of past eforts in this context come from private
companies, such as Microsoft ([
As we strongly believe in the utility of advancing in Vertical Search, we have recently undertaken a step towards the definition of a benchmark dataset for this task 11.
In this section, we describe the dataset we have defined and shared [ 28], as well as the process we followed for manually annotating each query term. Our dataset is composed of thousands of manually-labeled real-world queries in the movie domain for training and evaluating novel methods for Semantic Query Labeling.
The choice of working in the movie domain is motivated by the fact that movie streaming platforms are popular nowadays, but they still provide a sub-optimal search experience to their users. Moreover, structured search is fundamental in this context: as we assessed during our work described here, users tend to compose their queries referring to specific movie-related information, such as the name of an actor or a director, a movie genre, a topic, and others, which are usually available as metadata. By conducting a qualitative evaluation of the top 10 results returned by the search engine of one of the most popular movie streaming services, we assessed that it is not able to correctly retrieve movies even for simple queries. For example, “horror 2015” retrieved only one horror movie from 2015, many other results were neither horror movies nor movies from 2015. “2015 horror” did not retrieved any result at all. Neither “leone eastwood” nor “sergio leone clint eastwood” retrieved any result despite the presence on the platform of all the movies directed by Sergio Leone and starring Clint Eastwood at the time of the experiment.
11https://github.com/AmenRa/semantic-query-tagging-dataset
The first step in building a dataset suitable for studying Semantic Query Labeling is the query gathering. To collect the queries that are part of our dataset, we relied on a publicly available large-scale query log of the AOL Web search engine12, which was shared by Pass et al. [29]. This query set comprises queries issued by real users between March 1, 2006, and May 31, 2006. First of all, we defined a list of seed-terms for identifying movie-related queries: movie, movies, iflm , and films . Leveraging these terms, we extracted 39 635 unique queries. Then, we manually ifltered out all the queries that did not fall into our category of interest: keyword-based queries that resemble those used by users for searching movies on movie streaming platforms. As the large majority of the initially extracted queries were related to theaters’ movie listings — note that AOL ofers a general-purpose Web search engine — we ended up collecting 9 752 candidate queries. After removing the seed-terms used for gathering the queries, manually correcting misspellings, normalizing strings, removing the stop-words, and applying lemmatization, our dataset counts 6 749 unique queries.
The second step in the building process of our dataset was to define 1) the semantic label set to use for the creation of the ground truth and 2) the procedure to follow to assign the semantic labels to the query terms, ensuring the quality of the proposed dataset.
After an initial analysis of the harvested queries, we defined the following semantic classes to assign to each query term: Title, Country, Year, Genre, Director, Actor, Production company, Tag (mainly topics and plot features), Sort (e.g., new, best, popular, etc.). Following previews works in Natural Language Processing and Sequence Labeling [30], we used the IOB2 labeling format [31, 32] for manually assigning both semantic labels and segmentation delimiters. For example, the query “alien by ridley scott 1979” is labeled as follows: “alien B-TITLE by O ridley B-DIRECTOR scott I-DIRECTOR 1979 B-YEAR”, where the prefix B- indicates the beginning of a segment, the prefix I- indicates that the term is inside a segment, and the tag O is used to label terms with no semantic values, such as the preposition by in our example.
One of the main reasons for choosing to work in the movie domain is the public availability of movie-related information. We relied on this information to ensure the quality of the ground truth labels we manually assigned to the query terms. In this regard, we consulted many websites that contain movie-related information while labeling the queries, such as Wikipedia13, IMDb14, and many others. Furthermore, particular attention was paid in discerning actors from directors, as sometimes a single person is both an actor and a director, such as Ron Howard. 12https://www.aol.com 13https://www.wikipedia.org 14https://www.imdb.com In these cases, we followed a simple rule: if the query contains elements pointing towards a specific interpretation of the query, we labeled the query accordingly (e.g., in the query “1999 ron howard”, Ron Howard has been labeled as a Director as in 1999 he directed the movie EDtv and did not star in any movie), otherwise we assigned the most likely label based on the number of movies the person has directed or starred. Therefore, we can state that, where meaningful, we applied a contextual labeling.
To promote a realistic evaluation setting, we split the dataset into train, dev, and test sets temporally, using the queries issued in the first two months as train set, and those from the two subsequent two-weeks periods as dev set and test set. Temporal splitting also reduces query term overlaps between the splits: we noticed that queries issued by users in the same search session often share several terms. We also observed that not taking care of this aspect could yield unrealistic results when training with real-world data.
To build a fine-grained evaluation setting, we created three diferent scenarios of increasing dificulty by subsetting our benchmark dataset. The first scenario we built, Basic, comprises only queries containing the following semantic components: Actor, Country, Genre, Title, Year, and O. We then added the semantic components Director and Sort to create the Advanced scenario. Finally, we added Production Company and Tag to create the Hard scenario. The rationale behind these choices is as follows: the Basic scenario is composed of semantic components whose vocabularies are disjoint; the Advanced scenario introduces vocabulary overlaps (actors/directors), and a semantic class with few manually defined values; the Hard scenario introduces a semantic class often subject to omissions, e.g., Walt Disney Pictures → disney, and a class, Tag, afected by vocabulary overlaps with the others and vocabulary mismatch between queries and documents. Table 1 reports some statistics regarding the proposed scenarios.
In this manuscript, we described the building process of a novel benchmark dataset we have recently proposed. We hope our efort can stimulate research for the understudied task of Semantic Query Labeling and encourage other researchers in building datasets for other not very popular Information Retrieval tasks that could greatly benefit from the recent advancements in Deep Learning. [24] X. Li, Understanding the semantic structure of noun phrase queries, in: ACL 2010, Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, July 11-16, 2010, Uppsala, Sweden, The Association for Computer Linguistics, 2010, pp. 1337–1345. [25] N. Sarkas, S. Paparizos, P. Tsaparas, Structured annotations of web queries, in: Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, 2010, pp. 771–782. [26] J. Liu, X. Li, A. Acero, Y. Wang, Lexicon modeling for query understanding, in: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011, May 22-27, 2011, Prague Congress Center, Prague, Czech Republic, IEEE, 2011, pp. 5604–5607. [27] Z. Kozareva, Q. Li, K. Zhai, W. Guo, Recognizing salient entities in shopping queries, in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 2: Short Papers, The Association for Computer Linguistics, 2016. [28] E. Bassani, G. Pasi, Semantic query labeling through synthetic query generation, in: SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, ACM, 2021, pp. 2278–2282. [29] G. Pass, A. Chowdhury, C. Torgeson, A picture of search, in: Proceedings of the 1st International Conference on Scalable Information Systems, Infoscale 2006, Hong Kong, May 30-June 1, 2006, volume 152 of ACM International Conference Proceeding Series, ACM, 2006, p. 1. [30] E. F. Tjong Kim Sang, F. De Meulder, Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition, in: Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, 2003, pp. 142–147. [31] A. Ratnaparkh, Maximum entropy models for natural language ambiguity resolution, in: Ph.D. Dissertation in Computer and Information Science, University of Pennsylvania, 1998. [32] E. F. T. K. Sang, J. Veenstra, Representing text chunks, in: EACL 1999, 9th Conference of the European Chapter of the Association for Computational Linguistics, June 8-12, 1999, University of Bergen, Bergen, Norway, The Association for Computer Linguistics, 1999, pp. 173–179.