We present a query intent predictor that is based on Snorkel weak supervision approach. After using it on ORCAS dataset and conducting a series of experiments with a variety of machine learning models we found that the results produced by Snorkel were not outperformed by these competing approaches and can be considered state-of-the-art. The advantage of a rule-based approach like Snorkel's is its eficient deployment in an actual system, where intent classification would be executed for every query issued. When used as a component of TIRA/TIREX platform, our query intent predictor was shown to be applicable to other IR benchmark datasets. Also, we found out that the awareness of the intent overall can improve ranking results for informational intent and its subcategory factual intent.
To classify the queries in ORCAS dataset according to user intent we created a taxonomy based on
Broder’s [
• Navigational intent: the immediate intent is to reach a particular website [
To create labelling rules for Snorkel we used the characteristics suggested by Jansen et al. [
Also, we used Levenshtein similarity ratio to determine if the query is navigational. We wanted to understand whether the domain part of the URL and the query are similar. The Levenshtein similarity ratio is computed according to the following formula: || + || − Lev(, )
|| + ||
Here, Lev(, ) is Levenshtein distance (the minimum number of edits required change a one-word sequence into the other) and || and || are lengths of sequence a and sequence b respectively. A threshold on Levenshtein ratio was empirically established at 0.55, which means that if the query and the domain name were 55% or more similar they were classified as navigational.
According to Jansen et al., the queries that do not meet criteria for navigational or transactional have informational intent. Thus, we decided to make abstain subcategory part of the informational category. However, we could not establish consistent automatic characteristics for this group of queries because we could not find any reliable patterns in them.
More details about the intent labelling characteristics for ORCAS-I query intent predictor are provided
in [
The ORCAS dataset is part of the MS MARCO datasets (Microsoft) and is intended for non-commercial research purposes. It contains 18.8 million clicked query-URL pairs and 10.4 million distinct queries. The dataset has the following information: query ID, query, document ID and clicked URL. The documents that the URLs lead to come from TREC Deep Learning Track.
This dataset was aggregated based on a subsample of Bing’s 26-month logs to January 2020. The creators of the dataset applied several filters to the log. Firstly, they only kept query-URL pairs where the URL is present in the 3.2 million document TREC Deep Learning corpus. Secondly, they applied a k-anonymity filter and only kept queries that are used by k diferent users. Finally, ofensive queries such as hatred and pornography were removed.
ORCAS-I dataset is the version of ORCAS dataset annotated with user intent that we released [
In machine learning terms, our intent taxonomy could be represented as a two-level, multi-class classification problem. Snorkel has originally only been implemented to handle annotations for singlelevel classification problems. As our taxonomy is hierarchical, we needed to define two layers of Snorkel labelling functions.
We defined the first level of labelling functions to distinguish between navigational and transactional intents. All the queries that could not fit into one of these two categories were classified as informational intent in our taxonomy. Based on our user intent classification characteristics, we created four labelling functions for navigational queries and three functions for transactional queries.
On the second level, we defined labelling functions to cover factual and instrumental intents. Similar to the previous step, we designed nine factual and four instrumental labelling functions. All queries that were not assigned a label from the two layers of Snorkel got an abstain category.
We initially used Spacy’s en_core_web_lg language model to identify part of speech information and to detect named entities. After initial analysis, this proved to generate too many false negatives, especially for the detection of verbs. For example, the queries “change display to two monitors” and “export itunes library” were misclassified as abstain, because the verbs “change” and “export” were labelled as nouns. In the final version, we decided to use a list of the 850+ common verbs with which we obtained comparable coverage with fewer false positives. Eventually, we only have used Spacy for a labelling function where queries begin with the “-ing” form of the verb.
To obtain a final prediction score, we run independently two levels of Snorkel annotations. Based on our classification that all non-transactional, non-navigational queries are informational, for the second level prediction, we use all the queries which were assigned abstain from the first level. In order to conduct label aggregation, we experiment with both the LabelModel and MajorityLabelVoter methods implemented in Snorkel. LabelModel estimates rule weights using an unsupervised agreement-based objective. MajorityLabelVoter creates labels by aggregating the predictions from multiple weak rules via majority voting. We test their predictions on the test dataset using default hyperparameters. Results are presented in Table 1.
After analysis of results on the testset, MajorityLabelVoter achieved higher scores for all measures except macro average recall. Therefore, we decided to use it to obtain the final labels for the ORCAS-I2M dataset. MajorityLabelVoter has the additional benefit that it provides more explainable results, as for every query, the user can be presented with the raw aggregation of the single labelling functions.
Table 2 shows the label distribution in ORCAS-I dataset compared to the 40 test collections present in TIRA2. As for ORCAS dataset, the only underrepresented category compared to gold is the navigational intent, which contains 1.6% less items than in the manual labelling approach. The label distribution from Snorkel for ORCAS-2M and ORCAS-18M is comparable, so the 2M sample chosen from the full ORCAS dataset is representative.
As for the test collections present in TIRA, the categories that are comparable to ORCAS-I label distribution are instrumental and transactional. TIRA datasets contain much less navigational queries. Unlike ORCAS, the majority of those datasets do not have URLs, which can explain far fewer navigational queries. As for factual queries, ORCAS-I intent predictor often relies on the form of the query (e.g. starts with "what is", "who is"), which means that some datasets, like ClueWeb collections or BEIR Nutrition Facts Corpus would have fewer queries that belong to this intent.
Having a URL is crucial for the navigational intent, that is why better predictions are made in the post-retrieval stage. Comparing the label distribution in TREC Web test collections (Table 3) with and without URL shows us that no navigational intent is detected in the pre-retrieval stage. By contrast, it is detected for all three collections in the post-retrieval stage. It shows that all the elements that could help to disambiguate the intent of the query are important.
Model Snorkel Logistic regression SVM fastText BERT xtremedistil query query + URL query query + URL query query + URL query query + URL query query + URL query query + URL .771 .779 .701 .714 .735 .782 .694 .768 .742 .789 .725 .781 .648 764 .611 .689 .689 .759 .643 .753 .705 .764 .691 .765 .667 .770 .643 .700 .703 .767 .660 .758 .717 .774 .696 .772
We benchmark five diferent models by training them on ORCAS-I-2M: Logistic regression, Support
Vector Machine, fastText, BERT and xtremedistil (more details about the hyperparameters can be found
in [
We hypothesise that as we take URL features into account for the Snorkel classifier, models that train on queries and URLs will outperform the models that train on queries only. This hypothesis is confirmed for all the models, especially for fastText and xtremdistil. Also, SVM, BERT and xtremdistil show improvements on recall for query-only when compared to Snorkel. It indicates that the models learn well from the labels assigned by the Snorkel query and URL functions, even if they are trained only on queries.
None of our benchmark models significantly outperforms our Snorkel baseline when trained on queries and URLs. This could have been an expected behaviour when comparing two models, one being the teacher and the other the student who learned only from this one teacher, without any external knowledge. We also hypothesise that transformer-based models cannot express their full power because the input sequences are, on average, very short and lack context.
For ranking experiments we decided to focus on the informational intent in the taxonomy, because navigational and transactional intent are underrepresented in the datasets availiable in TIRA. Some collections do not have navigational and transactional queries at all (TREC Genomics 2004 and 2005, TREC 2018 Core track) some only have a few navigational and transactional queries (CLEF Longeval 2023). For the experiments we chose the TREC COVID collection and TREC Deep Learining track 2019 and 2020 passage collections because they provided suficcient samples of instrumental, factual and abstain queries for the evaluation of the models’ performance.
We tested the BM25 baseline against two ranking models. ColBERT [
After creating the ORCAS-I query intent predictor, we used it for TIRA test collections. In terms of the label distribution, TIRA datasets have fewer navigational queries than ORCAS-I dataset, as many of them do not have a URL field. Comparing the pre-retrieval with the post-retrieval intent for TREC Web collections showed that having a query and URL for navigational category provides better results than having only a query, which confirms our previous findings that Snorkel and benchmark models trained on it perform better when having a URL. As for ranking experiments, attention-based rankers performed better than the BM25 baseline for informational intent and its subcategory factual intent, which means that those models still benefit from the semantics of the queries despite the lack of context.
Although our query intent predictor is applicable to all the collections present in TIRA, it might not
be very useful for some of them. For the Touche collections that contain non-factoid queries, such as
argumentative questions (e.g. "should nuclear weapons be abolished?") it will not provide a classification
that will be fine-grained enough. A separate intent classification that labels non-factoid queries such as
the one suggested by [
This work was supported by the EU Horizon 2020 ITN/ETN on Domain Specific Systems for Information Extraction and Retrieval – DoSSIER (H2020-EU.1.3.1., ID: 860721) and by the OpenWebSearch.eu project (GA 101070014).