We present a unsupervised semantic search pipeline for the Causality-driven Adhoc Information Retrieval (CAIR-2021) shared task. The CAIR shared task expands traditional information retrieval to support the retrieval of documents containing the likely causes of a query event. A successful system must be able to distinguish between topical documents and documents containing causal descriptions of events that are causally related to the query event. Our approach involves aggregating results from multiple query strategies over a semantic and lexical index. The proposed approach leads the CAIR-2021 leaderboard and outperformed both traditional IR and pure semantic embedding-based approaches.
The Causality-driven Adhoc Information Retrieval (CAIR) shared task consists of retrieving
documents with the likely causes of a query event [
In this paper, we describe our solution for the CAIR shared task. We design a unsupervised semantic search pipeline, which aggregates results across several query strategies and indices. The pipeline leverages both a lexical index and a semantic index to retrieve causally relevant documents. Our approach both outperformed standard IR baselines and semantic baselines and was the top method on the CAIR-2021 task leaderboard.
Datta et al. [
Our approach focused on developing an unsupervised semantic search pipeline. Documents were indexed in two indices: a semantic index and a lexical index (see Section 3.1). Results from multiple queries across the two indices were then aggregated to return the most relevant documents. We additionally explored a post query filter step that aimed to identify documents that contained causal language in the context of the query event. This approach did not produce viable results and was not pursued. In this section, we will present our methodology and experimental setup in further detail.
Two document indices were created for our semantic search pipeline. The first was a lexical
index that treated documents as bags of words and was optimized for Okapi BM25 [
∑=1 ( ) ⋅ ( (
,)⋅( 1+1) ,)+ 1⋅(1−+⋅ ℎ
) ) is the inverse document frequency of the query token and ( , ) is the frequency is the length of the document (i.e. the total numis the average document length. Okapi BM25 scores are unbounded and larger scores indicate the retrieved document is more relevant compared to lower scored documents in the context of the query.
The second index was a semantic index where documents were represented fixed dimension vector embeddings generated by a sentence embedding model. The broad goal of the semantic index was to retrieve documents that are semantically similar to the query. Semantic relevance is measured by the cosine similarity between query embedding and document embedding which can be defined as: ( , ) = ⋅ ‖ ‖⋅‖ ‖ =
∑=0
√∑=1
2√∑=1 2
will likely result in poorer performance in the context of dense passage retrieval and ranking.
Additionally, there is an input asymmetry challenge where the query length is often shorter
than the relevant document that is to be retrieved. Finally, there may be limited lexical overlap
between the query text and the relevant document. As a result standard sentence embeddings
models like USE (Universal Sentence Encoder) [
Our semantic search pipeline (Figure 3) aggregates results from three distinct query strategies to produce the final set of relevant causal documents. Provided a topic consisting of a title and narrative (e,g, Figure 1), we treat the title as the query text and narrative as a source for causal keywords.
1 retrieves the 500 most semantically similar documents from the semantic index. This is accomplished by embedding the query text using the sentence embedding model, retrieving the closest document embeddings based on cosine distance, and then ranking the documents using cosine similarity scores between query embedding and document embedding.
2 retrieves the 500 most relevant documents from the lexical index, where the relevance is measured by the Okapi BM25 between the candidate documents tokens and query tokens.
3 also retrieves 500 results from the lexical index but uses causal keywords extracted
from the narrative description. The narrative text is first passed through a filter step which
removes any statements in the description that describes irrelevant documents. The filter uses a
simple keyword-based regex (e.g. not relevant, not considered, irrelevant, etc) to identify those
statements. Next, the filtered narrative is converted into a set of keywords using TopicRank [
1 , 2 , and 3 each produce a set of candidate documents (1 ′, 2 ′, and 3 ′ respectively). These results are sent to an aggregator module that deduplicates and re-ranks all the candidate documents. If a document appears in multiple results sets, its scores are summed. The top 500 documents are returned as the final result set.
We additionally explored a post query filtering step. This involved extracting causal relations (if any were found) from the candidate document. Candidate documents would have passed this ifltering stage if the extracted cause had an overlap with the query text and the extracted efect overlapped with the narrative causal keywords. This approach did not yield promising results on the train topics and was not explored further on the test topics. Often the causal documents did not mention the caused event as the document was reporting news that occurred before the query event. This filtering method would have failed to identify news reports of events in the past that lead to the query event because at the time of the reporting, the article did not know about the query event (as it would happen in the future).
In this section, we describe our implementation and experiment results.
4.1. Data
The CAIR dataset contains 303,291 Telegraph India news articles from 2001 to 2010 [
For the semantic index, we use the pretrained msmarco-distilbert-base-v4 sentence embedding
model from the SentenceTransformers library [
All the documents in the CAIR corpus were embedded using the msmarco-distilbert-base-v4 sentence embedding model and then stored in an index optimized for approximate nearest neighbors search. We used the ANNOY python library 3 to store the document embeddings and built a search index of 1000 trees.
We evaluated our approach against four diferent lexical and semantic baselines. All the baselines returned the top 500 relevant results which were evaluated against the gold document relevance set. Mean Average Precision (MAP) and Precision at 5 (P@5) metrics were used for evaluation. The first (Narrative Only Okapi BM25) baseline used returned results from the lexical index using the narrative text as the query. The second baseline (Query Only Okapi BM25) used the title as the query for lexical index. The third baseline (Query + Narrative Semantic) combined the query and title texts and retrieved the most relevant semantic results from the semantic index. Finally, the last baseline only used the title text to query the semantic index.
Experiment results can be found in Table 4.4. In addition to our baselines, we include the results of the best submission from the NITS team in the CAIR 2021 shared task. The test set contained 20 topics and a gold relevance set which identified causally relevant documents in the corpus. Our semantic search pipeline outperforms all the baseline methods and leads the shared task leader board. The semantic search pipeline posted a twenty-five percent increase in MAP and a fourteen percent increase in P@5 over the Narrative Only Okapi BM25 baseline.
Our semantic search pipeline uses the same lexical and semantic indexes as the baselines. However, the pipeline is better able to combine the lexical and semantic results to produce the most causally relevant documents. The aggregator module conceptually functions as an ensemble model and weights documents that appear in multiple query result sets higher. Each 1https://spacy.io 2https://github.com/dorianbrown/rank_bm25 3https://github.com/spotify/annoy query strategy utilized information from the topic diferently and the final result set reflected that.
Amongst the baselines, the Narrative Only Okapi BM25 baseline was the strongest. The narrative text contains the most useful information about what caused the query event and was expected to provide the best results amongst the baselines. However, the narrative input with a lexical index is still prone to returning topical documents that are not causally relevant. Figure 4 provides a qualitative comparison between the Narrative Only Okapi BM25 baseline and the Semantic Search Pipeline. The baseline models match on terms present in the narrative but the article is focused on accusing Modi of misconduct in the context of the IPL Kochi scandal. In contrast, the Semantic Search Pipeline correctly identifies a document that describes why Shashi Tharoor resigned in relation to scandal and his friend Sunanda Pushkar.
In this paper, we introduced a semantic search pipeline for the CAIR-2021 shared task. Our approach aggregated results from multiple query strategies across a lexical and semantic index. The semantic search pipeline outperformed the lexical and simple semantic baselines and was the top method on the CAIR 2021 leader board. This approach should serve as a stepping stone toward better causal information retrieval. Future work could explore developing a better model of causality and retrieving results using the query title only. The narrative text provides strong clues as the causal terms that would be in the causally relevant documents. A causal search system would have a better way identify and causally linking events.
This work was supported by Science Foundation Ireland under grants SFI/18/CRT/6223 (Centre for Research Training in Artificial Intelligence). Processing, Asian Federation of Natural Language Processing, Nagoya, Japan, 2013, pp. 543–551. URL: https://aclanthology.org/I13-1062. [10] T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, L. Deng, MS MARCO: A human generated machine reading comprehension dataset, CoRR abs/1611.09268 (2016). URL: http://arxiv.org/abs/1611.09268. a r X i v : 1 6 1 1 . 0 9 2 6 8 .