This notebook documents Team Macbeth's contribution to the CLEF 2021 shared task Touché: Argument Retrieval for Controversial Questions. Our approach consists of diferent configurations of a two-step retrieve and re-rank pipeline. We experimented with sparse and dense approaches for argument retrieval and trained query-document cross-encoders for argument re-ranking. Our findings suggest that a sparse retriever combined with a custom re-ranker performed the best out of all our approaches.
While standard information retrieval systems have focused largely on sparse
bag-of-wordsbased approaches such as BM25, recent trends in IR indicate the performant nature of a two-step
retrieval and re-ranking pipeline, where a sizeable number of candidate documents are first
retrieved using the aforementioned sparse representations, and then re-ranked using (trainable)
neural models [
Attempts are also being made to get rid of sparse representations altogether through the use
of dense retrieval systems [
Our submissions for the Touché shared task center around the above methods.
All our experiments were computed on a setup comprising of an Intel Xeon E5-2650 CPU (24 cores, 256 GB RAM) and 2 NVIDIA GTX 1080Ti GPU’s (24 GB VRAM). We also used Weights and Biases3 to track our experiments.
We pre-trained the entire args.me corpus on a Masked Language Modeling (MLM) task
introduced first by BERT [
Our motivation for pre-training was to make sure that our model first learns from the
domaininvariant representations present in RoBERTa-base, and then enhances these representations
through (continued) pre-training on our custom domain. This kind of domain-adaptive
pretraining has been known to ofer gains in task performance [
We used the hyper-parameters presented in the RoBERTa-base model and trained it for 10 epochs4, generating a domain-specific RoBERTa-base model with perplexity ≈ 4.1.
The organisers of Touché 2021 provide the participants with 2298 relevance judgements to allow
training/evaluation of their systems. These relevance judgments are the result of crowd-sourcing
eforts of Mechanical Turk 5 workers - a practice which has been criticised for its questionable
data quality [
Our initial plan was to use these annotations to train a sentence-pair classifier. After a closer look, however, we found that these annotations were riddled with errors and therefore, not suitable as a training set.
Instead of eliminating their use altogether, we decided to re-annotate all of the 2298 relevance judgements.6 We went through two rounds of annotation for each query-document pair, and achieved the following metric for inter-annotator agreement: Krippendorf’s alpha = 0.39 3https://wandb.ai/ 4https://wandb.ai/ragabet/’roberta-base’ 5https://www.mturk.com/ 6The annotations are available on our git repository.
Due to time constraints, the runs we submitted were trained only on the first round of annotations. The relatively low inter-annotator score suggests our runs would’ve turned out slightly diferent had we trained our models on an average of the two annotation rounds.
When it was first introduced, BERT set new state-of-the-art results on various NLP tasks, including question answering, sentence classification, and sentence-pair regression. A big disadvantage of the BERT’s network structure, however, was its inability to generate sentence embeddings based on single-input sequences.
To overcome the above issue, we used UKP Lab’s Sentence-BERT (or SBERT) [
The terminology used in SBERT is further refined by Humeau et al. [
The architecture introduced in SBERT is what is now known as a bi-encoder. Using a bi-encoder, each sentence can be encoded into an independent sentence embedding. The creation of these vector representations enables eficient document retrieval through the use of standard similarity measures (such as Euclidean distance/cosine-similarity) in the embedding space.
After the pre-training step (3.2), we trained a bi-encoder using the query-document annotations described in 3.3. This bi-encoder was used to generate document embeddings for the entire corpus, giving us an embedding space of size * , where is the embedding size and is the total number of documents. This embedding space was then indexed by a dense retriever as described in 3.5.2.
Note: Each document in the corpus consists of premises and a conclusion. To generate document embeddings, we ignore the conclusion and use only the premises.
A cross-encoder is analogous to the standard BERT design where full-attention is applied across tokens over an input sentence pair. While a bi-encoder takes two inputs and returns two representations (or embeddings), cross-encoders take two inputs and return a single decision directly. They outperform bi-encoders on pair-wise sentence scoring tasks at the cost of speed.
Since cross-encoders are slow and do not produce independent embeddings, they cannot be
used for retrieval tasks. We used them in the second step of our pipeline to re-rank documents
where a cross-encoder was trained (after MLM pretraining 3.2) on the annotations as described
in 3.3. As a baseline, we also made use of a cross-encoder pretrained on the MSMARCO dataset.
[
BM25 is a traditional bag-of-words-based retrieval function which scores the relevancy of documents for a given query using the frequencies of common terms between the query and document. As a variation of the TF-IDF function, it is sensitive to the token frequencies as well as their inverse document frequencies.
Due to its simplicity, computational eficiency, and performance, BM25 serves as a critical component of large-scale search applications and serves as the de facto industrial standard in IR tasks. To index our id-document pairs, we used the implementation available in Elasticsearch7 with the default settings enabled.
Despite its robustness, BM25 has several shortcomings. It sufers from the lexical gap problem
[
To overcome the above problems, we deployed BM25’s sparse retriever alongside a dense
retriever. Experimental results demonstrate that the contextual text representations from BERT
are more efective than BM25 on retrieval tasks [
Constructing a dense retriever was a two step process: first, we encoded the entire corpus into a dense vector space using the bi-encoder described in 3.4.1. Second, the representations 7https://lucene.apache.org/core/7_0_1/core/org/apache/lucene/search/similarities/BM25Similarity.html created by the bi-encoder were indexed using a library that implements approximate nearest neighbours search (hnswlib).8
Approximate nearest neighbour search is an important step in eficiently generating similar
document vectors for a given query vector. The alternative is to attempt cosine-similarity of
the query vector with every single document vector i.e. brute force. We chose hnswlib since
systems based on hierarchical navigable small world graphs (HNSW) [
For our data augmentation approach, we utilized the methodology described in the Augmented
SBERT paper [
The two-step pipeline of retrieve and re-rank has been known to work well on IR tasks. Given a search query, the first step is to retrieve a large list of candidate documents which are potentially relevant for the query. For the retrieval stage, we experimented with a sparse retriever (3.5.1), a dense retriever (3.5.2), and a combination of both (by simply appending the outputs of the two retrievers).
In the second step, we used a re-ranker based on a cross-encoder (3.4.2) that scores the relevancy of all the retrieved candidates (Figure 3). We experimented with a custom crossencoder trained on our annotations and a pre-trained cross-encoder trained on the MSMARCO dataset. For each query, 100 candidate documents were retrieved and sent to the cross-encoder 8https://github.com/nmslib/hnswlib 9http://ann-benchmarks.com/
Run 1 2 3 4 5
Retriever Sparse
Sparse Sparse + Dense
Dense Sparse + Dense
Augmenter
Reranker
Relevance
Quality No No No No Yes
Pretrained Cross Encoder Custom Cross Encoder Custom Cross Encoder Custom Cross Encoder Custom Cross Encoder for the purposes of re-ranking. After re-ranking, only the top 50 documents were included in the final submission file.
We performed evaluation using the relevance judgements10 and quality judgements11 provided by the organisers of the shared task. The metric used was nDCG@5. The results are in Table 1.
In this paper, we outlined Team Macbeth’s contribution to the CLEF lab Touché. Our central approach consisted of using tried-and-tested methods for information retrieval and re-ranking. We pre-trained the args.me corpus on a masked language modeling task, re-annotated the relevance arguments from Touché 2020, and attempted neural methods for both retrieval and re-ranking. The combination of a sparse retriever and a custom neural re-ranker stands out as the best method in terms of both argument relevance as well as argument quality. 10https://webis.de/events/touche-21/touche-task1-51-100-relevance.qrels 11https://webis.de/events/touche-21/touche-task1-51-100-quality.qrels