We report on our recent efforts to employ transformer-based models as part of an information retrieval pipeline, using argument retrieval as a benchmark. Transformer models, both causal and bidirectional, are independently used to expand queries using generative approaches as well as to densely embed and retrieve arguments. In particular, we investigate three approaches: (1) query expansion using GPT-2, (2) query expansion using BERT, and orthogonal to these approaches, (3) embedding of documents using Google's BERT-like universal sentence encoder (USE) combined with a subsequent retrieval step based on a nearest-neighbor search in the embedding space. A comparative evaluation of our approaches at the Touché lab on argument retrieval places our query expansion based on GPT-2 first on the leaderboard with a retrieval performance of 0.808 nDCG@5, improving over the task baseline by 6.878%.
Search has become the sine qua non tool of information access, and the gateway to the
World Wide Web. The users of web search engines meanwhile expect a high quality
of the search results in terms of their relevance to the queries submitted: If relevant
documents exist for a given query, they are usually found, and the most relevant ones
can be expected to be ranked highest. However, search engines are optimized for ad
hoc retrieval tasks, and a key assumption is that a single document suffices to satisfy
the information need underlying a query. That assumption falls apart when the topic of
interest is inherently subjective and nuanced, such as is the case for contentious issues.
Any one document will most likely argue one way or another, so that a user may need to
peruse many documents at a time to satisfy a deliberative information need. The current
search landscape is, however, not especially attuned to such a nuance, usually preferring
to let ”the stakeholders compete for what opinion ranks higher” [23]. The ability to
specifically handle arguments rather than the documents that might contain them is an
attempt to address that problem using computational argumentation analysis [23, 31].
Such an approach forms the basis of the args.me search engine which relies on a corpus
of more than 300,000 arguments mined from online debate portals [
We leverage transformer models to see how they might enrich argument retrieval
at various stages of the information retrieval (IR) pipeline. Attention-based transformer
models [30] have seen a recent surge in popularity as their readiness for massive
parallelization and the increasing availability of computational power on GPUs led to such
models claiming the state-of-the-art crown on many natural language processing and
understanding tasks [
Although the subject of arguments in a retrieval context is by no means a new
development (see, e.g., [26] and [28]), the field itself is still nascent. Lawrence and Reed
[18], Potthast et al. [23], and Wachsmuth et al. [31] provide a comprehensive overview
of the field, with the last two further putting forth the theoretical considerations that
underlie the development of a tool where arguments form the unit of retrieval, using the
args.me dataset [
The growing interest of natural language processing (NLP) researchers in
information retrieval and argument mining [27], as well as the impressive performance and ease
of use of transformer-based models [
The use of transformers in general, specifically BERT, to the field of document
retrieval was until very recently limited to frameworks where initial retrieval is
delegated to the Lucene-based Anserini retrieval toolkit [
The semantic prowess of transformers makes them prime candidates for enriching
IR pipelines that rely on query expansion [
We set out to instrumentalize for the retrieval of documents the ability of different
transformers to encode knowledge. The original transformer introduced by Vaswani
et al. [30] uses a standard seq2seq encoder-decoder architecture, whereby the encoder
learns a task-specific embedding, and the decoder learns a language model. Subsequent
transformer-based models do not necessarily follow that convention. BERT [
We use transformer decoders (i.e., GPT-2-like models) for query expansion via
text hallucination (Section 3.1), i.e., the generation of a text that reads as if it might
have come from the corpus, and transformer encoders (i.e., BERT-liked models) for
keyword-based query expansion (Section 3.2). Moreover, we consider transformer
encoders for document embedding (Section 3.3). Both query expansion approaches make
use of an Elasticsearch index where the documents of the args.me corpus were indexed
using a language model with Dirichlet priors (DirichletLM) [
Self-supervised pre-training on massive amounts of qualitatively diverse data is what enables transformer models to encode the knowledge that allows them to perform as well as they do on natural language processing (NLP) and natural language understanding (NLU) tasks. Therein also lies their promise for retrieval tasks. We make the conscious decision to only rely on the knowledge encoded into the models by these tasks; that is, we do not rely on any argumentation-specific fine-tuning. That allows us to gauge the performance of transformers for retrieval tasks in general. In particular, our investigation aims at a modular proof-of-concept approach, not to show the superiority of a certain method over others, nor was it our goal to optimize the ensuing pipeline for accuracy on the relevance judgments. 3.1
A lot of the recent media hype around transformer networks is centered around their ability to generate coherent text. Transformer decoders can be trained as causal language models (CLM), seeing as the representation of a given token can only depend on the tokens preceding it. Indeed, since transformer decoders like GPT-2 simulate language models, it is possible to iteratively generate sentences by sampling from the output distribution at a given time step and feeding that output back into the network at the following time step. By choosing an opening for a text (called prompt), it becomes thus possible to steer GPT-2 and exert some influence over the text it generates. Our approach for query expansion makes use of this capability to generate argumentative expansions of a given query with a positive, negative, and a neutral stance by prompting GPT-2 in turn with the six prompts shown in Table 1. The prompts are purposefully constructed so as to simulate an argumentative dialog that is to be completed by GPT-2.
The quality of generated sequences, and thus also of our retrieval results, is highly
contingent upon the way in which tokens are sampled from the network. It is easy
for neural language models to degenerate into incoherent, non-human sounding text
when using naive likelihood maximization [
This sampling method usually leads to low quality text.
– Beam Search is a heuristic that keeps a set number of candidate sequences until
they all reach the desired length and keeps the most likely candidate. Search for
the output sequence that actually maximizes likelihood is intractable, so that beam
search provides a reasonable alternative in practice.
– Temperature scaling reshapes the output distribution by skewing it either toward
high probability tokens or low probability tokens: the former improves generation
quality but hurts diversity [
For each one of the six prompts in Table 1, we use GPT-2 to generate four possible continuations up to a maximum of 100 tokens. To ensure the four continuations are different from each other, we chose different combinations of the aforementioned sampling strategies.1 Having framed the prompts as conversational (by virtue of the dashes and formatting), the ensuing generated text often tends to read like an argument. Having generated 24 such texts,2 we discard the original query and use these hallucinations as 24 queries against the DirichletLM index, which we combine additively to generate the final rankings. We only consider those documents which were returned by at least twelve of the queries simultaneously.3
For illustration purposes, consider the query “Can alternative energy effectively replace fossil fuels?”. Three of the 24 texts that our approach generates are: – Yes, because it has proven to be a significant and lasting improvement in fuel efficiency, carbon neutrality. The only other thing that could possibly help this energy is the need for nuclear reactors at low cost which would require more than 20 percent of current generation electricity by 2030 (currently under construction), plus less renewable resources like wind or solar power as well but with sufficient amounts of coal/solar panels if there are enough [sic] – No, because there is no evidence for that. There are a few possible alternative energy options available to people who would like to cut down fossil fuels, and I believe those include wind or solar power.“The main thing we want in our future climate policies has to be better use of resources instead on these things than if they weren’t used at the moment,” said Kieferlein. “We need clean air [sic] – Not sure. However, many scientists have made the point that alternative sources of power are already producing more carbon emissions than they would otherwise (and it seems like such a small number in our country). There has been some debate about whether this was actually true or if there simply wasn’t much coal available at all to replace fossil fuels and other forms thereof as an environmentally sustainable form. . . In fact, recent studies suggest we [sic] 3.2
Unlike transformer decoders, transformer encoders like BERT [
We leverage this ability of the model to “fill in the blank” to enrich the original query with a set of words that are contextually relevant to the topic at hand. To achieve that, we again augment the original query using the same strategy as outlined in the previous section, this time, however, we leave blanks for BERT to fill out in the form of the [MASK] token (see Table 2). For every [MASK] in every augmented seed text, we ask BERT to return the five most likely words, filtering out stop words, punctuation, and sub-words. This amounts to an average (min=206, max=473) of 340 thematic keywords per query. All keywords are then joined together into a space-separated list of keywords which is what we use to query the DirichletLM index, discarding the original query. For illustration, consider the query “Can Alternative Energy Effectively Replace Fossil Fuels?” and provide the resulting keywords when expanding the query with BERT: diesel, cost, nuclear, consumption, hydrogen, technologies, energy, future, electricity, pregnancy, coal, alternative, migration, emissions, efficiency, economics, technology, growth, wartime, earthquakes, green, environmental, accidents, costs, renewable, winter, development, pollution, new, stress, water, oil, accident, death, health, warming, sustainability, accidental, fires, competition 3.3
Our final approach also employs a transformer encoder in the form of the large variant
of Google’s universal sentence encoder (USE) [
These pre-training tasks make USE a great candidate for argument retrieval. Using USE, we embed each document in the args.me corpus into a 512-dimensional space. To retrieve arguments given a query, we embed the query using the same model into the same space, and perform exhaustive nearest-neighbor search,4 considering both L2 distance and inner-product (IP) distance for retrieval.
We carried out a small pilot experiment to make sure USE projects the args.me corpus in a semantically meaningful way by running k-means on the embedded corpus, choosing a cluster size of 100. The clusters obtained are both syntactically and semantically coherent in a way that is surprisingly meaningful. Some clusters are thematically coherent, encompassing topics, such as religion, politics, and economics. Others are both syntactically and semantically coherent, where all premises are of the form “X is better than Y” and covering themes such as video game consoles, superheroes, and consumer electronics. Further clusters are only syntactically coherent, where, for instance, all arguments consist of YouTube links or of repeated short idiosyncratic phrases one tends to find on online debate websites (e.g., “I agree.”). 4
The evaluation of our approaches to argument retrieval was carried out as part of the Touché shared task. In what follows, we briefly recap the experimental setup and overview the performance achieved. 4.1
The Touché shared task on argument retrieval [
Model GPT-2 Baseline (DirichletLM) BERT Team Aragorn USE (L2) Team Zorro USE (IP) corresponds to submitting software to be run on that virtual machine with the relevant inputs provided by TIRA at run time. Those inputs include the args.me corpus and a list of 50 topics (queries) on which the retrieval model is to be judged using crowdsourced relevance judgments. Though participant entries are ranked by nDCG@5 scores on the leaderboard,5 TIRA also returns nDCG@10, nDCG, and QrelCoverage@10, which we include in Table 3. 4.2
The results of all our runs are included in Table 3, where one can clearly see the considerable improvement over the official baseline by our GPT-2 query expansion approach, which comes out on top of the leaderboard. Our BERT query expansion model basically ties with the baseline of 0.756 as it manages to score an nDCG@5 score of 0.755. We speculate that this performance might be partly due to the fact that both the args.me corpus and the datasets on which BERT and GPT-2 are trained consist of user-generated internet data. Both embedding-based runs perform worse than the query expansion approaches and that is to be expected, as the only information signal afforded to those runs originates in the query itself, whereas the other approaches had the benefit of considerable added context through query expansion. Still, judging the embeddings on their own constitutes a useful baseline. A promising approach would combine these two orthogonal approaches. 5
This work showcases three possible uses of transformer models for the retrieval of relevant arguments from the args.me corpus in particular, and document retrieval from a corpus in general. Impressive results were achieved without hyperparameter tuning or optimization of any sort. A promising future continuation of this work would be an ablation study that judges the effect that the hyperparameters have on retrieval-optimized text generation. Such a continuation would be necessary to judge the quantitative merits of each approach.
It is also important to note that BERT and GPT-2 are merely the representatives of the transformer family of models. There exists a myriad of models6 that build on the foundations laid down by the works that introduced them, iterating, improving, and filling gaps those original models did not take into account. It would therefore be important to experiment with other models, perhaps also come up with new pre-training tasks that would make query expansion even more performant.
Furthermore, our approach leaves out any sort of natural language preprocessing of the corpus or fine-tuning of any of the used models. That some approaches perform as well as they do is a testament to the amount of linguistic and world knowledge encoded in the weights of pre-trained transformers. A future research direction might leverage the args.me and Project Debater corpora to add more argumentative awareness to the transformers and indubitably improve retrieval results.
Finally, we believe it crucial to sensibly modulate any research direction with the due ethical considerations of such projects. It is unclear whether to include user feedback, as including such signals would incur the risk of calcifying existing biases. While it might be useful to think of a search engine as an educational tool, it might prove dangerous to assume it is the prerogative of an information retrieval technology to monopolize the task of teaching its users how to think by conditioning them to blindly rely on it to populate their existing biases.