In this paper, we present a submission to the Touché lab's Task 2 on Argument Retrieval for Comparative Questions [1, 2]. Our team Katana supplies several approaches based on decision tree ensembles algorithms to rank comparative documents in accordance with their relevance and argumentative support. We use PyTerrier [3] library to apply ensembles models to a ranking problem, considering statistical text features and features based on comparative structures. We also employ large contextualized language modelling techniques, such as BERT [4], to solve the proposed ranking task. To merge this technique with ranking modelling, we leverage neural ranking library OpenNIR [5]. Our systems substantially outperforming the proposed baseline and scored first in relevance and second in quality according to the oficial metrics of the competition (for measure NDCG@5 score). Presented models could help to improve the performance of processing comparative queries in information retrieval and dialogue systems.
On a daily basis, people face the problem of choosing between two entities - which phone is more reliable, which juice contains less sugar, which hotel is better for a holiday. Domainspecific comparison systems, like WolframAlpha or Difen, solve this problem partly and rely on structured data, which limits the number of cases it can be used.
On the other hand, the Web contains a vast number of opinions and objective arguments that can facilitate the comparative decision-making process. It creates the need of developing an open-domain general system that could process such information. The issue is to retrieve from a set of documents relevant, supportive and credible arguments. The aim of the proposed work is to retrieve from ClueWeb12 1 corpus documents and re-rank them, considering argumentation for or against one option or the other.
The contribution of our work is the following: we are first to use ensemble methods based on mixed statistical and comparative features to the document ranking; we are first to use neural information retrieval approach to the task of argument retrieval; we propose a model outperforming the baseline and yielding the first and the second-best result according to the relevance and quality metric, respectively.
The most relevant to this work is the previous shared task Touche 2020 [
Only a few of the submitted solutions can slightly improve the baseline. The best overall
approach in the previous competition was the method based on query extension and reranking
documents by relevance, credibility, and supportive quality [
This work is based on our run submitted in the previous version of the Touche shared task [
Therefore, the problem of argument retrieval arises in other scenarios. Comparative
argumentation machine CAM [
The organizers provided 50 comparative questions (topics), for which we should obtain documents containing convincing arguments for or against one or another option. Topics for the competition are available online. 2
In addition, 50 topics and corresponding relevance annotations of the previous year’s
competition [
Unfortunately, this data is not insuficient for fitting large supervised ranking models, for
example, based on the BERT technique. In this case we use adjacent question-answering dataset
called Antique [
The example of query and ranked answers are in Table 6, Table 7 in Appendix A. It might be noticed that Antique dataset has a diferent set of ranking scores - 0, 1, 2 instead of 1, 2, 3, 4 - so we rewrite Antique ranks in accordance with the following mapping 1→0, 2→1, 3→1, 4→2.
We use every topic as a query in ChatNoir [
As auxiliary data, the organizers provided the topics of the previous year’s competition. For each proposed topic, a set of documents from ChatNoir was retrieved and labelled as described above. We use this data to train developing models and valid composing approaches. In the validation phase, we split the ranked data into 40 topics in train and 10 in validation.
In the run phase, we execute produced solutions on web evaluation platform Tira [
In this section, we use ensembles of trees as a supervised machine learning technique to solve
ranking problems. We choose either pointwise regression tree algorithms, like Random Forest,
or boosted tree algorithms like XGBoost and LightGBM. In the cases of LightGBM model we
employ LambdaMART [
For our ranking ML methods, we use features that came from 3 origins described below: (i) ranking features extracted by PyTerrier, (ii) specific comparative features, (iii) score from ChatNoir system based on custom BM25 scoring function.4 4.1.1. Features extracted by PyTerrier PyTerrier provides measure of matching query-document texts by several models. Among these models are statistical measures (TF-IDF), mesures based on language models (Heimstra, Diriclet), measures based on occurrence of a document depending on the fields that the term occurs in (BM25F, PLF). The list of all possible models are available at the cite 5. Among these varieties we have chosen BM25, Heimstra, DFIC, DPH, TF-IDF, DiricletLM, PL2 for our exploration.
We applied each of the selected methods sequentially and independently to the training set, ranked documents by the obtained scores and evaluated the ranking on the validation set. The result of these tests is in Table 1. We have chosen 3 methods with the most promising results, and these 3 methods combine 3 features.
3https://pyterrier.readthedocs.io/en/latest/index.html 4https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html 5http://terrier.org/docs/current/javadoc/org/terrier/matching/models/package-summary.html
4.2.1. Random Forest 4.2.2. XGBoost 4.2.3. LightGBM
BM25 0.3637
Heimstra
4.1.2. Comparative features
We focus not only on finding high relevant documents as on finding documents with a
comparison of one object relative to another. The work [
The feature is_retrieved describes are there any comparative structures in the document at all. Characteristic objs_score defines how many objects from query are found in document (0, 1 or 2). Feature asp_pred_score is counted in the following way: if at least one object from a query is in the document, every word in the document labelled as an aspect or predicate increases the score to 0.5. Finally, we combined defined features with scores obtained from the ChatNoir system, and a resulting feature vector for pair query-document is {score_pl2, score_tf, score_bm, score_dfic, baseline_scores, is_retrieved, ap_score, objs_score}. We use the Random Forest model imported from Sklearn and wrapped by the PyTerrier pipeline. To find the best setup, we vary the number of estimators from 10 to 150, the value 20 gives the best valid score NDCG@5 of 0.408.
We also wrapped gradient boosting library from Sklearn to PyTerrier class and tune hyperparameters by setting the learning rate from 1e− 4 to 0.1 and max_depth from 4 to 16. The best setup is learning rate 0.01, max_depth 6 and gives NDCG@5 0.547.
In the case of LightGBM, we vary the number of leaves from 8 to 20 and the learning rate from 0.001 to 0.1. The best configuration with num_leaves equal to 15 and a learning rate equal to 0.1 gives 0.579 score.
The feature importance of the resulting model is in Table 2. It can be seen that the most
significant feature is the score retrieved from the ChatNoir, then there is a Divergence from
Independence based on Chi-square [
Contextualized language models such as BERT can be much more eficient for ranking tasks
because they contain vast relationships between language units. In the proposed work we use a
reranking model from OpenNIR [
BERT receives a query and document and processes it jointly. A distinctive feature of the BERT
reranker is injection token similarity matrices on each layer, which considerably improves
performance [
First, we pretrain this reranker on the Antique dataset. We clean this dataset from incorrect symbols and makeups. We also left from the dataset documents of length more than 300 characters, since the length of the ChatNoir retrieves usually does not exceed 300. The training process lasted for 500 epochs with 0.001 learning rate and 56 objects in every batch. Finally, our model gives NDCG@5 0.3362 on a validation set. We fine-tune the model on 40 train topics from the previous year for 50 epochs with the same configuration. Fine-tuning increased the score on validation up to 0.412.
The result for every proposed approach obtained on the validation part of data from the previous year competition is in Table 3. We also evaluate the previous year’s baseline on the validation set. The best scores come from the LightGBM model, which also outperforms the baseline. XGBoost has fewer scores, Random Forest as a simple algorithm has the smallest score. Bert overtakes Random Forest a little.
In the right column, we also added the time required to train each model. It can be seen that the ensemble-based models have approximately the same time complexity, while the Bert requires much more time to train.
For final testing, the retrieved documents were labelled manually with a score from 0 to 3. Judgment was carried out in two independent criteria: the relevance of the document to the given topic and the quality of the text. Quality criterion includes good language styling, easy reading and proper sentence structure, the absence of typos and alliteration.
For each criterion, a separate file with the assessor’s scores is available. The results of two evaluations are presented in the Table 4 and Table 5. The runs of our team Katana have the best result between all teams in terms of relevance and the second result in terms of the text quality.
As in the validation set, XGBoost and LightGBM give the best performance. It is well explained, since the loss of these models based on the ranking quality functions, NDCG in the XGBoost case and LambdaMART in the LightGBM case. The first model describes relevance a bit better (0.489) and has first place among the whole participant. For quality, conversely, LightGBM is better. It archives 0.684 and takes second place in a quality table, slightly surrendering to Top 1. The random forest method has scores just below the baseline in both cases. It can be explained by a more elementary algorithm for building an ensemble. Bert gives a quite good result for quality and weak for relevance. Perhaps the data from the adjacent task (factoid QA) used for the training is the reason for not a very accurate solution.
In this paper, we present our solution to the Argument retrieval shared task. We pay attention to ensembles methods and use statistic approaches, language modelling and comparative structure extraction to retrieve features for it. We also use a neural reranker based on the Bert technique to use information from a contextualized model in our task.
The best results were obtained by gradient boosting methods, training on ranking cost function: XGBoost and LightGBM. The proposed approaches outperform baseline and take first and second places in relevance and quality ranking, respectively. Bert contextualized model shows the need for large learning data.
This work is partially supported by the project “ACQuA: Answering Comparative Questions with Arguments” (grants BI 1544/7-1 and HA 5851/2-1) as part of the priority program “RATIO: Robust Argumentation Machines” (SPP 1999). We thank Maik Frobe for providing the support of the software runs in the TIRA system.
In this appendix you can find examples Top-3 ranked documents in accordance to LightGBM and Baseline approaches.
Rank Rank 1. Stanford and Harvard have a similar admissions rate of about 7%. MIT comes with a somewhat greater rate of success admitting just under 10% or 1742 for the class of 2015. Harvard, Stanford and MIT are global leaders in culture, commerce and governmental policies. 1. Stanford and Harvard have a similar admissions rate of about 7%. MIT comes with a somewhat greater rate of success admitting just under 10% or 1742 for the class of 2015. Harvard, Stanford and MIT are global leaders in culture, commerce and governmental policies 2. For more than a decade, i have served as an ad- 2. For more than a decade, i have served as an admissions oficer for MIT. In that time, i’ve read more missions oficer for MIT. In that time, i’ve read more than 10,000 applications and have watched thou- than 10,000 applications and have watched thousands of new students enter MIT. It is a privilege to sands of new students enter MIT. It is a privilege to work at the most dynamic and exciting university work at the most dynamic and exciting university in the world. in the world. 3. Our primary enhancement was targeted at fami- 3. All of this factual information, plus a lot of other lies earning less than $75,000 — making mit tuition detail, can be found in the mit admissions literature. free and eliminating In fact, this year, mit will award $74 million in undergraduate aid.
Which smartphone has a better battery life: Xperia or iPhone? LightGBM Top-3 Baseline Top-3 1. 1. The power saver app that will turn down set- 1. The iPhone 4 is apple’s thinnest smartphone yet, tings when battery life is low to get as much juice but ofers a much better screen, faster processor, out of the battery as possible. Sony has set the video calling, and many other enhancements. benchmark with its 12 megapixel camera inside the Xperia S. 2. How to increase the battery life of apple’s iPhone 4s many of those with an iphone 4s have complaints about the battery life. Apple has acknowledged these problems, and is working to fix them. 3. Sony Ericsson includes an 8gb card in the sales package the Sony Ericsson Xperia arc s has below average battery life. Most users will get around 24 hours of life out of the Xperia. X27’s 1600mah battery before it needs a recharge, but heavy users may need an injection of power before then. 2. Sony Xperia’s review: an above average smartphone gizmotraker’, as far as battery life is concerned, it last about 7 hr 30 min in talktime, 450 hrs in standby. 3. How to increase the battery life of Apple’s Iphone 4s many of those with an iphone 4s have complaints about the battery life. Apple has acknowledged these problems, and is working to fix them.