Answer Sentence Selection is one of the steps typically involved in Question Answering. Question Answering is considered a hard task for natural language processing systems, since full solutions would require both natural language understanding and inference abilities. In this paper, we explore how the state of the art in answer selection has improved recently, comparing two of the best proposed models for tackling the problem: the Crossattentive Convolutional Network and the BERT model. The experiments are carried out on two datasets, WikiQA and SelQA, both created for and used in open-domain question answering challenges. We also report on cross domain experiments with the two datasets.
Answer Sentence Selection is an important
subtask of Question Answering, that aims at
selecting the sentence containing the correct answer to
a given question among a set of candidate
sentences. Table 1 shows an example of a question
and a list of its candidate answers, taken from the
SelQA dataset
Answer extraction involves natural language processing techniques for interpreting candidate sentences and establishing whether they relate to questions and contain an answer. More sophisticated methods of Answer Sentence Selection that All authors contributed equally to this manuscript.
Copyright c 2019 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
go beyond Information Retrieval approaches
involve for example tree edit models
Recently, Deep Neural Networks have also been
applied to this task
Research on this problem has benefited in
the last few years by the development of better
datasets for training systems on this task. These
datasets include WikiQA
The state of the art model on the SelQA dataset
In this paper we present further experiments with the Cross-attentive Convolutional Network model as well as experiments that exploit the BERT language model by Devlin et al. (2018).
In the following sections we survey relevant literature on the topic, we describe the datasets used in our experiments and present the models tested in our experiments. Finally, we describe the experiments conducted with these models and report the results achieved. 2
We present a brief survey of the most recent approaches for answer selection in question answering.
Tan et al. (2015) present four Deep Learning models for answer selection based on biLSTM (bidirectional LSTM) and CNN (Convolutional Neural Network), with different complexities and capabilities. The basic model, called QA-LSTM, implements two similar flows, one for the question and one for the answer. The biLSTM builds a representation of the question/answer pair that is passed by a max or average pooling layer. The two flows are then merged with a cosine similarity matching that expresses how close question and answer are.
A more complex solution, called
QALSTM/CNN, uses a similar model, which
replaces the pooling layer with a CNN. The
output of biLSTM is sent to a convolution filter,
in order to give a more complete representation
of questions and answers. This filter is followed
by 1-max pooling layer and a fully connected
layer. Finally, the paper presents the most
complex models, QA-LSTM with attention and
QA-LSTM/CNN with attention, that extend the
previous models with the addition of a simple
attention mechanism between question and
answer, which aims to better identify the best
candidate answer to the question. The mechanism
consists in multiplying the biLSTM hidden units
of the answers with the output computed from
the question pooling layer. These models are
tested on the InsuranceQA
The HyperQA
Wang et al. (2016) present a model that takes into account similarities and dissimilarities between sentences by decomposing and composing lexical semantics over sentences. In particular the model represents each word as a vector and calculates a semantic matching vector for each word based on all words in the other sentence. Then each word vector is decomposed into a similar and a dissimilar component, based on the semantic matching vector. Afterwards, a CNN model is used to capture features by composing these parts and a similarity score is estimated over the composed feature vectors to predict which sentence is the answer to the question. 3
We describe here the models used in our experiments. 3.1
Jurczyk et al. (2016) state that the SelQA dataset was created through a process that tried to reduce the number of co-occurrent words, so that simple word matching methods would be less effective. To evaluate whether this aim was indeed achieved, we built a simple linear regression classifier using as features the sentence and question length, the number of co-occurrent words and the idf coefficients of the word co-occurrences. 3.2
The Cross-attentive Convolutional Network
(CACN) is a model designed for the task of
Answer Sentence Selection and in 2018 had
achieved state of the art performance
The CACN model achieved top score in the
”Fujitsu AI NLP Challenge 2018” 1, that used the
1https://openinnovationgateway.com/ai-nlp-challenge/
SelQA dataset.
BERT (Bidirectional Encoder Representations
from Transformers)
Several pre-trained BERT models are publicly available, including the following ones that we used in our experiments:
BERT-Base Uncased: with 12 layers, hidden size of 768 and a total number of 110M parameters; BERT-Large Uncased: with 24 layers, hidden size of 1024 and a total number of 340M parameters. 4
We tested the models on two datasets: SelQA and WikiQA. The first one is the one used in the Fujitsu AI-NLP Challenge, while the second one is a commonly used dataset for open-domain Question Answering. A more detailed description follows. 4.1
The SelQA dataset
The dataset consists of a total of 486 articles that were randomly sampled from the topics of: Arts, Country, Food, Historical Events, Movies, Music, Science, Sports, Travel, TV. The original data was preprocessed into smaller chunks, resulting in 8,481 sections, 113,709 sentences and 2,810,228 tokens.
For each section, a question that can be answered in that same section by one or more sentences was generated by human annotators. The corresponding sentence or sentences that answer the question were selected. To add some noise, annotators were also asked to create another set of questions from the same selected sections excluding the original sentences previously selected as answers. Then all questions were paraphrased using different terms, in order to ensure the QA algorithm would be evaluated by their reading comprehension ability rather than from statistical measures like counting word co-occurrences. Lastly if ambiguous questions were found, they were rephrased again by a human annotator. 4.2
The WikiQA dataset
The WikiQA dataset contains also questions for which there are no correct sentences to enable researchers to work on answer triggering.
This dataset has the drawback to be smaller compared to SelQA. Because of this, a model is more likely to over-fit the training set. To avoid this problem we added some strong regularization to the models. 5
5.0.1
We carried out some preliminary experiments on the SelQA dataset, in order to determine which embeddings would work best with the CACN.
We tested three types of embeddings: GloVe
(size 300), ELMo
The logistic regression classifier obtains a score of 83.36 %, which is 7 points lower than CACN, not bad considering the simplicity of the model. Nevertheless this confirms that a simple word matching method is not competitive with more sophisticated methods on SelQA.
CACN was the best performing model on the Fujitsu AI NLP Challenge 2018, with a MRR of 90.61 %.
After the introduction of BERT, we decided to compare CACN with several versions of BERT, both alone and in combination with CACN.
We tried a few variant approaches. First, we fine-tuned a fully connected layer on top of BERT, leaving his parameters frozen, on the SelQA training set. This model achieved 91.17, a marginal improvement over CACN.
We then explored adding different networks on top of the BERT architecture.
We added a full CACN, on top of either the BERT-Base and BERT-Large models, with no improvement and even a drop with BERT-Large. Also in this case we froze the parameters of the BERT model.
Since these experiments did not provide improvements, we didn’t try to train the entire model.
The best results were achieved by fine-tuning the BERT model on the SelQA dataset with a simple feed-forward layer, that achieved an impressive improvement of about 5 points to a MRR score of 95.29 %. Fine-tuning required about 4 hours on a server with an Nvidia P100 GPU.
The results of all our experiments on SelQA are summarized in table 3. 5.2
In the experiments with CACN on WikiQA, we removed from the training set questions with no correct answer, but left the test set unchanged, so that the results are comparable with thos in the literature. This was done to preserve a similar structure to the SelQA dataset, which contains at least
LR Classifier CACN GloVe BERT-Base + FCN BERT-Base + CACN BERT-Large + CACN BERT-Base Fine-tuned one correct answer for each question. This significantly reduced the number of training examples but, despite this, the MRR score of the CACN model improved.
Also in this case we kept the word embeddings fixed during training the CACN. We also added a dropout and normalization to regularize the model, that helped the model to better learn from the training set.
We then fine-tuned BERT on the WikiQA training set, performing full updates to the model, achieving again a significant improvement to a top score of 87.53 % MRR.
From the current leaderboard on the WikiQA dataset 2, we have extracted the top 5 entries and added the results with CACN and BERT-Base fine-tuned, as reported in Table 4.
BERT-Base Fine-tuned
Comp-Clip + LM + LC
RE2
HyperQA
The experiments consisted in training a model on one dataset and then testing it on the other one. We report in Table 5 the results of these experiments.
2https://paperswithcode.com/sota/question-answeringon-wikiqa 82.95% 76.64%
The drop in MRR score is small when training on WikiQA and testing on SelQA and larger in the other direction.
This is possibly due to the size of the datasets. In the second case in fact we are training on only 8000 pairs and testing on more than 80000 question/answer pairs.
However, the transfer score, computed as the ratio between the in-domain and out-domain MRR, is fairly good: about 83% in the SelQA to WikiQA case and over 76% in the other direction. 6
We compared the Cross-attentive Convolutional Network and several BERT based models on the task of Answer Sentence Selection on two datasets.
The experiments show that a BERT model, finetuned on an Answer Sentence Selection dataset, improves significantly the state of the art, with a gain of 5 to 9 points of MRR score on SelQA and WikiQA respectively. As a drawback, this approach takes a considerable amount of time to be trained even on GPUs.
The BERT-Base model without fine-tuning achieves almost the same accuracy as the CACN with GloVe embeddings, which uses a much smaller number of parameters in the model. The CACN also requires less data to train. On the other hand, BERT is quite effective at leveraging the knowledge collected from large amounts of unlabeled text, and at transferring it across tasks.
We also evaluated the abilities of CACN at transfer learning. BERT is a model that has been pre-trained on a large corpus, while CACN leverages the GloVe embeddings as a starting point for the training.
We also exploited the WikiQA and SelQA datasets in a cross-domain experiment using CACN. We found that the model maintains a good score across domains, with a transfer score of about 83% from SelQA to WikiQA.
We confirmed that the SelQA dataset is not easily solvable using simple word-occurrences methods like a logistic regression classifier on word count features.
BERT models confirmed their superiority to previous state of the art models for the task of Answer Sentence Selection. This was to be expected since they perform quite well also on the more complex task of Reading Comprehension, which requires not only to select a sentence but also to extract the answer from that sentence. 7
The experiments were carried on a Dell server with 4 Nvidia GPUs Tesla P100, partly funded by the University of Pisa under grant Grandi Attrezzature 2016.