Community question answering and discussion platforms such as Reddit, Yahoo! answers or Quora provide users the flexibility of asking open ended questions to a large audience, and replies to such questions maybe useful both to the user and the community on certain topics such as health, sports or finance. Given the recent events around COVID-19, some of these platforms have attracted 2000+ questions from users about several aspects associated with the disease. Given the impact of this disease on general public, in this work we investigate ways to improve the ranking of user generated answers on COVID-19. We specifically explore the utility of external technical sources of side information (such as CDC guidelines or WHO FAQs) in improving answer ranking on such platforms. We found that ranking user answers based on question-answer similarity is not suficient, and existing models cannot efectively exploit external (side) information. In this work, we demonstrate the efectiveness of diferent attention based neural models that can directly exploit side information available in technical documents or verified forums (e.g., research publications on COVID-19 or WHO website). Augmented with a temperature mechanism, the attention based neural models can selectively determine the relevance of side information for a given user question, while ranking answers.
tween these entities to score answers. However, there are
some limitations of knowledge bases that would make
Question answering systems are key to finding relevant it dificult to use them for community Q&A for rapidly
and timely information about several issues. Community evolving topics such as disease outbreaks (e.g. ebola,
question answering (cQ&A) platforms such as Reddit, Ya- COVID-19), wild-fires or earthquakes. Firstly, knowledge
hoo! answers or Quora have been used to ask questions bases contain information about established entities, and
about wide ranging topics. Most of these platforms let do not rapidly evolve to incorporate new information
users ask, answer, vote or comment on questions present which makes them unreliable for novel disease outbreaks
on the platform. However, question answering platforms such as COVID-19 where information rapidly changes
are useful not only for getting public opinions or votes and its verification is time sensitive. Secondly, it may be
about areas such as entertainment or sports but can also hard to determine what even constitutes an entity as new
serve as information hot-spots for more sensitive topics information arrives about the topic. To overcome these
such as health, injuries or legal topics. Thus, it is imper- limitation in this work, we posit that external curated
ative that when the user visits sensitive topics content, free-text or semi-structured informational sources can
answer ranking also takes into account curated side infor- also be used efectively for cQ&A tasks.
mation from reliable (external) sources. Most prior work In this work, we demonstrate that free text or semi
on cQ&A has focused on incorporating question-answer structured external information sources such as CDC1,
similarity [
answer Ranking user submitted answers on community
answers from ranker question-answering platforms has been addressed with
members several approaches. Primary method is to determine the
relevance of the answer given an input question. Text
fever coughing and tiredness some serious ... based matching is one of the most common approaches
You can have all or some of the symptoms to rank answers. Researchers have used several
meththat can be caused by any other virus ... ods to compute similarity between a question and user
SWyHmOpto&mCsDofCvcirluaismFeuvpertoor14chdilalys,sC..o.ugh ... most relevant fgeeanteurraetebdasaendswqueressttioonde-atenrsmwienre mrealtecvhainncge.isFours eindsitnan[c1e],
with 17 features extracted from unigrams, bigrams and
the symptoms vary as the base virus mutates web correlation features using unstructured user search
Figure 1: Illustrative example of COVID-19 community an- logs to rank answers. It is worth noting that user features
swer ranking powered by side information in the form of and community features when incorporated may still
research papers, and information from verified sources (such yield further improvements in the performance of these
as CDC, WHO, and NHS). models but this is not the focus of our work. The authors
in [
The proposed approach in this work incorporates semistructured information directly with help of temperature regulated attention.
Finally, with the rise of COVID-19, researchers across
disciplines are actively publishing information and
datasets to share understanding of the virus and its
impact on people. Researchers routinely organize dedicated
challenges such as SemEval [
These questions/answer pairs are not user generated con- external source ⟨, ⟩=1.
tent, hence, do not reflect real user questions. We also
rely on recently released Q&A dataset from [
We attempt to model the relevance of each answer state ℎ− 1. Finally, the hidden state is input to a feed
forto its corresponding question using an external source ward layer with smaller dimension < to compress
which may contain free text or semi-structured informa- question encoding as follows:
tion. For example, the external source could consist of
information-seeking queries or questions 1, . . . , ℎ = (ℎ− 1, ), = (ℎ + )
related to a topic, with each linked to a set of rel- (1)
evant scientific articles or answers , where each
answer/document 1, . . . , may be judged for rele- Answer Encoding: Each word in the answer is also
vance by human judges [
We hypothesize that this semi-structured or free-text word embeddings. LSTM takes each token embedding as information may be valuable in identifying user answer input and updates hidden state ℎ. We also reduce the quality for certain kinds of questions, although not all. dimension of answer encoding with a feed forward layer We investigate this with our model to recover the true with dimension < as follows: labels for each user answer ∈ given its question , category information, and information from the ℎ = (ℎ− 1, ), = (ℎ + ) (2)
We concatenate the question and answer representations with respect to the question encoding. Temperature ( ) for further processing. parameter helps us control the uniformity of attention = [, ] (3) wlayeiegrhptesrcep.trFoinnaolvleyr, ltahbeeilnspaurtevpercetodrictedaunsdinthgealemarunletdiweighted average of side information ′. We use binary External source encoding: External sources of in- cross entropy loss to train the proposed model. formation can vary from task-to-task. We encode each segment of data individually. For instance, if there are ˆ = output([ ; ′]) two segments in the source (e.g. question/answer or query/document), our system encodes both segments in- where output uses sigmoid activation function. Since dividually. We use the same encoding architecture used community questions may often be entirely unrelated to for primary source question/answer encoding above. En- external sources, a key aspect of this approach is detercoding example for two segment external source is given mining whether the external source is useful, not merely below. attending to its entries that are most relevant. Temperature based attention mechanism is useful in controlling ℎ = (ℎ− 1, ), = (ℎ + w)hich external source entries are useful for user quesℎ = (ℎ− 1, ), = (ℎ + t)ions. It is worth noting that one will have to experiment (4) and tune the value of temperature such that ranking
performance improves.
We incorporate external source encoding with a temperature ( ) based variant of scaled dot-product attention, which provides a straightforward conditioning approach over a set of query-document pairs. Question
. If encoding vector serves as a query over keys two segments are present in the external source such as query/document, the model uses the attention weights over first segment (e.g. query) to determine the importance of the second segment (e.g. document) respectively. 4.1. Data It is easy to extend this framework to external sources with multiple segments. The two segment attention is described below.
⊤ = √
/ = ∑︀ / ′ = ∑︁
corresponding determine the relevance of each
We compiled two question answering datasets. The first
was collected from Yahoo! answers and the second was
recently released in [
COVID-19 track data with 50 queries which also contain
manually drafted query descriptions and narratives.
Expert judges have labeled over 5000 scientific documents
for these 50 queries from the CORD-19 dataset 8. These
documents contain coronavirus related research. Given
the documents are scientific literature, we initialize
document embeddings using SPECTER [
hub of WHO9 website to create a list of question-answer pairs. There are 147 question and answer pairs in this dataset where questions contain 13.28± 5.36 words and answers contain 133.2± 100.9 words respectively. 4.2. Baselines
Linear Attention (att) : When = 1.0, our model defaults to simple linear attention over all the information present in the external sources. This gives an indication of how well the model performs when its forced to look at all the information in the external source.
9https://www.who.int/emergencies/diseases/novelcoronavirus-2019/question-and-answers-hub Linear combination ( -sim) : We linearly combine similarities between Yahoo! question-answer and Trec query-answer as shown below: -sim = (, ) + (1 − ) max((, )) (6) where , and are Yahoo! answer, question and concatenated trec query, narrative and description embeddings respectively. This is a more crude version of temperature attention where controls the contribution of each component directly. We vary to determine the optimal combination. Question-Answer similarity (qasim) is similarity between question and answer embedding i.e. = 1. Both question and answer embeddings are obtained by averaging over their individual token embeddings. i.e. ( = 1) in the ranked list is indeed correct. It is defined as follows: 1 ∑|︁| ∑︀=1 I{ = 1} || =1 (7) where I{ = 1} indicates whether the answer at position is relevant to the ℎ question. • Recall (R@k): Recall at position evaluates the fraction of relevant answers retrieved from all the answers marked relevant for a question. We report recall averaged for all the queries in test set. For recall, we take a cutof as ( = 3), which evaluates whether the model is able to retrieve the correct answers in top 3 positions. It is defined as follows: || =1
|| || ∑︀ 1 ∑︁ =1 I{ = 1} (8) where || is the number of relevant answers for the th question. • MRR (MRR): evaluates the average of the reciprocal ranks corresponding to the most relevant answer for the questions in test set, which is given by: =
|| 1 ∑︁
1 || =1 (9) 4.3. Evaluation Metrics
popular ranking metrics, mainly Precision (P@1), Mean
Reciprocal Rank (MRR), and Recall (R@3). Each metric is
described below:
BERT Q&A (bert) : Large scale pre-trained
transformers [
Both primary datasets, Yahoo! ans and Infobot, were
divied into three parts: train (∼ 60%), validation and test
(20%) respectively. The baseline models -sim and
are initialized with glove embeddings 11 of 100
dimensions. We performed a parameter sweep over and
for -sim and -att models with step size of 0.1 between
{0, 1.0} respectively. We used base uncased model for
implementation. We fine-tuned the model between
1-10 epochs and found that 3 epochs gave the best
result on validation set. We used LSTM with 64 hidden
units to represent question, answer and all the
information in external datasets. We experimented with higher
embedding size and hidden units, but the performance
degraded significantly as the model tends to overfit on
• Precision (P@k): Precision at position eval- training data. Lastly we used batch size of 64 and trained
uates the fraction of relevant answers retrieved the model for 30 epochs with early stopping.
until position k. For, both datasets Yahoo! ans and
Infobot [
Category Entertainment (47) Health (62) Politics (143) Society (38) Family (20) efit cQ&A task. Since attention is dependent on the input query and key embedding lengths, it would be interesting to scale the computation in our model to incorporate several open external datasets to overcome this limitation in the future.
Yahoo! ans questions are also assigned categories by users. Category based breakdown of performance on test set is given in Table 6 and Table 5 respectively, where categories with largest number of questions in test set are listed. In all the categories, our model outperforms best -sim and qasim model respectively. The largest improvement happens for questions in Family category where our model achieves an improvement of 71% over the sim model. It seems that ranking answers for questions from society and politics are harder than other categories. All the models, however, are able to rank the top answer in first three positions efectively as Recall @3 is high for all the categories. qasim 0.59 0.645 0.587 0.42 0.65
external information in improving answer ranking for cQ&A task. Thus, we performed experiments to answer three main research questions listed below.
RQ1: Does external information improve answer ranking? RQ2: How does temperature ( ) compare with parameter? RQ3: What kind of queries/questions does the model attend to when ranking relevant/non-relevant answers?
parameter? We argued that linearly combining
similarities between question-answer in primary dataset and
between question-external source may not be suficient
to boost performance. We observe that in our results too
i.e. -sim models do not perform better than ( -att)
modRQ1: Does external information improve answer els. This clearly indicates that more sophisticated models
ranking? We evaluated diferent models for ranking can learn to combine this information directly from
trainanswers in Yahoo! ans and Infobot dataset in presence ing data. However, our experiments indicate that optimal
of TREC and WHO datasets respectively. We found that value of ( ) varies across primary datasets and external
temperature regulated attention models that incorpo- sources of information. For instance, ( -att) model
perrate external sources indeed outperform the baselines formed best when = 0.4 and = 0.9 for Yahoo! ans
as shown in Table 4 and Table 3 respectively. ( -att) and Infobot dataset respectively when TREC was used
model beats bert models by ∼ 30% in precision, ∼ 18% as external source. It performed best when = 0.1 and
in recall and ∼ 16% in MRR respectively on TREC data. = 0.5 for Yahoo! ans and Infobot dataset respectively
However, ( -att) does only marginally better than att when WHO was used as external source. We also tried to
model in precision and MRR on Infobot data. We sus- vary beyond 1.0 to determine whether it yielded a trend
pect that is due to the large set of query-document pairs as shown in Table 7. Higher values of temperature seem
in TREC-COVID data compared to fewer number of to degrade model performance. We found that optimal
question-answer pairs in Infobot dataset. Our results temperature range is between [0.1− 1]. Existing research
also clearly suggest that embedding based matching of in model distillation [
Src+ Ext Y! + TREC Y! + WHO Ibot + TREC Ibot + WHO 10 do not have a clear trend indicating that one needs to explore diferent values at the time of training for better performance. On the other hand, we observe that adding external information also helps the -sim models until a certain threshold. Overall, both sets of models show that free-text external information can be incorporated to improve answer ranking performance.
RQ3: What kind of queries/questions does the model attend to when ranking relevant/nonrelevant answers? Attention based models have a this external knowledge need not always be structured very unique feature: they can aid explaining the internal text. However, it is worth noting that curated and reliworkings of neural network models. We inspect what able external sources may not always be available for all kind of queries/questions in external datasets does our domains. We addressed a very niche task in this work, model pay attention to while ranking relevant or non- and further research is required to extend it to incorporelevant answers. Figure 5 shows one such example of rate multiple external sources. We posit that with scalYahoo! question and incorporation of TREC data. At the able attention mechanisms, this work can be easily made time of scoring relevant answer, the model gives higher tractable for large external sources containing thousands weight to some queries compared to others. In the exam- or millions of entries in the future. ple, for instance, it assigns more weight to queries associated with masks or COVID virus response to weather changes. We observe higher attention weights for ques- 6. Conclusion tions when relevant answers are ranked than when nonrelevant answers are scored. An example question, a Question answering platforms provide users with efecrelevant and non-relevant answer along with model at- tive and easy access to information. These platforms tention weights on TREC queries are shown from the also provide content on rapidly evolving sensitive topics Infobot data in Figure 6 respectively. It shows a simi- such as disease outbreaks (such as COVID-19) where it is lar trend where attention weights are high for external also useful to use external vetted information for ranking queries that are closely associated with the question an- answers. Existing work only exploits knowledge bases swer text. which have some limitations that makes it dificult to
Overall, our experiments show that curated external use them for community Q&A for rapidly evolving topinformation is useful for improving community ques- ics such as wild-fires or earthquakes. In this work, we tion answering task. Our experiments also indicate that tried to evaluate the efectiveness of external (free text or semi-structured) information in improving answer ranking models. We argue that simple question-answer text matching may be insuficient and in presence of external knowledge, but temperature regulated attention models can distill information better which in turn yields higher performance. Our proposed model with temperature regulated attention, when evaluated on two public datasets showed significant improvements by augmenting information from two external curated sources of information.
In future, we aim to expand these experiments to other categories such as disaster relief and scale the attention mechanism to include multiple external sources in one model.