Our long-term goal is to develop systems that are forthcoming with information. To be efective, such systems should be allowed to combine retrieval with language generation. To alleviate challenges such systems pose for today's IR evaluation paradigms, we propose EXAM, an evaluation paradigm that uses held-out exam questions and an automated questionanswering system to evaluate how well generated responses can answer follow-up questions-without knowing the exam questions in advance.
Often users want to learn about a topic they know very little about. Taylor and Belkin [1, 2] call this a “conscious information need” originating from an “anomalous state of knowledge” where the user knows too little about the topic to ask precise questions. As a result, web search and conversational search systems do not provide a satisfying user experience. Instead, users often turn to Wikipedia. date, incomplete, or missing. If this is the case, today’s users embark on a journey of exploratory search where they are required to manually compile relevant information from multiple search requests. trieval is developing novel search interfaces [3]. We consider a complementary avenue by aiming to provide the best possible response in a single interaction turn, by compiling an overview for a topic of the user’s choice. we aspire to retrieve-and-generate overview articles as found on Wikipedia. The objective of the third year of the track (CAR Y3) is to respond to the query with an article that is composed of existing paragraphs. are available in the online appendix.1
nEvelop-O (L. Dietz) (L. Dietz) CEUR htp:/ceur-ws.org
ISN1613-073
GPT-3 [5], T5 [6], and other natural language generation (NLG) models ofer a promising avenue for generating relevant text in combination with information retrieval (IR) systems. Recent models achieve great results with respect to grammar, flow of arguments, and readability. However, there are concerns whether these generated articles contain the most relevant information [7]. Integrating NLG with retrieval will help to instill influenced Darwin’s work?” The suggested evaluation the user’s trust in the faithfulness of the provided in- paradigm assesses the system’s ability to generate queryformation. The development of retrieve-and-generate relevant articles which ofer comprehensive information. systems is hindered by the lack of an accepted evaluation The goal is to preempt the user with answers to potential paradigm for a fair comparison. follow-up questions, thus alleviating the user from the burden of asking obvious questions that could have been
EXAM does not rely on relevance assessments or a Typical IR evaluation paradigms are based on relevance fixed corpus. Once a suficient question bank is created, assessments for texts from a known corpus in order to it can be reused to evaluate future systems without any quantify the relevance of a ranking. The evaluation manual involvements. EXAM can compare retrieval-only paradigm is directly applicable to predefined passages or systems as well as retrieve-and-generate systems. Since “legal spans” [8, 9]. However, when arbitrary text spans EXAM only assesses the information content, not the incan be retrieved or when retrieved text is modified, this formation source or document, it is a corpus independent evaluation paradigm needs to be adjusted. A common ap- metric that even allows the comparison of systems that proach is to predict if retrieved text is suficiently similar use the open-web as a corpus or neural NLG systems. to assessed text, as in character-MAP [10, 11], passageROUGE [12], or BERTscore [13]. In a preliminary study 1.4. The Chicken-and-Egg Problem we found BERTscore to be successful when the linguistic style is similar, e.g. both are Wikipedia paragraphs. The development of novel retrieve-and-generate systems However, when gold articles are linguistically diferent, and the development of suitable evaluation paradigms BERTscore was found to be less reliable. form a chicken-and-egg problem: New systems (the “egg”)
An open question is how to develop evaluation meth- cannot be studied without an established evaluation, ods that can identify relevant information without being while novel evaluation paradigms (the “chicken”) canafected by the linguistic style in which the information not be tested without established retrieve-and-generate is presented. systems. With this work we provide the “chicken” by studying the eficacy of the EXAM evaluation metric on 1.3. An Alternative Evaluation Approach retrieval-only systems with respect to an established IR benchmark. The eficacy study uses systems submitted In this work, we discuss an alternative evaluation paradigm to the TREC Complex Answer Retrieval track in Year 3 that directly assesses the usefulness of the retrieved in- for which a question bank is available through the TQA formation–instead of documents [14]. In particular, the collection which was created by the Allen Institute for AI evaluation paradigm is directly in service of our design (AI2). We demonstrate that the leaderboard ranking of goals: to educate the user about a topic they are not fa- systems under EXAM correlates highly with the oficial miliar with, while preemptively being forthcoming with track evaluation measures based on manual assessments answers. created by the National Institute of Science and Technol
We achieve this with a mostly2 automatic metric, called ogy (NIST). In contrast, using a collection of gold articles, the EXam Answerability Metric (EXAM). EXAM deter- we show that the system ranking under ROUGE does mines the quality of generated text by conducting an not correlate with the manual assessments, despite the exam that assesses the article’s suitability for correctly fact that corresponding gold articles contain the right answering a set of query-relevant follow-up questions as information and obtain a high EXAM score. depicted in Figure 1. Like an exam in school, the retrieve- Contributions of this paper are as follows. and-generate systems must identify relevant information without knowing the exam questions beforehand. An • Start a discussion on how to evaluate IR systems that external Q/A system will attempt to answer the follow- further process retrieved raw text. up questions; the more questions that can be answered • Suggest EXAM, an alternative evaluation paradigm to correctly with the article, the higher the system’s quality. complement existing evaluation strategies.
We suggest using exam questions that are relatively • Provide a study on TREC CAR Y3, by reusing exam obvious follow-up questions to the user’s request. For questions from the related TQA data set. We demonexample, when a user provides a query such as “Darwin’s strate a high correlation with traditional IR metrics, Theory of Evolution”, the generated comprehensive ar- even in cases where the linguistic style is too diferent ticle should directly answer some reasonable follow-up for ROUGE to work. questions such as “What species of bird did Darwin ob- • While our motivation arises from conscious informaserve on the Galapagos Islands?” and, “Which scientists tion needs, the EXAM paradigm is applicable to many
areas of IR, including ad hoc document retrieval and conversational search.
Outline. Section 2 provides an overview of related evaluation approaches. Section 3 introduces our EXAM evaluation paradigm. Section 4 outlines the experimental evaluation and discusses our results, before concluding in Section 5.
eries Corpus
Wikipedia paragraphs Retrieve & Generate
System Gold Article Our Evaluation
System Q/A System TREC CAR
Manual Assessment
+
Official Evaluation
EXAM evaluation score Exam estions Correct Answers
rics for text summarization, because the only human Figure 2: Pipeline of retrieve-and-generate systems and evalinvolvement is to create a reference summary. ROUGE uation with our proposed evaluation paradigm. and related metrics like METEOR [16] use n-gram overlap to quantify the similarity of phrases and vocabulary between two texts—one being the reference. Though 2.3. Information Retrieval Evaluation ROUGE is commonly used, it has some drawbacks, as suggested by Scialom et al. [17] and Deutsch et al. [18]. Dif- Information Retrieval is commonly evaluated with a poolfering word choice between summaries results in lower based Cranfield-style paradigm, where the top docuROUGE scores. Because of this, comparing two articles, ments are pooled and manually assessed for relevance by two diferent authors, both about the same topic, could [24]. Dietz and Dalton [8] automate manual IR assesshave very low ROUGE scores due to dissimilar word ment by deriving queries and a passage corpus from exchoice. We use ROUGE-1 as a baseline for our evaluation isting articles, then assess passages as relevant when they paradigm in Section 4. originated from the corresponding article and/or section.
Alternatively, some automated metrics use a trained This method does not allow information to deviate from similarity to detect text that is classified as relevant, such predefined passages. as Reval [19], BERTScore [13], and NUBIA [20]. To evaluate systems that retrieve passages of variable length, Keikha et al. [12] use a ROUGE metric to measure the n-gram overlap with ground truth passages. An 2.2. Summarization Evaluation with Q/A alternative approach is to use a character-wise MAP to Eyal et al. [21] compares the quality of generated sum- award credit for shared character sequences [10, 11]. maries to a reference summary, using a Q/A system and In this work we discuss an alternative evaluation paraquestions generated from entities in the reference sum- digm that focuses on retrieving information. mary. Deutsch et al. [18] focuses on automatic question generation from a reference summary. Huang et al. [22] 3. Approach: EXAM Evaluation develop a CLOZE-style training signal that automatically derives multiple-choice questions from reference sum- We propose the automatic EXam Answerability Metric maries. However, since not all phrases in a reference (EXAM) for evaluating the usefulness of systems which summary are equally important, such approaches can- retrieve and generate relevant articles in response to not guarantee that the evaluation will test for relevant topical queries. These articles are evaluated based on information. how many exam questions an automated Q/A system
Doddington [23] suggests evaluating machine trans- can answer correctly. Our metric does not use relevance lation by conducting an exam with questions. In 2002, judgments nor reference summaries. Instead, a benchthese were answered by human annotators (not a Q/A mark for EXAM consists of a set of queries with follow-up system), however inconsistencies between annotators led questions that a relevant article should answer. We use to issues in the evaluation, which was then discontinued. it to measure the relevance and completeness of compreWe hypothesize that automatic Q/A systems (albeit not hensive articles. perfect) ofer a fair comparison across systems. We first introduce our general evaluation approach, then explain customizations for evaluating CAR Y3 as used for evaluation.
score of a generated article from system over the question bank for the query as,
EXAM( |) =
correct answers in number of exam questions for 3.1.1. Resources Required for Evaluation The EXAM score of each retrieve-and-generate system is computed for each query (and hence generated artiWe reserve exam questions and disallow the retrieve-and- cle), then macro-averaged over multiple queries. Skipped generate systems under study to access them. Retrieve- queries are counted as zero score. EXAM awards no and-read systems are given: credit for unanswered or incorrectly answered questions, Queries: Given a free-text user query, such as “Dar- as these suggest that the generated article does not conwin’s Theory of Evolution”, systems must generate tain the right information.3 a comprehensive response. Queries can be a simple Similar to other proposals, e.g., of nugget-recall [29], keyword query or a more complex expression of infor- EXAM is a recall-oriented evaluation measure. To pemation needs, such as a conversation prompt, desired nalize large amounts of non-relevant information, the sub-topics, or usage contexts. article length can be restricted as in TREC CAR Y3, NTCIR One-click [30], or composite retrieval [31].
3.1.3. Normalizing EXAM with Gold Articles
EXAM does not require a predetermined corpus, unlike pool-based evaluations, such as the oficial TREC assessments [4]. Even corpus-free systems like GPT-3 can be evaluated.
resources to be available on a per-query basis: As introduced above, our EXAM score enables relative quality comparison among systems and baselines. However, questions which are too dificult for the Q/A system to answer, or that are irrelevant to the query, could result in an artificially lowered score. To correct for this, we propose a relative-normalized EXAM score that uses human-edited gold articles ∗ which are written to address the query and exam questions. This allows retrieveand-generate systems to be scored using the context of an expected best-case scenario.
Exam Questions and Answer Verification: A set of reasonably obvious follow-up questions about the query’s topic. Any question style (e.g. multiple-choice, free text, etc.) can be used as long as the underlying Q/A system is trained to answer them and the answer can be automatically verified.
Q/A System: A high-quality Q/A system that is trained n-EXAM() = ∑ EXAM( |) to answer exam questions. To be suitable, the Q/A sys- ∑ EXAM( ∗) tem must use the given article to identify evidence for the question. All systems must be evaluated using the Note that if the gold articles contain less informasame Q/A system for EXAM scores to be comparable. tion than the generated articles, or are written obtusely, the retrieve-and-generated articles could earn a higher
EXAM score than the gold articles, especially if the re
The evaluation process will use the above resources as trieve-and-generated articles express information clearer. depicted in Figure 2. We suggest using a multiple-choice This would result in an n-EXAM score above one. For Q/A system and an answer key to verify correctness. example, a Q/A system would have dificulty extracting However, many Q/A systems can be used in our paradigm, answers from a college textbook, when used as a gold such as the systems of Choi et al. [25], Nie et al. [26], article, as college-level reading material requires reader or Perez et al. [27]. To be suitable, some Q/A systems inference or significant logical deduction for full comwould need to be customized, e.g. restricting the sentence prehension. This is one way that human-written, gold selector of Min et al. [28]. articles could receive lower EXAM scores than generated articles sourced from more plainly written corpora. 3.1.2. EXAM Evaluation Scores
one article per query. The exam questions are only used during evaluation: the Q/A system attempts to answer all exam questions based on the content of the generated article. For a query , we measure the EXAM evaluation
algorithms that respond to complex information needs
ness of the generated article to a reader. Sub-topics Gold Article* mitted articles with respect to the query and sub-topics. Participants were encouraged to additionally submit paragraph rankings for each sub-topic. The oficial CAR Y3 evaluation was based on these rankings and the relevance assessments.4 3.2.2. Proposed Alternative: EXAM Evaluation
EXAM, we require a question bank of exam questions with an answer key. Queries are derived from titles of TQA textbook chapters which come with multiple-choice questions designed by the book author to test (human) students. We are using these multiple choice questions as a question bank for the EXAM metric to assess generated articles of participating systems. In particular, we use all provided non-diagram (i.e. not dependent on a picture) questions.
TQA textbook chapters as gold articles for the queries (also used by the ROUGE baseline as reference summary).
While the EXAM metric does not require gold articles, we report the EXAM score achieved by the gold article for reference and include n-EXAM scores as well. As gold by synthesizing longer answers by collating retrieved articles and questions were designed for middle school information, mimicking the style of Wikipedia articles. students, many answers are stated in an obtuse way and For the shared task to yield a reusable benchmark, par- cannot be answered by simple text matches. ticipant systems were restricted to use a corpus of five million predefined passages without modifications.
The Textbook Question-Answering (TQA) [32] dataset Used Question-Answering System: As a high-qualprovides textbook chapters and multiple-choice quesity Q/A system we use the Decomposable Attention Q/A tions designed for middle school students. In CAR Y3, system provided by the organizers of the TQA challenge. the queries were taken from titles of textbook chapters, The system is trained on the AI2 Reasoning Challenge and sub-topics were derived from headings. Sub-topics dataset (ARC) [33]. The model is adapted from Parikh are used as “nuggets” in the oficial assessment and were et al. [34], which performs the best on the SNLI [35] provided to participants. For each query, participants dataset, which contains questions similar to TQA queswere asked to produce relevant articles by selecting and arranging 20 paragraphs from the provided corpus of tions.
As inputs, the Q/A system requires a text and a set of Wikipedia paragraphs.
questions. The Decomposable Attention model searches For the example query “Darwin’s Theory of Evoluthe text for passages relevant to each question, then extion” (tqa2:L_0432), an excerpt of a textbook chapter is tracts answers by constructing an assertion per question depicted in Figure 3. Neither the questions nor the textand answer choice. Assertions without text support are book content were available to the CAR Y3 participant eliminated, the most likely assertion under the attention systems. The figure depicts an example of how parts of model is returned as the answer. If all assertions are rethe TQA dataset textbook chapters are used in the CAR jected, the question is not answered. Both unanswered Y3 benchmark versus held out for the EXAM evaluation. and incorrectly answered questions result in a reduced The connections between CAR Y3, retrieval systems under study, and our proposed evaluation paradigm are EXAM score. depicted in Figure 2. 3.2.1. Reference: Oficial CAR Y3 Evaluation
vided relevance annotations for all paragraphs in all sub
matic evaluation paradigm was only applicable to CAR Y1. -0.09 0.74 0.75 0.74 -0.07 0.57 0.57 0.56
Due to the chicken-and-egg problem, no established retrieve-and-generate benchmarks exist and no established retrieve-and-generate systems are available. We base the experimental evaluation on sixteen retrieval systems submitted to CAR Y3. We use 131 queries that have a total of 2320 questions in the TQA dataset. We use each query’s textbook chapter in the TQA dataset as a gold article. Dataset statistics are summarized in Table 3. Since these systems are not part of our work, we refer to the participant’s description of their systems in the TREC Proceedings5 and CAR Y3 Overview [4]. 4.1.1. Evaluating the Evaluation Measure
while mostly automatic—ofers the same high quality as a manual assessment conducted by NIST. Hence, our measure of success is to produce a system ranking (i.e., leaderboard) that is highly correlated with the oficial CAR Y3 leaderboard. Low or anti-correlation suggests that an evaluation measure would not agree with a user’s sense of relevance. Correlation of leaderboard rankings is measured in:
= ++−+ −−
a similar rank position under both leaderboards A, B: = 1 − 6 ∑ ( ( ()2−−1) ()) 2 Kendall’s Tau: High when the rank order of many system pairs 1, 2 is preserved ( +) versus swapped ( −): Under any evaluation metric some systems obtain a similar evaluation score within standard error. As this is unlikely to indicate significant diference, we define such system pairs as tied, and thus attribute any score diference to random chance. Therefore, we randomly break ties, which is necessary for Spearman’s rank, to produce the leaderboard and compute the rank correlation, repeating the process ten times. Results are presented in Tables 2a and 2b. 4.1.2. Metrics for System Quality
uses a Q/A system to evaluate generated articles (see Section 3.2). n-EXAM (ours): A relative-normalized version of EXAM that uses a set of gold articles to contextualize the EXAM score.
Oficial CAR Y3 Evaluation (reference): Systems in CAR Y3 are evaluated using Precision at R (Prec@R), Mean-Average Precision (MAP), and Normalized Discounted Cumulated Gain at rank 20 (nDCG20) as implemented in trec_eval.6 ROUGE-1 F1 (baseline): ROUGE evaluates via the similarity between a generated article and a gold article. ROUGE-1 F1 combines precision and recall of predicting words in the summary. Words are lowercased and lemmatized, with punctuation and stopwords removed. We include ROUGE as a baseline evaluation paradigm, because it is fully automated and widely used in NLG.
https://github.com/usnistgov/trec_eval
MAP
nDCG20
cial measures is higher than between EXAM and oficial measures.
trieve-and-generate systems ordered by EXAM score.
The trend is clear: Systems that rank high on the oficial ROUGE: By contrast, the leaderboard according to TREC CAR leaderboard also rank high on the EXAM ROUGE-1 F1 is uncorrelated to the oficial TREC CAR leaderboard, and systems that rank low on the oficial leaderboard. Ecnu-bm25-1, which has the best ROUGE-1 leaderboard also rank low on the EXAM leaderboard. For F1 score, is not in the top five of the oficial leaderboard. example, the rerank2-bert and dangnt-nlp participant Tables 2a and 2b demonstrate that the rank correlation systems are ranked as the top two systems by both the of ROUGE is near zero across all metrics, which is equivoficial leaderboard and the EXAM leaderboard. Some alent to a random ordering. systems achieved similar EXAM scores as they likely produce the same passages ordered diferently, afecting 4.3. Discussion oficial TREC CAR evaluations but not EXAM. Additionally, due to the setup described in Section 3.2.2, the best We discuss advantages and limitations of the paradigm. systems even slightly surpass the gold articles (see discussion in Section 4.3). Many systems are performing Resilience to Q/A system errors: Any real-world within standard-error of the gold articles (marked with Q/A system will make mistakes, most likely causing cor⋆)—these are all similar systems based on the BERT neu- rect answers contained within the generated article to ral language model. be missed. Indeed, the Q/A system is unable to correctly
Tables 2a and 2b display the Spearman’s and Kendall’s answer many questions with the gold article, in part berank correlations of the EXAM and the oficial TREC cause the article and questions were designed to be a CAR evaluation. Notably, EXAM achieves correlation of challenge for middle school students. However, despite 0.74 in terms of Spearman’s rank correlation and 0.56 in these Q/A errors, the study demonstrates EXAM reveals terms of Kendall’s Tau, both values can range from -1 to significant quality diferences between systems. If our ex1. These averages illustrate just how strong EXAM corre- periment had not been successful, we would not observe lates with oficial assessments, despite the much diferent any correlation between EXAM and the oficial CAR Y3 evaluation paradigm. Prec@R, MAP, and nDCG20 use assessments. the same relevance assessments and evaluation paradigm.
Hence it is not surpring that the correlation within ofiOvercoming linguistic diferences: We found when While exam questions cannot test all possible useful using the gold article with ROUGE-F1, the system rank- follow-up questions, we demonstrate that the available ing does not agree with manual assessments. The issue question bank is large enough to measure significant diforiginates from a diference in linguistic style, as gener- ferences between systems. To identify how little efort ated articles are constrained to use Wikipedia paragraphs, would still yield good results, we spent one hour to manubut the gold articles are sourced from TQA. Hence, it is ally create ten questions. While error bars are larger, the unlikely that gold articles would use the same phrases as results still correlate with the oficial leaderboard. (Study the generated articles—despite both covering the same, omitted due to space constraints). relevant topics.
In a previous (unpublished) study on CAR Y2 data we Benchmark reusability and comparability: Many found that ROUGE-F1 [12] obtains a reasonable correla- IR benchmarks mandate the use of unmodified elements tion (Kendall’s tau of 0.67, Spearman’s rank of 0.67) when from a fixed corpus. By contrast, EXAM uses a corpususing manually assessed relevant paragraphs instead of independent evaluation, and thus can be applied across gold articles. We conclude that ROUGE is struggling to diferent corpora and sources, including open web or overcome the linguistic diferences between generated NLG algorithms. Systems using diferent sources can all and gold articles. be evaluated and compared with each other using the
In contrast, when evaluated under the EXAM measure, EXAM evaluation. the gold article obtains the same score as the best participant system. Given the positive results, we conclude that EXAM is able to overcome the linguistic diferences. We 5. Conclusions and Future believe this is an important finding as the same issue is Directions likely to arise when a retrieve-and-generate system uses external sources or a fully generative model.
rate systems, which are systems that modify retrieved
Interpretation of N-EXAM: The n-EXAM metric can raw data before presentation. This poses a challenge for
exceed 1.0 when the gold article is written obtusely, but today’s IR evaluation paradigms. To facilitate empirical
the generated article explains relevant facts in accessible research on retrieve-and-generate systems, we discuss an
language. In our study, gold articles are designed for (hu- alternative evaluation paradigm, the EXam Answerability
man) students to carefully read the text and think about Metric (EXAM), that tests whether the system provides
the answer, which is challenging for the Q/A system. In relevant information rather than the right documents.
contrast, the submitted retrieval systems were allowed to EXAM uses a Q/A system and query-specific question
select content from Wikipedia passages, which are likely banks to evaluate whether the system response is capable
to clearly state answers to obvious follow-up questions. of answering some obvious follow-up questions, even
Hence, we suggest to consider EXAM scores on gold without being explicitly asked to do so. We verify that
articles as guidance, rather than a gold standard. Simi- leaderboards under the EXAM evaluation and the manual
lar dataset biases are known from work on Multi-Hop TREC CAR evaluation, agree with a Spearman’s Rank
Question Answering [
Our long-term goal is to develop systems to support users who do not (yet) know what exactly they are look- [6] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, ing for. We envision a system that synthesizes a com- M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the prehensive topical overview by collating retrieved text limits of transfer learning with a unified text-to-text with post-processing steps like natural language genera- transformer, Journal of Machine Learning Research tion. Permitting diferent linguistic styles and encourag- 21 (2020) 1–67. ing comprehensiveness, renders traditional IR evaluation [7] D. Ippolito, D. Duckworth, C. Callison-Burch, paradigms as very costly. These goals also pose chal- D. Eck, Automatic detection of generated text is lenges regarding benchmark reuse for a fair comparison easiest when humans are fooled, in: Proceedings across systems. The EXAM evaluation paradigm pro- of the 58th Annual Meeting of the Association for vides a new avenue for retrieve-and-generate research Computational Linguistics, 2020, pp. 1808–1822. to evaluate systems by information content. [8] L. Dietz, J. Dalton, Humans optional? Auto
However, EXAM can also evaluate many other infor- matic large-scale test collections for entity, passage, mation retrieval tasks: EXAM allows the comparison and entity-passage retrieval, Datenbank-Spektrum of ad hoc retrieval from fixed corpora with open-web (2020) 1–12. retrieval. EXAM ofers an alternative way to assess re- [9] W. R. Hersh, A. M. Cohen, P. M. Roberts, H. K. Rekadundancy for search result diversification. EXAM can palli, Trec 2006 genomics track overview., in: TREC, evaluate the information content of each turn of a con- volume 7, 2006, pp. 500–274. versational search system as well as the provided infor- [10] J. Kamps, M. Lalmas, J. Pehcevski, Evaluating relemation content over multiple turns. We believe that in vant in context: Document retrieval with a twist, in: general, evaluation paradigms that, like EXAM, penalize Proceedings of the 30th annual international ACM avoidable conversation turns will encourage information SIGIR conference on Research and development in systems that are forthcoming with answers. information retrieval, ACM, 2007, pp. 749–750. [11] C. Wade, J. Allan, Passage retrieval and evaluation, Technical Report, Massachusetts University Acknowledgments Amherst Center for Intelligent Information Retrieval, 2005.
We thank Peter Clark, Ashish Sabharwal, Tushar Khot [12] M. Keikha, J. H. Park, W. B. Croft, Evaluating anfrom the Allen Institute for AI for their help with the swer passages using summarization measures, in: TQA dataset and the provision of the Q/A System, and Proceedings of the 37th international ACM SIGIR the UNH TREMA group for their guidance in doing this conference on Research & development in informaresearch. tion retrieval, ACM, 2014, pp. 963–966. [13] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, References Y. Artzi, Bertscore: Evaluating text generation with bert, arXiv preprint arXiv:1904.09675 (2019). [1] R. S. Taylor, Question-negotiation and information [14] J. Allan, B. Croft, A. Mofat, M. Sanderson, Fronseeking in libraries, College & Research Libraries tiers, challenges, and opportunities for information 29 (1968) 178–194. retrieval: Report from swirl 2012 the second strate[2] N. J. Belkin, Anomalous states of knowledge as a gic workshop on information retrieval in lorne, in: basis for information retrieval, Canadian journal ACM SIGIR Forum, volume 46, ACM New York, NY, of information science 5 (1980) 133–143. USA, 2012, pp. 2–32. [3] T. Ruotsalo, J. Peltonen, M. J. Eugster, D. Głowacka, [15] C.-Y. Lin, ROUGE: A package for automatic evalP. Floréen, P. Myllymäki, G. Jacucci, S. Kaski, In- uation of summaries, in: Text Summarization teractive intent modeling for exploratory search, Branches Out, Association for Computational LinACM Transactions on Information Systems (TOIS) guistics, Barcelona, Spain, 2004, pp. 74–81. URL: 36 (2018) 1–46. https://www.aclweb.org/anthology/W04-1013. [4] L. Dietz, J. Foley, Trec car y3: Complex answer re- [16] S. Banerjee, A. Lavie, Meteor: An automatic mettrieval overview, in: Proceedings of Text REtrieval ric for mt evaluation with improved correlation Conference (TREC), 2019. with human judgments, in: Proceedings of the [5] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Ka- acl workshop on intrinsic and extrinsic evaluation plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas- measures for machine translation and/or summatry, A. Askell, et al., Language models are few-shot rization, 2005, pp. 65–72.
learners, arXiv preprint arXiv:2005.14165 (2020). [17] T. Scialom, S. Lamprier, B. Piwowarski, J. Staiano, tational Linguistics (Volume 1: Long Papers), 2017, Answers unite! Unsupervised metrics for rein- pp. 209–220. forced summarization models, in: Proceedings of [26] P. Nie, Y. Zhang, X. Geng, A. Ramamurthy, L. Song, the 2019 Conference on Empirical Methods in Nat- D. Jiang, Dc-bert: Decoupling question and docural Language Processing and the 9th International ument for eficient contextual encoding, ProceedJoint Conference on Natural Language Processing ings of the 43rd International ACM SIGIR Confer(EMNLP-IJCNLP), Association for Computational ence on Research and Development in Information Linguistics, Hong Kong, China, 2019, pp. 3246–3256. Retrieval (2020). URL: http://dx.doi.org/10.1145/ URL: https://www.aclweb.org/anthology/D19-1320. 3397271.3401271. doi:1 0 . 1 1 4 5 / 3 3 9 7 2 7 1 . 3 4 0 1 2 7 1 . doi:1 0 . 1 8 6 5 3 / v 1 / D 1 9 - 1 3 2 0 . [27] E. Perez, P. Lewis, W.-t. Yih, K. Cho, D. Kiela, Un[18] D. Deutsch, T. Bedrax-Weiss, D. Roth, Towards supervised question decomposition for question question-answering as an automatic metric for eval- answering, arXiv preprint arXiv:2002.09758 (2020). uating the content quality of a summary, arXiv [28] S. Min, V. Zhong, R. Socher, C. Xiong, Eficient preprint arXiv:2010.00490 (2020). and robust question answering from minimal con[19] R. Gupta, C. Orasan, J. van Genabith, Reval: A text over documents, in: Proceedings of the 56th simple and efective machine translation evaluation Annual Meeting of the Association for Computametric based on recurrent neural networks, in: tional Linguistics (Volume 1: Long Papers), 2018, Proceedings of the 2015 Conference on Empirical pp. 1725–1735.
Methods in Natural Language Processing, 2015, pp. [29] J. Lin, Is question answering better than infor1066–1072. mation retrieval? Towards a task-based evalua[20] H. Kane, M. Y. Kocyigit, A. Abdalla, P. Ajanoh, tion framework for question series, in: Human M. Coulibali, Nubia: Neural based interchange- Language Technologies 2007: The Conference of ability assessor for text generation, arXiv preprint the North American Chapter of the Association arXiv:2004.14667 (2020). for Computational Linguistics; Proceedings of the [21] M. Eyal, T. Baumel, M. Elhadad, Question answer- Main Conference, 2007, pp. 212–219. ing as an automatic evaluation metric for news ar- [30] T. Sakai, M. P. Kato, Y.-I. Song, Overview of ntcirticle summarization, in: Proceedings of the 2019 9, in: Proceedings of the 9th NTCIR Workshop Conference of the North American Chapter of the Meeting, 2011, 2011, pp. 1–7.
Association for Computational Linguistics: Human [31] H. Bota, K. Zhou, J. M. Jose, M. Lalmas, Composite Language Technologies, Volume 1 (Long and Short retrieval of heterogeneous web search, in: ProceedPapers), Association for Computational Linguis- ings of the 23rd international conference on World tics, Minneapolis, Minnesota, 2019, pp. 3938–3948. wide web, 2014, pp. 119–130.
URL: https://www.aclweb.org/anthology/N19-1395. [32] A. Kembhavi, M. Seo, D. Schwenk, J. Choi, doi:1 0 . 1 8 6 5 3 / v 1 / N 1 9 - 1 3 9 5 . A. Farhadi, H. Hajishirzi, Are you smarter than [22] L. Huang, L. Wu, L. Wang, Knowledge a sixth grader? Textbook question answering for graph-augmented abstractive summarization with multimodal machine comprehension, 2017 IEEE semantic-driven cloze reward, Proceedings of the Conference on Computer Vision and Pattern Recog58th Annual Meeting of the Association for Com- nition (CVPR) (2017) 5376–5384. putational Linguistics (2020). URL: http://dx.doi. [33] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Saborg/10.18653/v1/2020.acl-main.457. doi:1 0 . 1 8 6 5 3 / harwal, C. Schoenick, O. Tafjord, Think you have v 1 / 2 0 2 0 . a c l - m a i n . 4 5 7 . solved question answering? Try ARC, the AI2 rea[23] G. Doddington, Automatic evaluation of machine soning challenge, ArXiv abs/1803.05457 (2018). translation quality using n-gram co-occurrence [34] A. Parikh, O. Täckström, D. Das, J. Uszkoreit, statistics, in: Proceedings of the Second Interna- A decomposable attention model for natural lantional Conference on Human Language Technology guage inference, in: Proceedings of the 2016 Research, HLT ’02, Morgan Kaufmann Publishers Conference on Empirical Methods in Natural LanInc., San Francisco, CA, USA, 2002, p. 138–145. guage Processing, Association for Computational [24] E. M. Voorhees, The evolution of cranfield, in: In- Linguistics, Austin, Texas, 2016, pp. 2249–2255. formation retrieval evaluation in a changing world, URL: https://www.aclweb.org/anthology/D16-1244.
Springer, 2019, pp. 45–69. doi:1 0 . 1 8 6 5 3 / v 1 / D 1 6 - 1 2 4 4 . [25] E. Choi, D. Hewlett, J. Uszkoreit, I. Polosukhin, [35] S. R. Bowman, G. Angeli, C. Potts, C. D. Manning, A. Lacoste, J. Berant, Coarse-to-fine question an- A large annotated corpus for learning natural lanswering for long documents, in: Proceedings of the guage inference, arXiv preprint arXiv:1508.05326 55th Annual Meeting of the Association for Compu- (2015).