Exponential growth in digital information outlets and the race to publish has made scientific misinformation more prevalent than ever. However, the task to fact-verify a given scientific claim is not straightforward even for researchers. Scientific claim verification requires in-depth knowledge and great labor from domain experts to substantiate supporting and refuting evidence from credible scientific sources. TShceiFact dataset and corresponding task provide a benchmarking leaderboard to the community to develop automatic scientific claim verification systems via extracting and assimilating relevant evidence rationales from source abstracts. In this work, we propose a modular approach that sequentially carries out binary classification for every prediction subtask as in StchieFact leaderboard. Our simple classifier-based approach uses reduced abstract representations to retrieve relevant abstracts. These are further used to train the relevant rationale-selection model. Finally, we carry out two-step stance predictions that first diferentiate non-relevant rationales and then identify supporting or refuting rationales for a given claim. Experimentally, our sRyesrtreFmact with no fine-tuning, simple design, and a fraction of model parameters fairs competitively on the leaderboard against large-scale, modular, and joint modeling approaches. We make our codebase available hatttps://github.com/ashishrana160796/RerrFact.
(P. S. Rana) interpret numeric and statistical inferences. The second workshop on Scientific Document Understanding at AAAI ∗These authors contributed equally. component in the concerned task. Therefore, we present a computationally and architecturally simple pipelinedriven design for it.
sign withreduced evidence retrieval stage representations for modeling our systeRmerrFact’s subtask modules.
Misinformation is a modern day societal problem tihoauts use-cases 1[, 2, 3]. The most relevant amongst them has the potential to wreck havoc, especially with increisast-he FEVER shared task4][, which evaluates the veracingly many people having an online footprint withoiutty of human-generated claims from Wikipedia data. For adequate internet literacy. The problem grows intetnhseeFEVER task, there are two paradigms: one that take when science gets associated with disinformation aantdhree-step modular approach and the other which is provides a false sense of trustworthiness. Convincjionignt prediction approach for evidence retrieval & stance statements derived from general public opinions lipkreediction5[, 6]. Similarly, for thSeciFact task these ”Ginger consumption in food reduces the risk of getting
two paradigms have been used either with very large lanseverely infected with COVID-19” can efectively manip- guage models likeVerT5erini for modular architecture ulate the masses. It is hard to verify such misleadi[n7]gorARSJoint, JointParagraph for merged subtask archistatements from extensive scientific literature withteacpt-ure8[, 9]. In contrast to these diametrically opposite propriate reasoning even by providing relevant evidenpcaer.adigms, QMUL-SDS’s 1[0] partial binding between Also, it is a cumbersome task for experts to searchtfhoerabstract retrieval and rational selection stages ofers refuting or supporting argument rationales considearipnrgomising direction, which is also the inspiration for the amount of misinformation available on a plethoourracurrent work. Our experiments demonstrate that of outlets. Therefore, automatic fact-verification ttohoislspartial interdependence successfully introduces a are essential, especially for scientific knowledge wherfeorm of regularization, providing much-needed improvethe given system must understand scientific knowledgme,ents over precision and recall for the evidence retrieval
TF-IDF
Similarity Ranker Input Data claim C(ci ... n) & Abstract Corpus A(aj ... o) Claims C
Corpus A TF-IDF Vectorizer c1a1 ... cna1 ... ... ...
cna30 ... cna30 Top-K a Retrievals RoBERTa Large Binary Classifier Classified aj Retrievals for claim c [T/F] TC t1 ... tX T[SEP] t1 ... tY ...
...
...
Rationale Selection BioBERT Large Binary Classifier Relevant Sentences sk
for claim c [T/F]
Label Prediction BioBERT Large MNLI Enough Info Classifier
Not Enough Info rn Separated Out for claim c [T/F]
RoBERTa Large MNLI
Support Classifier Support/Contradict decision d
for claim c [T/F] TC t1 ... tX T[SEP] t1 ... tW
TC t1 ... tX T[SEP] t1 ... tU ...
...
...
...
...
... for the relevant abstract extraction subtask. After atrheatca,tegorized ays(c,a) {Supports, Refutes, NoInfo}. we use these retrieved abstracts for training the rSeactoion-d, the sentence selection task functionally retrieves nale selection model that adds a loose coupling efecthe relevant rationalre1s(c{,a), …, rm(c,a)} ∈ ℛ for the between the two evidence retrieval subtasks. Finally,gfiovren claimc for each abstraac.tThe performance of stance prediction, we first segregate oNuotI{nfo} ratio- both these tasks is evaluated wpitrehcision, recall, and nale instances and then predict stanceSfuoprpo{rts,
Refutes} rationalesR. errFact achieves the fourth rankfor the veracity verification task which is formulated as inSciFact leaderboard by using language models of difer-a stance prediction problem, labeSlusp{ports, Refutes} ent BERT-variants, choosing the best performing oneafroer considered as positive labels, anNdo{Info} is taken each subtask. Our experimental results demonstrateatshtehe negative label. importance of thliosose coupling phenomenon as we only stand after computationally expensive approaches tha3t. Methodology require much larger language models and optimization for various thresholding parameters for each subtasWk.e formulate each subtask for StchieFact task as a TC t1 ... tX T[SEP] t1 ... tQ ...
...
...
... ... ... binary classification problem and create corresponding
evant abstracts for 1,409 scientific claim11s].[ These The SciFact dataset consists of a corpus with 5,183 r3e.l1-. Abstract Retrieval abstracts can either support or refute a claim with mHaenrue-, we retrieve relevant abstracts from coar1,p.u..s, { ally annotated rationales. Each claim has a unique sinagj}le∈ for claimsc ∈ . First, we calculate the TF-IDF label, and no abstract has more than three rationalessi mfoilrarity of each clai miwcith all abstractos ian ∈ and a given claim. The natural claims derived from a papreerstrict to toKp(-K = 30) similar abstracts. Second, we and the papers cited in diferent paragraphs in it makcereatereduced abstract representations (ared j) from these the language modeling subtasks challenging especiallaybstracts which is given bayredj={title, s1, sn/2, sn}. These due to added contextual scientific nuance.
are empirically the most meaningful representations for
Total Abstract(atotajl) Dif-Size Abstracts, Five Sentences(adif-5 j) Dif-Size Abstracts, Three Sentences(adif-3 j)
RerrFact’s Reduced Abstract(aredj) 72.25 74.41 68.63 79.67
BioBERT large Oracle Retrieval Oracle Retrieval + No Evidence & Cited Oracle + No Evidence Cited + (-3)*TF-IDF
RerrFact’s Loose Coupling binary classification with input sequen<ceci,[SEP],ared BioBERT-MNLI (Multiclass) 74.09 j> for obtaining all the relevant abstracts. RoBREeRrTrAFa-Lcatr’sgNe-oMInNfoLI((BMiunlatricy)lass) 8776..5184
Additionally, we obtain the above-stated representa- RerrFact’s Supports/Refutes (Binary) 82.67 tion logic by permuting diferent combinations of ab- RerrFact Classifier(Two-Step Binary) 85.23 stract sentences. For all retrieval approaches, we appTeanbdle 3 the title with diferent lengths of abstract. Keeping Ft1h-escore performances ondev set for diferent comparative language model architecture constant, for the basestliannece prediction approaches. approach, we first feed the complete abstratcottalj with the title into the model. But while appending the whole abstract due to the limitation of BERT models to t[Ta/kFe]. Binding the abstract retrieval module to the ramaximum 512 tokens as input on an average, our inputtisonale selection module while model training helps in get truncated, which possibly results in some informatiimopnrovingco-reference identification performance and loss. gives special attention only to claim relevant data.
In the second approach, we divide our abstracts intoAlso, we further analyze diferent training mechanisms diferent groups based on their size{small (≤8 * sk), for the sentence selection subtask. First, we train our medium (>8 * sk & ≤14 * sk), large (>14 * sk & ≤24 * sk), baselines only by using oracle retrieved abstract. Further, extra-large (>24 * sk & ≤Lmax * sk)}, and for each group as a new variation, we add negative label sentences for of abstracts formed, we consider the top five relativecilna-ims with no supporting/refuting evidence but only dex positions of the most frequently occurring sentenrecsepsectivecited_doc_id in the abstract corpus. Second, we for each group and sequentially append those five send-ecide to add more negative samples by adding top-three tences after the titl(eadif-5 j) as our new input sequence tofalsely retrieved abstracts from initial TF-IDF similarity ifne-tune our language model. Also, we follow the sameretrieval. Finally, we try our loose-coupling approach by methodology but limit our sentences to only top-thbrieneding training to classified abstracts only. The results sentences appended after the tit(laedif-3 j) for observing from Table3 demonstrate the importance of the binding performance and computational trade-of variationsmoenchanism & emphasize that adding negative samples smaller representations. does not necessarily improve results.
The results from Tab1ledemonstrate our final reduced retrieval representations outperforming other represen
3.3. Stance Prediction tations with its best F1-score. Our manual analysis into workings of these representations shows thaatretdjh=e{ti- In this subtask, we use the predicted rationℛâl(ec,sa) = tle, s1, sn/2, sn} method captures qualitatively best portio{rn1s(c,a), …, rm(c,a)} from the evidence retrieval stage to of the introduction, methodology & conclusion on anparve-dict the veracity(c,ŷa) of the scientific claimsc ∈ . erage. More importantly, unlike other approachesW,ietformulate this subtask as a two-stage binary classifier avoids the abstract’s numeric & additional bulk inforpmroab-lem where the first classifier separates the rationales tion components, keeping the representations compwacitth ŷ(c,a)={NoInfo} with input sequenc<eci,[SEP],rm> & precise. and the second classifier predicts the stan(cce,aŷ)={Supports,
Refutes} with input representati<ocni,[SEP],rn>. We 3.2. Rationale Selection choose the pre-trained BioBERT-MNLI language model forEnough Information detection and pre-trained RoBERTAIn this subtask, relevant evidence rationℛâl(ecs,a) = Large-MNLI for predictinCglaim Veracity. {r1(c,a), ..., rm(c,a)} are retrieved, where eacr1h(c,a) com- Further, we explore three-way classification by trainprises of s{1(c,a), ..., sk(c,a)}. We use all sentences froming the individual models of thRerrFact veracity vereach retrieved abstract from the previous stage toificfinaet-ion two-step module. We train our multiclass lantune our pre-trained BioBERT large language mo1d3e]l [ guage model classifiers namely, BioBERT-MNLI & RoBERTawith input sequenc<eci,[SEP],sa k> and binary outputLarge-MNLI for directly predicting tShuepp{orts, ReSelection-only
R
Selection+Label
R
F1
Label+Rationale
R F1 P futes, NoInfo} labels. The results in Tabl2e demon- performance boost across all metricSsciFnact. This strates the advantage of using the two-step binary cmlaossdie-l is trained with batch size one for ten epochs. We ifcation process inRerrFact for theSciFact task. We achieve anF1-score of 79.67% against thdeev set, which attribute this performance increase to better prediscthiiognher than reported QMUL-SDSF’1s-score of 74.15% ofRefutes class, as multiclass classification models per-but lower thanVerT5erini’s 89.95% F1-score. Second, formed poorly for predicting this class due to its scarfcoirtytherationale selection subtask, the BioBERT-large lanin the dataset. HencRee,rrFact’s two-step classifica- guage model attains a higher recall score inSctihFeact tion approach avoids false positive predictionNsooInffo metrics because of the loose binding between the two class against thReefutes class and improves on the claimsubtasks for evidence retrieval as parRteorrfFact’s refuting rationale prediction. system design. Though ourF1-score performance for sentence selection was 69.57% which is again less than
4. Experiment and Results set supersedes all the systems, including the T5 language models ofVerT5erini. Based on our analysis of predicIn our experiments, we analyze the performance of vtairo-ns from abstract and sentence selection subtasks, this ious language models in a standalone manner for each
performance boost largely attributes to the regularizasubtask and attempt multiple permutation settings for
tion efect created by loosely binding the two evidence our systemRerrFact as shown in Tables1, 2 and3. Table retrieval stages leading to highly accurate sentence pre
dictions for the retrieved abstracts. models inRerrFact for each subtask iSnciFact against For the final stance prediction subtask, we train both the top leaderboard systems on bodtevh and test sets. our models in the two-step approach for 30 epochs with For evaluation and reporting performance odnevthse t, batch size 1. First, thNeo{ Info} detector language model all language models for each subtask are trained only
that eliminates evidence based on their unrelatedness on thetrain set. Table4 shows the evaluation resultsto the scientific claim, achieveFs1-score of 87.14%. The against thdeev set having 300 claims. And for evaluation
second stance predictor model for evidence that either against thteest set predictions, we train our models osnupports or refutes the claim, achievesF1a-nscore of thetrain set additionally combined with 75% of dtehve 82.67%. These two-step binary classifiers for neutral and set and validate our model results over the remaining 25%
support/refute evidence classification helps in achieving of thedev set. Table5 reports thReerrFact system’s significant relative performance improvements ondtevhe capabilities in terms o1f scores against 300 claims ofset, as shown in Tabl4e’s label prediction metrics. Also, thetest set. from Table5, we observe thaRterrfact’s performance
In theabstract retrieval subtask, we empirically ob-takes a relatively large dip in terms of prediction capaserve that threeduced abstract representations substan- bilities because of the relatively lower abilities to detect tially increase our retrieval performance, leading to a
Scientific Claim (Reasoning Type, Frequency %)
Wrongly Labeled Evidence (Stance Gold Label) 1/2000 in UK have abnormal PrP positivity. (Numeric, 27.7%) ...indicating an overall prevalence of 493 per million population (95% confidence interval 282 to 801 per million){.S..upport} Hypothalamic glutamate neurotransmission is crucial to energy balance(D.irectionality, 37.9%)
...secondary to impaired fasting-induced increases in the glucoseraising pancreatic hormone glucagon and{S..u.pport} Breast cancer development is determined exclusively by genetic factors(C.ausal Efect, 34.4%)
...women who developed breast cancer... established environmental risk factors...alcohol consumption{C).ontradict} true negatives for each subtask and wrong predictipornosve upon these limitations and further explore novel on scientifically exhaustive rationales. premise assimilation architectures to create qualitatively improved veracity verification systems.
Our manual analysis shows thRaetrrFact’s increase in
performance can be attributed to its ability to pro[c1e]ssJ. DeYoung, S. Jain, N. F. Rajani, E. Lehman,
scientific background knowledge andco-references more C. Xiong, R. Socher, B. C. Wallace, ERASER: A
accurately. First, the reduced abstract representationsbenchmark to evaluate rationalized NLP models,
help in qualitatively improving tchoe-references inference in: Proceedings of the 58th Annual Meeting of the
capabilities. Second, the dynamic biological pre-trained Association for Computational Linguistics,
Associaembeddings in classifier models help in increasing thscei- tion for Computa
6. Conclusion extraction and VERification (FEVER) shared task, in: Proceedings of the First Workshop on Fact ExIn this work, our proposed systeRmerrFact demon- traction and VERification (FEVER), Association for strates that reduced evidence retrieval representationsComputational Linguistics, Brussels, Belgium, 2018, and loosely binding the evidence retrieval stages for flex- pp. 1–9. URL: https://aclanthology.org/W18-55.01 ible regularization lead to better and concise retrieved rationale sentences. Additionally, combined wRietrhrFact’s [5] dYo.Ni:1ie0,.H18. 6C5h3e/nv,1M/W. 1B8a-n5s5a0l1,.Combining fact extractwo-step stance prediction approach, it outperforms all tion and verification with neural semantic matching the other veracity verification systems onStchiFeact networks, in: Proceedings of the AAAI Conferdev set. Also, forRerrFact, the performance especially ence on Artificial Intelligence, volume 33, 2019, pp. takes a relatively high dip on ttheest set, which can be 6859–6866. attributed to a high false-positive rate otnestthset & also thatSciFact metric penalizations requiring more[6] S. Chen, D. Khashabi, W. Yin, C. Callison-Burch,
sryesgtuelmarRiezrerdFparcetdircatnioksns4thfoorn etahceShcsiuFabtctaslkea.dOeurrboparordp,osed covering diverse perspectives about claims, in: Proceedings of the 2019 Conference of the North with 62.09% F1-score for the Sentence+Label prediction American Chapter of the Association for Compumodule, while the top-performing system has an F1-score tational Linguistics: Human Language Technoloof 67.21%. As future work, we would systematically im- gies, Volume 1 (Long and Short Papers), Associ