Fact Checking and Information Retrieval Matthew Lease School of Information, University of Texas at Austin, USA ml@utexas.edu INTRODUCTION. Designating October 2009 as National Infor- online narrative through only a single source, we can instead ex- mation Literacy1 Awareness Month, former U.S. President Barack plore how broader community discourse has shaped its develop- Obama promoted a key 21st century information challenge: “Though ment . . . [especially for] campaigns which flood social media with we may know how to find the information we need, we must also repeated stock phrases while obfuscating their . . . source.” know how to evaluate it. Over the past decade, we have seen a crisis This year, we proposed a graphical modeling approach to fact- of authenticity emerge. We now live in a world where anyone can checking which augments the efficiency and scalability of auto- publish an opinion or perspective, whether true or not, and have mated information retrieval (IR) with transparent, explainable ML that opinion amplified within the information marketplace.” [1]. Given a claim as query, the system first finds and retrieves Historically, we have relied upon Information Literacy educa- relevant articles from a variety of sources. It then infers the degree tion to teach key critical reading skills, use of multiple sources, to which each article supports or refutes the claim, as well as the and potential for source bias. However, today’s era of information reputation of each source. The system then aggregates this body of overload presents an historic stress test of traditional information evidence to predict the veracity of the claim, showing the user pre- literacy skills. Information tracking and sense-making has become cisely which information is being used and the various sources of increasingly difficult, along with effort required to consistently and model uncertainty underlying the overall claim prediction. We also effectively cross-check sources by hand. The rise of misinformation evaluated a hybrid variant in which the system integrates online – unwitting or deliberate – has further exacerbated this. crowd workers to further improve predictive accuracy. In response, researchers in natural language processing (NLP) In our most recent work (under review), we prototyped3 a mixed- and machine learning (ML) have proposed a variety of innovative initiative design for incorporating user knowledge and beliefs into new models to automatically fact-check claims. However, these model predictions. The model’s predicted source reputation and works have largely approached fact-checking as a fully-automated stance for each retrieved article is shown to the user and can be task focused whose primary goal it to maximize predictive accuracy. revised via simple sliders to reflect user beliefs and/or to correct While accurate predictions are important, someone skeptical of on- erroneousness model estimates. The overall claim prediction is then line information is likely to be equally skeptical of any fact-checking updated visually in real-time as the user interacts with the system. tool. Thus, a system should also be transparent (and auditable) in In a user study asking participants to assess claims using variant how it made a prediction so that a user can understand and trust the systems and interaction mechanisms, we found that users tend to model. In addition, an individual’s claim assessments will invariably highly trust model predictions, benefiting from the model when rely at least in part on that person’s prior world-views regarding the it is correct, but also (unfortunately) falling victim to its errors. perceived credibility of claims and sources. A fact-checking system Given the option to interact with these incorrect predictions, how- should be open-ended to integrate user beliefs, letting users easily ever, users were able to do so and improve their own performance, inject their own views and knowledge into the system. Finally, a emphasizing the need for interpretable, interactive models. model should communicate the uncertainty in its predictions while accounting for potential sources of errors, empowering users to IR RESEARCH QUESTIONS. What can IR bring to fact check- conduct their own in-depth inspection and reasoning. ing that NLP and ML do not, and what new questions does fact checking raise for IR? As an explicit or implicit task in IR, fact OUR WORK. In 2012, we prototyped2 a system for searching and checking has various implications for both system-centered and browsing memes underlying news: similar phrases which spread user-centered IR: which results should we return, how should we and evolved across sources [3]. Once detected, these latent memes present them, what modes of interaction should we provide, and were revealed to users via generated hypertext, allowing memes how should we evaluate success? While assessing the authority of to be recognized, interpreted, and explored in context. “Our vision pages for ranking and filtering is not new, fact checking presents a [was] to complement traditional forms of critical literacy education different framing of authority, with ranking and filtering decisions with . . . smarter browsing technology . . . Instead of understanding potentially impacting user trust of the system and fears of being manipulated4 . Beyond topical diversification of search results, how might we diversify political (or other forms of) bias to provide di- 1 en.wikipedia.org/wiki/Information_literacy verse perspectives, especially on controversial topics? How should 2 odyssey.ischool.utexas.edu/mb/ we rank and evaluate search results based on such diversity? In terms of personalized search, how do we balance giving users search results matching their existing beliefs vs. challenging those beliefs with alternative viewpoints, and without such challenges DESIRES 2018, August 2018, Bertinoro, Italy © 2018 Copyright held by the owner/author(s). 3 fcweb.pythonanywhere.com 4 fortune.com/2015/08/23/research-google-rig-election DESIRES 2018, August 2018, Bertinoro, Italy Matthew Lease prompting search engine switching behavior? Just as people choose different news outlets to follow having different political leanings, perhaps a new class of vertical search engines will soon arise which rank and filter search results to match a given audience’s views? While search results have traditionally been evaluated with re- spect to gain (i.e., how much the user benefits), recent work has explored the idea that, as with any technology, search engines and their results also have the potential to inflict harm on users [2]. How do we frame, measure, and address potential harm of search results including “alternative” facts, be they search result errors or intentional diversification? How do we evaluate traditional infor- mation gain alongside not only viewpoint diversification, but also potential varying severity of harm to varying numbers of users? Ultimately, what is the duty and opportunity for IR in fact check- ing? How do we effectively present model predictions to benefit users while also conveying uncertainty, supporting mixed-initiative decision making and creating interfaces inviting user exploration? REFERENCES [1] An T. Nguyen, Aditya Kharosekar, Matthew Lease, and Byron C. Wallace. 2018. An Interpretable Joint Graphical Model for Fact-Checking from Crowds. In AAAI. [2] Frances A Pogacar, Amira Ghenai, Mark D Smucker, and Charles LA Clarke. 2017. The Positive and Negative Influence of Search Results on People’s Decisions about the Efficacy of Medical Treatments. In ACM ICTIR. 209–216. [3] Hohyon Ryu, Matthew Lease, and Nicholas Woodward. 2012. Finding and Explor- ing Memes in Social Media. In ACM Hypertext. 295–304.