Inferring the Public Agenda from Implicit Query Data Laura Granka Stanford University Google, Inc granka@stanford.edu or granka@google.com ABSTRACT both costly and time-consuming to instrument and analyze. These Traditionally, implicit feedback measures are used to evaluate the measures are also susceptible to self-report or self-selection bias, performance of a particular information retrieval system. This particularly for questions assessing civic engagement or interest in research instead takes a distinctly applied approach to the use of public affairs [6, 7]. The ability to gauge aggregate changes in implicit feedback, and extends the inference from aggregate query public opinion and issue awareness, in an immediate and data to the social and political sciences. Using the three months inexpensive manner, is often the most desired alternative. One prior to the 2008 election as a test scenario, the analysis here currently untapped tool for this is publicly available search query assesses daily fluctuations in search coverage of candidates and data. One specific platform, Google Insights for Search [8], offers issues as predicted by the amount of news coverage, proximity to users the ability to access daily changes in query volume for election day, and public opinion poll ratings of the candidates. specific searches in a specific geography and time period. Existing Findings indicate that aggregate shifts in topical search queries research using this tool has shown how search volume is both may in fact be a useful, inexpensive indicator of political interest. indicative and predictive of external events, from flu outbreaks [9] to economic activity [10]. Categories and Subject Descriptors H.3.3 [Information Systems]: Information Search and Retrieval 2.1 Search Queries and Topical Interest – relevance feedback, search process; J.4. [Social and Online search is an active medium, meaning that a user has to Behavioral Sciences]: Sociology. explicitly make the effort to acquire information about a given topic by manually typing in a query. Because of this, online searches queries may be a strong behavioral indicator of what General Terms issues and topics are at the top of a user’s mind. This, coupled Measurement, Economics, Human Factors. with the lack of self-report bias makes search queries an attractive way to implicitly measure fluctuations and changes in political 1. INTRODUCTION issue interest over time. Information retrieval systems have frequently used measures of Existing political and media research has tracked changes in issue implicit feedback to evaluate both the performance of a retrieval interest over time, though as previously mentioned, through system and infer searcher satisfaction [1, 2]. Implicit feedback surveys or experiments. Research has repeatedly shown that refers to measures that are unobtrusively obtained from a user public perceptions of issue importance are shaped by the amount search session, such as clicks, queries, reading time, session of news coverage of that issue [11, 12]. In other words, the issues length, and page scrolling. Implicit feedback has been used most receiving the greatest news coverage are judged to be the most frequently to infer result relevance based on user click behavior important issues. Our first step is to conduct a systematic and reading time [3, 4], and has been validated with eyetracking evaluation of whether behavioral data obtained via search query [3]. To date, little research has applied implicit feedback to volume is also consistent with the conclusions of agenda setting situations beyond the actual retrieval system. As usage of online research. In other words, how do real world events and news search engines is only increasing [5], it is important to understand coverage motivate political search traffic? The issues covered how implicit search behavior can be applied to other domains to most prominently in the media are typically the issues that people understand broader user conditions. This research presents results judge to be most important; as such, we would expect to see these from one such analysis, and discusses additional ways in which perceptions of importance reflected in a greater volume of online implicit feedback can benefit the political and social sciences. searches. Overall, we hypothesize a strong level of convergence between 2. AGGREGATE USER FEEDBACK search queries and news volume. The more interesting insights in Much of the work done in the social sciences depends heavily on our analysis will likely be the deviant cases – instances where the survey and experimental research. Both of these methods, while search query volume for a topic or issue exceeds what might be extremely desirable when controlling for individual-level expected by its respective news coverage. Certain issues may be variables (e.g., education, age, gender, political affiliation), are marked by extended periods of search activity, potentially revealing the topics that sustain audience interest enough to pursue additional information past the peak of news coverage. Copyright is held by the author/owner(s). SIGIR'09, July 19-23, 2009, Boston, USA. 3. METHODS a given term, such as whether certain terms are more severely Standard surveys gauge public interest in political issues by first punctuated with spikes in traffic. assessing issue awareness, and secondly, measuring perceived The selected issues varied in degrees of their newsworthiness and importance (via a rating scale). Search queries have the advantage sensationalism. As measures of “hard news,” or substantive issues of being able directly measure the first dimension – issue to the US, we tracked occurrences of the terms Iraq, War, awareness. In order to perform a search, an individual already has Economy, Unemployment, Health Care, Taxes, and Education. To to know about the topic or individual being queried for. While we assess more sensationalist or “soft news” coverage, the terms Joe don't know the level of detailed knowledge an individual may the Plumber, Tina Fey, and Saturday Night Live were analyzed. have about this issue, we do know that the individual knows about News coverage and query volume for each candidate’s name – the topic and is making an effort to find out more about it. Obama, Biden, McCain, and Palin – were also obtained. Second, perceived importance is a bit trickier to measure through queries, but can still be done in a more indirect approach. The 4. RESULTS degree of importance attributed to a given issue can be inferred 4.1 Election Proximity, News, and Search from overall aggregate changes in query volume for that given For many campaign issues, the volume of news coverage topic. Deviation from the norm query volume can be easily significantly influenced subsequent search volume. Table 1 exemplified with seasonal examples – for instance, using Google presents regression results using news volume and proximity to trends, it is clear to see that in the United States, searches such as Election Day as predictors for search query volume. For most mittens or gingerbread increase in December. One would expect issues and candidates, there was a significant relationship between to see a similar phenomenon for political issues: query volume the issues covered in the news and the issues that people were will reflect the rise and fall of public interest. In sum, overall most interested in searching for. However, for the topics War, changes in query volume, or sudden spikes in query volume, are Unemployment, and Health Care, proximity to the election was two potential ways to assess how prominent or "important" an more influential than news coverage. In other words, searches for issue may be at any given time. these terms increased as Election Day grew closer, irrespective of news coverage. 3.1 Data Collection The data for this research was taken from the three months prior Table 1. Predicting Query Volume of Campaign Issues to the 2008 presidential election – the 92 days from August 1, Issue Econ War Unemp Taxes Iraq Hltcar Educ 2008 to October 31, 2008. Overall news coverage was measured by counting instances of issues and political candidates being 16.44 51.61 48.43 48.64 49.39 47.79 60.30 covered in transcripts from the three major US news networks Intrcpt (1.86) (1.72) (3.41) (2.11) (2.60) (4.21) (3.72) (ABC, CBS, NBC). Transcripts were obtained from the Vanderbilt transcript database. News 1.16** 0.24* 0.65 0.70** 0.98 ** 1.32# 2.42** Trans (0.11) (0.10) (0.79) (0.25) (0.23) (0.66) (0.57) Coverage for each candidate and issue was obtained on a daily basis, to ascertain the changing volume of news coverage for Electn 0.19** 0.43** 0.20** 0.24** 0.40** 0.31** 0.14 * every single day during this three-month period. As an additional Prox (0.49) (0.02) (0.07) (0.05) (0.04) (0.06) (0.06) step, the news coverage data was normalized according to the St. Er. same normalization scheme as the search query data (as described Reg 8.70 5.35 16.12 9.85 9.52 16.17 14.65 below), so that when necessary, means could be compared 2 between the two data sources. R 0.81 0.82 0.13 0.48 0.59 0.25 0.23 F Stat 189 200.3 6.47 40.82 64.48 14.8 13.48 Daily query volume data was downloaded from Google Insights for Search [8], which is publicly available online. The range of data collected was over the same three-month period, and limited Table 2. Query Volume for Candidates and Personalities to US websearch traffic. The purpose of this analysis is to Issue McCain Obama Palin Joe the Tina determine the domestic effects of the US presidential campaign, Plumber Fey so queries and news coverage were specifically chosen to Intercept ‐1.22 ‐74.40 5.09 ‐0.38 2.67 represent the US market. The query volume does not reflect the (46.39) (47.37) (2.55) (1.38) (3.60) actual number of queries that Google received; rather, the data is News 0.78** 0.45** 0.87** 2.97** 4.59** normalized according to the highest point in the data set, which Transcripts (0.12) (0.08) (0.09) (0.17) (0.81) receives a score of 100 (e.g., if there were 12 million searches for Election 0.15** 0.28** ‐0.01 0.02 0.18* Obama on September 3, that day would receive a score of 100. If Proximity (0.05) (0.08) (0.05) (0.03) (0.07) there were 6 million searches for Obama on August 1, that day would receive a score of 50). Other normalization factors are used Poll 0.09 1.69 __ __ 6.40 to account for base increases in search traffic over time due to Data (1.06) (1.06) growth in the online population. St.Error 11.72 10.09 __ __ 17.13 Regression The query distributions for individual issues and candidates can then be compared with network news coverage of that issue or R2 0.50 0.66 0.57 0.82 0.38 candidate. While the query means are not useful points of F‐Stat 29.81 57.91 57.94 197.4 27.2 comparison between issues (each query resides in its own Standard errors are reported in parentheses normalized set of data), the standard deviations may be useful, as No observations = 92 they are representative of how regular or irregular searches are for Significant p-values are indicated: ** p<.001, * p<.01, # p=.05 This may indicate that searches for these topics are driven by data was obtained from Pollster.com, which aggregates multiple interest or perceived importance, potentially signaling that these public opinion polls, and allows users to download data [13]. issues are important to searchers. An October, 2008 Gallup report The regressions show that (as with issue searches), there is a indicates that the key issues important to voters were the signification relationship between the volume of news queries and economy, gas prices, Iraq, healthcare, and terrorism [14]. While the volume of searches for both candidates and entertainment gas prices and terrorism were not included in this analysis, the personalities. As might be expected, the proximity to Election results from this study did compare with the Gallup results, as Day was only significantly influential for the two presidential searches for economy, Iraq and healthcare increased prior to the candidates. The hypothesis that high approval in public opinion election (Table 1). Additionally, it was clear that broadcast news polls might influence search query volume was not supported – did not equally cover the issues of public concern. Figure 1 shows external measures of presidential approval (i.e., polls) do not density plots of search queries and news coverage for two issues appear to translate into increased search activity. This is in our sample: economy and war. Economic news coverage fairly particularly interesting, as it hints that political searches may be consistently predicts queries for economy; however, a similar valence neutral; in other words, while it may be safe to say that trend does not exist when assessing news and queries for war. queries measure interest, we cannot make the jump to conclude that greater search traffic also leads to support or approval. Finally, in the final days leading up to the election, a number of searches increased. Searches for Obama spiked, as did searches Search queries for taxes. Prior to this, spikes in issue-based query traffic were limited to only one or two days, but immediately prior to the News coverage election, searches for these queries showed an increasing trend for multiple days. Recognizing how search volume changes directly before an electoral event could indicate the publics’ attached importance to the particular issue. 4.3 Differences in News and Query Volume Figures 2 and 3 present graphical differences between the news coverage and query volume of the presidential candidates and entertainment personalities. From these graphs, it is clear that search volume and news coverage are punctuated by key events in the campaign, such as political announcements and conventions. For some of these instances, particularly with individuals such as Sarah Palin and Joe the Plumber, who were previously unknown, the surge in query volume can also likely be attributed to novelty and curiosity – when a relative unknown comes on the scene, we may expect unsustainable spikes in query volume to learn about Search queries the newcomer. It is also evident that news coverage of issues does not always News coverage generate equivalent spikes in search traffic, and furthermore, sometimes the spikes in query volume last longer than the increases in news coverage. Specifically, on October 16th (the day following Joe the Plumber’s mention in the 3rd presidential debate), searches for Joe the Plumber surpassed online search activity for Obama and McCain, as people turned to the Internet to find out about this previously unknown individual. To quantitatively compare the difference between news coverage and query volume for each candidate and entertainment persona, we conducted Welch two-sample t-tests between the normalized transcripts and normalized query volume. There was a comparable Figure 1. Top: density of news coverage (black) and search amount of news coverage and query volume for Sarah Palin queries (green) for economy. Bottom: density of news (transcripts = 17.21, queries = 17.36, t=-0.06, p =.955) and Joe coverage (black) and search queries (blue) for war. The x-axis the Plumber (transcripts =3.76, queries = 4.73, t=-0.37, p=.71). represents the 92 days from Aug 1 – October 31, 2009. The same was true for Obama (transcripts = 35.32, queries 30.68, t=1.76, p=.08). 4.2 Candidate Queries There were significant differences between the amount of news Table 2 presents regression results predicting search queries for coverage and the level of query volume for John McCain and Tina the candidates (Obama, McCain, Palin) and entertainment Fey. While McCain received significantly fewer online searches personalities (Tina Fey, Joe the Plumber), again using news than what his news coverage might predict (news = 40.31, volume and proximity to the election as predictors. A variable queries = 28.06, t =4.55, p <.001), Tina Fey generated measuring public opinion approval, as assessed through polling significantly more online searches than what her news coverage data, was also included for the two presidential candidates. This might indicate (news = 7.61, queries = 16.46, t=-3.19, p =.002). Figure 2. News transcripts for Obama (blue), McCain (red), Figure 3. Search query volume for Obama (blue), McCain Fey (green), Joe the Plumber (purple) (red), Fey (green), Joe the Plumber (purple) Why might this be? For the case of Tina Fey, it is likely that search activity surpassed news coverage because individuals 6. ACKNOWLEDGMENTS wanted to watch (or re-watch) her SNL skits online. Searchers Thank you to Shanto Iyengar, Solomon Messing, and Hilary were not simply seeking out information, but additional media Hutchinson who provided valuable feedback on earlier drafts of content in the form of videos and comedy clips from the show. this paper. McCain may have generated fewer queries than news coverage because he was already an established Senator (whereas Obama 7. REFERENCES was largely an unknown), and individuals felt they needed to learn [1] Kelly, D. 2005. Implicit Feedback: Using Behavior to Infer less about him. Relevance. In eds, Spink, A & Cole, C. New directions in cognitive information retrieval, Springer: Netherlands. 5. FUTURE RESEARCH [2] Fox, S., Karnawat, K., Mydland, M., Dumais, S., White, T. The larger scope of this research effort is to take the first step at 2005. Evaluating Implicit Measures to Improve Web Search. assessing how implicit feedback from the search process can ACM Transactions on Information Systems, 23, 2, 147-168. effectively be applied towards the social sciences. The present study analyzed how fluctuations in query volume may be [3] Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, influenced by news coverage and external events. The degree of G. Gay. 2007. Evaluating the Accuracy of Implicit Feedback media influence on subsequent search activity is quite high, from Clicks and Query Reformulations in Web Search, ACM though in several cases (unemployment, war, healthcare), searches Transactions on Information Systems (TOIS), Vol. 25, No. 2. increased near Election Day irrespective of news coverage. [4] Radlinski, F., Kleinberg, R., Joachims, T. 2008. Learning A logical next step is to gather real world data (e.g., Diverse Rankings with Multi-Armed Bandits. International unemployment claims/ layoffs, Dow Index/ interest rates) to Conference on Machine Learning, Helsinki, Finland. compare changes in query volume with actual conditions. It will [5] Fallows, Deborah. Search Engine Use. Pew. Aug 2008. also be useful to gather public opinion data from National Election surveys to understand how search queries may fluctuate [6] Krosnick, J., 1999. Survey Research. Annual Review of with survey data about issue importance. Psychology, 50: 537-67. To fully assess the impact of news, a more specific time-series [7] Hovland, C.I. 1959. Reconciling conflicting results derived analysis comparing news volume to changes in search query from experimental and survey studies of attitude change. volume could be particularly informative: does media coverage American Psychologist, 14, 8-17. always precede queries? What is the lag time before a news item [8] Google Insights for Search. becomes popularized in search volume? A multimodal analysis http://www.google.com/insights/search/ would also be interesting to more rigorously compare the spikes in [9] Ginsberg, Mohebbi, Ginsberg, J., Mohebbi, M., Patel, R., query volume against the spikes in news coverage – for instance, Brammer, L., Smolinski, M., Brilliant, L. 2008. Detecting what is it about some media events or news that causes query influenza epidemics using search engine query data. Nature. volume to increase much more than one would expect given the amount of news coverage. While this paper used network news [10] Choi, H. & Varian, H. 2009. Predicting the Present through transcripts as the predictor for news, future analyses may attempt Google search queries. April 2. to show whether different news sources, such as newspapers or [11] McCombs and Shaw, 1972. The Agenda-Setting Function of web blogs, show stronger or weaker agenda setting effects. Mass Media. Public Opinion Quarterly, 26, 176-187. Finally, the only form of implicit data used in this paper was [12] Iyengar, S. & Kinder. 1984. News that Matters: Television aggregate query data. Subsequent analysis should also incorporate and American Opinion. Chicago: U. of Chicago Press. other typical measures of implicit feedback, such as reading time (to assess interest), clicks (from what sites did users acquire [13] Pollster.com http://pollster.com. Retrieved Dec 5, 2009. information), and query reformulation patterns. These additional [14] Newport, F. 2008. Obama has key edge on key election measures, combined with a better understanding of how the voting issues. Gallup Poll, June 24. Retrieved: electorate is represented in online search traffic will be useful in http://www.gallup.com/poll/108331/Obama-Has-Edge-Key- for making predictions about voter behavior or election results. Election-Issues.aspx