Information retrieval systems have frequently used measures of implicit feedback to evaluate both the performance of a retrieval system and infer searcher satisfaction [1,2]. Implicit feedback refers to measures that are unobtrusively obtained from a user search session, such as clicks, queries, reading time, session length, and page scrolling. Implicit feedback has been used most frequently to infer result relevance based on user click behavior and reading time [3,4], and has been validated with eyetracking [3]. To date, little research has applied implicit feedback to situations beyond the actual retrieval system. As usage of online search engines is only increasing [5], it is important to understand how implicit search behavior can be applied to other domains to understand broader user conditions. This research presents results from one such analysis, and discusses additional ways in which implicit feedback can benefit the political and social sciences.
Much of the work done in the social sciences depends heavily on survey and experimental research. Both of these methods, while extremely desirable when controlling for individual-level variables (e.g., education, age, gender, political affiliation), are both costly and time-consuming to instrument and analyze. These measures are also susceptible to self-report or self-selection bias, particularly for questions assessing civic engagement or interest in public affairs [6,7]. The ability to gauge aggregate changes in public opinion and issue awareness, in an immediate and inexpensive manner, is often the most desired alternative. One currently untapped tool for this is publicly available search query data. One specific platform, Google Insights for Search [8], offers users the ability to access daily changes in query volume for specific searches in a specific geography and time period. Existing research using this tool has shown how search volume is both indicative and predictive of external events, from flu outbreaks [9] to economic activity [10].
Online search is an active medium, meaning that a user has to explicitly make the effort to acquire information about a given topic by manually typing in a query. Because of this, online searches queries may be a strong behavioral indicator of what issues and topics are at the top of a user's mind. This, coupled with the lack of self-report bias makes search queries an attractive way to implicitly measure fluctuations and changes in political issue interest over time.
Existing political and media research has tracked changes in issue interest over time, though as previously mentioned, through surveys or experiments. Research has repeatedly shown that public perceptions of issue importance are shaped by the amount of news coverage of that issue [11,12]. In other words, the issues receiving the greatest news coverage are judged to be the most important issues. Our first step is to conduct a systematic evaluation of whether behavioral data obtained via search query volume is also consistent with the conclusions of agenda setting research. In other words, how do real world events and news coverage motivate political search traffic? The issues covered most prominently in the media are typically the issues that people judge to be most important; as such, we would expect to see these perceptions of importance reflected in a greater volume of online searches.
Overall, we hypothesize a strong level of convergence between search queries and news volume. The more interesting insights in our analysis will likely be the deviant cases -instances where the search query volume for a topic or issue exceeds what might be expected by its respective news coverage. Certain issues may be marked by extended periods of search activity, potentially revealing the topics that sustain audience interest enough to pursue additional information past the peak of news coverage.
Copyright is held by the author/owner(s). SIGIR'09, July 19-23, 2009, Boston, USA.
Standard surveys gauge public interest in political issues by first assessing issue awareness, and secondly, measuring perceived importance (via a rating scale). Search queries have the advantage of being able directly measure the first dimension -issue awareness. In order to perform a search, an individual already has to know about the topic or individual being queried for. While we don't know the level of detailed knowledge an individual may have about this issue, we do know that the individual knows about the topic and is making an effort to find out more about it.
Second, perceived importance is a bit trickier to measure through queries, but can still be done in a more indirect approach. The degree of importance attributed to a given issue can be inferred from overall aggregate changes in query volume for that given topic. Deviation from the norm query volume can be easily exemplified with seasonal examples -for instance, using Google trends, it is clear to see that in the United States, searches such as mittens or gingerbread increase in December. One would expect to see a similar phenomenon for political issues: query volume will reflect the rise and fall of public interest. In sum, overall changes in query volume, or sudden spikes in query volume, are two potential ways to assess how prominent or "important" an issue may be at any given time.
The data for this research was taken from the three months prior to the 2008 presidential election -the 92 days from August 1, 2008 to October 31, 2008. Overall news coverage was measured by counting instances of issues and political candidates being covered in transcripts from the three major US news networks (ABC, CBS, NBC). Transcripts were obtained from the Vanderbilt transcript database.
Coverage for each candidate and issue was obtained on a daily basis, to ascertain the changing volume of news coverage for every single day during this three-month period. As an additional step, the news coverage data was normalized according to the same normalization scheme as the search query data (as described below), so that when necessary, means could be compared between the two data sources.
Daily query volume data was downloaded from Google Insights for Search [8], which is publicly available online. The range of data collected was over the same three-month period, and limited to US websearch traffic. The purpose of this analysis is to determine the domestic effects of the US presidential campaign, so queries and news coverage were specifically chosen to represent the US market. The query volume does not reflect the actual number of queries that Google received; rather, the data is normalized according to the highest point in the data set, which receives a score of 100 (e.g., if there were 12 million searches for Obama on September 3, that day would receive a score of 100. If there were 6 million searches for Obama on August 1, that day would receive a score of 50). Other normalization factors are used to account for base increases in search traffic over time due to growth in the online population.
The query distributions for individual issues and candidates can then be compared with network news coverage of that issue or candidate. While the query means are not useful points of comparison between issues (each query resides in its own normalized set of data), the standard deviations may be useful, as they are representative of how regular or irregular searches are for a given term, such as whether certain terms are more severely punctuated with spikes in traffic.
The selected issues varied in degrees of their newsworthiness and sensationalism. As measures of "hard news," or substantive issues to the US, we tracked occurrences of the terms Iraq, War, Economy, Unemployment, Health Care, Taxes, and Education. To assess more sensationalist or "soft news" coverage, the terms Joe the Plumber, Tina Fey, and Saturday Night Live were analyzed. News coverage and query volume for each candidate's name -Obama, Biden, McCain, and Palin -were also obtained.
For many campaign issues, the volume of news coverage significantly influenced subsequent search volume. Table 1 presents regression results using news volume and proximity to Election Day as predictors for search query volume. For most issues and candidates, there was a significant relationship between the issues covered in the news and the issues that people were most interested in searching for. However, for the topics War, Unemployment, and Health Care, proximity to the election was more influential than news coverage. In other words, searches for these terms increased as Election Day grew closer, irrespective of news coverage. This may indicate that searches for these topics are driven by interest or perceived importance, potentially signaling that these issues are important to searchers. An October, 2008 Gallup report indicates that the key issues important to voters were the economy, gas prices, Iraq, healthcare, and terrorism [14]. While gas prices and terrorism were not included in this analysis, the results from this study did compare with the Gallup results, as searches for economy, Iraq and healthcare increased prior to the election (Table 1). Additionally, it was clear that broadcast news did not equally cover the issues of public concern. Figure 1 shows density plots of search queries and news coverage for two issues in our sample: economy and war. Economic news coverage fairly consistently predicts queries for economy; however, a similar trend does not exist when assessing news and queries for war.
Table 2 presents regression results predicting search queries for the candidates (Obama, McCain, Palin) and entertainment personalities (Tina Fey, Joe the Plumber), again using news volume and proximity to the election as predictors. A variable measuring public opinion approval, as assessed through polling data, was also included for the two presidential candidates. This data was obtained from Pollster.com, which aggregates multiple public opinion polls, and allows users to download data [13].
The regressions show that (as with issue searches), there is a signification relationship between the volume of news queries and the volume of searches for both candidates and entertainment personalities. As might be expected, the proximity to Election Day was only significantly influential for the two presidential candidates. The hypothesis that high approval in public opinion polls might influence search query volume was not supportedexternal measures of presidential approval (i.e., polls) do not appear to translate into increased search activity. This is particularly interesting, as it hints that political searches may be valence neutral; in other words, while it may be safe to say that queries measure interest, we cannot make the jump to conclude that greater search traffic also leads to support or approval.
Finally, in the final days leading up to the election, a number of searches increased. Searches for Obama spiked, as did searches for taxes. Prior to this, spikes in issue-based query traffic were limited to only one or two days, but immediately prior to the election, searches for these queries showed an increasing trend for multiple days. Recognizing how search volume changes directly before an electoral event could indicate the publics' attached importance to the particular issue.
Figures 2 and 3 present graphical differences between the news coverage and query volume of the presidential candidates and entertainment personalities. From these graphs, it is clear that search volume and news coverage are punctuated by key events in the campaign, such as political announcements and conventions. For some of these instances, particularly with individuals such as Sarah Palin and Joe the Plumber, who were previously unknown, the surge in query volume can also likely be attributed to novelty and curiosity -when a relative unknown comes on the scene, we may expect unsustainable spikes in query volume to learn about the newcomer.
It is also evident that news coverage of issues does not always generate equivalent spikes in search traffic, and furthermore, sometimes the spikes in query volume last longer than the increases in news coverage. Specifically, on October 16 th (the day following Joe the Plumber's mention in the 3rd presidential debate), searches for Joe the Plumber surpassed online search activity for Obama and McCain, as people turned to the Internet to find out about this previously unknown individual.
To quantitatively compare the difference between news coverage and query volume for each candidate and entertainment persona, Why might this be? For the case of Tina Fey, it is likely that search activity surpassed news coverage because individuals wanted to watch (or re-watch) her SNL skits online. Searchers were not simply seeking out information, but additional media content in the form of videos and comedy clips from the show.
McCain may have generated fewer queries than news coverage because he was already an established Senator (whereas Obama was largely an unknown), and individuals felt they needed to learn less about him.
The larger scope of this research effort is to take the first step at assessing how implicit feedback from the search process can effectively be applied towards the social sciences. The present study analyzed how fluctuations in query volume may be influenced by news coverage and external events. The degree of media influence on subsequent search activity is quite high, though in several cases (unemployment, war, healthcare), searches increased near Election Day irrespective of news coverage.
A logical next step is to gather real world data (e.g., unemployment claims/ layoffs, Dow Index/ interest rates) to compare changes in query volume with actual conditions. It will also be useful to gather public opinion data from National Election surveys to understand how search queries may fluctuate with survey data about issue importance.
To fully assess the impact of news, a more specific time-series analysis comparing news volume to changes in search query volume could be particularly informative: does media coverage always precede queries? What is the lag time before a news item becomes popularized in search volume? A multimodal analysis would also be interesting to more rigorously compare the spikes in query volume against the spikes in news coverage -for instance, what is it about some media events or news that causes query volume to increase much more than one would expect given the amount of news coverage. While this paper used network news transcripts as the predictor for news, future analyses may attempt to show whether different news sources, such as newspapers or web blogs, show stronger or weaker agenda setting effects.
Finally, the only form of implicit data used in this paper was aggregate query data. Subsequent analysis should also incorporate other typical measures of implicit feedback, such as reading time (to assess interest), clicks (from what sites did users acquire information), and query reformulation patterns. These additional measures, combined with a better understanding of how the voting electorate is represented in online search traffic will be useful in for making predictions about voter behavior or election results.