Traditionally, implicit feedback measures are used to evaluate the performance of a particular information retrieval system. This research instead takes a distinctly applied approach to the use of implicit feedback, and extends the inference from aggregate query data to the social and political sciences. Using the three months prior to the 2008 election as a test scenario, the analysis here assesses daily fluctuations in search coverage of candidates and issues as predicted by the amount of news coverage, proximity to election day, and public opinion poll ratings of the candidates. Findings indicate that aggregate shifts in topical search queries may in fact be a useful, inexpensive indicator of political interest.
Information retrieval systems have frequently used measures of
implicit feedback to evaluate both the performance of a retrieval
system and infer searcher satisfaction [
Much of the work done in the social sciences depends heavily on
survey and experimental research. Both of these methods, while
extremely desirable when controlling for individual-level
variables (e.g., education, age, gender, political affiliation), are
Copyright is held by the author/owner(s).
both costly and time-consuming to instrument and analyze. These
measures are also susceptible to self-report or self-selection bias,
particularly for questions assessing civic engagement or interest in
public affairs [
Online search is an active medium, meaning that a user has to explicitly make the effort to acquire information about a given topic by manually typing in a query. Because of this, online searches queries may be a strong behavioral indicator of what issues and topics are at the top of a user’s mind. This, coupled with the lack of self-report bias makes search queries an attractive way to implicitly measure fluctuations and changes in political issue interest over time.
Existing political and media research has tracked changes in issue
interest over time, though as previously mentioned, through
surveys or experiments. Research has repeatedly shown that
public perceptions of issue importance are shaped by the amount
of news coverage of that issue [
Overall, we hypothesize a strong level of convergence between search queries and news volume. The more interesting insights in our analysis will likely be the deviant cases – instances where the search query volume for a topic or issue exceeds what might be expected by its respective news coverage. Certain issues may be marked by extended periods of search activity, potentially revealing the topics that sustain audience interest enough to pursue additional information past the peak of news coverage.
Standard surveys gauge public interest in political issues by first assessing issue awareness, and secondly, measuring perceived importance (via a rating scale). Search queries have the advantage of being able directly measure the first dimension – issue awareness. In order to perform a search, an individual already has to know about the topic or individual being queried for. While we don't know the level of detailed knowledge an individual may have about this issue, we do know that the individual knows about the topic and is making an effort to find out more about it. Second, perceived importance is a bit trickier to measure through queries, but can still be done in a more indirect approach. The degree of importance attributed to a given issue can be inferred from overall aggregate changes in query volume for that given topic. Deviation from the norm query volume can be easily exemplified with seasonal examples – for instance, using Google trends, it is clear to see that in the United States, searches such as mittens or gingerbread increase in December. One would expect to see a similar phenomenon for political issues: query volume will reflect the rise and fall of public interest. In sum, overall changes in query volume, or sudden spikes in query volume, are two potential ways to assess how prominent or "important" an issue may be at any given time.
The data for this research was taken from the three months prior to the 2008 presidential election – the 92 days from August 1, 2008 to October 31, 2008. Overall news coverage was measured by counting instances of issues and political candidates being covered in transcripts from the three major US news networks (ABC, CBS, NBC). Transcripts were obtained from the Vanderbilt transcript database.
Coverage for each candidate and issue was obtained on a daily basis, to ascertain the changing volume of news coverage for every single day during this three-month period. As an additional step, the news coverage data was normalized according to the same normalization scheme as the search query data (as described below), so that when necessary, means could be compared between the two data sources.
Daily query volume data was downloaded from Google Insights
for Search [
The query distributions for individual issues and candidates can then be compared with network news coverage of that issue or candidate. While the query means are not useful points of comparison between issues (each query resides in its own normalized set of data), the standard deviations may be useful, as they are representative of how regular or irregular searches are for Issue Intrcpt News Trans Electn Prox St. Er.
Reg
R2 F Stat Issue Intercept ‐1.22
(46.39) News 0.78** Transcripts (0.12) Election Proximity Poll Data St.Error Regression
Econ 16.44 (1.86) 1.16** (0.11) 0.19** (0.49) 8.70 0.81 189
War 51.61 (1.72) 0.24* (0.10) 0.43** (0.02) 5.35 0.82 200.3 McCain 0.15** (0.05) 0.09 (1.06) 11.72 a given term, such as whether certain terms are more severely punctuated with spikes in traffic.
The selected issues varied in degrees of their newsworthiness and sensationalism. As measures of “hard news,” or substantive issues to the US, we tracked occurrences of the terms Iraq, War, Economy, Unemployment, Health Care, Taxes, and Education. To assess more sensationalist or “soft news” coverage, the terms Joe the Plumber, Tina Fey, and Saturday Night Live were analyzed. News coverage and query volume for each candidate’s name – Obama, Biden, McCain, and Palin – were also obtained.
For many campaign issues, the volume of news coverage significantly influenced subsequent search volume. Table 1 presents regression results using news volume and proximity to Election Day as predictors for search query volume. For most issues and candidates, there was a significant relationship between the issues covered in the news and the issues that people were most interested in searching for. However, for the topics War, Unemployment, and Health Care, proximity to the election was more influential than news coverage. In other words, searches for these terms increased as Election Day grew closer, irrespective of news coverage. R2 0.50 0.66 0.57 0.82 F‐Stat 29.81 57.91 57.94 197.4 Standard errors are reported in parentheses No observations = 92 Significant p-values are indicated: ** p<.001, * p<.01, # p=.05 Unemp 48.43 (3.41) 0.65 (0.79) 0.20** (0.07) 16.12 0.13 6.47 ‐74.40 (47.37) 0.45** (0.08) 0.28** (0.08) 1.69 (1.06) 10.09
Taxes 48.64 (2.11) 0.70** (0.25) 0.24** (0.05) 9.85 0.48 40.82 5.09 (2.55) 0.87** (0.09) ‐0.01 (0.05) __ __
Hltcar 47.79 (4.21) 1.32# (0.66) 0.31** (0.06) 16.17 0.25 14.8
Educ
60.30
(3.72)
2.42**
(0.57)
0.14 *
(0.06)
14.65
0.23
13.48
Tina
Fey
2.67
(3.60)
4.59**
(0.81)
0.18*
(0.07)
6.40
17.13
0.38
27.2
This may indicate that searches for these topics are driven by
interest or perceived importance, potentially signaling that these
issues are important to searchers. An October, 2008 Gallup report
indicates that the key issues important to voters were the
economy, gas prices, Iraq, healthcare, and terrorism [
News coverage
Search queries News coverage
Search queries
Table 2 presents regression results predicting search queries for
the candidates (Obama, McCain, Palin) and entertainment
personalities (Tina Fey, Joe the Plumber), again using news
volume and proximity to the election as predictors. A variable
measuring public opinion approval, as assessed through polling
data, was also included for the two presidential candidates. This
data was obtained from Pollster.com, which aggregates multiple
public opinion polls, and allows users to download data [
Figures 2 and 3 present graphical differences between the news coverage and query volume of the presidential candidates and entertainment personalities. From these graphs, it is clear that search volume and news coverage are punctuated by key events in the campaign, such as political announcements and conventions. For some of these instances, particularly with individuals such as Sarah Palin and Joe the Plumber, who were previously unknown, the surge in query volume can also likely be attributed to novelty and curiosity – when a relative unknown comes on the scene, we may expect unsustainable spikes in query volume to learn about the newcomer.
It is also evident that news coverage of issues does not always generate equivalent spikes in search traffic, and furthermore, sometimes the spikes in query volume last longer than the increases in news coverage. Specifically, on October 16th (the day following Joe the Plumber’s mention in the 3rd presidential debate), searches for Joe the Plumber surpassed online search activity for Obama and McCain, as people turned to the Internet to find out about this previously unknown individual.
To quantitatively compare the difference between news coverage and query volume for each candidate and entertainment persona, we conducted Welch two-sample t-tests between the normalized transcripts and normalized query volume. There was a comparable amount of news coverage and query volume for Sarah Palin (transcripts = 17.21, queries = 17.36, t=-0.06, p =.955) and Joe the Plumber (transcripts =3.76, queries = 4.73, t=-0.37, p=.71). The same was true for Obama (transcripts = 35.32, queries 30.68, t=1.76, p=.08).
There were significant differences between the amount of news coverage and the level of query volume for John McCain and Tina Fey. While McCain received significantly fewer online searches than what his news coverage might predict (news = 40.31, queries = 28.06, t =4.55, p <.001), Tina Fey generated significantly more online searches than what her news coverage might indicate (news = 7.61, queries = 16.46, t=-3.19, p =.002). Why might this be? For the case of Tina Fey, it is likely that search activity surpassed news coverage because individuals wanted to watch (or re-watch) her SNL skits online. Searchers were not simply seeking out information, but additional media content in the form of videos and comedy clips from the show. McCain may have generated fewer queries than news coverage because he was already an established Senator (whereas Obama was largely an unknown), and individuals felt they needed to learn less about him.
The larger scope of this research effort is to take the first step at assessing how implicit feedback from the search process can effectively be applied towards the social sciences. The present study analyzed how fluctuations in query volume may be influenced by news coverage and external events. The degree of media influence on subsequent search activity is quite high, though in several cases (unemployment, war, healthcare), searches increased near Election Day irrespective of news coverage. A logical next step is to gather real world data (e.g., unemployment claims/ layoffs, Dow Index/ interest rates) to compare changes in query volume with actual conditions. It will also be useful to gather public opinion data from National Election surveys to understand how search queries may fluctuate with survey data about issue importance.
To fully assess the impact of news, a more specific time-series analysis comparing news volume to changes in search query volume could be particularly informative: does media coverage always precede queries? What is the lag time before a news item becomes popularized in search volume? A multimodal analysis would also be interesting to more rigorously compare the spikes in query volume against the spikes in news coverage – for instance, what is it about some media events or news that causes query volume to increase much more than one would expect given the amount of news coverage. While this paper used network news transcripts as the predictor for news, future analyses may attempt to show whether different news sources, such as newspapers or web blogs, show stronger or weaker agenda setting effects. Finally, the only form of implicit data used in this paper was aggregate query data. Subsequent analysis should also incorporate other typical measures of implicit feedback, such as reading time (to assess interest), clicks (from what sites did users acquire information), and query reformulation patterns. These additional measures, combined with a better understanding of how the voting electorate is represented in online search traffic will be useful in for making predictions about voter behavior or election results.
Thank you to Shanto Iyengar, Solomon Messing, and Hilary Hutchinson who provided valuable feedback on earlier drafts of this paper.