=Paper=
{{Paper
|id=Vol-512/paper-7
|storemode=property
|title=Inferring the Public Agenda from Implicit Query Data
|pdfUrl=https://ceur-ws.org/Vol-512/paper07.pdf
|volume=Vol-512
|dblpUrl=https://dblp.org/rec/conf/sigir/Granka09
}}
==Inferring the Public Agenda from Implicit Query Data==
https://ceur-ws.org/Vol-512/paper07.pdf
Inferring the Public Agenda from Implicit Query Data
Laura Granka
Stanford University
Google, Inc
granka@stanford.edu or
granka@google.com
ABSTRACT both costly and time-consuming to instrument and analyze. These
Traditionally, implicit feedback measures are used to evaluate the measures are also susceptible to self-report or self-selection bias,
performance of a particular information retrieval system. This particularly for questions assessing civic engagement or interest in
research instead takes a distinctly applied approach to the use of public affairs [6, 7]. The ability to gauge aggregate changes in
implicit feedback, and extends the inference from aggregate query public opinion and issue awareness, in an immediate and
data to the social and political sciences. Using the three months inexpensive manner, is often the most desired alternative. One
prior to the 2008 election as a test scenario, the analysis here currently untapped tool for this is publicly available search query
assesses daily fluctuations in search coverage of candidates and data. One specific platform, Google Insights for Search [8], offers
issues as predicted by the amount of news coverage, proximity to users the ability to access daily changes in query volume for
election day, and public opinion poll ratings of the candidates. specific searches in a specific geography and time period. Existing
Findings indicate that aggregate shifts in topical search queries research using this tool has shown how search volume is both
may in fact be a useful, inexpensive indicator of political interest. indicative and predictive of external events, from flu outbreaks [9]
to economic activity [10].
Categories and Subject Descriptors
H.3.3 [Information Systems]: Information Search and Retrieval 2.1 Search Queries and Topical Interest
– relevance feedback, search process; J.4. [Social and Online search is an active medium, meaning that a user has to
Behavioral Sciences]: Sociology. explicitly make the effort to acquire information about a given
topic by manually typing in a query. Because of this, online
searches queries may be a strong behavioral indicator of what
General Terms issues and topics are at the top of a user’s mind. This, coupled
Measurement, Economics, Human Factors. with the lack of self-report bias makes search queries an attractive
way to implicitly measure fluctuations and changes in political
1. INTRODUCTION issue interest over time.
Information retrieval systems have frequently used measures of Existing political and media research has tracked changes in issue
implicit feedback to evaluate both the performance of a retrieval interest over time, though as previously mentioned, through
system and infer searcher satisfaction [1, 2]. Implicit feedback surveys or experiments. Research has repeatedly shown that
refers to measures that are unobtrusively obtained from a user public perceptions of issue importance are shaped by the amount
search session, such as clicks, queries, reading time, session of news coverage of that issue [11, 12]. In other words, the issues
length, and page scrolling. Implicit feedback has been used most receiving the greatest news coverage are judged to be the most
frequently to infer result relevance based on user click behavior important issues. Our first step is to conduct a systematic
and reading time [3, 4], and has been validated with eyetracking evaluation of whether behavioral data obtained via search query
[3]. To date, little research has applied implicit feedback to volume is also consistent with the conclusions of agenda setting
situations beyond the actual retrieval system. As usage of online research. In other words, how do real world events and news
search engines is only increasing [5], it is important to understand coverage motivate political search traffic? The issues covered
how implicit search behavior can be applied to other domains to most prominently in the media are typically the issues that people
understand broader user conditions. This research presents results judge to be most important; as such, we would expect to see these
from one such analysis, and discusses additional ways in which perceptions of importance reflected in a greater volume of online
implicit feedback can benefit the political and social sciences. searches.
Overall, we hypothesize a strong level of convergence between
2. AGGREGATE USER FEEDBACK search queries and news volume. The more interesting insights in
Much of the work done in the social sciences depends heavily on our analysis will likely be the deviant cases – instances where the
survey and experimental research. Both of these methods, while search query volume for a topic or issue exceeds what might be
extremely desirable when controlling for individual-level expected by its respective news coverage. Certain issues may be
variables (e.g., education, age, gender, political affiliation), are marked by extended periods of search activity, potentially
revealing the topics that sustain audience interest enough to
pursue additional information past the peak of news coverage.
Copyright is held by the author/owner(s).
SIGIR'09, July 19-23, 2009, Boston, USA.
3. METHODS a given term, such as whether certain terms are more severely
Standard surveys gauge public interest in political issues by first punctuated with spikes in traffic.
assessing issue awareness, and secondly, measuring perceived The selected issues varied in degrees of their newsworthiness and
importance (via a rating scale). Search queries have the advantage sensationalism. As measures of “hard news,” or substantive issues
of being able directly measure the first dimension – issue to the US, we tracked occurrences of the terms Iraq, War,
awareness. In order to perform a search, an individual already has Economy, Unemployment, Health Care, Taxes, and Education. To
to know about the topic or individual being queried for. While we assess more sensationalist or “soft news” coverage, the terms Joe
don't know the level of detailed knowledge an individual may the Plumber, Tina Fey, and Saturday Night Live were analyzed.
have about this issue, we do know that the individual knows about News coverage and query volume for each candidate’s name –
the topic and is making an effort to find out more about it. Obama, Biden, McCain, and Palin – were also obtained.
Second, perceived importance is a bit trickier to measure through
queries, but can still be done in a more indirect approach. The 4. RESULTS
degree of importance attributed to a given issue can be inferred 4.1 Election Proximity, News, and Search
from overall aggregate changes in query volume for that given
For many campaign issues, the volume of news coverage
topic. Deviation from the norm query volume can be easily
significantly influenced subsequent search volume. Table 1
exemplified with seasonal examples – for instance, using Google
presents regression results using news volume and proximity to
trends, it is clear to see that in the United States, searches such as
Election Day as predictors for search query volume. For most
mittens or gingerbread increase in December. One would expect
issues and candidates, there was a significant relationship between
to see a similar phenomenon for political issues: query volume
the issues covered in the news and the issues that people were
will reflect the rise and fall of public interest. In sum, overall
most interested in searching for. However, for the topics War,
changes in query volume, or sudden spikes in query volume, are
Unemployment, and Health Care, proximity to the election was
two potential ways to assess how prominent or "important" an
more influential than news coverage. In other words, searches for
issue may be at any given time.
these terms increased as Election Day grew closer, irrespective of
news coverage.
3.1 Data Collection
The data for this research was taken from the three months prior Table 1. Predicting Query Volume of Campaign Issues
to the 2008 presidential election – the 92 days from August 1,
Issue Econ War Unemp Taxes Iraq Hltcar Educ
2008 to October 31, 2008. Overall news coverage was measured
by counting instances of issues and political candidates being 16.44 51.61 48.43 48.64 49.39 47.79 60.30
covered in transcripts from the three major US news networks Intrcpt
(1.86) (1.72) (3.41) (2.11) (2.60) (4.21) (3.72)
(ABC, CBS, NBC). Transcripts were obtained from the
Vanderbilt transcript database. News 1.16** 0.24* 0.65 0.70** 0.98 ** 1.32# 2.42**
Trans (0.11) (0.10) (0.79) (0.25) (0.23) (0.66) (0.57)
Coverage for each candidate and issue was obtained on a daily
basis, to ascertain the changing volume of news coverage for Electn 0.19** 0.43** 0.20** 0.24** 0.40** 0.31** 0.14 *
every single day during this three-month period. As an additional Prox (0.49) (0.02) (0.07) (0.05) (0.04) (0.06) (0.06)
step, the news coverage data was normalized according to the
St. Er.
same normalization scheme as the search query data (as described Reg
8.70 5.35 16.12 9.85 9.52 16.17 14.65
below), so that when necessary, means could be compared 2
between the two data sources. R 0.81 0.82 0.13 0.48 0.59 0.25 0.23
F Stat 189 200.3 6.47 40.82 64.48 14.8 13.48
Daily query volume data was downloaded from Google Insights
for Search [8], which is publicly available online. The range of
data collected was over the same three-month period, and limited Table 2. Query Volume for Candidates and Personalities
to US websearch traffic. The purpose of this analysis is to Issue McCain Obama Palin Joe the Tina
determine the domestic effects of the US presidential campaign, Plumber Fey
so queries and news coverage were specifically chosen to Intercept ‐1.22 ‐74.40 5.09 ‐0.38 2.67
represent the US market. The query volume does not reflect the (46.39) (47.37) (2.55) (1.38) (3.60)
actual number of queries that Google received; rather, the data is News 0.78** 0.45** 0.87** 2.97** 4.59**
normalized according to the highest point in the data set, which Transcripts (0.12) (0.08) (0.09) (0.17) (0.81)
receives a score of 100 (e.g., if there were 12 million searches for
Election 0.15** 0.28** ‐0.01 0.02 0.18*
Obama on September 3, that day would receive a score of 100. If
Proximity (0.05) (0.08) (0.05) (0.03) (0.07)
there were 6 million searches for Obama on August 1, that day
would receive a score of 50). Other normalization factors are used Poll 0.09 1.69 __ __ 6.40
to account for base increases in search traffic over time due to Data (1.06) (1.06)
growth in the online population. St.Error 11.72 10.09 __ __ 17.13
Regression
The query distributions for individual issues and candidates can
then be compared with network news coverage of that issue or R2 0.50 0.66 0.57 0.82 0.38
candidate. While the query means are not useful points of F‐Stat 29.81 57.91 57.94 197.4 27.2
comparison between issues (each query resides in its own Standard errors are reported in parentheses
normalized set of data), the standard deviations may be useful, as No observations = 92
they are representative of how regular or irregular searches are for Significant p-values are indicated: ** p<.001, * p<.01, # p=.05
This may indicate that searches for these topics are driven by data was obtained from Pollster.com, which aggregates multiple
interest or perceived importance, potentially signaling that these public opinion polls, and allows users to download data [13].
issues are important to searchers. An October, 2008 Gallup report The regressions show that (as with issue searches), there is a
indicates that the key issues important to voters were the signification relationship between the volume of news queries and
economy, gas prices, Iraq, healthcare, and terrorism [14]. While the volume of searches for both candidates and entertainment
gas prices and terrorism were not included in this analysis, the personalities. As might be expected, the proximity to Election
results from this study did compare with the Gallup results, as Day was only significantly influential for the two presidential
searches for economy, Iraq and healthcare increased prior to the candidates. The hypothesis that high approval in public opinion
election (Table 1). Additionally, it was clear that broadcast news polls might influence search query volume was not supported –
did not equally cover the issues of public concern. Figure 1 shows external measures of presidential approval (i.e., polls) do not
density plots of search queries and news coverage for two issues appear to translate into increased search activity. This is
in our sample: economy and war. Economic news coverage fairly particularly interesting, as it hints that political searches may be
consistently predicts queries for economy; however, a similar valence neutral; in other words, while it may be safe to say that
trend does not exist when assessing news and queries for war. queries measure interest, we cannot make the jump to conclude
that greater search traffic also leads to support or approval.
Finally, in the final days leading up to the election, a number of
searches increased. Searches for Obama spiked, as did searches
Search queries for taxes. Prior to this, spikes in issue-based query traffic were
limited to only one or two days, but immediately prior to the
News coverage election, searches for these queries showed an increasing trend for
multiple days. Recognizing how search volume changes directly
before an electoral event could indicate the publics’ attached
importance to the particular issue.
4.3 Differences in News and Query Volume
Figures 2 and 3 present graphical differences between the news
coverage and query volume of the presidential candidates and
entertainment personalities. From these graphs, it is clear that
search volume and news coverage are punctuated by key events in
the campaign, such as political announcements and conventions.
For some of these instances, particularly with individuals such as
Sarah Palin and Joe the Plumber, who were previously unknown,
the surge in query volume can also likely be attributed to novelty
and curiosity – when a relative unknown comes on the scene, we
may expect unsustainable spikes in query volume to learn about
Search queries the newcomer.
It is also evident that news coverage of issues does not always
News coverage
generate equivalent spikes in search traffic, and furthermore,
sometimes the spikes in query volume last longer than the
increases in news coverage. Specifically, on October 16th (the day
following Joe the Plumber’s mention in the 3rd presidential
debate), searches for Joe the Plumber surpassed online search
activity for Obama and McCain, as people turned to the Internet to
find out about this previously unknown individual.
To quantitatively compare the difference between news coverage
and query volume for each candidate and entertainment persona,
we conducted Welch two-sample t-tests between the normalized
transcripts and normalized query volume. There was a comparable
Figure 1. Top: density of news coverage (black) and search amount of news coverage and query volume for Sarah Palin
queries (green) for economy. Bottom: density of news (transcripts = 17.21, queries = 17.36, t=-0.06, p =.955) and Joe
coverage (black) and search queries (blue) for war. The x-axis the Plumber (transcripts =3.76, queries = 4.73, t=-0.37, p=.71).
represents the 92 days from Aug 1 – October 31, 2009. The same was true for Obama (transcripts = 35.32, queries
30.68, t=1.76, p=.08).
4.2 Candidate Queries There were significant differences between the amount of news
Table 2 presents regression results predicting search queries for coverage and the level of query volume for John McCain and Tina
the candidates (Obama, McCain, Palin) and entertainment Fey. While McCain received significantly fewer online searches
personalities (Tina Fey, Joe the Plumber), again using news than what his news coverage might predict (news = 40.31,
volume and proximity to the election as predictors. A variable queries = 28.06, t =4.55, p <.001), Tina Fey generated
measuring public opinion approval, as assessed through polling significantly more online searches than what her news coverage
data, was also included for the two presidential candidates. This might indicate (news = 7.61, queries = 16.46, t=-3.19, p =.002).
Figure 2. News transcripts for Obama (blue), McCain (red), Figure 3. Search query volume for Obama (blue), McCain
Fey (green), Joe the Plumber (purple) (red), Fey (green), Joe the Plumber (purple)
Why might this be? For the case of Tina Fey, it is likely that
search activity surpassed news coverage because individuals 6. ACKNOWLEDGMENTS
wanted to watch (or re-watch) her SNL skits online. Searchers Thank you to Shanto Iyengar, Solomon Messing, and Hilary
were not simply seeking out information, but additional media Hutchinson who provided valuable feedback on earlier drafts of
content in the form of videos and comedy clips from the show. this paper.
McCain may have generated fewer queries than news coverage
because he was already an established Senator (whereas Obama 7. REFERENCES
was largely an unknown), and individuals felt they needed to learn
[1] Kelly, D. 2005. Implicit Feedback: Using Behavior to Infer
less about him.
Relevance. In eds, Spink, A & Cole, C. New directions in
cognitive information retrieval, Springer: Netherlands.
5. FUTURE RESEARCH
[2] Fox, S., Karnawat, K., Mydland, M., Dumais, S., White, T.
The larger scope of this research effort is to take the first step at
2005. Evaluating Implicit Measures to Improve Web Search.
assessing how implicit feedback from the search process can
ACM Transactions on Information Systems, 23, 2, 147-168.
effectively be applied towards the social sciences. The present
study analyzed how fluctuations in query volume may be [3] Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski,
influenced by news coverage and external events. The degree of G. Gay. 2007. Evaluating the Accuracy of Implicit Feedback
media influence on subsequent search activity is quite high, from Clicks and Query Reformulations in Web Search, ACM
though in several cases (unemployment, war, healthcare), searches Transactions on Information Systems (TOIS), Vol. 25, No. 2.
increased near Election Day irrespective of news coverage. [4] Radlinski, F., Kleinberg, R., Joachims, T. 2008. Learning
A logical next step is to gather real world data (e.g., Diverse Rankings with Multi-Armed Bandits. International
unemployment claims/ layoffs, Dow Index/ interest rates) to Conference on Machine Learning, Helsinki, Finland.
compare changes in query volume with actual conditions. It will [5] Fallows, Deborah. Search Engine Use. Pew. Aug 2008.
also be useful to gather public opinion data from National
Election surveys to understand how search queries may fluctuate [6] Krosnick, J., 1999. Survey Research. Annual Review of
with survey data about issue importance. Psychology, 50: 537-67.
To fully assess the impact of news, a more specific time-series [7] Hovland, C.I. 1959. Reconciling conflicting results derived
analysis comparing news volume to changes in search query from experimental and survey studies of attitude change.
volume could be particularly informative: does media coverage American Psychologist, 14, 8-17.
always precede queries? What is the lag time before a news item [8] Google Insights for Search.
becomes popularized in search volume? A multimodal analysis http://www.google.com/insights/search/
would also be interesting to more rigorously compare the spikes in
[9] Ginsberg, Mohebbi, Ginsberg, J., Mohebbi, M., Patel, R.,
query volume against the spikes in news coverage – for instance,
Brammer, L., Smolinski, M., Brilliant, L. 2008. Detecting
what is it about some media events or news that causes query
influenza epidemics using search engine query data. Nature.
volume to increase much more than one would expect given the
amount of news coverage. While this paper used network news [10] Choi, H. & Varian, H. 2009. Predicting the Present through
transcripts as the predictor for news, future analyses may attempt Google search queries. April 2.
to show whether different news sources, such as newspapers or [11] McCombs and Shaw, 1972. The Agenda-Setting Function of
web blogs, show stronger or weaker agenda setting effects. Mass Media. Public Opinion Quarterly, 26, 176-187.
Finally, the only form of implicit data used in this paper was [12] Iyengar, S. & Kinder. 1984. News that Matters: Television
aggregate query data. Subsequent analysis should also incorporate and American Opinion. Chicago: U. of Chicago Press.
other typical measures of implicit feedback, such as reading time
(to assess interest), clicks (from what sites did users acquire [13] Pollster.com http://pollster.com. Retrieved Dec 5, 2009.
information), and query reformulation patterns. These additional [14] Newport, F. 2008. Obama has key edge on key election
measures, combined with a better understanding of how the voting issues. Gallup Poll, June 24. Retrieved:
electorate is represented in online search traffic will be useful in http://www.gallup.com/poll/108331/Obama-Has-Edge-Key-
for making predictions about voter behavior or election results. Election-Issues.aspx