=Paper=
{{Paper
|id=Vol-1618/INRA_paper1
|storemode=property
|title=Measuring Actual Visitor Engagement in News Websites
|pdfUrl=https://ceur-ws.org/Vol-1618/INRA_paper1.pdf
|volume=Vol-1618
|authors= Razib Iqbal,Matthew Scott,Tarah Cleveland
|dblpUrl=https://dblp.org/rec/conf/um/IqbalSC16
}}
==Measuring Actual Visitor Engagement in News Websites==
https://ceur-ws.org/Vol-1618/INRA_paper1.pdf
Measuring Actual Visitor Engagement in News Websites
Razib Iqbal Matthew Scott Tarah Cleveland
Missouri State University Missouri State University Valley City State University
Springfield, MO, USA Springfield, MO, USA Valley City, ND, USA
riqbal@missouristate.edu matthew529@live.missouristate.edu tarah.cleveland@vcsu.edu
ABSTRACT online. According to [4], engagement refers to the degree of and
depth of visitor interaction on a website against a defined set of
As revenues from Internet advertising continue to grow, advertisers goals. Therefore, the obvious question is - How do we measure
seek popular news websites for placing advertisements in an effort visitor engagement? In [5], authors collect mouse tracking data to
to maximize profits. An important measure of how well a website correlate with visual attention. They examine the cursor behavior
is performing or how attractive it is to the advertisers is how to measure user engagement within a controlled environment.
engaged the web visitors are with that website. During our However, in reality, web visitors are in an uncontrolled
background study, we explored articles covering metrics to environment where they can be away from the computer screen or
measure online user activity and engagement. However, none of looking at another application while a specific news website is open
those proposed techniques address the need for advertisers and/or in the background.
website owners to detect tab/application switching or to find the
actual away time from a particular website. To address this need, in Besides academic research, companies in the private sector are
this paper, we propose two new metrics, focus ratio and active conducting research for marketers and suggest solutions for
ratio, along with a proof of concept tool to track web visitor optimizing web engagement in an effort to maximize advertisement
engagement more accurately. To the best of our knowledge, focus revenues. For example, in 2008, Forrester Research Inc. published
ratio and active ratio have not been proposed in the literature within the following five tools and technologies to measure engagement
the context of web analytics and visitor engagement. [7]: 1. A centralized customer data repository to establish a system
of record. 2. Analytics tools to help understand customer behavior.
3. Brand monitoring tools to amplify customer feedback cross
CCS Concepts media channels. 4. Ethnographic research to expand the breadth of
CCS → Human-centered computing → Human computer customer insight. 5. Measurement dashboards to share results with
interaction (HCI) → HCI design and evaluation methods a wide audience.
However, measurement of actual visitor engagement lacks
Keywords standardization and validated methods for measuring engagement,
especially in an online context. The current web engagement
Active ratio; focus ratio; online news; visitor engagement; web metrics include a variety of measures, e.g. session duration
analytics. (duration index), page views (click index), visits per visitor
(recency index), conversion rates, customer satisfaction, brand
1. INTRODUCTION index, interaction index, loyalty index, etc. [6]. Also, web user’s
activity can be tracked by means of teleporting, backpaging,
In the United States, Internet advertising revenues totaled hyperlinking, etc. These metrics give web advertisers a better
nearly $42.8 billion in 2013 which is a 17% increase from the year indication of the sites for best placement of their advertisements
2012 [1]. With advertising being a large part of the revenue of (ads for short).
websites, many tools have been made to track their effectiveness.
These tools have been able to track many things such as page views, Our preliminary study has revealed that there are some
bounce rates, clicks, scrolls and many more metrics. Also, fundamental problems with current measurements for web visitor
numerous methods have been proposed in the literature to measure engagement. For example, session duration which tracks visit time
user engagement (e.g. [2][3][5]), however, little has been done to on a web page does not take into consideration the physical away
assess the user engagement in terms of actual length of the user time of the web visitor from the computer or when a visitor
interaction. Therefore, our aim is to be able to track the actual switches to a different tab or application. While advertisers
interaction time of the web visitor from page load to when they commonly rely on a visitor’s time spent looking at the website,
finally close out of the page. traditional web analytics tools lack capability to measure this
accurately. Therefore, in order to address the need for an efficient
To better understand new methods for defining how web web engagement tool, in Section 3, we present two new metrics
visitor engagement is measured when tabs or applications are focus ratio and active ratio for accurately measuring session
switched, we briefly introduce the current definitions of user duration and web visitor engagement.
engagement. Authors in [9] described user engagement as the
emotional, cognitive and behavioral connection that exists between
a user and a resource. Therefore, we can say that higher user
2. ENGAGEMENT METRICS AND TOOLS
engagement refers to a website’s ability to hold the attention of a User engagement is measured at large-scale through analytic
visitor and/or motivate the visitor to participate in an activity tools assessing users’ depth of interaction with a website, which
Copyright is held by the authors. include metrics such as dwell time [9] and clickthrough rate [10].
Authors in [8] explored user comments on web pages to devise a
INRA 2016 Workshop, Halifax, Canada.
tool for measuring user engagement. Their initiative is focused on
human curiosity in online news engagement. In [9], authors counterpart of the focus method. This method is triggered when an
adopted a framework for user engagement through researching element loses focus. The blur method is used to remove focus from
existing information retrieval metrics, user engagement metrics, an element. Both focus and blur methods are supported by Chrome,
web analytics, and measures from immersion in gaming. Other web Internet Explorer, Firefox, Safari, and Opera web browsers.
engagement metrics in [4] and [6] include session duration, page Through further investigation, we found that the focus and blur
views per session, visits per visitor, loyalty, interaction, and brand methods give a lot of false positives. For example, if a user displays
index. a smaller window on top of the browser window, the browser
window loses focus, so the blur is raised. However, the user is still
Session duration is a measure of the time a visitor spends on a able to see the browser. Relying on user activity also gives a lot of
webpage in a given session. By subtracting the last timestamp on a false positive as well. Consider a user watching a video where the
given session from the first timestamp on the session, one can user may not move her mouse, click, or produce any key strokes,
calculate session duration [4]. Session duration neither takes into but the user is still active on the web page.
account the possibility of a web visitor leaving the computer with a
webpage still loaded in the browser nor does it indicate if the visitor Finally, the Page Visibility API [12] is a way for website
has changed tabs or switched applications. As long as the webpage designers to determine the visibility status of a particular web page.
is still loaded on the visitor’s screen, the session duration will It relies on the HTML5 specification. The Page Visibility API
increase - whether the visitor is engaged in the site or not. reveals when a webpage is visible or in focus. When a user switches
between tabs or minimizes the current tab, the API triggers the
Page views can be measured by dividing the number of page visibility change event to notify the developers when the visibility
views in a given period of time by the number of visits in a given state changes.
period of time [4]. This considers the number of clicks as well as
the total amount of information or content the visitor has viewed. Of the metrics and tools discussed so far, it is safe to say that
The issue with using the number of page views as a metric alone is a lot of time and research has been plugged into pinpointing a
that a visitor may be familiar with the site, and go directly to the definition of web engagement. The many tools and metrics we have
page he or she wishes to engage in. Whereas another visitor who is reviewed in our background study suggest a vast knowledge of
less familiar with the site may click through many pages before what the engaged visitor looks like compared to that of a less
finding the material they wish to engage in. The second type of engaged visitor. However, none of the definitions have addressed
visitors would show a higher engagement based on the page views the following questions - How can we determine if a visitor has
per session index, however, they may actually be much less switched tabs or applications? And, how long a user has actually
engaged in the webpage’s content. been engaged with a particular webpage?
Visits per visitor can be described as a measure of the loyalty,
frequency, and recency of a visitor to a site over a period of time 3. PROPOSED FOCUS RATIO AND
[4]. Visits per visitor is nearly a straight-forward count of the ACTIVE RATIO
number of times a visitor has returned to a webpage. Loyalty index
is used to describe visitors that have accounts with the website or We define focus ratio as the difference between the time a webpage
that visit the website at least three times per week [6]. Both visits has been loaded in a browser and the time that page was actually
per visitor and loyalty index metrics use a count of the times a visible in the active tab. For web advertisers, a higher focus ratio
visitor has been on a particular webpage. These measures tend to indicates a more attractive location for placing ads on. On the other
be less cryptic than previous metrics such as duration (time elapsed) hand, we define active ratio as the difference between the time a
and page depth (number of clicks in a page). At the same time, webpage is visible in the active tab and the user is actually
recency index and loyalty index fail to differentiate between the considered viewing or interacting with that page. Both focus ratio
visitors that are highly engaged but only visit the site once and the and active ratio will help advertisers to determine the worthiness of
visitors that visit the site frequently, but are lowly engaged in the placing an ad in a specific webpage at a relatively higher rate.
webpage’s content. The benefit of tracking the ratio of active time a user is on the site
Brand index measures the level of attention a web visitor is compared with just seeing how long they are active is that we can
giving to a site’s brand prior to landing on the site [6]. This is check areas of our websites that might be causing web visitor to
measured by examining incoming search phrases and comparing become inactive. If we have pages or news articles on our websites
those phrases to brand keywords. If the incoming phrase matches that have a very low active ratio, then we can take a look at these
the brand keyword or phrase, then the brand index measurement is pages and see what it is about these pages or news contents that
used. might be causing a user to leave the page. This is something we can
measure with our proposed active and focus ratios instead of just
Interaction index is the engagement that occurs when a visitor simply tracking the average active time of that page. Also, having
is directly interacting with the site through commenting, posting in an average active ratio for each webpage on a news website, we
forums, and/or engaging in discussion boards [6]. Like the brand will be able to find the contents that are the most appealing to the
index, interaction index will score positively if the visitor news readers. These will be the pages or contents that web visitors
completes any defined interactions during the session [4]. interact with the most, and thus we can promote these contents or
During our investigation phase, we set out to find resources categories to the prospective advertisers. Knowing which of our
that would serve as guidelines for implementing a new tool to webpages are the most active will also make the ads more effective
accurately detect web visitor engagement. Initially, we found two and better targeted. Therefore, high active and focus ratio pages
methods: focus and blur. The focus method is triggered when a will be a major selling points for ad spaces in an online news
visitor focuses on an element. This focus is often measured by website. Showing these metrics to the potential ad space buyers or
mouse movements or clicks [11]. The blur method is the bidders will help us to be able to charge a fair price for that space.
Visitor Engagement events: Active Ratio
Visitor Engagement events: Focus Ratio
Figure 1. Visitor Engagement events sent to Google Analytics
4. PROOF OF CONCEPT TOOL FOR function to increase the active time. In the clock() function of
Riveted, clockTime value is being increased by one every second
FOCUS RATIO AND ACTIVE RATIO as long as the user is considered active. If the clockTime is greater
In this section, we describe our code snippets written in JavaScript than zero and a multiple of predefined report interval, then an event
and PHP to show how various times can be measured and stored to is sent to Google Analytics with the clockTime. clockTime value
calculate and report the proposed focus and active ratios. The goal stops increasing when the user has not been active for the value in
here is to show a tool that reports these new metrics, focus ratio and idleTimeout. So, if idleTimeout was set to 30, then after 30 seconds
active ratio, and then provides an environment (in our case Google of inactivity clock() function would stop being called.
Analytics) where we can use these metrics to study the trends. To Page visibility listeners can detect if the page is the active tab in the
implement our concept, we used Riveted tool browser. We added two functionalities (hiddentTimeEvent() and
(http://riveted.parsnip.io/) for gathering user activity related data totalIdleTime()) to Riveted in order for it to keep track of how long
and to calculate the focus ratio and active ratio metrics. We then the page has been hidden. The first function is called when the
send this information to Google Analytics listener notices that the page is no longer visible, or becomes
(http://www.google.com/analytics/). hidden. This starts a timer that calls the function totalIdleTime()
every second and stores it in the variable hiddenTime.
4.1 Tracking active and hidden time totalIdleTime(), when called, increases hiddenTime variable by
With the help of the Riveted tool, we add listeners for keystrokes, one. If the user comes back to the page and it is visible again, then
mouse click, mouse movement, scroll, page visibility for each of we kill the timer.
the web pages in order to detect user activity. When it detects any
of these events it resets the idleTimeout value by calling the clock()
4.2 Tracking total time pages including a customizable dashboard to help users to get the
Tracking the total time since the page has been loaded is needed for most out of the information they are gathering. One major
us to calculate the active ratio of the user. To keep track of total improvement will be to gather focus and active ratios for certain
time, we added another two functions (totalTime() and sections of a webpage to accurately determine user engagements in
totalTimeEvent()). We call the totalTimeEvent() as soon as Riveted different sections of a webpage.
is started, i.e. the webpage has been loaded. The function
totalTime() increases visitTime value by one, and then calls the
6. REFERENCES
function totalTimeEvent(). The purpose of totalTimeEvent() is to [1] IAB Internet Advertising Revenue Report. Rep. Interactive
wait one second before calling totalTime(). This effectively stores Advertising Bureau (2014). Retrieved May 1, 2016 from
the total time since the page has been loaded, in seconds, in the http://www.iab.net/media/file/IAB_Internet_Advertising_Re
variable visitTime. venue_Report_FY_2013.pdf
[2] Lagun, D. & Lalmas, M., (2016), Understanding and
4.3 Sending ratios to Google Analytics Measuring User Engagement and Attention in Online News
Finally, we added a functionality to Riveted to calculate the Reading, International Conference On Web Search and Data
ratios based on the times we are keeping track of, and then send Mining (WSDM), San Francisco, USA.
those ratios to Google Analytics for further study of the webvisitor
[3] Barbieri, N., Silvestri, F., & Lalmas, M., (2016). Improving
behavior. We achieve this with a function that we call once when
Post-Click User Engagement on Native Ads via Survival
Riveted is started. The first part of this function adds an event to
Analysis. ACM international conference on World Wide Web
the page that waits for the page to be closed. We then calculate the
(WWW), Switzerland.
active ratio, and round it to a whole number, by taking the
clockTime and dividing it by the visitTime. This gives us the [4] Peterson, E. T. & Carrabis, J. Measuring the Immeasurable:
percentage of time the user was active on that site. Next, we Visitor Engagement. Web Analytics Demystified, (2008).
calculate the actual visible time to find the percentage of time the Retrieved May 1, 2016 from
page was visible, and not hidden. We subtract the hiddenTime from http://www.webanalyticsdemystified.com/downloads/Web_
visitTime to get the visibleTime. We then calculate focus ratio by Analytics_Demystified_and_NextStage_Global_-
taking the visibleTime and dividing it by the visitTime. This gives _Measuring_the_Immeasurable_-_Visitor_Engagement.pdf
us the percentage of time the page was visible. The last part of the [5] Arapakis, I., Lalmas, M. & Valkanas, G., (2014).
function sends these values to Google Analytics. We send an Understanding Within-Content Engagement through Pattern
‘Event’ with the category of ‘VisitorEngagement’, and label them Analysis of Mouse Gestures, ACM International Conference
as ‘Active’ or ‘Focus’ appropriately. on Information and Knowledge Management (CIKM).
In Figure 1, we show the screen captures of the sample Google [6] Beckett, L. Getting beyond Just Pageviews: Philly.com’s
Analytics dashboard containing our visitor engagement data for the Seven-part Equation for Measuring, (2010). Retrieved May
following website: http://www.squidlessgames.com/ over a period 1, 2016 from http://www.niemanlab.org/2010/10/getting-
of one week. In Figure 1.A., we show the total visitor engagement beyond-just-pageviews-philly-coms-seven-part-equation-for-
events reported versus how many users actually interacted with the measuring-online-engagement
website referring to an active ratio of 100%, 50%, and 0%. In [7] Haven, B., & Vittal, S. Five Tools and Technologies to
Figure 1.B., we show how many users (out of those who were Measure Engagement, (2008). Cambridge: Forrester
interacting with the website) departed right away (0%) or were Research Inc. Retrieved May 1, 2016 from
active for the full (100%) and half (50%) of their visit duration to https://www.adobe.com/enterprise/pdfs/five_tools_and_techn
the website. These active and focus ratios are also available for the ologies_engagement.pdf
individual webpages. For example, in the above website we hosted
two games: ‘Floe’ and ‘Vocabulisitics’. Based on the collected user [8] Arapakis, I., et al. (2014). User engagement in online News:
engagement data, we found that Floe has higher active and focus Under the scope of sentiment, interest, affect, and gaze.
ratio compared to any other pages. Journal of the Association for Information Science and
Technology. 65: 1988–2005. doi: 10.1002/asi.23096
5. CONCLUSION [9] Attfield, S., Kazai, G., Lalmas, M., & Piwowarski, B. (2011).
In this paper, we presented two new metrics and a proof of concept Towards a science of user engagement (Position Paper),
implementation to allow advertisers and/or news website owners to WSDM Workshop on User Modelling for Web Applications,
correctly detect when a visitor is physically away or changes Hong Kong.
applications in order to calculate the actual level of visitor
engagement. We do not use any pre-determined concepts to [10] Chakrabarti, D., Agarwal, D., & Josifovski, V., (2008).
characterize the user behavior or recommend news articles. Contextual advertising by combining relevance with click
However, focus and active ratio can be used to create an actual feedback, ACM international conference on World Wide Web
dwell time profiling of online news readers to help classify the news (WWW), New York, NY, USA, 417-426.
articles and rank news categories. This classification and rank [11] Kantor, I. Focus/Blur: methods and events, (2011). (Oracle)
information can be used by the news website owners to promote a JavaScript Tutorial. Retrieved May 1, 2016 from
specific category or news article to the advertisers in order to earn http://javascript.info/tutorial/focus-blur
a higher revenue. Advertisers can also choose a specific ad space, [12] Mozilla Developer Network. Using the Page Visibility API.
or place a higher bid on a particular page or news category for their (2015). Retrieved May 1, 2016 from
product promotion based on this information. Our tool detects user https://developer.mozilla.org/en-
activity by tracking the keyboard and mouse activities as well as US/docs/Web/Guide/User_experience/Using_the_Page_Visi
page visibility including tab switching in the browser. In the future, bility_API
we plan to add capabilities to compare the ratios among different