=Paper=
{{Paper
|id=Vol-1642/paper3
|storemode=property
|title=Gamification for WebSAIL
|pdfUrl=https://ceur-ws.org/Vol-1642/paper3.pdf
|volume=Vol-1642
|authors=Ioannis Karatassis,Norbert Fuhr
|dblpUrl=https://dblp.org/rec/conf/sigir/KaratassisF16
}}
==Gamification for WebSAIL==
https://ceur-ws.org/Vol-1642/paper3.pdf
Gamification for WebSAIL
Ioannis Karatassis Norbert Fuhr
Institute for Information Systems Institute for Information Systems
University of Duisburg-Essen, Germany University of Duisburg-Essen, Germany
karatassis@is.inf.uni-due.de norbert.fuhr@uni-due.de
1 Introduction
Since its establishment in 1991, the World Wide Web
Abstract
exceeded the enormous amount of 4 billion indexed
Recent studies reveal that most web search web pages1 that need to be usable and accessible to
engine users lack appropriate strategies for everyone all over the world. Web search engines help
finding relevant results efficiently and e↵ec- Internet users in satisfying information needs by of-
tively. They eventually miss out on important fering links to web pages that match a search query.
information, need significantly more time for Despite of frequent use of web search engines, the ma-
searching than high search literate users, and jority of users are little or no search literate at all which
overestimate their skills in the domain of web results in bad or unsatisfied search results. They might
search. In this paper, we introduce the on- not be even aware of better search results as search en-
going WebSAIL project (Web Search literacy, gines give almost to every query an answer.
Awareness, Interface complexity, and Long- Researchers constantly invent new or tweak present
term e↵ects) that tackles these problems. The algorithms to increase the e↵ectiveness of search en-
prime aim is to enhance web search literacy gines to satisfy a user’s information need as best it
(WSL) by teaching search strategies and basic can be. Instead of focusing on the system, WebSAIL
and advanced concepts of web search engines focuses on the user him/herself to make him/her a
in the form of a task based application. Users better searcher. The following section gives an insight
will be introduced to more complex search in- into the characteristics of WSL and gamification.
terfaces (containing more than a single query
box) as in some cases these would lead to bet- 2 Related Work
ter results but are hardly used due to the lack
2.1 Web search literacy
of motivation. Furthermore, WebSAIL fo-
cuses on long-lasting enhancements: acquired Under the term search literacy, we summarize skills re-
abilities should be sustainable instead of a quired to satisfy an information need through search-
temporary nature only. ing in well-known sources. It is a key concept of in-
formation literacy – the ability to know when there
The present project follows two approaches for
is a need for information, to be able to identify, lo-
enhancing WSL: online-tutorials and gamifi-
cate, evaluate, and e↵ectively use that information.
cation. After evaluating each method on its
We divide web search literacy into the fields below each
own, we will compare the results to answer 1)
of which targeting a specific aspect and thus helps in
whether WSL could be enhanced and 2) which
identifying and characterizing literate users.
approach suited best our goals. Moreover, by
evaluating same subjects after a certain pe-
Searchability This field encompasses knowledge
riod, we 3) will be capable of noticing whether
about the basic functioning of web search engines,
acquired skills remained sustainable.
i.e., the crawler, indexer, and searcher. Users
Copyright c by the paper’s authors. Copying permitted for should be aware of documents that are not in-
private and academic purposes. dexed and thus not searchable at all. We will
In: F. Hopfgartner, G. Kazai, U. Kruschwitz, and M. Meder teach our users the general aspects of indexing,
(eds.): Proceedings of the GamifIR 2016 Workshop, Pisa, Italy,
21-July-2016, published at http://ceur-ws.org 1 http://www.worldwidewebsize.com/ Accessed July 01, 2016.
e.g., linking, anchor texts, access rights, document sessions lasting more than 30 minutes. Furthermore,
types, topicality of the index, and language de- about 59% of users returned to search page after their
pendency. In summary, users are to be aware of first click. These findings show that there is a need to
which and why documents are searchable. elaborate on search skills of web search engine users.
One possible approach to increase web search lit-
Linguistic functions target word normalization (re-
eracy through a task based application is the use of
garding the use of capital and small letters, vari-
gamification which is presented in the following sec-
eties of spelling), lemmatization, nominal phrases
tion.
and composites, and synonyms.
Query language allows users for specifying their in- 2.2 Gamification
formation need more precisely to the system. A
query language enables the use of, e.g., disjunc- The concept turned into a hot topic in the past few
tion, negation, url operators, restrictions related years as it seems to be a promising approach both in
to time, language, or specific document fields like industry and academia to foster user engagement. It
the URL, title, text, or anchors. Furthermore it is commonly described as “the use of game design el-
allows searching for other document types than ements in non-game contexts” [9]. The idea behind
web, e.g., images, maps, videos, news, or shop- gamification sounds straightforward: rewarding users
ping. for completing tasks in a game-like and playful envi-
ronment with the objective of increasing motivation.
Ranking denotes the order in which search results are Gamification is most often manifested in the form
displayed to the users. But what exactly makes of points, badges, and leaderboards (also called the
a document appearing at the top position? Users PBLs) [9, 7] – game design elements that appear in
will learn about important concepts like PageR- almost every game. In fact, gamification is much more
ank, user clicks, or diversity to help for a deeper than PBLs. The key is to focus on deep game mechan-
understanding of the construction of result sets. ics and well constructed elements of strategy to create
Search tactics allow for continuing a search. Bates value and engage the user.
distinguishes between monitoring tactics to keep There are plenty of good case studies and exam-
the search on track and efficient, file structure tac- ples that confirm the positive impact of gamification
tics to thread one’s way though a file structure, [6]; online communities across enterprise, sales, ed-
search formulation tactics that aim at designing ucation, lifestyle, health, and financial services have
and redesigning a search formulation, and term experienced great success. Google, for instance, de-
tactics that help for selecting and revising specific signed a travel expense system and resulted in 100%
terms within a search formulation [3, 4]. We at- employee compliance within six months of its launch
tempt for teaching users the most relevant search [1]. Beat the GMAT, a large online community for
tactics for the web. students preparing for MBA programs, managed to
increase pages-per-visit by 195% and time users spent
Stratagems according to Bates, refer to the selection on site by 370% by following the implementation of
of the search domain and related operations, i.e., gamification techniques like Badgeville’s Social Fab-
the selection of a web search engine and the ap- ric [11]. Another notable enterprise example comes
plication of its specific search features. from the SAP community network [5] that regami-
fied its reputation system. Points and levels supported
Strategies comprise full plans for conducting a
their business goals but were not enough to encourage
search.
the behaviors that are beneficial to the community at
Web search engines make it easy to check for facts, large. After redesigning their gamification strategies,
but sometimes it is still difficult to find some specific they managed to increase activity by 400% and com-
information; users even struggle with finding an an- munity feedback by 96%.
swer to a yes-no question [19]. The study by Stark et. Gamification techniques should primarily refer to
al. [17] exposed that Internet users rather tend to over- the users’ (intrinsic) motivation in order to benefit. At
estimate their skills in the domain of web search. A its worst, it “is a mindless slapping of points, badges
large-scale study focusing on behavioral signals con- and leaderboards [. . . ] onto any boring and irrelevant
firms our assumption that users have trouble in dif- activity in a vain attempt to increase the corporate
ficult search tasks [2]. Teevan analysed Bing search bottom-line [. . . ]” [16]. The fact that subjects’ in-
sessions and found about 40% of them contained mul- trinsic desire is reduced is a consequence of shallow
tiple queries where 25% of queries came from multi- constructed gamification design techniques and the im-
session tasks [18]. Half of all search time was spent in proper use of extrinsic rewards.
Gamified applications make use of the fact that API2 and all of its provided search features are
games are fun [15] and aim therefore to invoke the integrated into our search interface. They allow
same psychological experiences as games do [14] in or- us for designing more complex tasks rather than
der to keep users motivated. Psychologists have iden- simply searching for websites.
tified three basic elements that support motivation,
all of which gamification designers can tweak to their Query Tuning is a new approach for showing users
benefit [8]: how good they actually are at searching. In each
task, an URL and specific information about the
Autonomy People gain motivation when they feel in document are provided. The goal is to craft a
charge for an activity. query and rank the URL sought at a good position
within the result list. After each query, the user
Value Assigning value to an activity increases moti- has two options. In case she is satisfied with the
vation. achieved position, the user can close the task and
Competence Skills improve when devoting enough continue to the next one. Otherwise she is free to
time to an activity. The better one gets at one either create a new query or optimize the present
activity, the more likely it is she will continue it. one to go for a better position. Thus users can see
how changes in their query a↵ect the results step
With this information in mind, tasks can be de- by step which, in our opinion, makes it easier to
signed in such a way that they motivate users in solv- understand the principle of ranking.
ing them to gain experience.
The core application consists of the three task types
3 The WebSAIL Application and can be enhanced by diverse approaches like gam-
ification or online-tutorials for eliciting the desire to
The focus of WebSAIL is to design an application fea- solve tasks.
turing di↵erent types of tasks each of which target-
ing at a specific aspect of WSL. Users are confronted 3.2 Goals
with various search engine features they have to use for
completing tasks. They gain experience while solving Di↵erent task types help users in better understand-
tasks as each task type gives other insights into the ing how web search engines work. By promoting WSL,
function of web search engines, e.g., the result con- we strictly speak about users being capable of creating
struction. We aim for enabling users to be capable good queries in order to improve the quality of search
of using acquired skills during their daily searches for results. Furthermore, the user should be able to iden-
improving the quality of search results. tify relevant items within the result list and find the
information sought in the document itself efficiently.
3.1 Task Types Web search engines provide both basic and advanced
features like a query language, complex search inter-
This section gives an overview of the available task faces, or a dialogue to choose the result type, for re-
types. We gave name to all the types representing the stricting the area of inquiry. The WebSAIL tasks will
core task respectively. be designed in such a way that they impart knowl-
edge about the domain of web search to support users
Quiz includes single and multiple choice questions
in gaining experience, developing strategies, and thus
regarding web search engines for tutoring users.
becoming better searchers.
Our goal here is to make the search progress trans-
Instead of simply solving tasks one by one, we use
parent to the user. She should be aware of what
gamification to turn this monotonous work process
happens in the background after entering a query
into a game. Well designed game mechanics reward
and how result lists are constructed.
users adequately for completing tasks and allow for
Search Hunt comprises fact finding quiz tasks where fostering user engagement. The next section describes
a user is asked a question and has to use a search how we extended the core application by gamification.
interface in order to find the correct solution. Be-
side the ability to create good queries, we aim 4 The gamification approach
to foster relevance judgements since users have to
Instead of task type, we henceforth use the notion of
determine and choose relevant items out of the
game mode which emphasizes the ludic character of
entire result list in order to find the requested in-
the application. We took the core application (as de-
formation. In addition, the user should be capa-
scribed in section 3) and enhanced it by gamification
ble of finding the requested information inside a
document. At this point, we use the Bing Search 2 https://datamarket.azure.com/dataset/bing/search
in order to motivate users to solve the given tasks. At one badge may exist with increasing difficulty to ob-
this early stage the gamified version comprises the ba- tain respectively. There might be an extension of our
sic game design elements points, level, leaderboards, badge represented through a more imposing icon that
and rewards in the form of badges. Each task rewards claims a user succeeded three times (in three di↵erent
the user with points when solved correctly. In prin- tasks) in ranking the provided URL at position one
ciple, the amount depends on the level of difficulty, the first attempt.
the time spent, and applied strategies (that vary from
Frederick-Recascino and Schuster-Smith have
game mode to game mode). Since each game mode hat
shown that competition positively motivates some
its individual style of play and strategies that need to
players [12]. It is supported in the form of leader-
be applied, the calculation may vary. Lets consider
boards that track a user’s progress against others. The
the query tuning game mode where users have to rank
drawback is that a shallow constructed leaderboard
a document at a good position within the result list.
could result in some users feeling driven to keep up
The calculation of points can be extended by the at-
with other users [10] which would reduce intrinsic
tempts the user required to achieve a good rank. A
motivation. As a consequence, the user activity would
user who needs only one attempt to rank the given
drop. Leaderboards should always be encouraging,
URL at the top position would get more points than a
never discouraging. Thus said, our leaderboard does
user who needs two or more attempts. We believe this
not show the top players unless the current player
encourages users to put more e↵ort into creating one
is among the top 10 players. Instead, the current
single query instead of trial and error to achieve good
player based on her amount of points as being smack
results.
in the middle of standings, regardless of where she
Another game design element our application fea-
actually ranks. Moreover it can be discouraging if the
tures is levels for adjusting the difficulty of tasks and
di↵erence to the next player is considered too great.
thus promoting advanced users. Levels are bound to
In this context, generated fake data could help to
game modes and determined by the number of points a
elicit the desire to continue by apparently reducing
user has earned. She starts at level one and advances
distances.
by reaching predefined thresholds. The difficulty of
tasks increases with each level and users gain more Each user has her own dashboard or profile that
experience as more complex strategies need to be de- displays user statistics for each game mode respec-
veloped and applied in order to solve a task. tively. It shows the total amount of points achieved,
Progress bars show progress and can encourage the amount of solved tasks, and average time spent on
users to not only complete, but compete. To add clar- a task by default. Rather than showing basic statis-
ity: they aim at extrinsic as well as intrinsic motivated tics only it displays statistics bound to game modes.
users. Our implementation represents the amount of For instance, in query tuning, we can show the average
point the user received during each current game mode amount of queries spent on a task.
respectively and on the other hand the points left to
Besides game mode statistics the profile comprises
proceed to the next level (see Figure 1).
a badge gallery (see Figure 2) showing all rewards that
can be unlocked. Badges that have already been un-
locked are marked in color. Conditions that must be
fulfilled are revealed by clicking on the corresponding
badge. We believe that this motivates some users in
spending more time on a task and thus improving or
inventing new strategies to unlock all rewards.
Figure 1: The representation of a user’s current
progress in each game mode respectively.
In this regard, we use images representing the game
modes and names given to each level that should
strengthen the sense of being in a game.
Users are rewarded with badges when reaching cer-
tain states or performing defined actions which vary
in each game mode. For instance, in query tuning, a
user can be rewarded with a badge claiming that she Figure 2: The badge gallery in a user’s profile. Already
has ranked the provided URL at position one right the unlocked badges are marked in color.
first attempt during a task. Moreover, extensions of
4.1 Usability Evaluation The next step is to evaluate the two versions on
their own to determine whether WSL can actually be
Before using the gamified application to measure web enhanced. After a certain period of time, we will rein-
search literacy, we carried out a usability test with vite subjects that will have to solve tasks of the same
N = 15 participants to test whether the application is complexity once again in order to determine whether
capable of motivating users. The application achieved the acquired skills during the first evaluation remained
a system usability score (SUS) around 90 which indi- sustainable or were of a temporary nature only.
cates it is highly e↵ective, efficient, and satisfacting.
It fosters the ability of users to complete tasks and in-
References
creases the quality of the output of those tasks. On
the other hand, we gained additional information on [1] Aon plc. Engaging participants through
how to improve game design techniques, e.g., the con- gamification. http://www.aon.com/
struction of leaderboards. attachments/human-capital-
consulting/Overview%20of%20Health%
20Improvement%20Gamification%
5 Long-term Goals 20White%20Paper-Final%20Clear.pdf.
Once we finished creating both versions of the Web- Accessed May 20, 2016.
SAIL application (online-tutorials and gamification) [2] A. Aula, R. M. Khan, and Z. Guan. How
we will perform a study to identify whether WSL has does search behavior change as search be-
been enhanced. By comparing the two approaches comes more difficult? In Proceedings of the
against each other, we will figure out the one having SIGCHI Conference on Human Factors in
the bigger impact. Computing Systems, CHI ’10, pages 35–44,
However, the drawback is we will not be able to dis- New York, NY, USA, 2010. ACM.
tinguish whether the acquired skills are long-lasting or
of a temporary nature only. We will address this prob- [3] M. J. Bates. Information search tactics.
lem by reinviting subjects after a certain period of time Journal of the American Society for Infor-
who will have to solve tasks of the same complexity mation Science, 30(4):205–214, 1979.
once again. The comparison of both the evaluation
phases should clarify whether knowledge and strate- [4] M. J. Bates. How to use information search
gies gained during the first evaluation phase remained tactics online. Online, 11(3):47–54, May
sustainable. 1987.
[5] L. Cetin. The sap community network:
6 Conclusion and Outlook How to use gamification to increase en-
gagement. http://www.enterprise-
Studies revealed the lack of good strategies and skills gamification.com/index.php?option=
in the domain of web search. Low literate users spend com content&view=article&id=160:
significantly more time than high literate users to solve the-sap-community-network-how-
a search task, discover less relevant documents, and to\\-use-gamification-to-increase-
eventually miss out on important information. Due to engagement&catid=15&Itemid=22&lang=
the lack of motivation and knowledge, complex search en. Accessed May 26, 2016.
interfaces are omitted during search which could even-
tually lead to better results. In order to head o↵ [6] Y. Chou. A comprehensive list of
these problems and to make users better searchers, we 90+ gamification cases with roi stats.
started the WebSAIL project. We use online-tutorials http://yukaichou.com/gamification-
and gamification and evaluate both approaches against examples/gamification-stats-figures/.
each other to find the one having the greatest impact. Accessed May 19, 2016.
The core application features three di↵erent task [7] Y. Chou. Points, badges, and leader-
types. Solving tasks one by one is considered to be boards: The gamification fallacy. http:
monotonous and boring. Gamification enriches the ap- //yukaichou.com/gamification-study/
plication with common game design elements in order points-badges-and-leaderboards-the-
to foster user engagement and elicit the desire to play, gamification-fallacy/. Accessed May 19,
and serves as a powerful motivator to continue. The 2016.
gamified application achieved during a usability evalu-
ation a SUS score around 90 and thus is very e↵ective [8] Y. Daisy. Three critical ele-
for approaching our goals. ments sustain motivation. http:
//www.scientificamerican.com/article/ International Workshop on Gamification for
three-critical-elements-sustain- Information Retrieval, GamifIR ’14, pages
motivation/, Nov 2012. Accessed May 17, 46–48, New York, NY, USA, 2014. ACM.
2016.
[17] B. Stark, D. Dörr, and S. Aufenanger.
[9] S. Deterding, D. Dixon, R. Khaled, and The Google-ization of information search
L. Nacke. From game design elements to – Search engines in the field of tension
gamefulness: Defining ”gamification”. In between usage and regulation. Manage-
Proceedings of the 15th International Aca- ment Summary. http://www.blogs.uni-
demic MindTrek Conference: Envisioning mainz.de/ifp/files/2013/08/
Future Media Environments, MindTrek ’11, Suchmaschinen Management Summary.pdf
pages 9–15, New York, NY, USA, 2011. (in German), 2013. Accessed May 4, 2016.
ACM.
[18] J. Teevan. The complicated task of making
[10] R. Farzan, J. M. DiMicco, D. R. Millen, search simple. Talk at the UMAP confer-
C. Dugan, W. Geyer, and E. A. Brownholtz. ence. http://research.microsoft.com/
Results from deploying a participation incen- en-us/um/people/teevan/publications/
tive mechanism within the enterprise. In Pro- talks/umap15.pptx. Accessed May 27,
ceedings of the SIGCHI Conference on Hu- 2016.
man Factors in Computing Systems, CHI ’08,
pages 563–572, New York, NY, USA, 2008. [19] R. White. Beliefs and biases in web search. In
ACM. Proceedings of the 36th International ACM
SIGIR Conference on Research and Devel-
[11] fastcompany.com. Gamification and opment in Information Retrieval, SIGIR ’13,
the power of influence. http: pages 3–12, New York, NY, USA, 2013.
//www.fastcompany.com/1840235/ ACM.
gamification-and-power-influence.
Accessed May 20, 2016.
[12] C. M. Frederick-Recascino and H. Schuster-
Smith. Competition and intrinsic motiva-
tion in physical activity: A comparison of
two groups. Journal of Sport Behavior,
26(3):240–254, 2003.
[13] N. Fuhr. Internet search engines (lec-
ture script). https://is.inf.uni-due.de/
courses/ir ss14/ISMs 1-7.pdf (in Ger-
man), September 2014. Accessed April 11,
2016.
[14] K. Huotari and J. Hamari. Defining gam-
ification: A service marketing perspective.
In Proceeding of the 16th International Aca-
demic MindTrek Conference, MindTrek ’12,
pages 17–22, New York, NY, USA, 2012.
ACM.
[15] K. Poels, Y. de Kort, and W. Ijsselsteijn. ”it
is always a lot of fun!”: Exploring dimen-
sions of digital game experience using focus
group methodology. In Proceedings of the
2007 Conference on Future Play, Future Play
’07, pages 83–89, New York, NY, USA, 2007.
ACM.
[16] M. Shovman. The game of search: What is
the fun in that? In Proceedings of the First