=Paper=
{{Paper
|id=Vol-1458/G01_CRC60_Karatassis
|storemode=property
|title=Introducing a Gamification Approach
for Enhancing Web Search Literacy
|pdfUrl=https://ceur-ws.org/Vol-1458/G01_CRC60_Karatassis.pdf
|volume=Vol-1458
|dblpUrl=https://dblp.org/rec/conf/lwa/KaratassisD15
}}
==Introducing a Gamification Approach
for Enhancing Web Search Literacy==
https://ceur-ws.org/Vol-1458/G01_CRC60_Karatassis.pdf
Introducing a Gamification Approach
for Enhancing Web Search Literacy
Ioannis Karatassis and Sebastian Dungs
University of Duisburg-Essen, Germany
{karatassis,dungs}@is.inf.uni-due.de
Abstract. Web search engines provide a rich feature set to users that
allows efficient satisfaction of information needs. Nevertheless, recent
studies show that Internet users do not know how to use Web search
engines effectively for satisfying information needs. The overall level of
Web search literacy leaves a lot to be desired and most users tend to
overestimate their abilities in the domain of Web search. In this paper,
we introduce a gamification approach with the aim of promoting search
literacy as well as the current state of our prototype application. We
present plans for future work to answer whether gamification is a viable
means to improve Web search literacy. Our goals include finding indi-
cators to differentiate between low and high literacy users and running
long-term user studies to investigate the sustainability of search literacy
improvements.
Keywords: gamification, search literacy, Web search
1 Introduction
Search literacy denotes the ability to locate and access desired information with
efficiency and effectiveness. It is, therefore, a subset of the much broader concept
information literacy which also encompasses evaluation, reuse of information,
and information synthesis. Instead of putting Web search on a level with infor-
mation retrieval, we look upon it as a lifelong learning process that we aim to
support in order to enable users orienting themselves in modern societies. Users
employ Web search engines not only for answering trivial information needs
but also trust in the machines and their own abilities when it comes to serious
decisions, e.g., health related issues or financial concerns.
In this paper, we draw attention to the problems arising from deficiencies in
aforementioned Web search literacy and introduce a potential means that aims at
increasing Web search literacy beyond traditional training methods like courses
or tutorials. Only few approaches exist in this regard. One notable exception
being A Google a Day 1 which features a fact finding search quiz. Users are
Copyright c 2015 by the papers authors. Copying permitted only for private and
academic purposes. In: R. Bergmann, S. Grg, G. Mller (Eds.): Proceedings of the
LWA 2015 Workshops: KDML, FGWM, IR, and FGDB. Trier, Germany, 7.-9. Oc-
tober 2015, published at http://ceur-ws.org
1
http://www.agoogleaday.com
340
encouraged to employ advanced features of the search engine to solve the tasks.
While the system includes a scoring system and rewards fast task completion, it
lacks key gamification elements like levels, achievement badges, or leaderboards.
Users solve tasks themselves and can not compete with other users.
We developed a gamification framework that features different types of tasks
(e.g., search and educational) to give users a deeper understanding of the func-
tioning of Web search engines. At the same time, our users are to learn and
develop skills that should help them in mastering their daily search tasks effi-
ciently. Furthermore, we plan to use the presented system as a basis for long-term
studies. The goal of these studies will be 1) to identify key factors that make
a user actually Web search literate, 2) to measure whether Web search literacy
was improved and by how much, and 3) to evaluate how sustainable these effects
are.
The remainder of this paper is structured as follows: First, we take a closer
look at related work regarding search literacy and gamification in Section 2.
The gamified application is presented in Section 3, covering game modes and
later focusing on the employed game design elements. In Section 4, we draw a
conclusion and provide an outlook for future work.
2 Related Work
2.1 Web Search Literacy
A recent study by Stark et al. [1] revealed that Internet users tend to overestimate
their capabilities in the domain of Web search. In fact, the overall Web search
literacy leaves a lot to be desired and common Web search engine users even
have problems with finding answers to yes-no questions [2]. Kogadoga et al.
[3] refer to the problems that arise from being low literate based on a study:
Participants with deficient Web search skills used to spend significantly more
time to complete a search task in comparison to high literate users and were
significantly less accurate. The main problem is that users do not know how
to use Web search engines effectively for satisfying their information needs [4].
Referring to this, Web search engines offer no feedback for users beyond query
completion or expansion that would help them in improving their skills.
In his recent talk at GamifIR’15 [17], Azzopardi raised the idea to make a
nature-nurture distinction when it comes to search behaviour. While this is an
interesting research approach, we expect nurture to a play considerable role in
users’ behaviour, allowing for potential improvements by promoting key search
skills.
According to Fuhr [5], a Web search literate user needs to know appropriate
search tactics and strategies in order to satisfy information needs effectively.
Users should be aware of the basic functioning of Web search engines as well as
the following key aspects:
Searchability In some cases where users try to find a specific open document
in the browser through Web search engines, they fail since not all online
resources are indexed. The language used in the search query, Website owner
341
restrictions (e.g., the robots.txt), the document type, and the recency of Web
pages are some of the reasons why resources can not be found.
Linguistic Functions A crucial problem in information retrieval is the lan-
guage itself since every natural language is both vague and ambiguous. To
deal with that problem and to represent a user’s information needs as best
as possible, Web search engines apply linguistic functions such as word nor-
malization, lemmatization, and phrase identification to search queries and
take composites and synonyms into account.
Query Language A specially designed language allows users to express com-
plex information needs and leads to more specific results since the latter are
restricted to a limited set. Search operators (e.g., Boolean operators, num-
ber ranges, facets, fields, and URL predicates) and search options (e.g., for
restricting the time, place, language, and document type of result items) are
commonly employed search features.
Ranking One search query leads to a set of results where the ranking is of
utmost importance. Hochstotter et al. [6] found out that users tend to look at
items on the first search result page and especially click on the first or second
item. Result items below the fold are seldom clicked on. Hence, users have
to formulate precise search queries to let search engines produce result sets
where the most relevant items are located on the first page and preferably
are visible without the need to scroll.
Strategies and Tactics The main goal of Web search engine users is to sat-
isfy their current information need. Complex information needs require a
series of search queries. Strategies are plans for performing a complex search
whereas tactics denote single operations to advance searches. Bates [7] dis-
tinguishes between the following types of tactics: monitoring, file structure,
search formulation, and term.
2.2 Gamification
In the book by Zichermann et al. [8] the term gamification is defined as the pro-
cess of game-thinking and game mechanics to engage users and solve problems.
The integration of game mechanics into non-game contexts invokes gameful and
ludic experiences to motivate users in solving monotonous tasks or for training
users in complex systems. Beyond that, the concept is a viable means to shape
users’ behavior and to enhance online services with motivational affordances [9].
At its worst, gamification is a “mindless slapping of points, badges and leader-
boards [. . . ] onto any boring and irrelevant activity in vain attempt to increase
the corporate bottom-line” [15]. When a person performs activities driven by
internal rewards, we say she is intrinsically motivated due to the enjoyment of
the activity itself. In contrast, users acting based upon extrinsic motivation aim
to earn external rewards or to avoid punishments. We focus on enhancing the
intrinsic motivation of users since it is known to be associated with the quality
of effort that people put into activities [16].
In [10, 11] university courses were gamified with great success. Gamification
helped in improving lecture attendance, content understanding, problem solving
skills, and general engagement of students. Achievement badges have been used
342
by Hulinen et al. [12] to reward students for solving interactive tasks. Results
show that the students’ motivation has been enhanced even when the badges
have had no impact on grading. Although there is still a lack of empirical evi-
dence on the side effects of the employed game elements, these findings lead to
the conclusion that gamification does not harm the intrinsic motivation at all
if gamification is meaningful enough to the user and applied in a user-centered
fashion [13]. Nevertheless, gamification designers should take social and contex-
tual factors into account as they may determine whether the employed game
elements diminish [14] or even suppress intrinsic motivation.
3 The Gamification Framework
3.1 Game Modes
Following the insights gained by literature review, we developed an application
for improving search skills which appears to the user in the form of a game. We
introduce the notion of game mode which emphasizes the playful character of
the framework and summarizes a set of tasks of a specific nature. In total, we
developed three game modes each of which aiming at a different aspect of Web
search literacy: Quiz, Search Hunt, and Query Tuning.
Quiz The quiz mode features single and multiple choice question answering
tasks (see Fig. 1). They allow for a deeper understanding of the functioning
of search engines. The main goal of this game mode is to familiarize a user
with advanced search engine functionality in a series of tasks. Furthermore,
the quiz acts as a means to measure a priori search knowledge.
Fig. 1. The demonstration quiz task asks the user to select all items identifying Web
search engines. Additional answer related information can be accessed by clicking on
the respective information icon.
Search Hunt This mode comprises typical fact finding tasks (see Fig. 2). Users
are asked to complete a task by issuing queries to one of the world’s leading
search engines which is directly included in the game by a proxy solution in
order to find the solution being sought. Search hunt primarily trains users to
343
formulate precise search queries, to identify relevant results, and to find the
desired content within the document. Furthermore, it promotes the ability
to judge accuracy of results. We exploit the search engine’s rich feature set
to provide a complete interface that contains all commonly employed search
functions to our users. The interface allows us to train users on how to use
specific features and can have more or less importance depending on the
task. In addition, tasks will be designed in a way to familiarize users with
commonly neglected search engine features and search strategies.
Fig. 2. In this task, users have to formulate a search query and to use certain functions
of the search interface in order to find the required image that includes the solution.
The task interface as seen in this figure offers hints and clues that users can request in
exchange of points as well as a function for skipping the current task if desired. The
hint can be a single word or a phrase that provides additional information and the clue
reveals the first and the last letter as well as the length of the solution.
Query Tuning is comprised of precision oriented tasks (see Fig. 3). Users are
again required to interact with the search engine but this time to produce a
result set that contains a specific site at a top position. Along with the target
site and a summary of the content comes a small set of search terms that are
not allowed to be used in the query to avoid trivial solutions (e.g., querying
for the URL of the site). Users formulate and reformulate queries until either
the given site is ranked at the top position or the search performance can not
be improved further. Hence, required skills for formulating precise queries
are enhanced within a step-by-step refining process. The main goal of this
mode is to form the understanding of ranking and to get a feeling how small
changes in the query can yield to major differences in the result set.
3.2 Game Design Elements
The core game mechanics of the application consist of points, levels, badges, and
leaderboards. Points are received for (partial) successful completion of tasks. The
amount is determined by the degree of correctness, the current user level as well
as the time needed to complete a task. Points act as the main performance
indicator in the application. Levels are used to define the user’s current state
and represent a task’s complexity. The next level up can be reached by exceeding
344
Fig. 3. Users enter a search query in a text field which is forwarded via a search proxy
to the connected Web search engine to produce a result set that holds the URL marked
in green at a top position. The words marked in red are terms or search operators that
must not appear in the search query. A user is free so close the task at any time. Points
are calculated based on the last search process: The position of the given Website
within the result list and the number of attempts have the biggest impact on scoring.
the corresponding point threshold. Badges are special rewards that are acquired
either for reaching a certain state or for various actions. They may come as
a surprise and with varying frequency and act as a motivator to explore the
application. Furthermore, badges can be used to “show off” individual user skills
to other users via a profile page (see Figure 4).
Fig. 4. The user’s profile page comprises game mode related statistics on the left side.
The main area gives an overview about the effort achieved in each game mode by
displaying the current level, a corresponding level description and the current score in
the form of a progress bar respectively. While already collected achievement badges are
depicted in color, the application displays all available badges to promote transparency.
Leaderboards are overviews of the top performing players in each game mode
and are represented as ordered lists with a points score beside each name to allow
simple comparisons and to engage users in competition. Again, these boards act
as a motivational means for continuing as well as an instrument along with levels
345
to indicate that users have more or less status or achievement in the game.
Besides the core mechanics, the application features different sound effects to
guide users and to introduce events, e.g., the beginning/completion of tasks and
the receipt of awards. A comprehensive logging system collects user data in the
background. The log data gives an insight into a user’s behavior and thus can
be used to create user profiles that reveal strategies and techniques being used
to solve tasks.
4 Conclusion and Future Work
In this paper, we presented a gamification framework for Web search. The system
in its current state features key gamification elements like points, badges, and
leaderboards. Furthermore, three different game modes, i.e., types of tasks, are
included. The system was tested regarding usability in a small user study (N =
15) with great success. The main goal of the system is to improve Web search
literacy among general Web search users. We believe that this will allow for
more time efficient and effective search sessions, which will lead to a higher task
completion rate. To accomplish this goal, we will address various challenges that
we are still facing with our prototype:
1. The actual search tasks and quizzes need to be tailored for the goal of im-
proving literacy. Therefore, tasks will be created that are challenging for an
average search engine user. Ideally, tasks should promote specific learning
goals, e.g., search strategies or search engine features.
2. After a larger collection of tasks has been created, we will run a long-term
study with a larger user base. This will not only allow us to tune game
balance. The results will also act as a ground truth for future experiments.
Observing many users completing the same tasks will allow us to “pool”
solution attempts and to generate an ideal solution for every task. Individual
users will then be scored by the closeness of their solution to the ideal one.
3. In the end, we will isolate key factors that make a user Web search literate
and find means to specifically promote these skills in a gamified environment.
The secondary goal of the long-term study will be to test the sustainability
of literacy improvements. Therefore, we will invite participants to reuse the
application with new tasks of similar complexity after a specific time period
and compare the outcomes of their endeavors.
References
1. Stark, B., Dörr, D., Aufenanger, S.: The Googleization of information search - Search
engines in the field of tension between usage and regulation. Management Summary
(in German), 2014. Available online at http://www.ifp.uni-mainz.de/Bilder_
allgemein/Suchmaschinen_Management_Summary.pdf; accessed 15-June-2015.
2. White, R.: Beliefs and biases in web search. In Proceedings of the 36th International
ACM SIGIR Conference on Research and Development in Information Retrieval,
SIGIR ’13, pages 312, New York, NY, USA, 2013. ACM.
346
3. Kodagoda, N., Wong., B.L.W.: Effects of low & high literacy on user performance
in information search and retrieval. In Proceedings of the 22Nd British HCI Group
Annual Conference on People and Computers: Culture, Creativity, Interaction -
Volume 1, BCS-HCI ’08, pages 173181, Swinton, UK, UK, 2008. British Computer
Society.
4. Bateman, S., Teevan, J., White, R.W.: The search dashboard: How reflection and
comparison impact search behavior. In Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems, CHI ’12, pages 17851794, New York, NY,
USA, 2012. ACM.
5. Fuhr, N.: Internet search engines - Lecture script for the course in SS 2014 (in
German), 2014. Available online at http://www.is.inf.uni-due.de/courses/ir_
ss14/ISMs_1-7.pdf; accessed 15-June-2015.
6. Höchstötter, N., Lewandowski, D.: What users see – structures in search engine
results pages. Inf. Sci., 179(12):17961812, May 2009. ISSN 0020-0255.
7. Bates, M.J.: Information search tactics. Journal of the American Society for Infor-
mation Science, 30(4):205214, 1979.
8. Zichermann, G., Cunningham, C.: Gamification by Design: Implementing Game
Mechanics in Web and Mobile Apps. O’Reilly Media, Inc., 1st edition, 2011.
9. Deterding, S., Dixon, D., Khaled, R., Nacke, L.: From game design elements to
gamefulness: Defining gamification. In Proceedings of the 15th International Aca-
demic MindTrek Conference: Envisioning Future Media Environments, MindTrek
’11, pages 915, New York, NY, USA, 2011. ACM.
10. Iosup, A., Epema., D.: An experience report on using gamification in technical
higher education. In Proceedings of the 45th ACM Technical Symposium on Com-
puter Science Education, SIGCSE ’14, pages 2732, New York, NY, USA, 2014.
ACM.
11. O’Donovan, S., Gain, J., Marais, P.: A case study in the gamification of a university-
level games development course. In Proceedings of the South African Institute for
Computer Scientists and Information Technologists Conference, SAICSIT ’13, pages
242251, New York, NY, USA, 2013. ACM.
12. Hakulinen, L., Auvinen, T., Korhonen, A.: Empirical study on the effect of achieve-
ment badges in trakla2 online learning environment. In Proceedings of the 2013
Learning and Teaching in Computing and Engineering, LATICE ’13, pages 4754,
Washington, DC, USA, 2013. IEEE Computer Society.
13. Nicholson, S.: A User-Centered Theoretical Framework for Meaningful Gamifica-
tion. Paper Presented at Games+Learning+Society 8.0, Madison, WI, June 2012.
Available online at http://scottnicholson.com/pubs/meaningfulframework.pdf;
accessed 15-June-2015.
14. Mekler, E.D., Brühlmann, F., Opwis, K., Tuch, A.N.: Do points, levels and leader-
boards harm intrinsic motivation?: An empirical analysis of common gamification
elements. In Proceedings of the First International Conference on Gameful Design,
Research, and Applications, Gamification ’13, pages 6673, New York, NY, USA,
2013. ACM.
15. Shovman, M.: The game of search: What is the fun in that? In Proceedings of the
First International Workshop on Gamification for Information Retrieval, GamifIR
’14, pages 4648, New York, NY, USA, 2014. ACM.
16. Ryan, R., Deci, E.: Intrinsic and extrinsic motivations: Classic definitions and new
directions. Contemporary educational psychology 25, 1 (2000), 5467.
17. Kazai, G., Hopfgartner, F., Kruschwitz, U., Meder, M.: ECIR 2015 Workshop on
Gamification for Information Retrieval (GamifIR’15). SIGIR Forum 49(1): 41-49
(2015)
347