=Paper=
{{Paper
|id=Vol-1404/paper26
|storemode=property
|title=Gamification in Information Retrieval: State of the Art, Challenges and Opportunities
|pdfUrl=https://ceur-ws.org/Vol-1404/paper_26.pdf
|volume=Vol-1404
|dblpUrl=https://dblp.org/rec/conf/iir/MunteanN15
}}
==Gamification in Information Retrieval: State of the Art, Challenges and Opportunities==
https://ceur-ws.org/Vol-1404/paper_26.pdf
Gamification in Information Retrieval:
State of the art, Challenges and Opportunities
Cristina Ioana Muntean, Franco Maria Nardini
ISTI-CNR, Pisa, Italy
{name.surname}@isti.cnr.it
Abstract. Gamification aims at applying game design principles and
elements, such as points, badges, feedbacks or leader boards in non-
gaming environments. An interesting goal of gamification is to combine
and exploit the fun factor for targeting other aspects like achieving more
accurate work, more cost effective solutions and better retention rates.
The application of gamification techniques to IR tasks poses interesting
research challenges. In this paper, we propose an analysis of the state of
the art in this field and we summarize interesting challenges and oppor-
tunities for the near future.
1 Gamification in IR
Gamification relates with the use of game thinking and techniques in non-game
contexts to enable the engagement of users with solving problems and to increase
users’ self contributions [15]. It is defined in [5] as the use of game-play mechanics
for non-game applications. For this reason, any application, task, process or
context can theoretically be gamified [12].
The main goal of gamification is to increase the engagement of users by us-
ing game-like techniques such as scoreboards and personalized fast feedback [7],
making people feel more ownership and purpose when engaging with tasks [13].
Gamification also implies a social game and interaction with other participants.
It employs game mechanics, i.e., points, levels, challenges, virtual goods, leader-
boards, gifting and charity, and aesthetics (motivations), i.e., reward, status,
achievement, self expression, competition, altruism to enact such interactions
[14]. There are precise gamification effects that one may wish to stimulate and
clear game mechanics that address these changes. For example in order to stim-
ulate the sense of achievement, challenges can be proposed to the user. Similarly
status can be stimulated through levels, competition through leaderboards and
rewards through virtual goods and points.
In this rule-bounded, goal-oriented play, valuable content is also an important
incentive. According to Groh [9] it is important for a product to offer real benefits
and real value rather than just rewards which in the long run may seem less
appealing to users. Thus by stripping a site down of its point and badges there
is still meaningfull content in it. Gamification is about making an application or
a task more fun, rather than actually stimulating playfulness.
Gamification can be applied in several domains, typically for increasing the
engagement of a user, for teaching, entertaining, measuring, and to improve the
perceived ease of use of information systems. Successful examples are Foursquare,
Twitter, Stack Overflow, Hacker News. While both Foursquare and Twitter em-
ploy gamification for increasing their user engagement, Stack Overflow1 and
Hacker News2 are powerful platforms for question answering that promotes also
authoritativeness of trustworthy users with both reward and punishment mech-
anisms. Gamification is thus used for stimulating and promoting users to be
authoritative and active. In gamification it is important to understand users
and create scenarios that appeal to their personality types: explorers, achievers,
socializers and/or killers. Gamification can bring various benefits like increased
engagement, loyalty, time spent, influence, fun or productivity. A possible ben-
efit for IR is the understanding that the contribution of user will help raise the
quality of the service and content, thus indirectly the user satisfaction. Bartle
argues that although game design is an art form, gamification is an application
of psychology, thus one of the most important aspects of gamifying Information
Retrieval (IR) tasks relies on understanding of the human motivation [2].
Relevance Assessment, Feedback and Ranking
An interesting application in IR is relevance assessment. Chamberlain intro-
duces a model for rewarding and evaluating users using retrospective validation,
with only a small gold standard required to initiate the system [4]. The model
is not based on the quantity of tasks performed but rather is focused on an
agreement-based reward which prizes the quality of the solutions. The evalu-
ation of the theoretical model indicated that the reward mechanism succeeds
in awarding the high quality answers, but in practice it is not such a strong
signal for predicting the user performance. Harris exploits groups’ ability to as-
sess relevance of documents and images and also rank their choices [10]. During
their experiment participants are divided in two groups, one making judgements
based on their own assessments, while the other makes judgements based on the
estimate of consensus decision. To motivate participant, financial rewards are
offered. When participants use consensus opinion as a guide, relevance assess-
ment are homogenous probably due to the fact that they are more conservative.
Another paper that uses player feedback is [3]. Here, authors’ objective is im-
proving image recognition accuracy. The process is divided in three steps: first
the systems performs recognition, then the information on which they wish to
retrieve feedback is compiled and lastly they retrieve the feedback form the play-
ers through a crowd-sourced gamified approach. They also propose a method for
leveraging ambigous feedback by introducing a measure of certainty. They con-
clude that due to the feedback received from the players the accuracy of the
recognition systems improves. Fort et al. propose a Game With a Purpose that
allows annotation of corpora with dependency syntax [8]. Students in linguistics
are trained to annotate certain phenomena whereas a previous pre-annotation
step gives an idea of whether there are significant inconsistencies between the
two. They do not motivate users by claiming it is useful for research instead
they focus on the playful aspect of the game and user personal interests. The
1
http://stackoverflow.com/
2
https://news.ycombinator.com/
quality of the data implies: the trustworthiness of players and the assessment of
the correctness of analyses.
Web Search
Azzopardi et al. proposed a gamification-enhanced sequel of Page-Fetch, a
game where participants, given a web page, must enter the query that they con-
sider most suitable for that page [1]. The shorter the query the higher the score
they receive for the task. Users are also time constrained, they gain points, are
ranked in leaderboards and receive badges. The system allows the evaluation of
the quality of competing search engines or evaluating the capacity of players for
performing a task. Fernandez-Luna et al. focus on gamifing a collaborative infor-
mation seeking system (CIS), defined as a process of information seeking “that
is defined explicitly among the participants, interactive, and mutually benefi-
cial”[6]. They proposed several ways to gamify a CIS system that could end up
in intensifying a seeker’s engagement. He et al. apply gamification to crowsourc-
ing tasks to make them more appealing and so making the users play, rather
than work. Nevertheless, differences in task design and incentives elicit different
player behavior [11]. The proposed solution simulates user behavior when per-
forming a search task. Authors propose a faceted interface, which diminishes the
rank bias present in a typical SERP and is preferred by the users.
2 Challenges and Opportunities
In this paper we discussed the latest research results employing gamification
approaches within IR tasks. We believe that the application of gamification to
IR tasks are still in a preliminary stage. It opens the way to many research
challenges, in particular for tasks that are difficult to quantify or qualify, based
on crowdsensing, or requiring user evaluation. Here we highlight some promising
directions that we believe could be natural scenarios where gamification can be
successfully exploited.
User profiling: modern Web search engines track the activities of the users in
order to derive their profile to be used and then exploited again during search,
for personalizing the search experience. This activity is done by distilling im-
plicit feedback from clicked results, query logs, etc. Gamification approaches
that enable users to select preferences on specific domains, i.e., documents, im-
ages, videos, etc. contribute to building the user profile explicitly, that in turn
allows deriving more precise and detailed information.
Document annotation: modern Web search engines deeply rely on machine
learning to perform several important tasks ranging from document and query
classification to document ranking and spam detection. Machine learning needs
labeled data during the training phase. Labeled data is produced by employing
hundreds of human assessors judging documents returned by a query and who
assign a relevance label. Human labeling is a complex, hard and time-consuming
task. For this reason, we believe that by exploiting gamification, and thus by
adding fun and competitiveness to an annotation platform, will lead to more
motivated annotators producing higher quality results. The annotation task can
be referred to: indicating a certain class of a document (domain, part-of-speech
etc.), offering a relevance judgement (true or false) or ranking objects according
to preference, correctness etc.
Diversification and Query Intent Discovery: diversification of Web search
results is an important research field studying the best way to answer ambiguous
or “multi-faceted” search queries. Roughly, diversification aims at covering all
(the majority of) the possible meanings behind a search query in a single results
page. We believe gamification approaches could help in discovering the most
popular interpretations behind a given ambiguous query. The same approach
could help in determining the quality of a diversified results page.
Gamification opens the way to many research challenges that has been only
partially addressed so far, especially in the IR field. The proposed literature
review also revealed that more rigorous methodologies ought to be used in further
research on gamification.
References
1. Azzopardi, L., Bevc, M., Gardner, A., Maxwell, D., Razzouk, A.: Pagefetch 2:
Gamification the sequel. In: Proc. GamifIR ’14. ACM
2. Bartle, R.A.: Information reconstruction: Unpicking the gamifir call for papers. In:
Proc. GamifIR ’14. ACM
3. Brenner, M., Mirza, N., Izquierdo, E.: People recognition using gamified ambiguous
feedback. In: Proc. GamifIR ’14. ACM
4. Chamberlain, J.: The annotation-validation (av) model: Rewarding contribution
using retrospective agreement. In: Proc. GamifIR ’14. ACM
5. Deterding, S., Dixon, D., Khaled, R., Nacke, L.: From game design elements to
gamefulness: Defining ”gamification”. In: Proc. MindTrek ’11. ACM
6. Fernández-Luna, J.M., Huete, J.F., Rodrı́guez-Avila, H., Rodrı́guez-Cano, J.C.:
Enhancing collaborative search systems engagement through gamification. In:
Proc. GamifIR ’14. ACM
7. Flatla, D.R., Gutwin, C., Nacke, L.E., Bateman, S., Mandryk, R.L.: Calibration
games: making calibration tasks enjoyable by adding motivating game elements.
In: Proc ACM UIST’11. pp. 403–412. ACM
8. Fort, K., Guillaume, B., Chastant, H.: Creating zombilingo, a game with a purpose
for dependency syntax annotation. In: Proc. GamifIR ’14. ACM
9. Groh, F.: Gamification: State of the art definition and utilization. Institute of
Media Informatics Ulm University 39 (2012)
10. Harris, C.G.: The beauty contest revisited: Measuring consensus rankings of rele-
vance using a game. In: Proc. GamifIR ’14. ACM
11. He, J., Bron, M., Azzopardi, L., de Vries, A.: Studying user browsing behavior
through gamified search tasks. In: Proc. GamifIR ’14. ACM
12. Muntean, C.I.: Raising engagement in e-learning through gamification. In: Proc.
6th International Conference on Virtual Learning ICVL. pp. 323–329 (2011)
13. Pavlus, J.: The game of life. Scientific American 303(6), 43–44 (2010)
14. Viola, F.: Gamification - I Videogiochi nella Vita Quotidiana. Fabio Viola (2011),
http://books.google.it/books?id=nEiwmXn9J2EC
15. Zichermann, G., Cunningham, C.: Gamification by design: Implementing game
mechanics in web and mobile apps. ” O’Reilly Media, Inc.” (2011)