The creation of a labelled dataset for Information Retrieval (IR) purposes is a costly process. For this reason, a mix of crowdsourcing and active learning approaches have been proposed in the literature in order to assess the relevance of documents of a collection given a particular query at an affordable cost. In this paper, we present the design of the gamification of this interactive process that draws inspiration from recent works in the area of gamification for IR. In particular, we focus on three main points: i) we want to create a set of relevance judgements with the least effort by human assessors, ii) we use interactive search interfaces that use game mechanics, iii) we use Natural Language Processing (NLP) to collect different aspects of a query. 1
Italiano
La creazione di una collezione
sperimentale per l’Information Retrieval (IR) e` un
processo costoso sia dal punto di vista
economico che in termini di sforzo umano.
Per ridurre i costi legati all’attribuzione
dei giudizi di rilevanza ai documenti di
una collezione, sono stati proposti
approcci che integrano tecniche di
crowdsourcing e active learning. In questo
paper viene presentata un’idea basata
sull’utilizzo della gamification
(‘ludicizzazione’) in IR per l’attribuzione di giudizi
di rilevanza in maniera semi-automatica.
1This paper is partially an extended abstract of the
paper “Gamification for Machine Learning: The Classification
Game” presented at the GamifIR 2016 Workshop co-located
with SIGIR 2016
In Information Retrieval (IR), the performance of
a system is evaluated using experimental test
collections. These collections consist of a set of
documents, a set of queries, and a set of relevance
judgments, where each judgement explains whether a
document is relevant or not to each query. The
creation of relevance judgements is a costly,
timeconsuming, and non-trivial task; for these
reasons, the interest in approaches that generate
relevance judgements with the least amount of effort
has increased in IR and related areas (i.e.,
supervised Machine Learning (ML) algorithms). In the
last years, mixed approaches that use
crowdsourcing
The effectiveness of a search and the satisfaction
of users can be enhanced through providing
various results of a search query in a certain order
of relevance and concern. The technique used
to avoid presenting similar, though relevant,
results to the user is known as a diversification of
search results
Most information retrieval systems operate by
performing a single retrieval in response to a
query. Effective results sometimes require
several manual reformulations by the user or
semiautomatic reformulations assisted by the system.
Diaz presents an approach to automatic query
reformulation which combines the iterated nature
of human query reformulation with the automatic
behavior of pseudo relevance feedback
Gamification is defined as “the use of game
design elements in non-game contexts”
IR has recently dealt with gamification, as
witnessed by the Workshop on Gamification for
Information Retrieval (GamifIR) in 2014, 2015 and
2016. In
In this paper, we want to apply game mechanics to the problem of relevance assessment with three goals in mind: i) we want to create a set of relevance judgements with the least effort by human assessors, ii) we use interactive search interfaces that use game mechanics, iii) we use NLP to collect different aspects of a query. In this context, we can define our web application as a Game with a Purpose (GWAP), that is a game which presents some purposes, usually boring and dull for people, within an entertaining setting, in order to make them enjoyable and to solve problem with the aid of human computation. The design and the implementation of this interactive interface will be used as a post-hoc analysis of two Text REtrieval Conference (TREC)2 2016 tracks, namely the Total Recall Track and the Dynamic Domain Track. These two tracks are interesting for our problem since they both re-create a situation where we need to find the best set (or the total amount) of relevant documents with the minimum effort by the assessor that has to judge the documents proposed by the system given an information need. 2
In this first pilot study, we will implement a
simple game based on a visual interpretation of
probabilistic classifiers
The ‘original game’
P (ojc2) < m P (ojc1) +q | {yz } | {xz } (1) where P (ojc1) and P (ojc2) are the likelihoods of the object o given the two categories, while m and q are two parameters that depend on the misclassification costs that can be assigned by the user to compensate for either the unbalanced classes situation or different class costs.
If we interpret the two likelihoods as two
coordinates x and y of a two dimensional space, the
problem of classification can be studied on a
twodimensional plot. The decision of the
classification is represented by the ‘line’ y = mx + q that
splits the plane into two parts, therefore all the
points that fall ‘below’ this line are classified as
objects that belong to class c1 (see Figure 1 for
an example). Without entering into the
mathematical details of this approach
The classification game can be easily adjusted into a relevance assessment game if the two classes are ‘relevant’ and ‘non-relevant’ (we assume only binary relevance assessment for the moment). How3http://www.venetonight.it/padova/ 4https://unity3d.com 5https://www.madewithmarmalade.com/ ever, while in the classification game we already have a set of labelled documents and the goal is to find the optimal classifier, in this new game we need to find the relevant documents. To this purpose, we will follow the idea of the works described in the following subsections: i) building assessment by varying the description of the information need, ii) using an interactive interface that suggests the amount of relevant information that has to be judged, iii) using NLP approaches to generate variations of a query. 3.1
In
In practice, in order to build a set of relevance assessments for q, we generate a number of query aspects using a single IR system. Then, the union of the top k documents retrieved for each aspect constitutes a list of pseudo-relevance assessments for the query q. 3.2
Building different aspects of the same information
need is not an easy task. As explained in
Umemoto et al. propose an interactive interface, named ScentBar, that helps searchers to visualize the amount of missing information for both the search query and suggestion queries in the form of a stacked bar chart. The interface, a portion of which is shown in Figure 2, visualizes an estimate of missing information for each aspect of a query that could be obtained by the searcher. When the user collects new information during the browsing of the results, the bars of the different query aspects change color to indicate the amount of effort that the system estimates necessary to find most of the relevant information. The estimates of the required effort to complete a task are formalized as as a set-wise metric were the gain for each aspects is represented by a conditional probability. 3.3
The last part of the design of the query aspects
game involves the use of natural language
processing techniques to propose variations of a query to
express the same information need. This problem
has been studied for more than twenty years in IR.
In
Some examples of advanced techniques of
phrase extraction, including extended N-grams
and syntactic parsing, attempt to uncover
concepts, which would capture underlying semantic
uniformity across various surface forms of
expression. Syntactic phrases, for example, appear
reasonable indicators of content since they can
adequately deal with word order changes and other
structural variations. In the literature, there are
examples of query reformulation using NLP
approaches for example to the modification and/or
expansion of both parts thematic and
geospatial that are usually recognized in a geographical
query
In this work, we presented the requirements of the
design of an interactive interface that uses game
mechanics together with NLP techniques to
generate variation of an information need in order to
label a collection of documents. Starting from the
successful experience of the gamification of a
machine learning problem