The creation of a labelled dataset for machine learning purposes is a costly process. In recent works, it has been shown that a mix of crowdsourcing and active learning approaches can be used to annotate objects at an a↵ordable cost. In this paper, we study the gamification of machine learning techniques; in particular, the problem of classification of objects. In this first pilot study, we designed a simple game, based on a visual interpretation of probabilistic classifiers, that consists in separating two sets of coloured points on a two-dimensional plane by means of a straight line. We present the current results of this first experiment that we used to collect the requirements for the next version of the game and to analyze i) what is the 'price' to build a reasonably accurate classifier with a small amount of labelled objects, ii) and compare the accuracy of the player to the state-of-the-art classification algorithms.
Supervised machine learning algorithms require
labelled examples to be trained and be evaluated
properly. However, the labelling process is a costly,
timeconsuming and non-trivial task. Manual annotation
by experts is the obvious choice [
In this first pilot study, we designed a simple game
based on a visual interpretation of probabilistic
classifiers [
To summarize, this experiment has been designed to study: • how the gamification of a classification problem can be used to understand what is the ‘price’ a user is willing to pay to build a classifier; • how the performance of a ‘human’ classifier compares to the state-of-the-art algorithms on a small scale training dataset.
The first goal is related to the problem of minimizing
the cost of labelling the dataset and at the same time
building a reasonably accurate classifier. The second
goal is related to the problem of classification
performance given the presence of small sample sizes and
little training [
The paper is organized as follows: in Section 2, we present some background literature on gamification in Information Retrieval (IR). In Section 3, we describe in detail the game, the rules and the interactive application. Section 4 discusses the pilot study and the initial results. Section 5 is dedicated to the requirements and suggestions for future improvements collected from the players during this study. In Section 6, we give our final remarks. 2
Gamification is defined as “the use of game design
elements in non-game contexts” [
In recent years, also IR has dealt with
gamification, as witnessed by the Workshop on Gamification
for Information Retrieval (GamifIR) in 2014, 2015 and
2016. In [
The game is based on the two-dimensional
representation of probabilities [
P (o|c2) < m P (o|c1) +q | {yz } | {xz } (1) where P (o|c1) and P (o|c2) are the likelihoods of the object o given the two categories, while m and q are two parameters that depend on the misclassification costs that can be assigned by the user to compensate for either the unbalanced classes situation or di↵erent class costs.
If we interpret the two likelihoods as two
coordinates x and y of a two dimensional space, the problem
of classification can be studied on a two-dimensional
plot. The decision of the classification is represented
by the ‘line’ y = mx + q that splits the plane into
two parts and all the points that fall ‘below’ this line
are classified as objects that belong to class c1 (see
Figure 1 for an example). Without entering into the
mathematical details of this approach [
The game is organized in 10 levels, which are presented from the easiest to the most dicult and which correspond to the di↵erent classification tasks of the top 10 classes of the Reuters 21578 dataset1. A level is difficult when it is hard to linearly separate the positive and the negative class (class c1 and c2 in Equation 1), i.e. when the positive and negative class partially overlap, and/or when there are few examples of objects in the positive class. We used the standard Reuters ModApte split to obtain the training and test documents for the ten classes. An objects in the training set can be used during the game either as a training example or a validation sample, but not both. In this way we are setting the machine learning problem in terms of a hold-out method that, despite being not very accurate compared to a cross-validation, is much easier to use in a game.
1http://www.daviddlewis.com/resources/ testcollections/reuters21578/
The goal of each level (an in general of the game) is to find the best classifier, i.e. the one which maximizes the F1 score, with the least amount or resources. Resources (we intentionally did not use the word credits or money) can be spent to increase the training and the validation set. At the beginning of the game, the players has already a free 10% of the collection annotated: 5% used for training and 5% used for validation. At any point in the game, the player can use some resources to buy additional training or validation objects. When he/she selects the training option, an additional 5% of the collection is added to the training, this action will set the points more precisely (because probabilities are estimated more accurately), while when he/she chooses the validation option, an additional 5% is included in the validation set and more points will appear in the plot. Notice that, when the player starts the game, he/she is provided with a sucient amount of resources to ‘buy’ the whole dataset for each level. In this way, the player does not have to think about saving resources for difficult levels. There is no limit to the time the user can spend in a level since the important part of the game is to understand the amount of resources that the player consider sucient to solve the problem, not how fast he/she can do it. Once the player has found what he/she considers the best classifier, he/she can proceed with the test, thus the classifier is tested on the test set and the F1 score is computed. At this point, the level is completed and the player is forced to go to the next level or conclude the game. 3.2
We have implemented the game of classification with
the Shiny package in R [
In Figure 1, you can see the layout of the web application for this pilot study. The interface can be divided in three main areas: the left panel, which contains buttons and controls to adjust the parameters of the classifier, the top right panel, which displays the plot of the two classes, and the bottom right area, which summarizes the employed resources and the F1 scores.
In the left side panel, there is a text field that allows the player to choose a username, and two buttons to start the game and to move to the next category. Be2https://github.com/gmdn/Classification low these two buttons, there is a field which indicates the remaining resources, and the available ‘clicks’, to training/validate the model for the current level. The player can utilize the positive and negative check boxes to visualize only the positive and negative points respectively. The player can center the line by clicking directly on the plot image and then shift and rotate it by using the sliders, which corresponds to the q and m parameter of Equation 1. Finally, we added a button that allows the player to go back to the best parameters setting that he/she has found during the current level. The color of the button indicates whether it is active (blue) or disabled (red).
In the top right panel, there is the main window that shows the two setsw of points: light blue for the positive class and red for the negative one. The dark blue line is the decision line used to split the two classes. In the left upper corner of the plot, there are three numbers: the first one in yellow is the current F1 score, the second one in blue is the best F1 score obtained by the user in the current class with the amount of training and validation set, and the red one is the value that the player should ‘beat’ 3.
Finally, below the plot area, on the right bottom box, there is a summary of the user actions and results. The histogram describes the amount of resources spent in training actions, with the green color, and validation actions, with the pink color. For each class, the F1 score obtained on the test set is reported below the corresponding column. At the end of the game, i.e. when the user has completed all the levels, he/she will see his/her mean F1 score, averaged on each category, and the mean goal score, which is the average of the F1 scores obtained by the automatic algorithm on each category. 4
A pilot study was carried out to test this preliminary version of the game and to collect opinions and suggestions regarding possible improvements of the game. The experiment was conducted on a sample of 20 student and researchers of the University of Padua. As future work, we aim at spreading this game through the use of social media in order to collect a bigger and a more diversified sample of users.
The majority of the users that participate in this
test had just a na¨ıve idea of what machine learning is
and how an automatic classifier operates. Since the
majority of our users were not machine learning
expert, we provided them a brief explanation of the
prob3This value corresponds to the best score of the Bernoulli
Na¨ıve Bayes classifier trained on whole dataset. We used the
results reported in [
In our experiments, for each player we collected the F1 score for each class and the amount of resources used. For a comparison with the state-of-the-art, we trained on the exact same training and validation set used by each player for each class a SVM with linear kernel using the ‘kernlab’ package in R4 together with the ‘caret’ package5. For a fair comparison, since the players acted as ‘optimizers’ of the Na¨ıve Bayes classifier decision, we also optimized the SVM by adjusting the cost parameter C within the range [0.01, 0.05] (smaller or larger values of C did not produce any significant change).
In Table 1, we report the F1 measure for each player on each level. The name of the column are the original names of the top 10 Reuters-21578 classes. The last three column shows the average of the F1 measure across the classes for each player, the average F1 measure of the SVM across the data of each player, the percentage of resources used by each player in a game. The last two rows represents: the average F1 score for each class for the players and for the SVM. 4https://cran.r-project.org/web/packages/kernlab/ 5https://cran.r-project.org/web/packages/caret/ Together with the users who participated to this pilot study, we started to sketch some possible scenarios of • Plants and gardening: we have a field sown with di↵erent types of seeds, but we do not know exactly where these seeds are. Some of the seeds will grow into edible plants, others will grow into weeds. The goal is to build a fence that separated the field in a way that “our” part of the field will contain the most of the edible plants and the least of the weeds. We can ask some help to our animal granivore friends (like pigeons and doves) to fly over and check some part of the fields to see whether the seeds are good or not. • Gold mines: there is an area with a lot of gold nuggets as well as useless common stones and you are the first explorer to mine this area. Your resources are limited, and you can only choose one part of the area while the rest remains untouched. The goal is to choose the area with loads of gold and the smallest number of stones. You have a friend who is an expert in gold mining and can probe the area to understand whether there is gold or stone. • Aerial warfare: in this war scenario, we have an army that is involved in a military zone, and we are forced to perform a raid to seize an area in order to secure the zone. The goal is to send our air forces to clear the area that contains the most of the enemies and the least of our ground troops and civilians. Before the raid, we can send our helicopters to explore the area and check the current situation. • Plastic and Glass Recycling: after a music festival in a park, you have to collect all the bottles and 5.2
We received some very good feedback about game controls and interaction and how to improve them to obtain a better feeling of the game. 5.2.1
Most of the players would prefer a full screen window to see the points more clearly without any distraction. The amount of credits used per class is not very relevant for their game when they play as well as the score obtained in previous classes. It would be better to have a window with the ranking of the scores that players can open when they need to see their status. 5.3
Players began to understand the use of the sliders after a few attempts. In particular, the rotation of the line was not immediately clear since the plot is centred around the minimum and maximum values of the coordinates, while the slope is computed given the intercept of the line with the y axis. The thing that puzzled the players was the non-intuitive rotation around a point far from the plot limits. For this reason they suggested two alternatives: • to select a fixed point within the plot (i.e. the center of the line) and to rotate the line along this point; • to maintain the slope fixed and rotate the plane instead of the line.
Moreover, the control of the slope of the line would be easier with a “knob” rather than a slider6. There are also new ideas about the interactions with the game in terms of touch screen technologies. In fact, it would be much easier for the players to interact with the game with the gestures that are now “natural” on mobile devices: rotation, swipe, zooming in and out, may enhance the user experience and bring the game to a di↵erent level. Finally, line controls should be overlaid on top of the main window instead of being on one side, in this way the eye of the player does not have to move from one side to the other of the screen every time the line has to be adjusted. 5.4
From the live interaction with the players during the
game sessions, it was clear that one of the strongest
motivations to replay the game was to have ranking of
the players with the scores obtained, to know whether
the friends/colleagues performed better or worse. At
the time of the pilot experiments, we could give them
hints about their performance compared to the other
players and just that information was enough for them
to sparkle their sense of competition. Some of them
were willing to play a second time to just beat their
competitors. This is in line with the literature on
gamification [
Since the competitiveness is one of the main motivations which encourages the users to play the game and reach high performances, it is very important to find the definition of a formal criterion to rate and rank players by taking into account the F1 score and the resources spent. As described in section 3.1, the goal of the game is multiobjective, indeed the main task consists in defining, at the same time, a classifier which is e↵ective, i.e. it reaches high values of accuracy and precision, and ecient, i.e. it uses a few amount of resources in terms of training and validation.
Let C = {c1, . . . , ci, . . . , cN } be the set of categories, we denote with si the F1 score obtained on the test set of the i-th category by the player, and with gi the F1 score obtained by the automatic algorithm that we 6See for example the gallery of this type of control realized with d3js https://radmie.github.io/ng-knob/ called ‘goal’ score on the same category. We indicate with ti and vi the amount of resources spent by the user in training and validation documents respectively, and R is the total amount of resources provided at the beginning of the game, 1800 in our game. We define the user rating J as
N J = a X si + b 1 i=1 gi
N
1 X(ti + vi)
R
i=1
!
(2)
where a and b are two parameters, which range in
the interval [
In this first pilot study of the gamification in machine learning, we set up a simple game, based on a visual interpretation of probabilistic classifiers, that consists in separating two sets of coloured points on a twodimensional plane by means of a straight line. The 20 players that participated in this study already gave us important suggestions in order to improve both the game mechanics and the game controls. We believe that, with the right game scenario (plants or gold mine for example), this game could be easily played by users that do not need any information about training/validation.
Moreover, the classification results of the game were also very high compared to the small amount of labelled objects and we also found a very promising relation with the results of the SVM trained on the same labelled dataset. We are currently study a new version of the game with some options that implements cross validation, which would bring the machine learning aspect to a new di↵erent level.