In this paper, we describe a set of experiments that turn the machine learning classi cation task into a game, through gami cation techniques, and let non expert users to perform text classi cation without even knowing the problem. The application is implemented in R using the Shiny package for interactive graphics. We present the outcome of three di erent experiments: a pilot experiment with PhD and post-doc students, and two experiments carried out with primary and secondary school students. The results show that the human aided classi er performs similarly and sometimes even better than state of the art classi ers.
The creation of a labelled dataset for supervised learning is slow and expensive.
In the last years, mixed approaches that use crowd-sourcing and interactive
machine learning [
Nowadays, gami cation spreads through a wide range of disciplines and its applications are implemented in di erent areas. For instance, an increasingly common feature of online communities and social media sites is a mechanism for rewarding user achievements based on a system of badges and points. They have been employed in many domains, including educational sites like Khan Academy3, and tourist review sites like Tripadvisor4. At the most basic level, these game elements serve as a summary of a users key accomplishments; however, experience with these sites also shows that users will put in non-trivial
amounts of work to achieve particular badges, and as such, badges can act as
powerful incentives [
The use of gami cation in academic research areas has been introduced very
recently and its potential is still to be explored and validated. Information
Retrieval (IR) has recently dealt with gami cation, as witnessed by the GamifIR in
2014, 2015 and 20165. In [
In this paper, we present the recent studies of gami cation in text classi
cation and the development of a Web application written in R with the package
Shiny [
In this section, we present the re nements of a visualization approach of
probabilistic text classi ers that was transformed into a game. The application was
implemented with the Shiny package in R that allows to build interactive
graphics [
The game is based on the two-dimensional representation of probabilities, also
known as Likelihood Spaces [
can be found in [
P (ojc2) < m P (ojc1) +q | {yz } | {xz } (1) where P (ojc1) and P (ojc2) are the likelihoods of the object o given the two categories, while m and q are two parameters that can be either set automatically, for example by optimizing a measure of classi cation accuracy, or semiautomatically by asking to a user to suggest the initial conditions based on a visual inspection of the problem. In fact, if we interpret the two likelihoods as two coordinates x and y of a two dimensional space, the problem of classi cation can be studied on a two-dimensional plot where: i) the decision of the classi cation is represented by the line y = mx + q that splits the plane into two parts, ii) the points that fall `below' this line belongs to class c1. 2.2
The initial version of the interface7, shown in Figure 1, was designed to be used by experts to understand how to optimize the search of the optimal parameters.
In the \gami ed" version of this problem, players have to nd the best combination of m and q having a xed amount of resources available to train and validate the algorithm. The game is organized in N levels (that corresponds to the binary classi cation problems), which are presented from the easiest to the most di cult and which correspond to the di erent classi cation tasks of the top N classes of the Reuters 21578 dataset8. A level is di cult when it is hard to linearly separate the positive class c1 and the negative class c2. An object can be used during the game either as a training example or a validation sample, but not both. The goal of each level (and in general of the game) is to nd the best classi er, i.e. the line which best separates the two categories, c1 and c2 and therefore which maximizes the F1 score, with the least amount or resources.
Resources can be used to increase the number of objects of the training and/or the validation set. At any point in the game, the player can use some resources to buy additional training or validation objects. By doing so, an additional 5% of the collection is added to either the training set (more precise) or the validation set (more objects on the screen). Once the player has found what he/she considers the best classi er, he/she can proceed with the test, thus the classi er is tested on the test set and the F1 score is computed. At this point, the level is completed and the player is forced to go to the next level or conclude the game. 3
In the previous section, we presented how the players can interact with the classi cation game by \investing" a limited amount of resources to buy training and validation data and, consequently, to nd a better combination of the two parameters m and q.
In this section, we present the results of three di erent experiments of the gami cation of text classi cation that involved di erent users and di erent interfaces. 3.1
A second version of the interface was designed for PhD and post-doc students9
and a pilot study was carried out to test this preliminary version of the game
and to collect opinions and suggestions regarding possible improvements of the
game. In this rst experiment, we were positively surprised by two results (a
complete description of the results can be found in [
25% of the annotated dataset) performed as well as the same SVM trained on the whole dataset. This results are very promising since, the gami cation of text classi cation may give a reliable indication about when to stop the labelling process and use the annotated dataset to train with good classi cation performances a state-of-the-art-algorithm. This second part will require a deep analysis and further experiments to con rm the statistical signi cance of this process. 3.2
During the European Researcher's Night at the University of Padua in September 2016, we designed a new interface to make the game easier for kids of primary and secondary schools who played the application. The interface, shown in Figure 2, lets users play only three levels (each level corresponds to a di erent category) and give feedback about the current performance whenever the line is adjusted. In this experiment, we also added some incentives like a public leaderboard that was displayed and regularly updated and chocolate candies for the top scorer. A total of 28 players used the interface.
Considering that these users did not know anything about machine learning or text classi cation, the results in terms of classi cation performance were even more surprising compared to the rst experiment. In Table 1, we compare the average results of the classi cation performance of the players (column manual ) with the classi cation performance of a Nave Bayes classi er (NB) and a Support Vector Machine (SVM) as well as the `goal' performance. You can notice that the results obtained by participants are very close to the one obtained with the NB and in the case of the second class, the users achieves better performances than NB. On average, the classi er with the human contribution is performing better than NB and worse than SVM. 3.3
The rst week of April 2017, during an event at one of branches of Banca d'Italia in Padua for the brand new 50 euro note, we presented a third version of the game that was available for the public a whole week. For this study, we decided to make the layout cleaner, see Figure 3, and add keyboard controls to change the decision line instead of using sliders. We kept the same game incentives, chocolate candies and leaderboard, and we added an instructional presentation of the problem to help the player to understand what `machine learning' and `training set' are.
A total of 27 participants played with the game and their results are reported in Table 2. Even in this case the human aided classi er achieves good results and the interaction of users with the algorithm through the gami ed approach reached performances close to SVM and often better than NB. In this case the results are much closer to SVM than NB even if the amount of resources used was comparable to the second experiments: players tend to consider the performance of the classi er satisfactory when 30% of the resources are used.
Finally, notice that the algorithms were trained on a di erent amount of data during the game, the scores in Table 1 and Table 2 are not directly comparable. This explains the di erent results reported for NB and SVM in Table 1 and Table 2. In this paper, we presented the ongoing work on gami cation for text classi cation that involves non expert users in the task of labelling data and produce an estimate of the monetary cost of creating the training dataset. Considering the very abstract game (a line and some dots), the rst three preliminary studies were successful in terms of participation and initial results. The goals of these studies is to have feedback and collect enough data to study how to design the game in order to make it open to the general public; in addition, we want to understand whether a `serious' game can be implemented in order to gather labelled data for machine learning.
Future work aims at extending the proposed game and transform it in an application for di erent mobile devices. Therefore, further e ort is needed to design the interface of the mobile application with integrated environments like Unity10. Moreover, considering that the players are not expert in classi cation the rules of the game should be presented clearly and some concepts, as for example the validation phase, need to be explained in an easier way.
Finally, we aim at investigating a di erent game mode with two players collaborating together to reach a common goal. For instance, the users can share the controls so they need to cooperate to nd the best solution, or an alternative is to assign di erent tasks to each user, one user will control the classi cation line while the other user will assess documents to help him or her to get more training examples. 10 https://unity3d.com