Many interesting social studies can be done by asking people about what they think, feel or experience at the moment when certain events occur in their daily lives. These studies can be conducted based on the event-contingent protocol of the Experience Sampling Method. We have implemented this protocol in Crowdpinion - our software tool consisting of a web panel where the researchers can set up and control their studies and a mobile app for the responders. In order to extend the users' motivation beyond the will to contribute to research, we have applied some gamification elements based on fostering curiosity by gradually revealing the big picture from the overall study. In this paper we describe the concepts of Crowdpinion as a research tool, our approach to gamification, how we tested it in the beta version and present plans for future improvements
H.5.1 [Information Interfaces (e.g.,HCI)]: [Miscellaneous]; Computing]: [Games] gamification, crowdsourcing, information retrieval
There are various situations in business and research when obtaining the momentary opinions of some group of people is useful and important. It can in fact prove to Copyright © 2015 for the individual papers by the paper's authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. be more valuable than surveying people about longterm overall experience. For example, it is quite common in big businesses to periodically ask the employees to give an opinion about their working environment. The employees are then given a long survey sheet with many questions about how have they felt at work in last e.g. 6 months. This method has a couple of downsides. It is heavily based on participants’ long term memory and is prone to being affected by some contemporary biases, e.g. if such survey is conducted at a stressful month before deadlines, the results are likely to be worse than if it was conducted shortly after summer holidays or company-wide salaries rise. In an experience sampling study, if the employees are asked to answer small sets of questions a couple of times during a day in a period of one or two months, the results of the study will give more reliable and unbiased results [1].
In the classic approach to Experience Sampling Method (ESM) studies [2] the participants would be given some sort of signaling devices that would “beep” at certain moments and give the participants the signal to answer the questions from a paper answer sheet. Later solutions introduced using more advanced electronic devices like palmtops to collect the data faster [3]. Some tools include some context-aware elements, attempting to guess the possible current state of the participants using electronic sensors like GPS location [4]. In our work we decided to focus on a new variant of the ESM protocol.
We created Crowdpinion - a tool for event-based Experience Sampling [5] studies in the crowd. Our tool allows asking a big population (the crowd) to answer some sets of questions when something particular happens or an event occurs in their day. The user voluntarily notifies Crowdpinion about the event and answers a set of questions. Users are free to notify the occurrence of an event when he/she has the “urge to give individual opinion” about issues that they feel affects a larger group. For instance, this could be events in services such as public transport or in a working environment. The tool consists of a web panel for configuring the studies and analyzing results and a mobile application for the studies’ participants. This way the tool can be easily distributed and reach a wide group of study participants and the researchers can manage their studies easily whenever they have access to the Internet. At this stage both core components of Crowdpinion - the web panel and the app are developed, but some functionalities (e.g. the summary of responses and the communication between the app and the server) need further development. Because of it, we could not conduct a full long-term study using the tool by the time of writing this paper.
Facing the challenge of people losing interest in regular responding to questions, we have used gamification elements to ensure higher level of participants’ engagement. Gamification in Crowdpinion is based on participants’ curiosity – by responding to questions in a study the participants unlock access to pieces of information about the study results and if they are really active, they can even add their own questions to the batch.
Plenty of research has been done in the field of gamification used in crowdsourcing of solving scientific problems. Scientific discovery games [6] aim to deliver complicated scientific problems split and converted to tasks that can be solved by unskilled users in the crowd. In Crowdpinion the core functionality is different, as it is a tool for surveying and social studies rather than for distributed solving of problems. Our motivation is closer to the serious games concept [7], where games aim to raise users’ awareness and change the way they think about some social issues. Crowdpinion aims to make people think about various related situations and issues and give opinions on them. These opinions can later be used in research to make a real change. In Crowdpinion we have added gamification elements to a surveying tool instead of creating a game. Similar situation can be found in [8], where the author describes how gamification can be added to search engines.
In some ways in Crowdpinion the researchers crowdsource the job of collecting data about the study issues to the responders. There are no financial incentives for the participants and gamification is one of the main means of motivation.
The combination of game mechanics with a
crowdsourcing task has been explored in several other
research studies. One of the first ones in this direction
was from Eickhoff and colleagues[
In Crowdpinion we have implemented the
eventcontingent protocol of the Experience Sampling
method [
Crowdpinion consists of a web panel for managing studies and a mobile app for studies’ participants. The researchers log in the web panel and create studies. Each study consists of question groups. Each question group is assigned to an event and contains one or more questions. A question should be fairly short and come together with the positive and the negative answer.
We assume that people are rather willing to share their opinion, especially if they do it for a good research purpose, they can do it anonymously and with little effort. On the other hand, in experience sampling studies in real world there is a risk that the participants will be too busy, too concerned or too tired to answer the questions systematically. Therefore we decided that we should provide the participants with additional motivation by including some gamification elements. The first element is based on assumption that the participants are curious about responses of other people and the overall results of the studies. The fact that they agreed to take part in a study suggests that they are interested in the case, so we believe our assumption is right. When a participant subscribes to a study, they have access to a summary of responses to one of the study’s questions. There is also an information that in order to unlock the summary of another question, the user has to give 10 responses themselves. With each . unlocked question the amount of required responses shall increase – e.g. 10 – 25 – 50 – 80 – 120 responses. The participants that have given a big number of responses shall be allowed to add their own question to the study. This element also derives from the assumption that the participants are interested in the case and would like to learn some more facts. The questions created by the participants shall be moderated by the researcher. Questions that are offensive, too personal or irrelevant to the main focus of the study shall be eliminated. The number of responses that allows participants to ask their own question should be set quite high so that the number of questions asked by participants is kept lower than the number of questions asked by the researchers. Furthermore, if the additional questions can be asked only by the most active users, there’s a big chance that they will be quite relevant to the study. However, if these measures prove to be insufficient to stop the users from adding low-quality questions to the set, we can go further and crowdsource the quality assurance of the studies’ content within the app. We could for example add a Report or Mark as irrelevant button to each question, so that the participants can contribute to high quality of the studies.
The questions asked by participants, being outside of the core questions of the study, do not need to be scientifically important or appropriately formulated for a serious study. Having a couple of questions that are intended to entertain the responders and attract their attention by introducing a bit of fun to the study can have a good effect on participants’ engagement. The researchers themselves shall be advised to add a couple of such questions to their studies apart from the questions that are actually important for the study. Another element of gamification in Crowdpinion is a leaderboard that is present in many computer games. The participants have access to a ranking where they can compare themselves to other participants. This is supposed to give the users motivation to be better (more involved) than other users and a feeling of accomplishment when they reach a high position in the ranking.
During the design phase of Crowdpinion development we created a theoretical use case – evaluation of attitude at work throughout the day. We have chosen Simula Research Laboratory as an example of a medium-sized workplace. In the pilot study every employee shall be asked to install the Crowdpinion app on their smartphones. The researcher conducting the study shall set it up in the web panel and send the subscription code to all participants.
As the study aims to analyze employees’ emotions during various moments of a working day, the list of events can include: Travelling to work Arriving in the office Just before a meeting Just after a meeting Completing a task Coffee break Lunch break Chat with a colleague Mailing Getting ready to go home Travelling home The participants shall be asked to open the app every time when one of the events from the list occurs, select the event from the list and answer the questions that will appear on the screen. In real life there is a risk that the employees will not have time or will to answer the questions every time when an event occurs. However, if they react to most of the events, we think that the results will still be valuable.
A set of questions asked at one of the events, e.g. the “Just after a meeting” event can consist of following questions: Do you think the meeting was productive? (Neg.: Waste of time; Pos: Very productive) Are you motivated to work in the remaining hours? (Neg.: Demotivated; Pos: Motivated) Are you sleepy? (Neg.: Sleepy; Pos.: Full of energy) The two first questions clearly are a part of the study. They can provide the researcher with meaningful data about the employees’ motivation and perception of the meetings. The results can be even more interesting if similar questions are asked e.g. before and after a meeting. If such study shows that employees’ motivation tends to drop after meetings and the meetings are mostly evaluated as “waste of time”, then the management should rethink the way the meetings are organized or even reduce the number of meetings. The third question (“Are you sleepy?”) is not a part of the study. It is one of the questions that are meant to provide some entertainment to the participants and attract their attention. While the researchers obtain valuable data about meetings’ effectiveness, the employees may find it entertaining to express how sleepy they are after a meeting and check if the other employees feel the same. This question could’ve been added to the study by the researcher or by one of the participants as one of the gamification bonuses. As described in Section 4, while answering more and more questions the participants gain access to summary of responses to the questions. For example, at the beginning they can see the responses to one question (e.g. “Do you think the meeting was productive? – Just after a meeting”) and information about the number of responses required to unlock another summary (e.g. “Give 10 more responses and unlock summary of ‘Are you satisfied with the effects of your todays work? – Travelling home’.”). The issues included in the study are interesting for the employees, so they are motivated to respond questions in order to see new summaries. The architecture of the system, which guarantees immediate or almost immediate sending the responses to the database over the Internet and provides the researchers with constant access to the web panel makes the analysis of the responses quick and convenient. The researchers do not need to wait until the study is finished, because they can access the current set of responses whenever they need to.
In order to evaluate our approach to gamification in Crowdpinion, we have conducted a short survey with 10 participants, using the early beta version of the app. We have presented the app to nine students of the University of Oslo and one postdoctoral researcher at the University of Trento. None of the participants have been involved in the project before. Two of the participants were female, eight were male. Six had a background in IT. In the evaluation we described the purpose of the app, brief ESM studies theory, the use case and made a quick walkthrough. After this each of the evaluation participants has been given the following instruction: Imagine you are asked to take part in this study for a month, every day when you are at work. Please answer the questions: 1. Would you still be motivated to respond to the events' questions regularly? 2. Would the ranking and the unlocking of the results help to keep you motivated? 3. Are there any ways, which would work better for your motivation? The general trend in the responses have been quite positive for our idea of adding gamification to Crowdpinion. Half of the responders stated that they would probably be motivated to take an active part in such study for a long period of time. What is more significant, most of them (8 out of 10) said that the ranking and the unlocking of the summaries of responses would play a major role in building the motivation. There were several interesting inputs regarding motivation. One responder admitted that even if he would be motivated, he might be forgetting to respond regularly. Another person highlighted the feeling of obligation to respond after subscribing to a study as the key of her motivation. One responder stated that she is not used to mobile apps and therefore would not like to use Crowdpinion regularly. A couple of people mentioned that the topic of a study must be interesting to them if they are to be motivated and one said that he would be motivated, if he saw that the study brings consequences (a real change) in his environment. Asked about what else could improve the participant’s motivation, one person emphasized the technical quality of the app and it’s interface as a motivating factor, another said that (e.g. in a company context) it would be interesting to divide responders into groups (e.g. by teams or professions) and compare responses between the groups.
One of the responders suggested extended gamification – a game, where the participants answer questions while travelling through a maze. Each question would be located in a room with 5 closed doors and by choosing an answer the participant would open one of the doors and go to the next room. At the end of a study the responder would reach some point or escape the maze. During the game the participants would also be able to see who else is following the same path – who is giving the same or similar answers. We find this idea interesting, because it would definitely improve the responders’ immersion in the studies.
In order to complete our work on Crowdpinion, we need to develop the system from a working beta version which it is now to fully developed system that can be used by anyone in a variety of studies. This requires many more iterations of designing, implementation and testing. Testing should include security testing, because even if the information about users is minimal, it must still be protected. Apart from technical approach to data protection, the effort must be put on making the users logically anonymous. The responses shall be separated from the responders in the researchers’ panel and in the summaries in the app, but still there may be some cases when the responders can be identified by people from within the group. For example, if only 3 people attended the meetings in a studied company in a given week and the average responses to the meetings-related questions decreased noticeably, the managers can easily guess that the people who attended the meetings in that week complained about them in Crowdpinion. Some measures have to be taken to prevent this undesired situations.
Most of the new features which we would like to add to Crowdpinion in later versions can be divided into two categories: making a better research tool and making gamification more effective. The first extension from the first category would be social targeting, meaning that the app shall ask the participants to provide some personal information about themselves, which would then allow the researchers to target the questions to particular groups of participants (e.g. ask young living in Oslo about their awareness of cancer risk factors) and filter the responses based on similar social categories (e.g. compare responses from women with higher and professional education).
Another feature that looks promising in terms of
improving Crowdpinion as a research tool is the
context-awareness [
Context awareness together with some other factors
can allow us to create an algorithm for evaluating the
quality of responses. Such algorithm would combine
various types of information, e.g. location, user’s
schedule, time interval between consecutive responses,
user’s history, validity of personal details provided by
the user etc. and calculate the value of quality of the
response [
We have emphasized two primary improvements that can be done in gamification. In one approach Crowdpinion could be extended to become a more social game. It could become a limited virtual reality where each responder has an avatar and interacts with other users while giving responses needed by the researchers. Gamification in Crowdpinion could also largely benefit from some applications of augmented reality. For example, using the participant’s phone’s camera, GPS, compass and accelerometer the app could display questions in the participant’s surrounding (e.g. in the office, in the canteen) in a shape of some tokens that need to be collected, which would encourage people to actively search for possibilities to respond to questions.
The element, which we need most at this stage, is multiple real-world studies to apply and test Crowdpinion. Once we reach a sufficiently advanced release, we will give the tool to selected researchers and conduct several pilot studies. 8.
In this paper we presented the way in which we added gamification to our experience sampling research tool. In Crowdpinion we used ranking and awards system for users that are most active in responding to questions in various studies. We have also encouraged the researchers and the participants to add pieces of fun content to the studies.
Crowdpinion is now at an early beta stage, so we have limited the evaluation to a short walkthrough and a survey with the users. However, we believe that once ready, Crowdpinion will become a useful surveying tool and that the gamification elements that we applied will make its users engaged in the studies, so that the researchers that do their studies with the tool will be obtaining valuable research data. We will continue developing Crowdpinion in two main areas: improved research features and improved users’ engagement via gamification.