The Impact of Gamification on Socio-technical Communities: A Case for Network Analysis Antti Knutas Timo Hynninen Lero, the Irish Software Research Centre Lappeenranta University of Technology Dublin, Ireland Lappeenranta, Finland antti.knutas@dcu.ie timo.hynninen@lut.fi ABSTRACT The gamification of learning in software engineering educa- Designing for motivating and engaging experiences is at the tion has been studied earlier [25, 14, 3], mostly at the level of core of gamification. The results of gamification are often eval- individual students. However, software engineering education uated with user experience testing involving recordings, sur- classes and software engineering are communities, or socio- veys, and interviews. However, in multi-user socio-technical technical entities [17], which are interconnected by personal environments the benefits of gamification are often realized in and technical communication channels. In this context, we interactions between users. We propose that social network define a socio-technical system as a complex system which analysis should be used more to analyze the impact of gam- involves both physical–technical elements and networks of ification at community level. To demonstrate the approach, interdependent actors [10]. These technological structures are we present a study where a gamified computer-supported col- not only neutral instruments, but also shape users’ perceptions, laborative learning system was introduced to a course, and behavioural patterns, and activities [17]. Computer-supported compare the course to a previous instance. Furthermore, we collaborative learning tools as software systems always in- present several examples of how social network analysis can clude certain inbuilt assumptions, interaction methods, and be used with hypothesis testing and discuss the benefits of the rules. An issue tracker such as GitHub enables collaboration approach. in a different way than for example a forum software. When gamification is added into the mix, it adds explicit interaction ACM Classification Keywords rules. Embedding a technical element into the social environ- H.1.2. User/Machine Systems: Human Factors; I.2.1. Appli- ment changes not only the experience of an individual users, cations and Expert Systems: Games; K.3.1. Computer Uses but has the potential to impact interaction patterns and rules in Education: Collaborative Learning of social interaction. Author Keywords We propose that social network analysis (SNA) is a fitting re- gamification, social network analysis, hypothesis testing, search approach for investigating the impact of new technolo- computer-supported collaborative learning, socio-technical gies in socio-technical systems, such as in the case of applying systems gamification. Social network analysis can be summarized to be an interdisciplinary research field about who communicates INTRODUCTION with whom, in which social relationships are viewed in terms Collaborative learning is a learning method where students of the network theory [26]. In social network analysis, com- have a symmetry of action, knowledge and status, and have munication between individual or social units are mapped into a low division of labor [15]. Computer-supported collabora- a communication matrix and then modeled as graphs that are tive learning (CSCL) facilitates the interaction with software composed of nodes and edges. Nodes represent individuals tools and increases potential for creative activities and social and edges the connections or communications between them. interaction [30]. Collaborative learning is a commonly used These graphs can be used to visualize communication patterns method in software engineering education as it prepares stu- in socio-technical systems systems. Additionally, in the graph dents to work in software teams as independent experts. In theory there are different mathematical tools available, which recent studies, it has been shown that students can be guided can be used for example to estimate the relative influence of towards educational goals like collaboration by using gamifi- nodes in the graph or analyze the graph by the connection cation [25], which is the application of game-like elements to patterns of the nodes [2, 4, 29]. non-game environments [13]. Currently many social network analysis -based studies rely only on descriptive statistics and omit hypothesis formulation and evaluation. We make a case that hypothesis testing is an essential part of empirical studies that use quantitative methods, such as SNA. Wohlin et al. [32, p. 12] summarize the the importance of hypothesis testing for generating new GHItaly18: 2nd Workshop on Games-Human Interaction, May 29th, 2018, Castiglione knowledge as follows. "In science, physical phenomena are della Pescaia, Grosseto (Italy). addressed by putting forward hypotheses. The phenomenon is Copyright © 2018 for the individual papers by the papers’ authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its observed and if the observations are in line with the hypothesis, editors. this becomes evidence for the hypothesis. Experiments are More recently SNA has been utilized by Wise & Cui [31], important to test the hypothesis and in particular the predictive for example, to distinguish social relationships in MOOC dis- ability of the hypothesis. If the new experiments support cussion forums. Their study showed connections between the hypothesis, then we have more evidence in favor of the network structure and discussion practices. The study also hypothesis." If the study remains at the level of descriptive pointed out that interactions involving course content related statistics, no evidence is created to support the presented new discussion and other discussion should be examined sepa- knowledge. 1 rately. To illustrate the social network analysis as a research method SOCIAL NETWORK ANALYSIS AS A RESEARCH and how it can be used in hypothesis testing, we present a study METHOD of how embedding technical elements into a collaborative Social network analysis is based on graph theory, which is learning course creates a new socio-technical system. The a concept related to mathematics and information sciences goal is to evaluate how the new elements affect the structure of [18]. Central concepts in graph theory are network, node, and collaboration in and between student teams. First, we present edge. A network is composed of actors, or nodes, between descriptive statistics generated by social network analysis and which are connections, or edges. A common method in social then test several hypotheses about the impact based on that network analysis is to store graphs as adjacency matrices. data. We conclude the paper with discussion of the study From these matrices one can calculate different statistics and results and what kind of insight social network analysis can characteristics with graph theory and matrix algebra. Several provide for other studies. applications have been developed to automate the analysis, such as UCINET [6] and Gephi [4]. A common method to RELATED WORK ON ANALYSIS OF GAMIFIED COMMUNI- visualize the graph matrix is a sociogram, which is is a visual representation of the nodes and and their edges. Most often TIES the nodes are represented as circles and edges as connecting Social network analysis has been utilized by social scientists lines. to explain different phenomena. Borgatti et al. [7] divide the various themes that can be examined by SNA into the Descriptive Statistics for Graphs and Nodes similarities of actors, their social relations, interactions and The degree is most essential of node characteristics. The node flows (transferring physical or intellectual property between degree indicates the number of edges connected to a node, or actors). the number of connections the node has to other nodes. The A number of authors have previously conducted research in weighted degree also takes into account the different values the group and peer learning activities in software engineering ed- edges might have. Additionally, the most simple measure of ucation using social network analysis [21]. This is because centrality, the degree centrality, is derived from node degrees. the effects of social networking in an educational setting can Degree centrality is calculated from the sum of edges a node have a substantial impact on learning performance [11]. For has. example, the study by De Laat et al. [24] presents an overview Graph density is derived from node degrees. It is a value of how social network analysis can be applied in the field between zero and one, and it indicates how many connections of CSCL. The authors build a case for utilizing SNA to in- have been established of all possible connections. If all nodes vestigate the group dynamics in a CSCL environment. The are connected to all other nodes, it is 1. If the graph has no study emphasizes that in order to fully understand student par- edges, then network density is 0. ticipation in CSCL environments, we must analyze who are the actors in the collaborative learning task, and if actors are A social network can be composed of one or several compo- participating actively or peripherally. It is also important to nents. A component is an element of a social network where a understand how the different methods of participation change path can be traced from one node to another. The number of over time. connected components can be evaluated how fragmented the network is. In well-functioning socio-technical systems the In terms of social network analysis in gamified CSCL context, desired number of connected components is just one. de-Marcos et al. [12] used SNA to investigate a gamified e- learning course. In the study different network metrics were Centrality is a method to measure nodes’ relative importance assessed in how well they can predict academic performance. in a graph [29]. Application areas for centrality include finding The investigation found potential in the use of network metrics influencers in social media or finding the most active collabo- to predict learning achievement but also found limitations to rators in learning. Degree centrality is the easiest to calculate their applicability beyond central nodes in the social network. and was covered earlier in this section. Betweenness centrality This study is one of the few studies which have evaluated the is the measure of all the shortest paths that go through the node. impact of gamification on socio-technical communities from a Other, more advanced methods such as eigenvector centrality network analysis perspective. are beyond the scope of this article. Regression Analysis and Comparisons in Social Network 1 The authors acknowledge that this is a very positivist view and other philosophies of science or qualitative approaches are just Analysis as valid. However, in positivist-quantitative approaches hypothesis Traditional hypothesis testing, such as regression analysis, is testing is essential. difficult to use on networks, because edges exist as dyads between two nodes. The Multiple Regression Quadratic As- Software Artifact signment Procedure (MRQAP) method was developed to ad- The software artifact for computer-supported collaboration in dress this issue and it allows analyzing the effect of external software engineering student teamwork, originally presented variables on network structure [23]. Correlation between net- in [19], was created following along the following principles: works can be analyzed with a related method, the Quadratic Increased team collaboration, extended collaborative commu- Assignment Procedure (QAP) [22]. nication between students, and explicit goal communication that supports shared goal setting and goal achievement. CASE STUDY: ANALYZING THE IMPACT OF GAMIFICA- These tenets were realized as a collaboration platform that TION ON STUDENT COLLABORATION concentrated around setting team goals and issues, viewing In this section, we present a case study where we perform com- the status of other teams and using a chat-based tool, Slack, parative social network analysis of two subsequent teamwork- to communicate and share information regarding the goals based computer-supported collaborative courses in software and issues. The information and views concentrated around engineering at master’s level. The research data has been three main views, with a sidebar and notifications showing collected in earlier studies [19, 20]. The two courses were additional information when needed. The guidelines of re- arranged in subsequent years with the main difference being sponsive design were followed in the implementation of the that in the second year a gamified, computer-supported collab- views, with the system working equally well on desktops or oration platform was introduced to the course. In this paper, mobile phones. The selected chat system also had a mobile we take the analysis further with hypothesis testing, with an client application for iOS and Android, enabling continuous emphasis on evaluating the impact of the artifact introduced collaborative communication for the community. in the second study. Social Network Analysis Results In the first study [20], we observed 17 students over a five day We performed descriptive social network analysis with the long intensive format and collaborative software engineering Gephi software [4] and hypothesis testing with the R statistical course, arranged in the Code Camp format [27]. The course programming language [28]. First, we produced descriptive had 10 hours of lecturing and 40 to 64 hours (depending on statistics for the two networks for comparison purposes, such the student team) of collaborative teamwork around a set task. as the number of components in the network, various indica- The topic for the course was to develop a new mobile or tablet tors for network size, and the average degree for the nodes. application before the deadline on the fifth day. The teams Then we tested hypotheses of whether the distribution of node were free to choose their own path to a solution within the attributes, such as degree and centrality, were different in the theme and how to achieve it. The students had no other courses networks, and how well being in the same group predicts two during the week. The students spent their time in the same students having a collaborative connection. shared computer classroom, with each student team sitting at their own table group. After the kickoff event and a technology Descriptive Statistics tutorial, the teachers were available to advise when requested We analyzed the social networks of two studies, classroom and during the rest of the week and to facilitate inter-team col- online communication, and produced descriptive statistics of laboration. Most of the course participants were master’s the networks as presented in Table 1. The online system had a level students with previous experience of other programming little more nodes and a lot more recorded interactions. Both projects. All recorded collaborative communications occurred networks had only one connected component, which means offline in the classroom in this study. that there was a path from each node to any other node. The online network was a little less connected, which is signified All student interactions that occurred in the classroom were by an increased network diameter and lower network density. recorded during the first study. The video and audio recordings were combined into multi-angle and surround sound videos Despite the lower network density, on average a node had more that allowed the researchers reviewing the video to analyze connections to other nodes in the online network as signified by several concurrent interactions. This resulted in 40 hours of the average degree. However, the average weighed degree was video, from which 3366 interactions were coded for analysis. much higher in the classroom, implying that students in the The data analysis presented in this paper is sourced from the online system were slightly better connected and classroom- quantitative analysis results of the video recordings. based students established stronger and more even ties. The second study [19], with the computer-supported collab- Hypothesis testing orative learning system, was similar to the first one and also Descriptive statistics are not enough to evaluate proof of evi- arranged in the Code Camp format [27]. The only differences dence when for example making claims about the differences were the introduction of the computer-supported collabora- in social networks. For this, we need statistical hypothesis tive learning system, the theme of the course, and the number testing which can give us the probability of the evidence sup- of participants. This time there were 22 participants divided porting our hypothesis being significant, and the effect size of into six teams. The topic of the course was using open data the evidence [9]. An obtained p-value represents the probabil- for sustainability. In this study, all recorded collaborative ity of observing the current data due to chance when the null communications occurred online in the computer-supported hypothesis is true. Effect size allows us to evaluate how sub- collaboration system. The data analysis presented in this paper stantial the evaluated phenomenon is, after it has been shown was sourced from the system log files. that the phenomenon probably exists. Graph No. of Recorded No. of connected Network Average Average degree Network nodes interactions components diameter degree (weighted) density Classroom 18 703 1 3 9.39 374 0.55 Online, gamified 23 3216 1 4 10.5 54.5 0.46 Table 1. Descriptive social network analysis statistics We first used the Mann–Whitney U test [32] to test the dif- Despite the medium effect size and statistical significance, ference in distributions between the datasets because of its in both cases the R2 value is low. The same-team variable suitability for non-normal data. A continuity correction was predicts only 11 to 12% of the variance, which means that a enabled to compensate for non-continuous variables [5]. The social connection is mainly predicted by other factors. Bonferroni correction was used to adjust the p-value to com- pensate for the family-wise error rate in multiple comparisons Discussing the Analysis Results [1]. We calculated effect size r using guidelines by Fritz et When evaluating the differences in the networks, we made al. [16] for the Mann-Whitney U test. We evaluate the effect the following findings: The online network from the second size as proposed by Cohen [9] that in r a large effect is .5, a study was slightly larger than the classroom collaboration net- medium effect is .3, and a small effect is .1. work in the first study. The online network had vastly more Our null hypotheses is that there is no difference in the distribu- interactions, but the nodes had stronger ties in the classroom tions of node degrees or betweenness centralities between the network in a statistically significant manner. In both networks, networks. Node degree and two node centrality variables were there was the same distribution of betweenness centralities, tested to establish differences in the distribution of connections which means there were the same distribution of central nodes and whether there was overall differences in how the relative mediating communication. According to the MRQAP results, importance of nodes were distributed. The Mann-Whitney U being in the same team was a stronger predictor for collabo- test results are summarized in Table 2, with the sum of ranks ration in the online network, which means that the classroom denoted by the U-value, the probability by the p-value, and ef- network had stronger inter-team collaboration. A summary of fect size by r. When evaluating node degrees and betweenness the tested hypotheses and outcomes are presented in Table 3. centrality we fail to reject the null hypothesis, with there being At a first look the gamified online communication platform no statistically significant difference between the networks appears to increase collaborative communication between stu- in these aspects. When evaluating weighted degrees we can dents. The three essential goals of the platform were to in- reject the null hypothesis and accept an alternative hypothe- crease communication, enable stronger inter-team communica- sis that there is a difference in distributions between the two tion, and to reduce the hierarchy in communication. However, networks in this aspect. after testing our hypotheses with social network analysis data, Finally, we want to evaluate the impact of external factors we found that compared to plain classroom communication, on the network. For this purpose we use MRQAP to apply the online platform failed to enable stronger social ties, in- regression analysis to evaluate the effect of external variables creased inter-team communication, or more diverse patterns of on dyads, or the connections between nodes. For the MRQAP collaboration. As the CSCL platform had an increased number analysis we used the R sna library [8] and its network multiple of interactions, these outcomes might not have been discov- regression function. ered without in-depth social network analysis and statistical evaluation. In our first regression analysis we evaluate whether being in the same team predicts a connection in the classroom. The regres- These discoveries were enabled by statistical hypothesis test- sion model using MRQAP and ordinal least squares regression ing, which allowed us to evaluate the strength of the evidence statistically significantly predicts collaborative connections, F in regard to statistical significance and effect size. For ex- (1, 304) = 38.38, p < .001, adj. R2 = .11. Regression coeffi- ample, the beneficial differences in betweenness centralities cient estimate of 0.41 on the same team variable means that might seem plausible when presented side by side as descrip- being in the same team increases the possibility of a connec- tive statistics. However, by statistically testing our hypotheses, tion by 41%. Cohen’s f2 for effect size is 0.13, which means a we were able to conclude that the difference is not significant medium effect [9]. and the effect size was trivial. In the second regression analysis we evaluate whether being in the same team predicts a connection in the online plat- DISCUSSION AND CONCLUSION form. The regression model using MRQAP and ordinal least In this paper, we demonstrated how social network analysis squares regression statistically significantly predicts collab- can be used in hypothesis testing -based evaluation of socio- orative connections, F (1, 504) = 72.28, p < .001, adj. R2 technical communities. In the demonstration we evaluated = .12. Regression coefficient estimate of 0.56 on the same the impact of a new software artifact. The artifact at a glance team variable means that being in the same team increases the appeared to increase communication between participants of possibility of a connection by 56%. Cohen’s f2 for effect size the community, but when evaluating social ties, the artifact’s is 0.14 which means a medium effect [9]. impact was more complex than reflected by simple usage statistics, and not fully beneficial. Variable U- Mdn Mdn p-value Adjusted p (Bonferroni) / Effect size value (Classroom) (Online) significance (r) Degree 251.5 20.00 21.08 0.37 1 / no 0.13 (small) Weighted degree 20 748.00 108.97 0.000000670.000002019891 / yes 0.72 (centrality) (large) Betweenness 218 7.24 6.35 0.97 1 / no 0.007 centrality (none) Table 2. Mann-Whitney U test results Test Hypothesis Outcomes Mann- The distribution of node attributes is different in the Node degrees and betweenness centralities have no Whitney two datasets. significant differences. Weighed degrees have. U MRQAP Being in the same team is a statistically significant Statistically significant relationship exists between the predictor for the strength of connection between the variables. In the online case the relationship is somewhat nodes. stronger. Table 3. Summary of tested hypotheses and outcomes Social network analysis is a research method worth consid- REFERENCES ering when studying the impact of technological artifacts on 1. Hervé Abdi. 2007. Bonferroni and Šidák corrections for socio-technical communities, such as gamified systems. It multiple comparisons. Encyclopedia of measurement and is a powerful technique, especially when combined with hy- statistics 3 (2007), 103–107. pothesis testing. It can reveal patterns of actions in complex, 2. Ajith Abraham and Aboul Ella Hassanien. 2010. interlinked systems that involve multiple interdependent ac- Computational Social Network Analysis: Trends, Tools tors. The number of these socio-technical systems will only and Research Advances. Springer. increase with the ongoing digital transformation in our society. 3. Manal M Alhammad and Ana M Moreno. 2018. Some studies already use social network analysis to evaluate Gamification in software engineering education: a the impact of gamification on the educational communities, systematic mapping. Journal of Systems and Software such as the line of research by de-Marcos et al. [11, 12]. (2018). However, in gamification, computer-supported collaboration, and socio-technical research this is still in the minority [3, 14, 4. Mathieu Bastian, Sebastien Heymann, and Mathieu 21, 25]. Additionally, many social network analysis studies do Jacomy. 2009. Gephi: An open source software for not use hypothesis testing or regression analysis and rely just exploring and manipulating networks. In International on descriptive statistics. We recommend publishing tutorials, AAAI conference on weblogs and social media, Vol. 2. reviews, and methodological demonstrations of the benefits 5. Reinhard Bergmann, John Ludbrook, and Will PJM of more in-depth and rigorous research approaches in social Spooren. 2000. Different outcomes of the network analysis. Social network analysis has potential as a Wilcoxon—Mann—Whitney test from different statistics research method in the field of gamification and beyond. As packages. The American Statistician 54, 1 (2000), 72–77. future work, aspects of social network analysis not covered by this paper, such as community detection and more advanced 6. Stephen P Borgatti, Martin G Everett, and Linton C approaches to centrality, should be demonstrated in regard to Freeman. 2002. Ucinet for Windows: Software for social compatibility with hypothesis testing. network analysis. (2002). 7. Stephen P Borgatti, Ajay Mehra, Daniel J Brass, and ACKNOWLEDGMENTS Giuseppe Labianca. 2009. Network analysis in the social The work of the first author was supported by the Ulla Tuomi- sciences. science 323, 5916 (2009), 892–895. nen Foundation. This study was partially funded by the European Union Regional Development Fund grant number 8. Carter T. Butts. 2016. sna: Tools for Social Network A70554, “Kyberturvallisuusosaamisen ja liiketoiminnan kehit- Analysis. https://CRAN.R-project.org/package=sna R täminen”, administrated by the council of Kymenlaakso. This package version 2.4. work was supported, in part, by Science Foundation Ireland 9. Jacob Cohen. 1992. A power primer. Psychological grant 13/RC/2094 and co-funded under the European Regional bulletin 112, 1 (1992), 155. Development Fund through the Southern & Eastern Regional Operational Programme to Lero - the Irish Software Research 10. Hans De Bruijn and Paulien M Herder. 2009. System and Centre (www.lero.ie). actor perspectives on sociotechnical systems. IEEE Transactions on systems, man, and cybernetics-part A: Systems and Humans 39, 5 (2009), 981–992. 11. Luis de Marcos, Eva Garcia-Lopez, and Antonio 21. Antti Knutas, Jouni Ikonen, and Jari Porras. 2015. Garcia-Cabot. 2016a. On the effectiveness of game-like Computer-Supported Collaborative Learing in Software and social approaches in learning: Comparing Engineering Education: A Systematic Mapping Study. educational gaming, gamification & social networking. Journal on Information Technologies & Security 7 (2015), Computers & Education 95 (2016), 99–113. 4. 12. Luis de Marcos, Eva García-López, Antonio 22. David Krackardt. 1987. QAP partialling as a test of García-Cabot, José-Amelio Medina-Merodio, Adrián spuriousness. Social networks 9, 2 (1987), 171–186. Domínguez, José-Javier Martínez-Herráiz, and Teresa 23. David Krackhardt. 1988. Predicting with networks: Diez-Folledo. 2016b. Social network analysis of a Nonparametric multiple regression analysis of dyadic gamified e-learning course: Small-world phenomenon data. Social networks 10, 4 (1988), 359–381. and network metrics as predictors of academic performance. Computers in Human Behavior 60 (2016), 24. Maarten de Laat, Vic Lally, Lasse Lipponen, and 312–321. Robert-Jan Simons. 2007. Investigating patterns of interaction in networked learning and 13. Sebastian Deterding, Dan Dixon, Rilla Khaled, and computer-supported collaborative learning: A role for Lennart Nacke. 2011. From Game Design Elements to Social Network Analysis. International Journal of Gamefulness: Defining "Gamification". In Proceedings of Computer-Supported Collaborative Learning 2, 1 (March the 15th International Academic MindTrek Conference: 2007), 87–103. Envisioning Future Media Environments (MindTrek ’11). ACM, New York, NY, USA, 9–15. 25. Fiona Fui-Hoon Nah, Qing Zeng, Venkata Rajasekhar Telaprolu, Abhishek Padmanabhuni Ayyappa, and Brenda 14. Darina Dicheva, Christo Dichev, Gennady Agre, and Eschenbrenner. 2014. Gamification of education: a Galia Angelova. 2015. Gamification in education: a review of literature. In International Conference on HCI systematic mapping study. Journal of Educational in Business. Springer, 401–409. Technology & Society 18, 3 (2015), 75. 26. Evelien Otte and Ronald Rousseau. 2002. Social network 15. Pierre Dillenbourg. 1999. What do you mean by analysis: a powerful strategy, also for the information collaborative learning? Collaborative-learning: sciences. Journal of Information Science 28, 6 (Dec. Cognitive and computational approaches 1 (1999), 1–15. 2002), 441–453. 16. Catherine O Fritz, Peter E Morris, and Jennifer J Richler. 27. J. Porras, K. Heikkinen, and J. Ikonen. 2007. Code camp: 2012. Effect size estimates: current use, calculations, and a setting for collaborative learning of programming. Adv. interpretation. Journal of experimental psychology: Technol. Learn. 4, 1 (Jan. 2007), 43–52. General 141, 1 (2012), 2. 28. R Core Team. 2017. R: A Language and Environment for 17. Frank W Geels. 2004. From sectoral systems of Statistical Computing. R Foundation for Statistical innovation to socio-technical systems: Insights about Computing, Vienna, Austria. https://www.R-project.org/ dynamics and change from sociology and institutional theory. Research policy 33, 6-7 (2004), 897–920. 29. John Scott. 2012. Social network analysis. SAGE Publications Limited. 18. Frank Harary. 1969. Graph theory. Addison-Wesley, Reading, MA. 30. Gerry Stahl, Timothy Koschmann, and Dan Suthers. 2006. Computer-supported collaborative learning: An 19. Antti Knutas. 2016. Increasing Beneficial Interactions in historical perspective. Cambridge handbook of the a Computer-Supported Collaborative Environment. learning sciences 2006 (2006), 409–426. Number 718 in Acta Universitatis Lappeenrantaensis. Lappeenranta University of Technology. 31. Alyssa Friend Wise and Yi Cui. 2018. Learning communities in the crowd: Characteristics of content 20. Antti Knutas, Jouni Ikonen, Dario Maggiorini, Laura related interactions and social relationships in MOOC Ripamonti, and Jari Porras. 2016. Creating Student discussion forums. Computers & Education (2018). Interaction Profiles for Adaptive Collaboration Gamification Design. International Journal of Human 32. Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Capital and Information Technology Professionals Ohlsson, Björn Regnell, and Anders Wesslén. 2012. (IJHCITP) 7, 3 (2016), 47–62. Experimentation in software engineering. Springer Science & Business Media.