=Paper=
{{Paper
|id=Vol-2246/GHItaly18_paper_08
|storemode=property
|title=The Impact of Gamification on Socio-technical Communities: A Case for Network Analysis
|pdfUrl=https://ceur-ws.org/Vol-2246/GHItaly18_paper_08.pdf
|volume=Vol-2246
|authors=Antti Knutas,Timo Hynninen
|dblpUrl=https://dblp.org/rec/conf/avi/KnutasH18
}}
==The Impact of Gamification on Socio-technical Communities: A Case for Network Analysis==
https://ceur-ws.org/Vol-2246/GHItaly18_paper_08.pdf
The Impact of Gamification on Socio-technical
Communities: A Case for Network Analysis
Antti Knutas Timo Hynninen
Lero, the Irish Software Research Centre Lappeenranta University of Technology
Dublin, Ireland Lappeenranta, Finland
antti.knutas@dcu.ie timo.hynninen@lut.fi
ABSTRACT The gamification of learning in software engineering educa-
Designing for motivating and engaging experiences is at the tion has been studied earlier [25, 14, 3], mostly at the level of
core of gamification. The results of gamification are often eval- individual students. However, software engineering education
uated with user experience testing involving recordings, sur- classes and software engineering are communities, or socio-
veys, and interviews. However, in multi-user socio-technical technical entities [17], which are interconnected by personal
environments the benefits of gamification are often realized in and technical communication channels. In this context, we
interactions between users. We propose that social network define a socio-technical system as a complex system which
analysis should be used more to analyze the impact of gam- involves both physical–technical elements and networks of
ification at community level. To demonstrate the approach, interdependent actors [10]. These technological structures are
we present a study where a gamified computer-supported col- not only neutral instruments, but also shape users’ perceptions,
laborative learning system was introduced to a course, and behavioural patterns, and activities [17]. Computer-supported
compare the course to a previous instance. Furthermore, we collaborative learning tools as software systems always in-
present several examples of how social network analysis can clude certain inbuilt assumptions, interaction methods, and
be used with hypothesis testing and discuss the benefits of the rules. An issue tracker such as GitHub enables collaboration
approach. in a different way than for example a forum software. When
gamification is added into the mix, it adds explicit interaction
ACM Classification Keywords rules. Embedding a technical element into the social environ-
H.1.2. User/Machine Systems: Human Factors; I.2.1. Appli- ment changes not only the experience of an individual users,
cations and Expert Systems: Games; K.3.1. Computer Uses but has the potential to impact interaction patterns and rules
in Education: Collaborative Learning of social interaction.
Author Keywords We propose that social network analysis (SNA) is a fitting re-
gamification, social network analysis, hypothesis testing, search approach for investigating the impact of new technolo-
computer-supported collaborative learning, socio-technical gies in socio-technical systems, such as in the case of applying
systems gamification. Social network analysis can be summarized to
be an interdisciplinary research field about who communicates
INTRODUCTION with whom, in which social relationships are viewed in terms
Collaborative learning is a learning method where students of the network theory [26]. In social network analysis, com-
have a symmetry of action, knowledge and status, and have munication between individual or social units are mapped into
a low division of labor [15]. Computer-supported collabora- a communication matrix and then modeled as graphs that are
tive learning (CSCL) facilitates the interaction with software composed of nodes and edges. Nodes represent individuals
tools and increases potential for creative activities and social and edges the connections or communications between them.
interaction [30]. Collaborative learning is a commonly used These graphs can be used to visualize communication patterns
method in software engineering education as it prepares stu- in socio-technical systems systems. Additionally, in the graph
dents to work in software teams as independent experts. In theory there are different mathematical tools available, which
recent studies, it has been shown that students can be guided can be used for example to estimate the relative influence of
towards educational goals like collaboration by using gamifi- nodes in the graph or analyze the graph by the connection
cation [25], which is the application of game-like elements to patterns of the nodes [2, 4, 29].
non-game environments [13]. Currently many social network analysis -based studies rely
only on descriptive statistics and omit hypothesis formulation
and evaluation. We make a case that hypothesis testing is
an essential part of empirical studies that use quantitative
methods, such as SNA. Wohlin et al. [32, p. 12] summarize
the the importance of hypothesis testing for generating new
GHItaly18: 2nd Workshop on Games-Human Interaction, May 29th, 2018, Castiglione knowledge as follows. "In science, physical phenomena are
della Pescaia, Grosseto (Italy). addressed by putting forward hypotheses. The phenomenon is
Copyright © 2018 for the individual papers by the papers’ authors. Copying permitted
for private and academic purposes. This volume is published and copyrighted by its
observed and if the observations are in line with the hypothesis,
editors.
this becomes evidence for the hypothesis. Experiments are More recently SNA has been utilized by Wise & Cui [31],
important to test the hypothesis and in particular the predictive for example, to distinguish social relationships in MOOC dis-
ability of the hypothesis. If the new experiments support cussion forums. Their study showed connections between
the hypothesis, then we have more evidence in favor of the network structure and discussion practices. The study also
hypothesis." If the study remains at the level of descriptive pointed out that interactions involving course content related
statistics, no evidence is created to support the presented new discussion and other discussion should be examined sepa-
knowledge. 1 rately.
To illustrate the social network analysis as a research method SOCIAL NETWORK ANALYSIS AS A RESEARCH
and how it can be used in hypothesis testing, we present a study
METHOD
of how embedding technical elements into a collaborative
Social network analysis is based on graph theory, which is
learning course creates a new socio-technical system. The
a concept related to mathematics and information sciences
goal is to evaluate how the new elements affect the structure of
[18]. Central concepts in graph theory are network, node, and
collaboration in and between student teams. First, we present
edge. A network is composed of actors, or nodes, between
descriptive statistics generated by social network analysis and
which are connections, or edges. A common method in social
then test several hypotheses about the impact based on that
network analysis is to store graphs as adjacency matrices.
data. We conclude the paper with discussion of the study
From these matrices one can calculate different statistics and
results and what kind of insight social network analysis can
characteristics with graph theory and matrix algebra. Several
provide for other studies.
applications have been developed to automate the analysis,
such as UCINET [6] and Gephi [4]. A common method to
RELATED WORK ON ANALYSIS OF GAMIFIED COMMUNI- visualize the graph matrix is a sociogram, which is is a visual
representation of the nodes and and their edges. Most often
TIES
the nodes are represented as circles and edges as connecting
Social network analysis has been utilized by social scientists
lines.
to explain different phenomena. Borgatti et al. [7] divide
the various themes that can be examined by SNA into the Descriptive Statistics for Graphs and Nodes
similarities of actors, their social relations, interactions and The degree is most essential of node characteristics. The node
flows (transferring physical or intellectual property between degree indicates the number of edges connected to a node, or
actors). the number of connections the node has to other nodes. The
A number of authors have previously conducted research in weighted degree also takes into account the different values the
group and peer learning activities in software engineering ed- edges might have. Additionally, the most simple measure of
ucation using social network analysis [21]. This is because centrality, the degree centrality, is derived from node degrees.
the effects of social networking in an educational setting can Degree centrality is calculated from the sum of edges a node
have a substantial impact on learning performance [11]. For has.
example, the study by De Laat et al. [24] presents an overview Graph density is derived from node degrees. It is a value
of how social network analysis can be applied in the field between zero and one, and it indicates how many connections
of CSCL. The authors build a case for utilizing SNA to in- have been established of all possible connections. If all nodes
vestigate the group dynamics in a CSCL environment. The are connected to all other nodes, it is 1. If the graph has no
study emphasizes that in order to fully understand student par- edges, then network density is 0.
ticipation in CSCL environments, we must analyze who are
the actors in the collaborative learning task, and if actors are A social network can be composed of one or several compo-
participating actively or peripherally. It is also important to nents. A component is an element of a social network where a
understand how the different methods of participation change path can be traced from one node to another. The number of
over time. connected components can be evaluated how fragmented the
network is. In well-functioning socio-technical systems the
In terms of social network analysis in gamified CSCL context, desired number of connected components is just one.
de-Marcos et al. [12] used SNA to investigate a gamified e-
learning course. In the study different network metrics were Centrality is a method to measure nodes’ relative importance
assessed in how well they can predict academic performance. in a graph [29]. Application areas for centrality include finding
The investigation found potential in the use of network metrics influencers in social media or finding the most active collabo-
to predict learning achievement but also found limitations to rators in learning. Degree centrality is the easiest to calculate
their applicability beyond central nodes in the social network. and was covered earlier in this section. Betweenness centrality
This study is one of the few studies which have evaluated the is the measure of all the shortest paths that go through the node.
impact of gamification on socio-technical communities from a Other, more advanced methods such as eigenvector centrality
network analysis perspective. are beyond the scope of this article.
Regression Analysis and Comparisons in Social Network
1 The authors acknowledge that this is a very positivist view
and other philosophies of science or qualitative approaches are just Analysis
as valid. However, in positivist-quantitative approaches hypothesis Traditional hypothesis testing, such as regression analysis, is
testing is essential. difficult to use on networks, because edges exist as dyads
between two nodes. The Multiple Regression Quadratic As- Software Artifact
signment Procedure (MRQAP) method was developed to ad- The software artifact for computer-supported collaboration in
dress this issue and it allows analyzing the effect of external software engineering student teamwork, originally presented
variables on network structure [23]. Correlation between net- in [19], was created following along the following principles:
works can be analyzed with a related method, the Quadratic Increased team collaboration, extended collaborative commu-
Assignment Procedure (QAP) [22]. nication between students, and explicit goal communication
that supports shared goal setting and goal achievement.
CASE STUDY: ANALYZING THE IMPACT OF GAMIFICA- These tenets were realized as a collaboration platform that
TION ON STUDENT COLLABORATION concentrated around setting team goals and issues, viewing
In this section, we present a case study where we perform com- the status of other teams and using a chat-based tool, Slack,
parative social network analysis of two subsequent teamwork- to communicate and share information regarding the goals
based computer-supported collaborative courses in software and issues. The information and views concentrated around
engineering at master’s level. The research data has been three main views, with a sidebar and notifications showing
collected in earlier studies [19, 20]. The two courses were additional information when needed. The guidelines of re-
arranged in subsequent years with the main difference being sponsive design were followed in the implementation of the
that in the second year a gamified, computer-supported collab- views, with the system working equally well on desktops or
oration platform was introduced to the course. In this paper, mobile phones. The selected chat system also had a mobile
we take the analysis further with hypothesis testing, with an client application for iOS and Android, enabling continuous
emphasis on evaluating the impact of the artifact introduced collaborative communication for the community.
in the second study.
Social Network Analysis Results
In the first study [20], we observed 17 students over a five day We performed descriptive social network analysis with the
long intensive format and collaborative software engineering Gephi software [4] and hypothesis testing with the R statistical
course, arranged in the Code Camp format [27]. The course programming language [28]. First, we produced descriptive
had 10 hours of lecturing and 40 to 64 hours (depending on statistics for the two networks for comparison purposes, such
the student team) of collaborative teamwork around a set task. as the number of components in the network, various indica-
The topic for the course was to develop a new mobile or tablet tors for network size, and the average degree for the nodes.
application before the deadline on the fifth day. The teams Then we tested hypotheses of whether the distribution of node
were free to choose their own path to a solution within the attributes, such as degree and centrality, were different in the
theme and how to achieve it. The students had no other courses networks, and how well being in the same group predicts two
during the week. The students spent their time in the same students having a collaborative connection.
shared computer classroom, with each student team sitting at
their own table group. After the kickoff event and a technology Descriptive Statistics
tutorial, the teachers were available to advise when requested We analyzed the social networks of two studies, classroom and
during the rest of the week and to facilitate inter-team col- online communication, and produced descriptive statistics of
laboration. Most of the course participants were master’s the networks as presented in Table 1. The online system had a
level students with previous experience of other programming little more nodes and a lot more recorded interactions. Both
projects. All recorded collaborative communications occurred networks had only one connected component, which means
offline in the classroom in this study. that there was a path from each node to any other node. The
online network was a little less connected, which is signified
All student interactions that occurred in the classroom were by an increased network diameter and lower network density.
recorded during the first study. The video and audio recordings
were combined into multi-angle and surround sound videos Despite the lower network density, on average a node had more
that allowed the researchers reviewing the video to analyze connections to other nodes in the online network as signified by
several concurrent interactions. This resulted in 40 hours of the average degree. However, the average weighed degree was
video, from which 3366 interactions were coded for analysis. much higher in the classroom, implying that students in the
The data analysis presented in this paper is sourced from the online system were slightly better connected and classroom-
quantitative analysis results of the video recordings. based students established stronger and more even ties.
The second study [19], with the computer-supported collab- Hypothesis testing
orative learning system, was similar to the first one and also Descriptive statistics are not enough to evaluate proof of evi-
arranged in the Code Camp format [27]. The only differences dence when for example making claims about the differences
were the introduction of the computer-supported collabora- in social networks. For this, we need statistical hypothesis
tive learning system, the theme of the course, and the number testing which can give us the probability of the evidence sup-
of participants. This time there were 22 participants divided porting our hypothesis being significant, and the effect size of
into six teams. The topic of the course was using open data the evidence [9]. An obtained p-value represents the probabil-
for sustainability. In this study, all recorded collaborative ity of observing the current data due to chance when the null
communications occurred online in the computer-supported hypothesis is true. Effect size allows us to evaluate how sub-
collaboration system. The data analysis presented in this paper stantial the evaluated phenomenon is, after it has been shown
was sourced from the system log files. that the phenomenon probably exists.
Graph No. of Recorded No. of connected Network Average Average degree Network
nodes interactions components diameter degree (weighted) density
Classroom 18 703 1 3 9.39 374 0.55
Online, gamified 23 3216 1 4 10.5 54.5 0.46
Table 1. Descriptive social network analysis statistics
We first used the Mann–Whitney U test [32] to test the dif- Despite the medium effect size and statistical significance,
ference in distributions between the datasets because of its in both cases the R2 value is low. The same-team variable
suitability for non-normal data. A continuity correction was predicts only 11 to 12% of the variance, which means that a
enabled to compensate for non-continuous variables [5]. The social connection is mainly predicted by other factors.
Bonferroni correction was used to adjust the p-value to com-
pensate for the family-wise error rate in multiple comparisons Discussing the Analysis Results
[1]. We calculated effect size r using guidelines by Fritz et When evaluating the differences in the networks, we made
al. [16] for the Mann-Whitney U test. We evaluate the effect the following findings: The online network from the second
size as proposed by Cohen [9] that in r a large effect is .5, a study was slightly larger than the classroom collaboration net-
medium effect is .3, and a small effect is .1. work in the first study. The online network had vastly more
Our null hypotheses is that there is no difference in the distribu- interactions, but the nodes had stronger ties in the classroom
tions of node degrees or betweenness centralities between the network in a statistically significant manner. In both networks,
networks. Node degree and two node centrality variables were there was the same distribution of betweenness centralities,
tested to establish differences in the distribution of connections which means there were the same distribution of central nodes
and whether there was overall differences in how the relative mediating communication. According to the MRQAP results,
importance of nodes were distributed. The Mann-Whitney U being in the same team was a stronger predictor for collabo-
test results are summarized in Table 2, with the sum of ranks ration in the online network, which means that the classroom
denoted by the U-value, the probability by the p-value, and ef- network had stronger inter-team collaboration. A summary of
fect size by r. When evaluating node degrees and betweenness the tested hypotheses and outcomes are presented in Table 3.
centrality we fail to reject the null hypothesis, with there being At a first look the gamified online communication platform
no statistically significant difference between the networks appears to increase collaborative communication between stu-
in these aspects. When evaluating weighted degrees we can dents. The three essential goals of the platform were to in-
reject the null hypothesis and accept an alternative hypothe- crease communication, enable stronger inter-team communica-
sis that there is a difference in distributions between the two tion, and to reduce the hierarchy in communication. However,
networks in this aspect. after testing our hypotheses with social network analysis data,
Finally, we want to evaluate the impact of external factors we found that compared to plain classroom communication,
on the network. For this purpose we use MRQAP to apply the online platform failed to enable stronger social ties, in-
regression analysis to evaluate the effect of external variables creased inter-team communication, or more diverse patterns of
on dyads, or the connections between nodes. For the MRQAP collaboration. As the CSCL platform had an increased number
analysis we used the R sna library [8] and its network multiple of interactions, these outcomes might not have been discov-
regression function. ered without in-depth social network analysis and statistical
evaluation.
In our first regression analysis we evaluate whether being in the
same team predicts a connection in the classroom. The regres- These discoveries were enabled by statistical hypothesis test-
sion model using MRQAP and ordinal least squares regression ing, which allowed us to evaluate the strength of the evidence
statistically significantly predicts collaborative connections, F in regard to statistical significance and effect size. For ex-
(1, 304) = 38.38, p < .001, adj. R2 = .11. Regression coeffi- ample, the beneficial differences in betweenness centralities
cient estimate of 0.41 on the same team variable means that might seem plausible when presented side by side as descrip-
being in the same team increases the possibility of a connec- tive statistics. However, by statistically testing our hypotheses,
tion by 41%. Cohen’s f2 for effect size is 0.13, which means a we were able to conclude that the difference is not significant
medium effect [9]. and the effect size was trivial.
In the second regression analysis we evaluate whether being
in the same team predicts a connection in the online plat- DISCUSSION AND CONCLUSION
form. The regression model using MRQAP and ordinal least In this paper, we demonstrated how social network analysis
squares regression statistically significantly predicts collab- can be used in hypothesis testing -based evaluation of socio-
orative connections, F (1, 504) = 72.28, p < .001, adj. R2 technical communities. In the demonstration we evaluated
= .12. Regression coefficient estimate of 0.56 on the same the impact of a new software artifact. The artifact at a glance
team variable means that being in the same team increases the appeared to increase communication between participants of
possibility of a connection by 56%. Cohen’s f2 for effect size the community, but when evaluating social ties, the artifact’s
is 0.14 which means a medium effect [9]. impact was more complex than reflected by simple usage
statistics, and not fully beneficial.
Variable U- Mdn Mdn p-value Adjusted p (Bonferroni) / Effect size
value (Classroom) (Online) significance (r)
Degree 251.5 20.00 21.08 0.37 1 / no 0.13
(small)
Weighted degree 20 748.00 108.97 0.000000670.000002019891 / yes 0.72
(centrality) (large)
Betweenness 218 7.24 6.35 0.97 1 / no 0.007
centrality (none)
Table 2. Mann-Whitney U test results
Test Hypothesis Outcomes
Mann- The distribution of node attributes is different in the Node degrees and betweenness centralities have no
Whitney two datasets. significant differences. Weighed degrees have.
U
MRQAP Being in the same team is a statistically significant Statistically significant relationship exists between the
predictor for the strength of connection between the variables. In the online case the relationship is somewhat
nodes. stronger.
Table 3. Summary of tested hypotheses and outcomes
Social network analysis is a research method worth consid- REFERENCES
ering when studying the impact of technological artifacts on 1. Hervé Abdi. 2007. Bonferroni and Šidák corrections for
socio-technical communities, such as gamified systems. It multiple comparisons. Encyclopedia of measurement and
is a powerful technique, especially when combined with hy- statistics 3 (2007), 103–107.
pothesis testing. It can reveal patterns of actions in complex,
2. Ajith Abraham and Aboul Ella Hassanien. 2010.
interlinked systems that involve multiple interdependent ac-
Computational Social Network Analysis: Trends, Tools
tors. The number of these socio-technical systems will only
and Research Advances. Springer.
increase with the ongoing digital transformation in our society.
3. Manal M Alhammad and Ana M Moreno. 2018.
Some studies already use social network analysis to evaluate
Gamification in software engineering education: a
the impact of gamification on the educational communities,
systematic mapping. Journal of Systems and Software
such as the line of research by de-Marcos et al. [11, 12].
(2018).
However, in gamification, computer-supported collaboration,
and socio-technical research this is still in the minority [3, 14, 4. Mathieu Bastian, Sebastien Heymann, and Mathieu
21, 25]. Additionally, many social network analysis studies do Jacomy. 2009. Gephi: An open source software for
not use hypothesis testing or regression analysis and rely just exploring and manipulating networks. In International
on descriptive statistics. We recommend publishing tutorials, AAAI conference on weblogs and social media, Vol. 2.
reviews, and methodological demonstrations of the benefits
5. Reinhard Bergmann, John Ludbrook, and Will PJM
of more in-depth and rigorous research approaches in social
Spooren. 2000. Different outcomes of the
network analysis. Social network analysis has potential as a
Wilcoxon—Mann—Whitney test from different statistics
research method in the field of gamification and beyond. As
packages. The American Statistician 54, 1 (2000), 72–77.
future work, aspects of social network analysis not covered by
this paper, such as community detection and more advanced 6. Stephen P Borgatti, Martin G Everett, and Linton C
approaches to centrality, should be demonstrated in regard to Freeman. 2002. Ucinet for Windows: Software for social
compatibility with hypothesis testing. network analysis. (2002).
7. Stephen P Borgatti, Ajay Mehra, Daniel J Brass, and
ACKNOWLEDGMENTS
Giuseppe Labianca. 2009. Network analysis in the social
The work of the first author was supported by the Ulla Tuomi-
sciences. science 323, 5916 (2009), 892–895.
nen Foundation. This study was partially funded by the
European Union Regional Development Fund grant number 8. Carter T. Butts. 2016. sna: Tools for Social Network
A70554, “Kyberturvallisuusosaamisen ja liiketoiminnan kehit- Analysis. https://CRAN.R-project.org/package=sna R
täminen”, administrated by the council of Kymenlaakso. This package version 2.4.
work was supported, in part, by Science Foundation Ireland
9. Jacob Cohen. 1992. A power primer. Psychological
grant 13/RC/2094 and co-funded under the European Regional
bulletin 112, 1 (1992), 155.
Development Fund through the Southern & Eastern Regional
Operational Programme to Lero - the Irish Software Research 10. Hans De Bruijn and Paulien M Herder. 2009. System and
Centre (www.lero.ie). actor perspectives on sociotechnical systems. IEEE
Transactions on systems, man, and cybernetics-part A:
Systems and Humans 39, 5 (2009), 981–992.
11. Luis de Marcos, Eva Garcia-Lopez, and Antonio 21. Antti Knutas, Jouni Ikonen, and Jari Porras. 2015.
Garcia-Cabot. 2016a. On the effectiveness of game-like Computer-Supported Collaborative Learing in Software
and social approaches in learning: Comparing Engineering Education: A Systematic Mapping Study.
educational gaming, gamification & social networking. Journal on Information Technologies & Security 7 (2015),
Computers & Education 95 (2016), 99–113. 4.
12. Luis de Marcos, Eva García-López, Antonio 22. David Krackardt. 1987. QAP partialling as a test of
García-Cabot, José-Amelio Medina-Merodio, Adrián spuriousness. Social networks 9, 2 (1987), 171–186.
Domínguez, José-Javier Martínez-Herráiz, and Teresa 23. David Krackhardt. 1988. Predicting with networks:
Diez-Folledo. 2016b. Social network analysis of a Nonparametric multiple regression analysis of dyadic
gamified e-learning course: Small-world phenomenon data. Social networks 10, 4 (1988), 359–381.
and network metrics as predictors of academic
performance. Computers in Human Behavior 60 (2016), 24. Maarten de Laat, Vic Lally, Lasse Lipponen, and
312–321. Robert-Jan Simons. 2007. Investigating patterns of
interaction in networked learning and
13. Sebastian Deterding, Dan Dixon, Rilla Khaled, and computer-supported collaborative learning: A role for
Lennart Nacke. 2011. From Game Design Elements to Social Network Analysis. International Journal of
Gamefulness: Defining "Gamification". In Proceedings of Computer-Supported Collaborative Learning 2, 1 (March
the 15th International Academic MindTrek Conference: 2007), 87–103.
Envisioning Future Media Environments (MindTrek ’11).
ACM, New York, NY, USA, 9–15. 25. Fiona Fui-Hoon Nah, Qing Zeng, Venkata Rajasekhar
Telaprolu, Abhishek Padmanabhuni Ayyappa, and Brenda
14. Darina Dicheva, Christo Dichev, Gennady Agre, and Eschenbrenner. 2014. Gamification of education: a
Galia Angelova. 2015. Gamification in education: a review of literature. In International Conference on HCI
systematic mapping study. Journal of Educational in Business. Springer, 401–409.
Technology & Society 18, 3 (2015), 75.
26. Evelien Otte and Ronald Rousseau. 2002. Social network
15. Pierre Dillenbourg. 1999. What do you mean by analysis: a powerful strategy, also for the information
collaborative learning? Collaborative-learning: sciences. Journal of Information Science 28, 6 (Dec.
Cognitive and computational approaches 1 (1999), 1–15. 2002), 441–453.
16. Catherine O Fritz, Peter E Morris, and Jennifer J Richler. 27. J. Porras, K. Heikkinen, and J. Ikonen. 2007. Code camp:
2012. Effect size estimates: current use, calculations, and a setting for collaborative learning of programming. Adv.
interpretation. Journal of experimental psychology: Technol. Learn. 4, 1 (Jan. 2007), 43–52.
General 141, 1 (2012), 2.
28. R Core Team. 2017. R: A Language and Environment for
17. Frank W Geels. 2004. From sectoral systems of Statistical Computing. R Foundation for Statistical
innovation to socio-technical systems: Insights about Computing, Vienna, Austria. https://www.R-project.org/
dynamics and change from sociology and institutional
theory. Research policy 33, 6-7 (2004), 897–920. 29. John Scott. 2012. Social network analysis. SAGE
Publications Limited.
18. Frank Harary. 1969. Graph theory. Addison-Wesley,
Reading, MA. 30. Gerry Stahl, Timothy Koschmann, and Dan Suthers.
2006. Computer-supported collaborative learning: An
19. Antti Knutas. 2016. Increasing Beneficial Interactions in historical perspective. Cambridge handbook of the
a Computer-Supported Collaborative Environment. learning sciences 2006 (2006), 409–426.
Number 718 in Acta Universitatis Lappeenrantaensis.
Lappeenranta University of Technology. 31. Alyssa Friend Wise and Yi Cui. 2018. Learning
communities in the crowd: Characteristics of content
20. Antti Knutas, Jouni Ikonen, Dario Maggiorini, Laura related interactions and social relationships in MOOC
Ripamonti, and Jari Porras. 2016. Creating Student discussion forums. Computers & Education (2018).
Interaction Profiles for Adaptive Collaboration
Gamification Design. International Journal of Human 32. Claes Wohlin, Per Runeson, Martin Höst, Magnus C.
Capital and Information Technology Professionals Ohlsson, Björn Regnell, and Anders Wesslén. 2012.
(IJHCITP) 7, 3 (2016), 47–62. Experimentation in software engineering. Springer
Science & Business Media.