=Paper=
{{Paper
|id=Vol-2256/SWEPHD18_paper_04
|storemode=property
|title=Rubric for Measuring Indicators of Commitment in Computer-Supported Collaborative Student Teams
|pdfUrl=https://ceur-ws.org/Vol-2256/SWEPHD18_paper_04.pdf
|volume=Vol-2256
|authors=Antti Knutas,Jouni Ikonen
}}
==Rubric for Measuring Indicators of Commitment in Computer-Supported Collaborative Student Teams==
https://ceur-ws.org/Vol-2256/SWEPHD18_paper_04.pdf
Rubric for Measuring Indicators of Commitment in Computer-
Supported Collaborative Student Teams
Antti Knutas Jouni Ikonen
School of Engineering Science School of Engineering Science
LUT University LUT University
Lappeenranta, Finland Lappeenranta, Finland
antti.knutas@lut.fi jouni.ikonen@lut.fi
pages.
ABSTRACT
Student collaboration supported by online tools has been shown
to be beneficial in many contexts in computer science education. 1 INTRODUCTION
However, according to literature, little research has been devoted Collaborative learning, or cooperative activity of students
to individual analysis of factors that affect collaboration working together towards a specific learning goal with the
processes either negatively or positively. In this study, a teacher as a facilitator [3, 5, 10], has become an increasingly
grounded theory analysis was performed on three engineering important topic in education [13]. This collaborative approach to
education courses, investigating factors that affect the selection education has been shown to develop critical thinking, deepen
of collaboration tools and their use in student cooperation. The the level of understanding, and increase shared understanding of
presence of internal team motivation and commitment of team the material [8, 10]. Computer-supported collaborative learning
members was found to be an essential theme in relation to the (CSCL) facilitates this collaboration by using computer-mediated
success of online planning and collaboration. In this paper we communication tools to either enable new communication
present a rubric developed for measuring commitment to shared methods between students or to extend the range of
team goals in environments where team planning or interaction
communication beyond a single classroom [12, 14].
occurs through online collaborative tools. This metric, developed
The extension of collaboration with computer-supported
by generating an evaluation rubric from a grounded theory
collaborative learning allows increased knowledge building
analysis, enables the comparative analysis of different between a wider range of participants, more flexible teaching
collaborative approaches. We also discuss the relationship structures independent of place or time, better monitoring of
between the indicators and the collaborative outcomes in teams. student understanding by instructors, and improved student
productivity and satisfaction [14]. However, the nature of CSCL
CCS CONCEPTS has to be taken into account from the first planning stages when
• Human-centered computing~Collaborative and social designing courses and it has to be clearly explained to the
computing theory, concepts and paradigms students [21]. If not implemented properly, poorly designed
• Applied computing~Collaborative learning CSCL setup will be a drawback instead of a benefit.
While there has been extensive research on the benefits and
KEYWORDS drawbacks of collaborative learning approaches on higher
Collaborative learning, computer science education, computer education [2, 12, 14], there has been less research on evaluating
supported collaborative learning, metrics, computer-mediated how the individual aspects of teamwork affect collaborative
communication outcomes [14]. A link between student attitudes to teamwork,
team cohesion and collaborative learning outcomes has already
ACM Reference format: been established [16, 20]. However, we are interested if there are
Antti Knutas and Jouni Ikonen. 2018. Rubric for Measuring Indicators of more factors that affect student commitment to teamwork than
Commitment in Computer-Supported Collaborative Student Teams. In initial student attitudes. More specifically, we want to identify
Proceedings of the 2018 Workshop on PhD Software Engineering Education: and measure individual factors that affect team collaboration and
Challenges, Trends, and Programs (SWEPHD2018). St. Petersburg, Russia, 6
commitment to shared team goals.
Copyright © 2018 for the individual papers by the papers' authors. Copying Our research questions in this study are:
permitted for private and academic purposes. This volume is published and
copyrighted by its editors.
1. Which factors affect individual commitment to shared
The 2018 Workshop on PhD Software Engineering Education: Challenges, Trends, team goals and team collaboration?
and Programs, September 17th, 2018, St. Petersburg, Russia 2. How can these factors be expressed as a rubric for
comparing student team collaboration success?
In order to develop the metric, we studied three engineering
SWEPHD2018, September 17th, 2018, St. Petersburg, Russia A. Knutas et al.
courses, two of which were longer in duration (28 and 13 weeks) applied to analyzing student motivations in collaboration. Deci
and one was a weeklong intensive course. Two of the courses and Ryan present in their self-determination theory [4] that
involved a software project, one course arranged in Italy and the three intrinsic motivations for humans are autonomy,
second one in Finland. The third course, arranged in Finland, was competence and relatedness.
multidisciplinary with electrical engineers, mechanical
engineers, industrial management and business science students.
The main data source for the study were team interviews, which 3 RESEARCH METHODOLOGY
were analyzed using a limited version of the Grounded Theory We conducted the research by directly observing the courses and
(GT) [7] research methodology. Using the analysis results we then interviewing the students. The notes from observations and
created a rubric to evaluate and compare student team the interviews were coded and analyzed by using the Strauss-
commitment. Corbin version of the Grounded Theory methodology [17].
3.1 Overview of the Observed Courses
2 RELATED RESEARCH ON STUDENT
All of the three observed courses were teamwork-based courses,
COLLABORATION with an emphasis on independent teamwork, collaboration and
According to secondary studies computer-supported problem-based learning. The two longer courses were major
collaborative learning in general has been a topic of many events in their curriculum, or so called capstone courses [6].
studies when it comes to establishing its benefits in classroom Capstone courses are large problem- and teamwork -based
and educational settings [12, 14]. However, a study by Resta and courses that challenge students to work on problems and in
Lafarriére [14] points out that while in general the benefits of environments that are similar to their field of industry. The
CSCL in education has been established, the specific success students are also given multidisciplinary problems and skillsets
factors have not yet been explored in detail, or what exact and they are expected to cooperate on solving the assignments.
factors affect collaborative outcomes in CSCL. The study further While all the courses had some tutoring at the beginning, the
proposes that future research should concentrate less on students were expected to independently form their teams,
comparing computer-supported collaborative learning methods regulate the teamwork and solve the problems independently.
to other educational methods and instead future research should Although the three courses had the same kind of work and
begin to compare different computer-supported collaborative problem setup, they varied in topic and the required student
learning methods to each other. Furthermore, Gress et al. [9] skillsets. The list of course names, duration and theme are
write in their paper that many studies do not go into enough presented in the Table 1.
detail in analyzing collaboration variables in CSCL.
The effects and outcomes of collaborational group work have 3.2 Application of Grounded Theory
also been examined from an educational psychology perspective.
The interviews and other material gathered from the courses
In a study by Boekaerts & Minnaert [1] a correlation was found
were analyzed using the Grounded Theory [7] research
between student motivation for collaboration, competence level,
methodology by Strauss-Corbin [17], using additional guidelines
autonomy granted and social relatedness. Their research also
for computer science education by Kinnunen and Simon [11].
indicates that Deci & Ryan’s self-determination theory [4] can be
Grounded Theory is a method which has been said to be a well-
Table 1. Observed courses
Course ID; Course description Country;
name duration (ECTS); students
A: Melting A graduate-level multidisciplinary capstone project course, which allows students to Finland;
Pot Project work on an industry project, which is equivalent in challenge to their future tasks as 28 weeks (6-7 ECTS);
Course professionals. After attending the course the students are expected to be able to use 64 students
their learned knowledge to solve business challenges in cooperation with
professionals in other disciplines.
B: Online A graduate-level non-compulsory course where students learn the basic of designing Italy;
Game and managing multiplayer online games, from the initial idea to the final product. At 13 weeks (6 ECTS);
Design the end of the course students are supposed to demo a prototype of a game. After 14 students
attending the course, students are expected to use the achieved knowledge to design,
implement, and manage indie-level games on a number of platforms and technologies.
C: dotNET A short-term hands on course where students work together on their projects based Finland;
Code Camp on selected topic of the course. After the course students are expected to be able to 1 intensive week, 2 standard
use the achieved knowledge on the topic in their work and to implement other weeks (4 ECTS); 14 students
projects with selected platform and technology
Rubric for Measuring Indicators of Commitment in Computer-
WOODSTOCK’18, June, 2018, El Paso, Texas USA
Supported Collaborative Student Teams
suited analysis method for phenomena, which involve multiple categories of concepts affecting individual commitment in
human interaction factors, especially if the phenomenon is not collaboration, which concerned tools, success factors, preventing
well-known or strictly definable [17]. At the start of the study a issues and processes. In the following subsections we go into
non-committed literature review was performed, presented in further detail of how these were analyzed and how the
section two, for the purposes of theoretical sensitizing [18]. categories affect each other and individual commitment.
Theoretical sensitizing is a method for reviewing existing
literature to see what is considered a significant contribution to 4.1 Data Analysis
the field of science while not committing to follow any existing In open coding we analyzed sixteen group interviews with a
theory or framework [18]. A committed comparison to other total of 26 interviewees participating. We did not make a
theories is presented in section six. distinction between courses while discovering categories and
The aim of grounded theory is not only to describe a performing open coding in order to get a wide view of categories
phenomenon, but also to provide an explanation of relevant present in collaboration issues. Instead we built a table of teams,
conditions, how actors respond to the conditions and with the tools, issues each individual team faced and used
consequences of the actors’ actions [17]. Grounded Theory collaborative methods in order to perform a comparative
supports a wide variety of collection methods and the analysis of collaboration approaches in later research steps. The
methodology concentrates on analyzing the data. For data table was coded, and these results were used as additional
analysis it has a systematic set of procedures that support the material in constant comparison, refinement of categories and
development of theory that is inductively derived and discovery of casual relationship as a part of the axial coding
continuously tested against empirical data through constant phase.
comparison [17]. We applied the first two steps of Grounded The codification process resulted in 59 initial concepts in a
Theory for qualitative data analysis as summarized by Kinnunen total 201 quotations after finishing the open coding phase. The
and Simon [11] from Strauss-Corbin’s approach [17]. The concepts that were not relevant to the main categories were left
selective coding phase is omitted, because this research out from subsequent analysis. In the axial coding phases these
concentrates more on identifying the phenomenon, its factors, were further abstracted and condensed, resulting in four main
and causal conditions between phenomena instead of forming a categories. The discovered main categories are: Collaboration
full theory. This research approach of using a partial Strauss- tools, collaboration (success) factors, collaboration (preventing)
Corbin Grounded Theory process to analyze processes is also issues and collaboration processes.
further discussed by Rodon and Pastor [15]. In the second part of the Grounded Theory analysis process
The first step we took using the Grounded Theory analysis we used axial coding to discover aspects of collaboration present
was open coding, where data is broken down, given conceptual in the courses and to analyze their relationships. The results of
labels and compared with each other. The result is an initial view axial coding are presented in this section.
of the content of the data and an initial set of categories and The observed aspects of the collaboration were divided into
codes. The second step in analysis was axial coding, where four main categories, which are collaboration processes,
categories are developed further and causal conditions between collaboration (preventing) issues, positive collaboration factors and
categories are specified. Additionally, axial coding allows collaboration tools. The following subsections explain each
discovering context for the phenomenon and the actors. This category in detail and the most important codes in each section.
step resulted in refined categories, specified casual conditions
and dependencies. Additionally, we studied actor strategies and Collaborative tools. All students in the study used some
consequences for the strategies while constantly comparing and collaborative tools or method to organize. The main tools
grounding the analysis with the raw data. identified were project management, communication tools,
The same person who performed the interviews and meeting in person, repositories and document management
observations also did the Grounded Theory analysis in order to software. Several of these tools were evaluated based on
retain the richness of the data as much as possible, following the previous experience and convenience. These tools contributed to
best practices of Grounded Theory analysis [19]. The coding, information distribution, change management, goal tracking and
constant comparison and grounding processes were reviewed by contributed to effective communications, which was mentioned to
the research team at the end of each data collection and coding be a major factor in successful cooperation and cooperative work.
phase in order to avoid bias in the qualitative analysis and to
improve the depth of analysis. Collaboration factors. The second category that was
discovered related to positive factors that result from the use of
collaboration tools. Some of the benefits were simple, but they
4 RESEARCH FINDINGS had ripple effects that affected several aspects of cooperative
The main research approach in this study is using the Grounded work. The main benefits were effective communication that
Theory method [17] to find out the factors affecting student resulted from proper change management and the use of personal
commitment and collaboration processes in order to find or shared communication tools. These indirectly contributed to
indicators and construct metrics for them. We found four major goal assignment, goal tracking and the proper functioning of
SWEPHD2018, September 17th, 2018, St. Petersburg, Russia A. Knutas et al.
cooperative work. The collaboration tools also allowed the team The first indicator is cooperative goal setting processes, which is
to benefit from external support, increasing motivation and crucial in longer-term teamwork. It describes both the team’s
individual competence in some occasions. A major factor that decision of what the overall goals are, and how the tasks based
also affected team’s collaboration was shared goals, which were on these goals are divided among the team members. It is crucial
attributed to cooperative goal setting in collaborative processes to motivation that the students perceive this process to be fair
and were often indirectly related to efficient communications. and that they feel that they have been able to affect the direction
of teamwork. This furthers individual ownership of team goals,
Collaboration issues. Goal achievement was the major issue in because it allows the team members to feel that they have
several of the teams. The problem is basic, that students did not participated in setting the goals. A process that allows the
achieve their goals, which can cause frustration because of the students to solve disputes also furthers individual commitment
mismatch between shared goals and achieved goals. The lack of to goals.
goal achievement was attributed to lack of experience, lack of The second indicator is goal achievement and tracking and it is
commitment to shared goals which caused a drop in a team related to effective communications. It measures how well the
member’s motivation to work and a mismatch between the team follows who has achieved their goals and whether the team
team’s task schedule and the actual time it took to achieve the balances workloads. Goal tracking allows the individuals to
goals. In cases where goal tracking did not function well and the relate their individual progress to shared team progress and see
status of the team was not communicated effectively, the how their efforts promote the advancement of the shared, overall
mismatch led frustration and a loss of commitment. goal.
The third indicator is effective communication, which is
Collaboration processes. Task assignment was an aspect of the another important aspect for group cohesion. Communication is
collaborative process that all teams did to some extent. Some had not only important for organizing teamwork, but also
clearly defined leadership that assigned tasks to individuals and maintaining social cohesion. Teams are always social units to
others relied on individual initiative, where team members took some extent and if individuals feel that other team members are
ownership of tasks based on their own decisions. Most teams passive, there is a possibility that they feel being passive is
were a combination of this, where the teams had regular acceptable for them as well. This means student teams with slow
meetings either online or online where they decided on shared or erratic communication can start to drift apart both in social
goals and at the same time discussed the task assignment. Goal cohesion and goal direction.
tracking is something that was essential to task assignment and The fourth indicator is the level of collaboration. Collaboration
completing shared goals. It was also done less systematically in this context means mutual support towards shared learning
than task assignment in many of the groups. Almost none of the goals instead of just cooperating to achieve individual student
groups combined goal tracking to effective scheduling, where the goals. It is another important aspect of teamwork and mutual
task progression would be systematically compared against the support, and collaboration is what separates a group of
deadlines set by the course. individuals from a learning, working team. Good collaboration
and mutual support can also increase individual motivation.
Using these indicators and the categories found in the
5 A METRIC FOR MEASURING INDICATORS grounded theory analysis we defined a rubric for evaluating the
OF STUDENT COMMITMENT level of each indicator. The rubric variables are presented in the
Grounded Theory analysis enables the describing of phenomena, Table 2. Each variable is evaluated using the following scale:
describing actor strategies and identifying factors affecting those Does not meet expectations (0), meets expectations (1) and
strategies, but at its core it is a qualitative data analysis method exceeds expectations (2). This means that the minimum amount
and does not allow building metrics. Because of this, we chose a of points awarded from each category (marked with an alphabet
mixed method approach, where we identify and describe the and in bold text in the table) is zero and maximum eight.
phenomena using the first two steps of Grounded Theory. After Maximum amount of points awarded from the rubric is 32. The
the initial analysis we build an evaluation rubric using the most rubric is printed out in full in the Online Appendix1.
important codes identified in the analysis. This allows building
an evaluation metric for comparing different collaboration
situations, enables more lightweight analysis in future case
studies, and allows comparison of team outcomes between case
studies. In the next subsection we describe four variables, built
on the GT categories that are related to team commitment
according to our research, and how these individual indicators
relate to commitment.
4.1 Defining the Indicators and Rubric
1 http://doi.org/10.5281/zenodo.546087
Rubric for Measuring Indicators of Commitment in Computer-
WOODSTOCK’18, June, 2018, El Paso, Texas USA
Supported Collaborative Student Teams
regulation. Effective communication (C) is also connected to
successful team regulation and the intrinsic motivation of
relatedness. The level of collaboration (D) is more complex to
Table 2. A list of variables used in the rubric for evaluating relate, because it is an overall indicator of a complex
indicators of student commitment, sorted by indicators
phenomenon. However, according to the study by Boekaerts &
Minnaert [1], successful collaboration relates to competence
A: B: Goal C: Effective D: Level of
level, autonomy and social relatedness. Essentially good
Cooperative achievement communication collaboration
goal setting and tracking collaboration requires mutual support towards learning goals
processes and communication [5].
The presented metric extends measuring student commitment
1. There is a 1. Team 1. Team has 1. Team beyond direct inquiry about student motivation. It does so by
formal members default members
know what communication work towards using several indicators of commitment that were detected in the
process for
goal setting tasks are channels shared goals qualitative study of the three courses. The metric uses an
2. The being worked 2. Team members 2. Team observation-based approach and qualitative observations as a
process is on know how to members data source. The metric can be used to find issues in an
perceived to 2. Team reach all other work
members can members in a cooperatively
individual team’s work or used as an average measure to
be fair
request timely manner towards goals evaluate different versions of course arrangements.
3. There is a
discussion of support from 3. Communication 3. Team The main limitation of the metric is that it requires a qualified
shared goals each other is efficient and recognizes observer and a detailed analysis of the data. While the metric
before 3. Proactive supports team learning goals
and the indicators can be expressed in a relatively simple
deciding on stance on activities as valid goals
goal tracking 4. Communicating 4. Team manner, it requires a wealth of background material to produce.
the overall
team task 4. Process to status and issues members The second limitation is the scope of testing. While the metric is
4. Ability to track and re- has no single collaborate based on a wide study from three courses in two universities, it
allocate point of failure (support
solve still requires a lot of further testing and comparisons to existing
disputes workload in and takes no extra others in
case of issues effort achieving metrics for validation. This testing and evaluation of how widely
learning applicable the indicators are, is a critical direction for future
goals) research.
REFERENCES
5 DISCUSSION AND CONCLUSION
In this study, we identified several factors that affect individual [1] Boekaerts, M. and Minnaert, A. 2006. Affective and motivational outcomes
of working in collaborative groups. Educational Psychology. 26, 2 (2006),
commitment to team goals and collaboration and present a 187–208.
metric for measuring them. In the process of creating the metric [2] Bratitsis, T. and Demetriadis, S. 2013. Research Approaches in Computer-
Supported Collaborative Learning. Int. J. e-Collab. 9, 1 (Jan. 2013), 1–8.
we found several identifiers that are connected to individual DOI:https://doi.org/10.4018/jec.2013010101.
commitment in addition to initial motivation. These are goal [3] Bruffee, K.A. 1995. Sharing Our Toys: Cooperative Learning Versus
setting processes, goal tracking, effective communication and Collaborative Learning. Change: The Magazine of Higher Learning. 27, 1
(1995), 12–18. DOI:https://doi.org/10.1080/00091383.1995.9937722.
level of collaboration. They do not have direct causality with [4] Deci, E.L. and Ryan, R.M. 1985. Intrinsic motivation and self-determination
individual commitment but are indirect indicators of it. For in human behavior. Plenum, New York.
[5] Dillenbourg, P. 1999. What do you mean by collaborative learning?
example, having a successful cooperative goal setting process Collaborative-learning: Cognitive and computational approaches. (1999), 1–
requires a certain level of organization and effort from the team. 19.
Serrano-Camara et al. [16] discuss the several types of [6] Dunlap, J.C. 2005. Problem-based learning and self-efficacy: How a
capstone course prepares students for a profession. Educational Technology
motivation in learning from intrinsic motivations of Deci & Research and Development. 53, 1 (Mar. 2005), 65–83.
Ryan’s self-determination theory [4] to external motivation like DOI:https://doi.org/10.1007/BF02504858.
[7] Glaser, B. and Strauss, A.L. 1967. The discovery of grounded theory:
rewards and regulation, and state that fostering intrinsic Strategies for qualitative research. Aldine.
motivation is essential in collaborative learning environments. In [8] Gokhale, A.A. 1995. Collaborative Learning Enhances Critical Thinking.
Journal of Technology Education. 7, 1 (1995).
their study and their review of the literature they establish a link [9] Gress, C.L.Z., Fior, M., Hadwin, A.F. and Winne, P.H. 2010. Measurement
between intrinsic motivation and positive consequences. When and assessment in computer-supported collaborative learning. Computers
comparing their research [16], theories on motivation [1, 4] and in Human Behavior. 26, 5 (Sep. 2010), 806–814.
DOI:https://doi.org/10.1016/j.chb.2007.05.012.
the presented metric, similarities can be found between aspects [10] Johnson, D.W. and Johnson, R.T. 1999. Making cooperative learning work.
of teamwork in the metric and factors that promote intrinsic Theory Into Practice. 38, 2 (1999), 67–73.
DOI:https://doi.org/10.1080/00405849909543834.
motivation or team regulation. Cooperative goal setting [11] Kinnunen, P. and Simon, B. 2010. Building Theory About Computing
processes (A) are related to the intrinsic motivation of Education Phenomena: A Discussion of Grounded Theory. Proceedings of
autonomy. Goal achievement and tracking (B) are related to both the 10th Koli Calling International Conference on Computing Education
Research (New York, NY, USA, 2010), 37–42.
intrinsic motivation of competence and successful team
SWEPHD2018, September 17th, 2018, St. Petersburg, Russia A. Knutas et al.
[12] Kirschner, P.A. 2001. Using integrated electronic environments for
collaborative teaching/learning. Learning and Instruction. 10, (2001), 1–9.
[13] Okamoto, T. 2004. Collaborative technology and new e-pedagogy. IEEE
International Conference on Advanced Learning Technologies, 2004.
Proceedings (Sep. 2004), 1046–1047.
[14] Resta, P. and Laferrière, T. 2007. Technology in support of collaborative
learning. Educational Psychology Review. 19, 1 (2007), 65–83.
[15] Rodon, J. and Pastor, J.A. 2007. Applying grounded theory to study the
implementation of an inter-organizational information system. Electronic
Journal of Business Research Methods. 5, 2 (2007), 71–82.
[16] Serrano-Cámara, L.M., Paredes-Velasco, M., Alcover, C.-M. and Velazquez-
Iturbide, J.Á. 2014. An evaluation of students’ motivation in computer-
supported collaborative learning of programming concepts. Computers in
Human Behavior. 31, (Feb. 2014), 499–508.
DOI:https://doi.org/10.1016/j.chb.2013.04.030.
[17] Strauss, A. and Corbin, J.M. 1990. Basics of qualitative research: Grounded
theory procedures and techniques. Sage Publications, Inc.
[18] Urquhart, C. and Fernandez, W. 2006. Grounded theory method: the
researcher as blank slate and other myths. ICIS 2006 proceedings. (2006), 31.
[19] Urquhart, C., Lehmann, H. and Myers, M.D. 2010. Putting the ‘theory’back
into grounded theory: guidelines for grounded theory studies in
information systems. Information systems journal. 20, 4 (2010), 357–381.
[20] Williams, E.A., Duray, R. and Reddy, V. 2006. Teamwork Orientation,
Group Cohesiveness, and Student Learning: A Study of the Use of Teams
in Online Distance Education. Journal of Management Education. 30, 4
(Aug. 2006), 592–616. DOI:https://doi.org/10.1177/1052562905276740.
[21] Williams, S. and Roberts, T.S. 2002. Computer-supported collaborative
learning: strengths and weaknesses. Computers in Education, 2002.
Proceedings. International Conference on (2002), 328–331.