Student collaboration supported by online tools has been shown to be beneficial in many contexts in computer science education. However, according to literature, litle research has been devoted to individual analysis of factors that afect collaboration processes either negatively or positively. In this study, a grounded theory analysis was performed on three engineering education courses, investigating factors that afect the selection of collaboration tools and their use in student cooperation. The presence of internal team motivation and commitment of team members was found to be an essential theme in relation to the success of online planning and collaboration. In this paper we present a rubric developed for measuring commitment to shared team goals in environments where team planning or interaction occurs through online collaborative tools. This metric, developed by generating an evaluation rubric from a grounded theory analysis, enables the comparative analysis of diferent collaborative approaches. We also discuss the relationship between the indicators and the collaborative outcomes in teams.
• Human-centered computing~Collaborative and social computing theory, concepts and paradigms • Applied computing~Collaborative learning Collaborative learning, computer science education, computer supported collaborative learning, metrics, computer-mediated communication ACM Reference format:
Collaborative learning, or cooperative activity of students working together towards a specific learning goal with the teacher as a facilitator [3, 5, 10], has become an increasingly important topic in education [13]. iThs collaborative approach to education has been shown to develop critical thinking, deepen the level of understanding, and increase shared understanding of the material [8, 10]. Computer-supported collaborative learning (CSCL) facilitates this collaboration by using computer-mediated communication tools to either enable new communication methods between students or to extend the range of communication beyond a single classroom [12, 14].
hTe extension of collabora tion with computer-supported collaborative learning allows increased knowledge building between a wider range of participants, more eflxible teaching structures independent of place or time, beetr monitoring of student understanding by instructors, and improved student productivity and satisfaction [14]. However, the nature of CSCL has to be taken into account from the first planning stages when designing courses and it has to be clearly explained to the students [21]. If not implemented properly, poorly designed CSCL setup will be a drawback instead of a benetfi.
While there has been extensive research on the benefits and drawbacks of collaborative learning approaches on higher education [2, 12, 14], there has been less research on evaluating how the individual aspects of teamwork aefct col laborative outcomes [14]. A link between student aittudes t o teamwork, team cohesion and collaborative learning outcomes has already been established [16, 20]. However, we are interested if there are more factors that afect student commitment to teamwork than initial student aittudes. More specifically, we want to identify and measure individual factors that afect team collaboration and commitment to shared team goals.
Our research questions in this study are: 1. Which factors afect individual commitment to shar ed team goals and team collaboration? 2. How can these factors be expressed as a rubric for comparing student team collaboration success? In order to develop the metric, we studied three engineering courses, two of which were longer in duration (28 and 13 weeks) and one was a weeklong intensive course. Two of the courses involved a software project , one course arranged in Italy and the second one in Finland. The third course, arranged in Finland, was multidisciplinary with electrical engineers, mechanical engineers, industrial management and business science students. hTe main data source for the st udy were team interviews, which were analyzed using a limited version of the Grounded Theory (GT) [7] research methodology. Using the analysis results we created a rubric to evaluate and compare student team commitment.
According to secondary studies computer-supported collaborative learning in general has been a topic of many studies when it comes to establishing its benetfis in classroom and educational seitngs [12, 14]. However, a study by Resta and Lafarriére [14] points out that while in general the benefits of CSCL in education has been established, the specific success factors have not yet been explored in detail, or what exact factors afect collaborative outcomes i n CSCL. The study further proposes that future research should concentrate less on comparing computer-supported collaborative learning methods to other educational methods and instead future research should begin to compare diefrent computer -supported collaborative learning methods to each other. Furthermore, Gress et al. [9] write in their paper that many studies do not go into enough detail in analyzing collaboration variables in CSCL.
hTe efects and outcomes of collaborational group work have also been examined from an educational psychology perspective. In a study by Boekaerts & Minnaert [1] a correlation was found between student motivation for collaboration, competence level, autonomy granted and social relatedness. Their research also indicates that Deci & Ryan’s self-determination theory [4] can be applied to analyzing student motivations in collaboration. Deci and Ryan present in their self-determination theory [4] that three intrinsic motivations for humans are autonomy, competence and relatedness.
We conducted the research by directly observing the courses and then interviewing the students. The notes from observations and the interviews were coded and analyzed by using the StraussCorbin version of the Grounded Theory methodology [17]. 3.1
All of the three observed courses were teamwork-based courses, with an emphasis on independent teamwork, collaboration and problem-based learning. The two longer courses were major events in their curriculum, or so called capstone courses [6]. Capstone courses are large problem- and teamwork -based courses that challenge students to work on problems and in environments that are similar to their field of industry. The students are also given multidisciplinary problems and skillsets and they are expected to cooperate on solving the assignments. While all the courses had some tutoring at the beginning, the students were expected to independently form their teams, regulate the teamwork and solve the problems independently. Although the three courses had the same kind of work and problem setup, they varied in topic and the required student skillsets. The list of course names, duration and theme are presented in the Table 1. 3.2
hTe interviews and other material gathered fr om the courses were analyzed using the Grounded Theory [7] research methodology by Strauss-Corbin [17], using additional guidelines for computer science education by Kinnunen and Simon [11].
Grounded Theory is a method which has been said to be a wellA graduate-level multidisciplinary capstone project course, which allows students to work on an industry project, which is equivalent in challenge to their future tasks as professionals. After attending the course the students are expected to be able to use their learned knowledge to solve business challenges in cooperation with professionals in other disciplines.
A graduate-level non-compulsory course where students learn the basic of designing and managing multiplayer online games, from the initial idea to the final product. At the end of the course students are supposed to demo a prototype of a game. After attending the course, students are expected to use the achieved knowledge to design, implement, and manage indie-level games on a number of platforms and technologies. A short-term hands on course where students work together on their projects based on selected topic of the course. After the course students are expected to be able to use the achieved knowledge on the topic in their work and to implement other projects with selected platform and technology Country; duration (ECTS); students Finland; 28 weeks (6-7 ECTS); 64 students
suited analysis method for phenomena, which involve multiple human interaction factors, especially if the phenomenon is not well-known or strictly definable [17]. At the start of the study a non-commited literature review was performed, presented in section two, for the purposes of theoretical sensitizing [18]. hTeoretical sensitizing is a me thod for reviewing existing literature to see what is considered a significant contribution to the field of science while not commiting to follow any existing theory or framework [18]. A commited comparison to other theories is presented in section six.
hTe aim of grounded theory is not only to describe a phenomenon, but also to provide an explanation of relevant conditions, how actors respond to the conditions and consequences of the actors’ actions [17]. Grounded Theory supports a wide variety of collection methods and the methodology concentrates on analyzing the data. For data analysis it has a systematic set of procedures that support the development of theory that is inductively derived and continuously tested against empirical data through constant comparison [17]. We applied the first two steps of Grounded hTeory for qualitative data analysis as summarized by Kinnunen and Simon [11] from Strauss-Corbin’s approach [17]. eTh selective coding phase is omited, because this research concentrates more on identifying the phenomenon, its factors, and causal conditions between phenomena instead of forming a full theory. iThs research approach of using a partial Strauss Corbin Grounded Theory process to analyz e processes is also further discussed by Rodon and Pastor [15].
hTe first step we took using the Grounded Theory analysis was open coding, where data is broken down, given conceptual labels and compared with each other. The result is an initial view of the content of the data and an initial set of categories and codes. The second step in analysis was axial coding, where categories are developed further and causal conditions between categories are speciefid . Additionally, axial coding allows discovering context for the phenomenon and the actors. This step resulted in refined categories, specified casual conditions and dependencies. Additionally, we studied actor strategies and consequences for the strategies while constantly comparing and grounding the analysis with the raw data.
hTe same person who performed the interviews and observations also did the Grounded eThory analysis in order to retain the richness of the data as much as possible, following the best practices of Grounded Th eory analysis [19]. eTh coding, constant comparison and grounding processes were reviewed by the research team at the end of each data collection and coding phase in order to avoid bias in the qualitative analysis and to improve the depth of analysis.
The main research approach in this study is using the Grounded Theory method [17] to find out the factors affecting student commitment and collaboration processes in order to find indicators and construct metrics for them. We found four major categories of concepts affecting individual commitment in collaboration, which concerned tools, success factors, preventing issues and processes. In the following subsections we go into further detail of how these were analyzed and how the categories affect each other and individual commitment. 4.1
In open coding we analyzed sixteen group interviews with a total of 26 interviewees participating. We did not make a distinction between courses while discovering categories and performing open coding in order to get a wide view of categories present in collaboration issues. Instead we built a table of teams, with the tools, issues each individual team faced and used collaborative methods in order to perform a comparative analysis of collaboration approaches in later research steps. The table was coded, and these results were used as additional material in constant comparison, refine ment of categories and discovery of casual relationship as a part of the axial coding phase.
hTe codification process resulted in 59 initial concepts in a total 201 quotations after finishing the open coding phase. The concepts that were not relevant to the main categories were left out from subsequent analysis. In the axial coding phases these were further abstracted and condensed, resulting in four main categories. eTh discovered main categories are : Collaboration tools, collaboration (success) factors, collaboration (preventing) issues and collaboration processes.
In the second part of the Grounded Th eory analysis process we used axial coding to discover aspects of collaboration present in the courses and to analyze their relationships. The results of axial coding are presented in this section.
hTe observed aspects of the collaboration were divided into four main categories, which are collaboration processes, collaboration (preventing) issues, positive collaboration factors and collaboration tools. The following subsections explain each category in detail and the most important codes in each section. Collaborative tools. All students in the study used some collaborative tools or method to organize. The main tools identified were project management, communication tools, meeting in person, repositories and document management software. Several of these tools were evaluated based on previous experience and convenience. These tools contributed to information distribution, change management, goal tracking and contributed to effective communications, which was mentioned to be a major factor in successful cooperation and cooperative work. Collaboration factors. The second category that was discovered related to positive factors that result from the use of collaboration tools. Some of the benefits were simple, but they had ripple effects that affected several aspects of cooperative work. The main benefits were effective communication that resulted from proper change management and the use of personal or shared communication tools. These indirectly contributed to goal assignment, goal tracking and the proper functioning of cooperative work. The collaboration tools also allowed the team to benefit from external support, increasing motivation and individual competence in some occasions. A major factor that also affected team’s collaboration was shared goals, which were attributed to cooperative goal setting in collaborative processes and were often indirectly related to efficient communications. Collaboration issues. Goal achievement was the major issue in several of the teams. The problem is basic, that students did not achieve their goals, which can cause frustration because of the mismatch between shared goals and achieved goals. The lack of goal achievement was attributed to lack of experience, lack of commitment to shared goals which caused a drop in a team member’s motivation to work and a mismatch between the team’s task schedule and the actual time it took to achieve the goals. In cases where goal tracking did not function well and the status of the team was not communicated effectively, the mismatch led frustration and a loss of commitment.
Collaboration processes. Task assignment was an aspect of the collaborative process that all teams did to some extent. Some had clearly defined leadership that assigned tasks to individuals and others relied on individual initiative, where team members took ownership of tasks based on their own decisions. Most teams were a combination of this, where the teams had regular meetings either online or online where they decided on shared goals and at the same time discussed the task assignment. Goal tracking is something that was essential to task assignment and completing shared goals. It was also done less systematically than task assignment in many of the groups. Almost none of the groups combined goal tracking to effective scheduling, where the task progression would be systematically compared against the deadlines set by the course.
Grounded eThory analysis enables the describing of phenomena, describing actor strategies and identifying factors afecting those strategies, but at its core it is a qualitative data analysis method and does not allow building metrics. Because of this, we chose a mixed method approach, where we identify and describe the phenomena using the rfist two steps of Grounded Theory . After the initial analysis we build an evaluation rubric using the most important codes identified in the analysis. This allows building an evaluation metric for comparing diefrent collaboration situations, enables more lightweight analysis in future case studies, and allows comparison of team outcomes between case studies. In the next subsection we describe four variables, built on the GT categories that are related to team commitment according to our research, and how these individual indicators relate to commitment. 4.1
hTe rfist indicator is cooperative goal seitng processes, which is crucial in longer-term teamwork. It describes both the team’s decision of what the overall goals are, and how the tasks based on these goals are divided among the team members. It is crucial to motivation that the students perceive this process to be fair and that they feel that they have been able to aefct the direction of teamwork. This furthers individual ownership of team goals, because it allows the team members to feel that they have participated in seitng the goals. A process that allows the students to solve disputes also furthers individual commitment to goals.
hTe second indicator is goal achievement and tracking and it is related to eefctive communications . It measures how well the team follows who has achieved their goals and whether the team balances workloads. Goal tracking allows the individuals to relate their individual progress to shared team progress and see how their eforts promote the advancement of the shared, overall goal.
hTe third indicator is efective communication , which is another important aspect for group cohesion. Communication is not only important for organizing teamwork, but also maintaining social cohesion. Teams are always social units to some extent and if individuals feel that other team members are passive, there is a possibility that they feel being passive is acceptable for them as well. iThs means student teams with slow or erratic communication can start to drift apart both in social cohesion and goal direction.
hTe fourth indicator is the level of collaboration. Collaboration in this context means mutual support towards shared learning goals instead of just cooperating to achieve individual student goals. It is another important aspect of teamwork and mutual support, and collaboration is what separates a group of individuals from a learning, working team. Good collaboration and mutual support can also increase individual motivation.
Using these indicators and the categories found in the grounded theory analysis we defined a rubric for evaluating the level of each indicator. The rubric variables are presented in the Table 2. Each variable is evaluated using the following scale: Does not meet expectations (0), meets expectations (1) and exceeds expectations (2). This means that the minimum amount of points awarded from each category (marked with an alphabet and in bold text in the table) is zero and maximum eight. Maximum amount of points awarded from the rubric is 32. The rubric is printed out in full in the Online Appendix1. Rubric for Measuring Indicators of Commitment in ComputerSupported Collaborative Student Teams indicators of student commitment, sorted by indicators
In this study, we identiefid several factors that aefct individual commitment to team goals and collaboration and present a metric for measuring them. In the process of creating the metric we found several identifiers that are connected to individual commitment in addition to initial motivation. These are goal seting processes, goal tracking, efective communication and level of collaboration. eThy do not have dir ect causality with individual commitment but are indirect indicators of it. For example, having a successful cooperative goal seting process requires a certain level of organization and efort from the team.
Serrano-Camara et al. [16] discuss the several types of motivation in learning from intrinsic motivations of Deci & Ryan’s self-determination theory [4] to external motivation like rewards and regulation, and state that fostering intrinsic motivation is essential in collaborative learning environments. In their study and their review of the literature they establish a link between intrinsic motivation and positive consequences. When comparing their research [16], theories on motivation [1, 4] and the presented metric, similarities can be found between aspects of teamwork in the metric and factors that promote intrinsic motivation or team regulation.
Cooperative goal seitng processes (A) are related to the intrinsic motivation of autonomy. Goal achievement and tracking (B) are related to both intrinsic motivation of competence regulation. Efective communication (C) is also connected to successful team regulation and the intrinsic motivation of relatedness. The
level of collaboration (D) is more complex to relate, because it is an overall indicator of a complex phenomenon. However, according to the study by Boekaerts & Minnaert [1], successful collaboration relates to competence level, autonomy and social relatedness. Essentially good collaboration requires mutual support towards learning goals and communication [5].
hT
e presented metric extends measuring student commitment beyond direct inquiry about student motivation. It does so by using several indicators of commitment that were detected in the qualitative study of the three courses. eTh metric uses an observation-based approach and qualitative observations as a data source. The metric can be used to find issues in an individual team’s work or used as an average measure to evaluate diefrent versions of course arrangements.
hTe main limitation of the metric is that it requires a qualiefid observer and a detailed analysis of the data. While the metric and the indicators can be expressed in a relatively simple manner, it requires a wealth of background material to produce. hTe second limitation is the scope of tes ting. While the metric is based on a wide study from three courses in two universities, it still requires a lot of further testing and comparisons to existing metrics for validation. This testing and evaluation of how widely applicable the indicators are, is a critical direction for future research. DOI:https://doi.org/10.4018/jec.2013010101.
Our Toys: Cooperative Learning Versus Collaborative Learning. Change: The Magazine of Higher Learning. 27, 1
Deci, E.L. and Ryan, R.M. 1985. Intrinsic motivation and self-determination in human behavior. Plenum, New York.
Dillenbourg, P. 1999. What do you mean by collaborative learning? Collaborative-learning: Cognitive and computational approaches. (1999), 1– 19.
Dunlap, J.C. 2005. Problem-based learning and self-efficacy: How a capstone course prepares students for a profession. Educational Technology Research and
Development.
53, 1 (Mar.
2005), 65–83.
DOI:https://doi.org/10.1007/BF02504858.
Glaser, B. and
Strauss, A.L. 1967. The discovery of grounded theory: Strategies for qualitative research. Aldine.
Gokhale, A.A. 1995. Collaborative Learning Enhances Critical Thinking. Journal of Technology Education. 7, 1 (1995). in
Human
Behavior.
26, DOI:https://doi.org/10.1016/j.chb.2007.05.012.
Theory
Into
Practice.
DOI:https://doi.org/10.1080/00405849909543834.
5 38, Johnson, D.W. and Johnson, R.T. 1999. Making cooperative learning work. (Sep.
2010),
806–814. 2 (1999), 67–73.
Kinnunen, P. and Simon, B. 2010. Building Theory About Computing Education Phenomena: A Discussion of Grounded Theory. Proceedings of the 10th Koli Calling International Conference on Computing Education Research (New York, NY, USA, 2010), 37–42. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]