In recent years, computer science education teachers needed to incorporate challenge-based and teambased collaboration projects to enhance learning and prepare students for their future careers. The successful use of team-based work structures depends on the ability of team members to work together with each team developing its own pattern of work. Using afinity propagation as an algorithm, we study the patterns of collaborative behavior from Trello data obtained from 16 teams collaborating on business cases. All the teams were given the same instructions and were asked to use Trello to organize and monitor tasks. Actions of the group members in Trello were categorized as diferent contribution types, namely activities involving planning, coordination, further input, deletion, or updates on tasks. Sequences of those actions were first created for each group and later used to explore the diferences in the working process between the groups. We analyze data and interpret patterns of collaborative work during the entire collaboration on the project, as well as patterns of work that emerged in the first 50 actions in teamwork, since literature indicates that the first actions in teamwork are important for the creation of a “team memory”. We present both the initiating sequences along with entire sequences for all teams, and some first results for the diferent collaboration patterns we observed. This study is the very first exploratory research covering collaboration patterns with Trello data. This kind of pattern analysis could provide teachers with a means to identify teams that need further support in their teamwork. Our data suggest that future studies could complement the analysis with data also coming from other channels used by students for communication and organization of teamwork triangulating data with them.
Collaborative learning is considered superior to traditional learning, as it not only promotes
student engagement and learning, it also advances the development of teamwork skills, such
as communication and collaboration, both considered important for a successful transition to
the workplace [
There are disciplinary diferences in the perception of collaborative skills between engineering
and business students, and these diferences require diferent approaches in the development
of a curriculum [
Despite the recognized benefits of team-based and cross-functional projects, the dificulty of
designing and implementing teamwork, especially in computer science education, significantly
slows down the difusion of team-based education practices [
Moreover, during the first few trials of a redesigned course, teachers often experience a high
workload, since the design of the activity still has to be optimized, and teachers are insecure
about meeting the expected learning outcomes. Transition from one to another instructional
method requires adjustment on the part of the student as well as the instructor and first trials
may, therefore, not always meet expectations, so several design iterations may be needed [
Teams can be defined as workgroups created and structured to achieve a common goal
[
Challenge-based education provides a context for problem-solving, and (self-regulated) teamwork by providing student teams with authentic problems that are introduced by stakeholders from industry [17]. In challenge-based learning, researchers and teachers also participate as mentors. They provide feedback about the knowledge formation process and monitor the development of skills and competences [18]. In such a context, the instructors constantly monitor the level of knowledge the learners assimilate during the study process and at the same time facilitate efective collaboration with the stakeholders [ 19]. Recent research findings indicate that monitoring multiple groups at the same time and making sure that teams engage in productive interaction is dificult without dedicated training in technological support systems [20, 21]. For teachers, it is important to continuously adapt methods of teaching/training as well as content and tasks to continuously optimize the course and facilitate high levels of learning in student teams. When more information is available about how efective teams collaborate, teachers and curriculum developers are better equipped to make the required changes.
Use of large-scale data generated in educational context is more and more common, and
the field of educational data mining is on rise [ 22, 23, 24]. The new frontier is finding suitable
methods to analyze student data, especially in the case of collaborative learning among students
to better understand students’ problems in education [
The contribution of this paper is two-fold: first, we propose a new approach to analyze collaborative activity using Trello log files and data to detect collaborative patterns, as a tool for teachers to identify dysfunctional teams, and second, we provide initial results, identifying the patterns we observed in a small Trello data set. The dataset is rather limited in size, so the authors want to stress the importance of further studies to complement and enrich the identification of patterns.
Following the principles of challenge-based education, in this paper we analyze data coming from an Innovation and Entrepreneurship (I&E) course where students work on a case provided by a company. They trace their project work and advances on Trello boards. The case may be related to considering alternative business models or go-to-market scenarios in relation with the innovation or entrepreneurial contexts, fed by exploration in some specific areas: business environment, competition, suppliers, partners, environmental, sustainability issues, etc. Prior to solving the case, students acquire concepts and tools pertaining to the assessment of the impact of technology on industry, market and/or organization, and business research. This course is a perfect playground to develop the skills necessary to work in international teams located in diferent countries, a requirement for modern jobs nowadays [26, 27].
Trello (https://trello.com/) is a web-based project management and collaboration tool. Trello is a commercial product owned by Atlassian company, operating on a web application, MacOS, Windows OS, iOS 12+, and Android 5.1 (source: About Trello. (2021). Trello. https://help.trello.com/category/ 697-category). Trello ofers three options, namely Free, Business Class, and Enterprise. The latter two options are paid. In this study, the Trello Free option was used because most of the necessary features for project work are already available in this option. The tool can be described as a sort of shared virtual pinboard where users can post, assign, and modify tasks. A Trello board can be opened for each project; users can create lists (basically a set of columns) on the board that can be used to organize sub-tasks. These sub-tasks are represented by cards, that can be moved around the board. The cards resemble to-do items or little tasks assigned to a team member. It is also possible to add due date information. Figure 1 shows an example of a Trello board. You can distinguish the diferent tasks on the board. It also clearly indicates, who is assigned to a task and what the progress is on the task. Trello visualizes the goals and tasks within the project, finished tasks can be checked of, allowing for easy progress checking, while also keeping team members accountable for the assigned tasks.
The data for the diferent student teams working on business cases are presented in Table 1, while an example of a board is presented in Figure 1.
Trello allows for easy export of the data from the Trello boards. Teams can create their own Trello board and organize it in a way that fits both the team and the project. Sixteen teams participated in the course described in this paper. At the beginning of the course, each team was asked to create their own Trello board and invite the teachers and business-case representatives to their Trello board. The Trello boards provided a shared space where updates could easily be traced, followed and monitored by all involved stakeholders, team members, teachers, and case representatives.
Trello ofers the opportunity to collect logs of all the actions performed on the board (i.e., who created a “card,” at what time the “card” was moved, etc.). Trello data can be exported in JSON format (a lightweight data-interchange format that is easy to parse and extract data from) which will be further analyzed. Each action performed on the board is an “action” inside the JSON object with a timestamp, member id of the participant who took the action, and further details about the action (if a task was moved between two lists, then the “action” element will contain data identifying the original and destination lists of the moved task, if a due date was assigned, if the task was assigned to a member, etc.).
The first and most important step was data preprocessing. For each team, we downloaded the sequence of actions performed on the board and assembled the actions into sequences based on their temporal order. The longest sequence accounted for 752 actions, and the shortest one contained 84 actions in total. We identified the diferent actions that students performed in Trello, like “creation of task”, “move task between lists”, “assign due date”, “assign the task to member” etc., and then put each action in order of time sequence and tried to characterize diferent types of teams and identify their distinctive patterns of work on tasks. We identified more than 25 diferent Trello actions on the board, some examples include: ’Update card’, ’Add a member to card’, ’Create card’, etc. We coded the 25 Trello actions into the following 5 main categories, activities involving planning, coordination of tasks, activities that require further input for a task, deletion of a task, and update of a task. The coding schemes for Trello actions are provided in Table 2. Using this coding scheme, we analyzed the data from 16 Trello boards and we converted each action into a value from 0 to 4, each number representing a distinct activity, and then we composed the final matrix (data set) that we used for the clustering algorithms (next section).
The Trello JSON is structured in a way that one can query the information through diferent lenses, and obtain, for instance, all the information about all the ’cards’, through the dedicates ’cards’ array of elements, ’members’ array, where one can find all the information about all the members, or for instance ’actions’ array where one can find more information about every individual action performed on the Trello board. This latest array is the information we used to construct the sequences, each ’action’ element inside the ’actions’ array is composed of an id field of the action, id of the member creator, date when performed, and type of action, and additional data. The ’type of action’ fields correspond to those presented as Trello actions in Table 3, that we later coded into categories.
Create card Copy card
Create list
Add member to card Remove member from card
Add member to board
Make admin of board Unconfirmed board invitation
Delete card Remove check list from card Delete attachment from card
Delete custom field
Comment card Add attachment to card
Add checklist to card Copy comment card
Update custom card Update custom field to card
Update list Update board
Update card
Update check list Update check item state on card
Enale plugin Disable plugin
This approach allowed us to collect “in situ” data that provides an objective account of student activity over time [28] and is considered superior to self-report or questionnaire data. Machine learning algorithms are used to identify patterns in team collaboration.
Next, we categorize the sequences into diferent clusters, where each cluster is composed of similar group activity patterns. The sequences are for team, and not individual participation to the group work, because there wasn’t any requirement for equal participation in the group work by the diferent team members and the students were given the freedom to self organize in the ways that worked best for them.
For the clustering we used two diferent algorithms, namely DBSCAN and Afinity
propagation. We made this decision based on previous literature [
We decided to use relative cluster validation, which evaluates the clustering structure by varying parameter values for the same algorithm, the clusters remained the same until they reached really high values of eps (0.9) when they started taking diferent shape. We have a really small sample to perform any internal cluster validation approaches, and potential external cluster validation can be performed once other and bigger datasets are available in the literature. Still, the novelty of our work is in the proposition of the approach for analysis of Trello data, it is up to further studies to confirm, and as our intuition hints extend the types of patterns observed.
The diferent sequences and how they fit into the clusters are presented in Figure 2. This figure presents all 16 sequences and their fit with the clusters as result of the afinity propagation algorithm. Each sequence is presented in terms of the actions the team performed namely, coordination actions are presented in yellow update actions in red, specifying further input in green, planning actions in blue and delete actions in grey.
Cluster A: 2 teams Cluster B: 1 team Cluster C: 6 teams Cluster D: 3 teams Cluster E: 4 teams Legend
Coordination actions Update actions Specifying further input Planning actions
Delete actions
The patterns show that that teams were primarily clustered on the total number of distinctive actions that they performed, and in the distribution of diferent types of actions they performed on their boards. In cluster A, for instance, we see teams that predominantly performed update actions, and these update actions happened one after the other in the sequence order (Cluster A). The sequence with 752 actions presents its own cluster (Cluster B). Cluster E is composed, on average, of long sequences, containing diverse types of actions and not primarily only of update activities or planning activities, while cluster D is composed of small sequences with diverse actions. It seems that, for instance, in the sequences in cluster B teams use update actions and they frequently perform actions on the board, while cluster A contains teams that do not perform actions too frequently and when they do, they use update actions (and not other types of actions), while cluster E contains teams that perform diferent activities across the spectrum and again they do not perform these actions too frequently.
The clustering of the sequences based on the first 50 actions resulted in four diferent clusters, presented in Figure 3. Cluster A denotes the teams with long internal sequences of planning actions (marked in blue). Cluster B contains sequences where teams performed predominantly update actions (marked in red). In cluster C, there are the teams with mixed actions, while cluster D contains teams that had longer sub-sequences of actions of a single type (first only planning for instance, then followed by a long sequence of update actions, then followed by coordination actions).
In this paper, we presented the results from an exploratory study on a challenge-based course of student teamwork. While tracing their project work in Trello, we presented a new approach to detect patterns of student behaviour exhibited during their team work.
This kind of analysis can help teachers to further understand diferent patterns of team collaboration that develop in the process of project work and provide adequate support to teams that need help. We believe that using “in situ” data, to understand the dynamics of collaboration in teams over time is superior to asking participants for their opinions, as the student activity is measured objectively without the potential bias that questionnaires might bring. Our approach has the potential to advance team literature in general as well as provide input for education curriculum development.
This kind of data analysis derived from teamwork could be especially useful in the context of Massive Open Online courses (MOOCs), where teachers must support large number of teams. The type of analysis we propose in this paper could help instructors detect groups that have problems in their teamwork and groups that need further instructor attention. Additionally, onsite courses with large numbers of student teams and limited teaching capacity could benefit from such a data-driven large scale collection of data. Teaching staf could use the data to identify teams that present poor patterns and need further attention, and thus optimize their teaching presence.
Previous research shows that the whole project collaboration process impacts group performance [29]. Understanding more about group process patterns during the whole collaboration project could ofer useful insights into the potential group performance by the team. These variations in the whole project collaboration, therefore set a conceptual framework for further research on how the collaboration patterns relate to group performance.
Generally, the first 50 team actions can provide insights on the tendency of the whole project collaboration. If the team in the first 50 actions focuses mostly on updating actions (Cluster B) or mixed actions (Cluster C) at the beginning, the team presents a risk of ending up their whole project collaboration with few actions. The teams that have 50 first actions mainly on planning would have the highest potential to keep their work pattern for the whole project.
We feel that it is premature to discuss student team work archetypes for both of these results, and further research is needed to draw strong conclusions on diferent styles. Our results are aimed at ofering an initial idea of the possible conclusions one can draw from analysis such as ours.
The research has several practical implications. Firstly, the findings of the whole project collaboration pattern complements prior research on online group patterns. While most of the prior studies focused on the communication aspect and social interaction of group work dynamics [30, 31, 32, 33], this research focused on the group patterns related to task orientation, particularly challenge-based learning. Moreover, these actions are studied in sequence, inline with research that suggest that not specific actions sort efects, but sets of actions [ 34]. To our knowledge this is one of the first exploitative study in suggesting the tendency of collaboration patterns based on Trello data.
Second, previous literature suggested that the initial actions of teams would shape the development of the whole project collaboration. Particularly, student teams focusing on update actions or mixed types of actions at the beginning phase would likely have a whole project collaboration pattern with a small number of actions. The teams that have first actions mostly on a particular action type with long sequences would have a large total number of actions. Hence, our findings suggest a basis for teachers to identify the teams with a small or large number of actions as those that will require additional support and interventions to guide teams to desired whole project collaboration patterns.
One of the limitations of our paper is that the is the small number of teams and participants. While the results clearly indicate that teams followed diferent approaches in the coordination of their project, due to the small number of groups, patterns cannot be linked to the success of the teams. In future research with more data points, we hope to identify patterns that contribute to successful teamwork. Due to the COVID-19 pandemic, students needed to coordinate and communicate virtually even more frequently than normal, although through diferent channels, primarily via WhatsApp and Google Meet. The use of other communication channels might explain why some teams were less active on Trello, they may have communicated through alternative channels. Due to this, we believe that a future study should complement data coming from Trello as we did, and also analyze qualitatively why the observed patterns appeared in the first place. The discovered patterns gave a first indication of the level of activity within the teams and the type of actions that were performed in Trello. The colored overview allows for a quick inspection of the intensity and type of activity. In future these types of overviews could be implemented dashboards that increase the instructors’ understanding of the team process at a glance [35].
This is a pilot study, with practical applications and the meaning of the team patterns must still be better understood. Future pilots are planned and foreseen with diferent challenge-based courses.
The research aims to propose a new way to analyze teamwork collaboration of student teams in a challenge-based course using the project management tool, Trello. The study has indicated some emerging clusters of whole project collaboration with the first 50 actions in teams and suggested the respective whole project collaboration development tendency.
This study complements prior research on online group patterns and sets the initial exploratory ground for studying collaboration patterns with Trello data. Our initial results suggest that students teams experience diferences in the use of Trello, with distinguishable patterns. In future it can contribute practically to the teachers’ understanding of how to identify teams in need of further support, to guide teams to favorable project collaboration.
The authors would like to acknowledge and thank all the students and the business case providers for their participation in the course and thus made this work possible. In addition, we want to thank Manuel Dalcastagnè for the help provided help provided during the analysis of the data. [14] E. Wenger, Communities of practice and social learning systems: the career of a concept, in: Social learning systems and communities of practice, Springer, 2010, pp. 179–198. [15] A. Meier, H. Spada, N. Rummel, A rating scheme for assessing the quality of computersupported collaboration processes, International Journal of Computer-Supported Collaborative Learning 2 (2007) 63–86. [16] J. Morrison, Team memory: Information management for business teams, in: [1993] Proceedings of the Twenty-sixth Hawaii International Conference on System Sciences, volume 4, IEEE, 1993, pp. 122–131. [17] G. Pisoni, J. Segovia, M. Stoycheva, M. Marchese, Distributed student team work in challenge-based innovation and entrepreneurship (i&e) course, in: International Symposium on Emerging Technologies for Education, Springer, 2019, pp. 155–163. [18] L. Félix-Herrán, A. E. Rendon-Nava, J. M. N. Jalil, Challenge-based learning: an i-semester for experiential learning in mechatronics engineering, International Journal on Interactive Design and Manufacturing (IJIDeM) 13 (2019) 1367–1383. [19] K. Kohn Rådberg, U. Lundqvist, J. Malmqvist, O. Hagvall Svensson, From cdio to challengebased learning experiences–expanding student learning as well as societal impact?, European Journal of Engineering Education 45 (2020) 22–37. [20] A. van Leeuwen, J. Janssen, A systematic review of teacher guidance during collaborative learning in primary and secondary education, Educational Research Review 27 (2019) 71–89. [21] G. Pisoni, H. Gijlers, A pilot study to inform the design of a supportive environment for challenge-based collaboration, in: International Conference in Methodologies and intelligent Systems for Techhnology Enhanced Learning, Springer, 2020, pp. 216–225. [22] C. Romero, S. Ventura, M. Pechenizkiy, R. S. Baker, Handbook of educational data mining,
CRC press, 2010. [23] K. Koedinger, K. Cunningham, A. Skogsholm, B. Leber, An open repository and analysis tools for fine-grained, longitudinal learner data, in: Educational Data Mining 2008, 2008. [24] R. S. Baker, K. Yacef, The state of educational data mining in 2009: A review and future visions, JEDM| Journal of Educational Data Mining 1 (2009) 3–17. [25] S. Strauß, N. Rummel, F. Stoyanova, N. Krämer, Developing a library of typical problems during collaborative learning in online courses, International Society of the Learning Sciences, Inc.[ISLS]., 2018. [26] S. C. Kong, T.-W. Chan, P. Grifin, U. Hoppe, R. Huang, Kinshuk, C. K. Looi, M. Milrad, C. Norris, M. Nussbaum, et al., E-learning in school education in the coming 10 years for developing 21st century skills: Critical research issues and policy implications, Journal of Educational Technology & Society 17 (2014) 70–78. [27] G. Pisoni, Collaborative learning in a shared course between two universities, in: Proceedings of the 31st Australian Conference on Human-Computer-Interaction, 2019, pp. 534–537. [28] P. Guo, N. Saab, L. S. Post, W. Admiraal, A review of project-based learning in higher education: Student outcomes and measures, International Journal of Educational Research 102 (2020) 101586. [29] J. M. Assbeihat, The impact of collaboration among members on team’s performance,
Management and Administrative Science Review 5 (2016) 248–259. [30] K. Mattingly, A.-L. Ponsonby, A consideration of group work processes in modern epidemiology, Annals of epidemiology 24 (2014) 319–323. [31] M. Oren, S. Gilbert, Convocons: a tool for building afinity among distributed team members, in: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, 2012, pp. 1213–1222. [32] S. McComb, D. Kennedy, R. Perryman, N. Warner, M. Letsky, Temporal patterns of mental model convergence: Implications for distributed teams interacting in electronic collaboration spaces, Human Factors 52 (2010) 264–281. [33] I. Oliveira, L. Tinoca, A. Pereira, Online group work patterns: How to promote a successful collaboration, Computers & Education 57 (2011) 1348–1357. [34] E. H. Eshuis, J. ter Vrugte, A. Anjewierden, L. Bollen, J. Sikken, T. de Jong, Improving the quality of vocational students’ collaboration and knowledge acquisition through instruction and joint reflection, International journal of computer-supported collaborative learning 14 (2019) 53–76. [35] A. van Leeuwen, N. Rummel, Comparing teachers’ use of mirroring and advising dashboards, in: Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, 2020, pp. 26–34.