Proposal of a system of indicators to assess teamwork using log-based learning analytics

Abstract

One of the main lines of research in the field of learning analytics focuses on the identification of adequate data extracted from Learning Management Systems' (LMS) databases that may help predicting different behaviors and outcomes, such as academic achievement or student performance. Prior research has investigated these indicators at an individual level. The situation is more complex in collaborative settings involving teamwork, such as project-based learning, where student assessment considers the group as a single entity, disregard of individual contributions to the team. Furthermore, most often only the final deliverable is taken into account when assessing teamwork in collaborative learning, leading to a loss of perspective about the whole process and whether or not teamwork is effectively happening. This research aims to provide a comprehensive selection of log-based information from LMS databases that could serve as potential indicators to perform learning analytics and assess teamwork in online learning. The proposal of this novel and theory-grounded framework understands the multidimensional nature of teamwork, considers different sets of indicators for each of its dimensions-communication, cooperation, coordination, and monitoring and tracking-and incorporates the temporal dimension of activity data. This proposal sets the basis for future software development to effectively transform LMS log-based data and provide actionable measures of teamwork using learning analytics.

Keywords

• Learning Analytics
• Teamwork
• Learning Management Systems
• Logs

-

Online collaboration is at the heart of new networked organizations. Job posts evolve toward pervasive connectivity embedded in networked organizations that replace traditional hierarchical structures [ 1,2 ]. The origin of this change in organizational models has three main causes: the relation between collaborative work and productivity [ 3,4 ], flexibility and teamwork structures that maximize the talent of employees.

Copyright © 2017 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.

Organizational initiatives to improve employees’ teamwork skills include hiring of experts to manage and coordinate teamwork building projects [ 5 ]. However, in many occasions these initiatives do not achieve the expected outcomes due to contextual, individual or motivational factors [ 6,7 ]. Therefore, there is a great interest in investigating the factors that are characteristic of high-performance teams [ 8 ].

The demand for professional workers who have developed their teamwork skills also affects educational institutions [ 9 ]. The number of graduates joining the workforce is a proxy measure of an institution’s success, which has led to the integration of teamwork training in most undergraduate and graduate courses [ 10 ], primarily through the implementation of project-based learning approaches. Higher education is also incorporating the use of Information Technologies such as Learning Management Systems (LMS) as tools to support learning in general, and group activities in particular, allowing for anywhere-anytime, synchronous and asynchronous participation. LMS keep track of the contributions from each student–or group/team member–, providing evidences for effective assessment, as well as information about the learning process, helping instructors to effectively monitor and support students.

Following similar approaches from prior literature [ 11 ], this conceptual research aims to develop a learning analytics-based classification and to propose a system of indicators that uses student activity traces in the LMS to provide instructors and teachers with the most relevant information about student teamwork assessment in LMS.

This system may be used to facilitate decision-making about course instruction and student and group assessment in project-based learning. In order to do so, the study proposes a framework of learning interactions for the analysis of teamwork in virtual learning environments. . 2

Conceptualization of teamwork

This study defines teamwork as a collaborative process, details its characteristics, and builds on Fuks et al’s 3C collaboration model [ 12 ], which considers communication, cooperation and coordination as the main components of collaboration, and further expands it with a fourth dimension: tracking/monitoring. 2.1

Definition

Teamwork refers to a behavioral pattern between two or more individuals [ 13,14 ] who interact dynamically [ 15 ], establish a regular and constant negotiation to reach agreements [ 16 ] through knowledge exchanges and problem solving [ 17 ], while keeping a steady pace and coordinating efforts [ 18 ] in order to achieve their shared goals [ 19 ].

Teamwork is observable when a task is being performed [ 16,20,21 ], and thus presents a behavioral pattern that can be recognized through observation and is different from other group actions [ 22 ]. Teamwork is also stable, extending to other tasks and contexts [ 23 ], even though group member changes may alter the level of success of outcomes [ 24 ]. Finally, and most important, teamwork is a cause and predictor of outcomes, given an established behavioral pattern or interaction model [ 19,22,25,26 ]. 2.2

Group teamwork and individual teamwork

Because this study focuses on observable aspects of teamwork, it is necessary to limit the concept and highlight the differences between group teamwork and the individual work of team members. When observed closely, both an individual task and teamwork assignment share some inputs–goal definition and available resources–, process–task execution– and outputs–delivered result or outcome [ 27 ]. Individual and team work even share the most basic sub-tasks, to the extent that some assignments that are carried out in teams could easily be done by a single individual.

The main difference between individual and team work is that, when working in teams, both the final goal and intermediate goals and objectives are shared among the team members. In other words, their actions are interrelated. Thus, the interdependence among group members is one of the key elements that make individual and team work different [ 28 ].

This interdependence translates to interactions between members. Member interactions are an essential element of teamwork and lead to observable behaviors because, in order to achieve the shared goals, the team has to show cooperation and social skills that are not necessary to perform an individual task [ 14,29,30 ].

In sum, teamwork is a multidimensional and interdependent concept. In the context of this research, teamwork corresponds to the regular communication between two or more people who coordinate their effort in a period of time during which they cooperate sharing ideas, knowledge and information, paying attention to the progress of the different tasks in order to achieve a common goal. Therefore, teamwork is the result of different behaviors: communication, coordination, cooperation and monitoring. These behaviors are complementary, combine with one another, are observable, include recurring activities, and are developed by every team member. 2.3

Dimensions of teamwork

Teamwork is a complex and dynamic concept. The multidimensional and interdependent nature of teamwork implies that the observation of one single dimension or behavior does not determine that teamwork is happening; on the contrary, all the different behaviors have to be observed in order to confirm that teamwork exists. Nonetheless, this does not mean that all the different behaviors occur with the same frequency, intensity or duration.

The first observable dimension when a teamwork task begins is communication, or interaction between all team members. This interaction is present during the whole process, even though with different levels of intensity, and in LMS it becomes manifest in the form of message exchanges [ 12 ] in message boards and chats, with immediate, lengthy and timely replies [ 31 ] between all participant members.

As long as all team members are participating, communication happens in a closed loop between emitter and receivers [ 31,32 ], confirming that if teamwork is happening, high levels of interaction must occur [ 12 ]. High levels of interaction are also related to cooperation, as they drive an increasing number of contributions of members to the common task [ 14,31 ] in shared workspaces, such as a wiki, glossary or workshop in LMS. Together with the dimensions of communication and cooperation, teamwork involves coordination. Coordination becomes manifest when every team member communicate with each other and cooperate in a synchronous way.

Finally, for teamwork to develop effectively, there must be some kind of control of performed and pending tasks. This supervision includes monitoring and tracking activities [ 33 ], and constitutes the fourth dimension of teamwork. Monitoring and tracking activities must be defined from the beginning of the activity, and one of their defining characteristics is their regularity or consistency [ 12,34 ]. Monitoring and tracking requires a timely reading of other members’ contributions [ 35 ]. Tracking represents assessment of performed and remaining tasks, as well as of time and resource availability, and it reflects effort and commitment to the team [ 33 ].

In sum, an adequate conceptualization of teamwork must include the following four dimensions: 1) communication, defined in terms of message exchanges with a structure of reply and confirmation–active listening–; 2) cooperation, as long as team members share and exchange information in order to complete a shared goal; 3) coordination that translates in synchronicity and constant pace during the execution of the different tasks; and 4) monitoring and tracking, by adequately keeping up to date record of communications and interactions. 2.4

Temporal dimension

Each and every interaction in LMS generates a record or digital footprint that links the data to a unique identifier: the user–in this case, a student. Every session login, message, query, update or click in the LMS has a corresponding record in the database.

Besides the association of each interaction with the user, the fact that in LMS all records have a timestamp, opens a door to the analysis of time-related factors for the application of learning analytics [ 36 ]. Teamwork is a temporal series of behaviors [ 37 ] that allows to assess learning progress [ 38,39 ]. Additionally, and given the iterative nature of teamwork behaviors [ 38,39 ], it is necessary to analyze the recurrence of these behaviors during the execution of the common task. 3

Interactions in LMS and teamwork dimensions

The previous section has already highlighted the relevance of interactions. Broadly speaking, an interaction refers to an action that two objects, people or agents execute reciprocally. When a team of people work together toward a common goal, interactions refer to every interpersonal observable behavior aiming to synchronize and coordinate resources and tasks in order to achieve the goal in a limited time [ 40 ], during a period covering two arbitrary time points [ 41,42 ].

In the context of LMS, interactions are the basic unit of data in learning analytics [ 11 ], as traces of user activity in LMS are stored in real time as records in the system database. According to this, collaborative activities–e.g. teamwork–in LMS comprise groups of records that represent the reciprocal actions–interactions–between team members in a given period of time.

While there is no existing classification of collaborative interactions in LMS–the only available classifications offer a study at the individual level [ 11,43 ]–, prior research has investigated teamwork interactions in face-to-face learning, using observation through recording and analysis of conversations. The classifications resulting from these analyses include meta-analysis of team efficacy [ 44 ], necessary behaviors for teamwork in higher education [ 45 ], or organizational approaches focusing on team structures as predictors of success in project management [ 19,22,25,26,46 ]. The main conclusion of these studies is that effective teams show higher frequency and consistency in their interactions during the execution of the task, with higher levels of interaction during the first days of activity and in the days before a deliverable is due.

Addressing teamwork from a group perspective requires the unit of analysis to be comprised of groups of interactions that include at least an individual–but not isolated– interaction of each member, while considering also time and space concurrence [ 47 ]. However, it is not enough to group interactions as an aggregate–i.e. the aggregation of individual contributions and results–; that is, team interactions will be defined by the group of interactions of all team members that happen in a temporal span [ 41,48 ].

The following subsections will detail the proposal of interaction indicators for the different dimensions of teamwork. 3.1

Communication (Cm)

In online contexts, communication between team members involves message and information exchanges [ 12,49 ]. Therefore, database records informing about message creation, publication or updating in message boards or chats are potential candidates to measure communication.

Regarding message exchanges, prior research has noted the positive relation between message length [ 50 ] or number of messages exchanged in a period of time [ 51,52 ] and student outcomes. These records can be considered as part of the communication when there is reciprocity [ 53,54 ]; that is, when there is a quick and timely reply from other team members. From an individual perspective, reciprocity can be measured by the average in-reply and out-reply time [ 19 ].

Communication has to be persistent: each team member has to show constant implication in the interactions [ 54 ]. The levels of persistence can be measured as the ratio between actual potential activity time intervals–e.g. real time used and total time available [ 19 ].

From the above, our proposal includes the following observable indicators of communication, at both individual and group levels: • Individual: ─ Individual message exchanges [ 12,49 ]: Ratio between number of messages sent and total team messages. ─ Individual message length [ 50 ]: Average length of sent messages. ─ Individual frequency of messages [ 51,52 ]: Number of messages sent by the individual by each task-dependent time unit. ─ Individual out-reciprocity [ 15,54 ]: Temporal distance of an individual replies’ to other team members. ─ Individual in-reciprocity [ 19,35,50 ]: Temporal distance of other members’ replies to messages sent by an individual. ─ Individual consistency [ 51,52 ]: Time interval between messages sent and interval variability. • Group: ─ Team message exchanges [ 55 ]: Total number of messages exchanged within the team. ─ Team message length [ 50 ]: Average length of team messages and length variability among team members. ─ Team message frequency [ 56 ]: Number of team messages by time unit (hour, day, week, month): ─ Team reciprocity [ 19,35,50 ]: Balance between promptness and timeliness between messages sent and replies received among team members. ─ Team consistency [ 51,52 ]: Time interval between team messages and interval variability. 3.2

Cooperation (Cp)

Cooperation involves sharing knowledge and, the same as communication, is positively related to learning outcomes [ 57 ]. Cooperation in LMS is observable through the collection and combination of all members’ contributions to a shared workspace; in other words, it reflects the team workload by showing alternating contributions of the different group members [ 12,14 ]. Production, in terms of knowledge interactions between members in a shared workspace, can also be seen as a form of cooperation [ 35 ]. Therefore, cooperation includes data related to contributions in a shared workspace–such as a wiki, workshop or glossary in Moodle–, comprising both original contributions and content updates.

Cooperation means working together, and it is characterized by: its presence during the execution of the whole task–consistency [ 58 ]–, how early the members start contributing [ 17,18 ], timely contributions [59] and on-time delivery of outcomes. Execution time makes the relation between delays and poor time management evident, both at an individual and group level [60-62].

In summary, data related to cooperation in an LMS are those that have been generated through interaction in the shared workspaces that provide support to the creation of the deliverables, including the time occurred between contributions, and taking into account start and end dates.

From the above, our proposal includes the following indicators of cooperation: • Individual: ─ Individual contributions [ 12,14 ]: Number of individual contributions. ─ Combination of individual contributions [ 35 ]: Ratio between individual workload and team workload. ─ Individual consistency [ 58 ]: Distribution of contributions in the time available to do the task. ─ Individual earliness [ 17,18 ]: Time between the moment an activity starts and the first contribution. ─ Individual delay [59]: Time between last contribution and deadline to complete the activity. ─ Individual cooperative interaction [ 57 ]: Every interaction recorded in the shared workspace. • Group: ─ Team contributions [ 12,14 ]: Total contributions made by all team members. ─ Combination of team contributions [ 35 ]: Average team workload and variability across members. ─ Team consistency [ 58 ]: Average time separation between contributions of all members, and variability across members. ─ Team earliness [ 17,18 ]: Average distance between every team members’ first contribution and variability across team members. ─ Team delay [59-62]: Average time distance between the last contribution of the team and deadline to complete the activity, and variability across members. ─ Team cooperative interaction [ 35,57 ]: Ratio between all the interactions of team members and contributions made in the shared workspace. 3.3

Coordination (Cd)

Teamwork coordination consists on the synchronization of member interactions [ 14 ], which translates to a constant task execution pace and regularity [63]. Concurrency of interactions is easily observable in LMS through specific information in three record fields: timestamp, virtual space where the interaction occurs and user id.

In an online context, teamwork coordination offers a link between communication and cooperation that helps harmonizing and integrating individual efforts to achieve shared goals [ 12,33 ]. As a harmonization tool, message exchanges in earlier stages of task completion usually serve as an indication of successful task completion [64]. Relationship with outcomes aside, it is during the coordination process where communication and cooperation occur and may be observed through task completion progress and message exchanges, as that involves that team members are reaching agreements. In other words, sudden changes in the frequency of message exchanges and task execution breaks indicate that the team is experiencing difficulties and an intervention may be required [ 31 ]. Therefore, from a database perspective, coordination becomes evident if the temporal distance between messages and contributions presents sudden changes.

As an effort-integrating tool, coordination becomes manifest when the work involves all group members. Global efforts may be detected by observing the time allocated to available resources [65], time spent in sending messages and time dedicated to each contribution [ 33 ].

Another characteristic that may affect coordination is the fact that teamwork has a temporal limitation. The existence of delivery dates and deadlines makes it easier to observe that constant exchanges and contributions help completing the task on time [ 53,62,66 ] or that delays and last-minute delivery are an indication of poorly coordinated teams [60,61]. Along this line of reasoning, time of task execution relative to available time and the interval between delivery and deadline measure the degree of coordination and organization of the team [67].

From the above, our proposal includes the following indicators to measure teamwork coordination at individual and group levels: • Individual: ─ Quantitative individual synchronicity [ 38 ]: Correspondence between individual active interactions and active interactions of the rest of team members. ─ Spatial individual synchronicity [ 14 ]: Concurrency in the same virtual space with the rest of team members. ─ Temporal individual synchronicity [ 31,39 ]: Temporal distance between each member interactions and the interactions of the rest of team members. ─ Individual communication coordination [ 33,64 ]: Ratio between individual time spent publishing messages and total time spent publishing messages by all team members. ─ Individual cooperation coordination [ 33 ]: Ratio between time spent in individual contributions and total time spent in contributions by all team members. ─ Individual monitoring coordination [65]: Ratio between individual time spent reading messages and contributions and overall team spent reading messages and contributions by all team members. ─ Individual delivery date coordination [67,68]: Temporal distance between the latest reading, messaging and contribution actions and delivery date relative to the rest of team members. ─ Individual pace [63]: Total individual time spent in a task relative to time spent in the task by the rest of team members. • Group: ─ Quantitative team synchronicity [ 38 ]: Correspondence between total effort (global messages and contributions): and the number of team members, including variability between members. ─ Spatial team synchronicity [ 14 ]: Overall concurrence of team members in a given virtual space. ─ Temporal team synchronicity [ 31 ]: Temporal distribution of synchronous interactions between team members during the time available to task execution. ─ Team communication coordination [64]: Average time dedicated to message exchanges and variability among members. ─ Team cooperation coordination [60,61]: Average time dedicated to contributions and variability among members. ─ Team monitoring coordination [65]: Average time dedicated by all team members to reading activities in the specified period of time available to complete the task. ─ Team delivery date coordination [ 53,62,66,67 ]: Temporal distance between the latest interaction of each team member and delivery date. ─ Team pace [63,67]: Total team time spent in a task relative to time available for task completion. 3.4

Monitoring and tracking (MT)

Monitoring and tracking refers to the process of observing team actions, detecting errors and differences of opinion related to the task being performed, which promotes the generation of suggestions and corrections as feedback for every team member [ 48 ]. Therefore, monitoring and tracking has a strong association with both communication and cooperation interactions. In LMS, monitoring and tracking correspond to what [ 11 ] identify as passive interactions.

As with other dimensions, one characteristic of monitoring and tracking is its regularity. More specifically, monitoring is evident in early stages of the task by observing when team members access the guidelines and resources, information that is also related to academic performance [ 37,69 ]. Checking resources–access, file downloads, etc.–is considered a predictor of better results during task execution [70], even though the required level of resource access is contingent on the difficulty of the task [71], and changes depend on the number of resources available. This study primarily considers as monitoring and tracking indicators those that provide information about observation/reading activities in the beginning of the task–temporal distance to first access– and also during the execution of the task relative to the number of resources [69].

In relation to communication and cooperation, monitoring and tracking in online teamwork is a process that includes reviewing and reflecting upon the task being performed, and also upon the changes and agreements made [ 31 ]. Reviewing may be noticed by observing passive interaction records, such as time spent in communication– message board, chat–or cooperative–wiki, workshop, glossary, etc.–spaces. An important aspect to consider is that the duration of observations/reading must be proportional to the quantity of information that is being accessed [72]: if this duration–difference between access and leaving–is very short or too long–automatic session logout–, these records do not provide relevant information about cooperation [73].

Following this discussion, this research proposes the following indicators as potential candidates to measure monitoring and tracking for teamwork in LMS: • Individual: ─ Individual reading [ 31,33,48 ]: Ratio between number of passive interactions and active interactions of a team member. ─ Individual message reading [ 31,33,48 ]: Ratio between number messages read and sent/updated by a team member. ─ Individual contribution reading [ 31,33,48 ]: Ratio between contributions read and created/modified by a team member. ─ Individual resource reading [ 31,33,48 ]: Ratio between the number of accesses to resources of a team member and total number of resources available. ─ Individual monitoring consistency [ 37 ]: Distribution of passive interactions in a given period of time. ─ Individual promptness to access the guidelines [69]: Temporal distance between first access of an individual to the task guidelines and the moment the guidelines were made available. ─ Individual average message tracking time [70]: Ratio between time spent reading a message board and total number of messages published by all team members. ─ Individual average contribution tracking time [70]: Ratio between time spent reading contributions and total number of contributions made by all team members. ─ Individual monitoring frequency [ 37 ]: Ratio between the number of passive interactions and time available to perform the task. • Group: ─ Team reading [ 31,33,48 ]: Average number of passive interactions of all team members and variability across members. ─ Team message reading [ 31,33,48 ]: Average number of messages read by all team members and variability across members. ─ Team contribution reading [ 31,33,48 ]: Average number of contributions read by all team members and variability across members. ─ Team resource reading [ 31,33,48 ]: Average number of accesses to resources by all team members and variability across members. ─ Team monitoring consistency [ 37 ]: Average temporal distance of all monitoring interactions in the team and variability across members. ─ Team promptness to access the guidelines [69]: Average temporal distance between first access to the task guidelines of all team members and the moment the guidelines were made available, and variability across members. ─ Team average message tracking time [70]: Average time spent reading messages by all members and variability across members. ─ Team average contribution tracking time [70]: Average time spent reading contributions by all members and variability across members. ─ Team monitoring frequency [ 37 ]: Average number of passive interactions relative to time available to complete the task and variability across team members. 4 4.1

Conclusion

Implications for theory and practice

LMS log-based learning analytics has traditionally investigated the usefulness of database records as a source of meaningful and actionable information for learning analytics purposes at an individual level–primarily focusing on students–, in the search for variables that help predicting certain aspects of relevance in learning, such as student success, at-risk students, drop-out rates, etc. However, collaborative learning behaviors have generally been neglected in this kind of studies.

This research provides a novel, theoretically grounded approach to the identification of information stored in LMS databases that may allow performing learning analytics and offer further insight of teamwork in collaborative learning processes. To do so, the study provides a systematic identification of different aspects of teamwork as observable behaviors that may be registered in LMS databases. The system of indicators corresponding to the four dimensions of teamwork presented in this study open a door to further development in this field. The research also introduces the temporal component, missing from most log-based learning analytics studies, which is essential to better understand learning processes.

It should be pointed out that this research is not a purely theoretical exercise. On the contrary, our intention is to provide a common framework to the study of teamwork in online collaborative learning contexts. Because of the differences between the formats of the information stored across different types of LMS, the operationalization of some of the indicators might differ slightly from one LMS to another, or even be impossible at this moment because of missing information. Therefore, the proposal developed in this study might help LMS developers to improve their design regarding information collection and database design in order to enhance the learning analytics capabilities of their systems. Furthermore, some of the indicators proposed in this research are not directly available in the LMS logs, but they can be obtained through data transformation. In this sense, the research also presents software developers and learning analytics research teams with some ideas and opportunities for future tool development and information about approaches that might be required to improve our understanding and assessment of teamwork in online learning. Finally, it is evident that the true value of this research will only become evident after empirical validation of the system of indicators in real settings. Testing the adequacy of the framework, offering detailed description about the dynamics of teamwork processes in online collaborative learning and further observing the relation of the indicators with other variables of interest requires a successful implementation and operationalization of the framework proposed here. In this regard, this study is a starting point for teamwork assessment using learning analytics approaches. 4.2

Limitations and further research

This proposal focuses on observable actions occurring in LMS, and therefore registered in the systems’ databases and susceptible to be reduced to quantitative indicators. However, the proposal excludes additional information about teamwork dynamics from the interactions that may or may not be registered in the database, such as content and discourse information–which refer to the semantic aspect of collaboration–, or to antecedents and specific characteristics of the different team members–e.g. historic record of interactions in other courses, past grades, personality traits. The proposal further omits the potential effect of instructional and technological changes–e.g. introduction of a new learning method or new software, or the effect of feedback from instructors. A holistic approach that included all these aspects would greatly help to offer further insight about teamwork processes in online education. However, empirical validation of the indicators proposed in this study–and depuration of those indicators that might not provide meaningful information–is highly recommended before incorporating these additional elements to the analysis.

Acknowledgements

The authors thank the Spanish Ministry of Economy, Industry and Competitiveness for the support of the SNOLA Network of Excellence (TIN2015-71669-REDT). 59. Hollenbeck, J. R., Beersma, B., Schouten, M. E. Beyond Team Types and Taxonomies: A Dimensional Scaling Conceptualization for Team Description. Academy of Management Review, 37(1), 87-106 (2012). 60. Klassen, R. M., Ang, R. P., Chong, W. H., Krawchuk, L. L., Huan, V. S., Wong, I. Y.F.

Yeo, L. S. Academic Procrastination in Two Settings: Motivation Correlates, Behavioral Patterns, and Negative Impact of Procrastination in Canada and Singapore. Applied Psychology, 59(3), 361–379 (2010). 61. Levy, Y., Ramim, M. M. A study of online exams procrastination using data analytics techniques. Interdisciplinary Journal of E-Learning and Learning Objects, 8(1), 97-113 (2012). 62. You, J. W. Examining the Effect of Academic Procrastination on Achievement Using LMS

Data in e-Learning. Educational Technology & Society, 18(3), 64-74 (2015). 63. Kozlowski, S. W., Chao, G. T., Chang, C. H., Fernandez, R. Team dynamics: Using “big data” to advance the science of team effectiveness. Big data at work: The data science revolution and organizational psychology. Routledge Academic, New York, NY (2016). 64. Jiang, S., Williams, A., Schenke, K., Warschauer, M., O'dowd, D. Predicting MOOC performance with week 1 behavior. In Educational Data Mining 2014 (2014). 65. Arnold, K. E., Pistilli, M. D. Course signals at Purdue: Using learning analytics to increase student success. Proceedings of the 2nd international conference on Learning Analytics and Knowledge (LAK’12). ACM, New York (2012). 66. Wang, J., Sperling, R. A. Haspel, P. Patterns of procrastination, motivation, and strategy use across class contexts and students’ abilities. Journal of Psychology and Behavioral Science, 3(2), 61-73 (2015). 67. Baker, R. S., Lindrum, D., Lindrum, M. J., Perkowski, D. Analyzing Early At-Risk Factors in Higher Education E-Learning Courses. International Educational Data Mining Society (2015). 68. Yang, Y. F. Engaging students in an online situated language learning environment. Computer Assisted Language Learning, 24(2), 181-198 (2011). 69. Corrigan, O., Glynn, M., McKenna, A., Smeaton, A. F., Smyth, S. Student data: data is knowledge – putting the knowledge back in the students’ hands. In: 14th European Conference on e-Learning ECEL-2015, 29-30 Oct 2015, University of Hertfordshire, Hatfield, UK (2015). 70. Kotsiantis, S., Tselios, N., Filippidi, A., Komis, V. Using learning analytics to identify successful learners in a blended learning course. International Journal of Technology Enhanced Learning, 5(2), 133-150 (2013). 71. Greene, J. A., Moos, D. C., Azevedo, R., Winters, F. I. Exploring differences between gifted and grade-level students’ use of self-regulatory learning processes with hypermedia. Computers & Education, 50(3), 1069-1083 (2008). 72. Cocea, M., Weibelzahl, S. Can log files analysis estimate learners' level of motivation? In: Lernen-Wissensentdeckung-Adaptivität, 9-11 October 2006, Hildesheim (pp. 32-35) (2006). 73. Vermeulen, A. E. Improving Student Engagement Through Visualization of Course Activities (Doctoral dissertation) (2014).

References

1. Wastiau , P. , Blamire , R. , Kearney , C. , Quittre , V., Van de Gaer , E., Monseur , C. The use of ICT in education: a survey of schools in Europe . European Journal of Education , 48 ( 1 ), 11 - 27 ( 2013 ).

2. Fundación Telefónica . El trabajo en un mundo de sistemas inteligentes ( 2015 ).

3. Hackman , J. R. Why teams don't work . Leader to leader, 1998(7) , 24 - 31 ( 1998 ).

4. Pita , C. , Pizarro , E. Cómo ser competente. Competencias profesionales demandadas en el mercado laboral ( 2013 ).

5. Ellis , A. P. , Bell , B. S. , Ployhart , R. E. , Hollenbeck , J. R. , Ilgen , D. R. An evaluation of generic teamwork skills training with action teams: effects on cognitive and skill‐based outcomes . Personnel psychology , 58 ( 3 ), 641 - 672 ( 2005 ).

6. Aguado , D. , Arranz , V. , Valera-Rubio , A. , Marín-Torres , S. Evaluación de un programa blended-learning para el desarrollo de la competencia trabajar en equipo . Psicothema , 23 ( 3 ), 356 - 361 ( 2011 ).

7. Maynard , M. T. , Mathieu , J. E. , Rapp , T. L. , Gilson , L. L. Something(s) old and something(s) new: Modeling drivers of global virtual team effectiveness . Journal of Organizational Behavior , 33 ( 3 ), 342 - 365 ( 2012 ).

8. Bloom , B. , Engelhart , M. , Furst , E. Taxonomía de los objetivos de la educación. Ateneo, Buenos Aires ( 1990 ).

9. Martínez Leyet , O. L. , Puentes Puentes , Ú., Hernández , B. , Sahara , M. Las tecnologías de la información y la comunicación y su integración a la educación universitaria: una mirada al futuro ( 2015 ).

10. Johnson , L., Adams Becker , S. , Estrada , V. , Freeman , A. NMC Horizon Report: Edición Educación Superior 2015 . The New Media Consortium , Austin, Texas: ( 2015 ).

11. Agudo-Peregrina , Á. F. , Iglesias-Pradas , S. , Conde-González , M. Á. , Hernández-García, Á. Can we predict success from log data in VLEs? Classification of interactions for learning analytics and their relation with performance in VLE-supported F2F and online learning . Computers in Human Behavior , 31 , 542 - 550 ( 2014 ).

12. Fuks , H. , Raposo , A. , Gerosa , M. A. , Pimentel , M. , Lucena , C. J. The 3C collaboration model . In: Ned Kock (ed.), The Encyclopedia of E-Collaboration, 637 - 644 . IGI Global, Hershey PA ( 2007 ).

13. Boyatzis , R. E. The competent manager: A model for effective performance . John Wiley & Sons ( 1982 ).

14. Stevens , M. J. , Campion , M. A. The knowledge, skill, and ability requirements for teamwork: Implications for human resource management . Journal of Management , 20 ( 2 ), 503 - 530 ( 1994 ).

15. Salas , E. , Dickinson , T. L. , Converse , S. A. , Tannenbaum , S. I. Toward an understanding of team performance and training . In: Swezey RW , Salas E (Eds.), Teams: Their Training and performance (pp. 3 - 29 ). Norwood, NJ: ABLEX ( 1992 ).

16. Hobson , C. J. , Strupeck , D. , Griffin , A. , Szostek , J. , Rominger , A. S. Teaching MBA students' teamwork and team leadership skills: An empirical evaluation of a classroom educational program . American Journal of Business Education (Online) , 7 ( 3 ), 191 ( 2014 ).

17. Torrelles , C. , Coiduras Rodríguez , J. , Isus , S. , Carrera , F. X. , París Mañas , G. , Cela , J. M. Competencia de trabajo en equipo: definición y categorización . Revista de Profesorado , 15 ( 3 ) ( 2011 ).

18. Villa , A. , Poblete , M. Aprendizaje basado en competencias. Una propuesta para la evaluación de las competencias genéricas . Mensajero, Bilbao ( 2007 ).

19. Loparev , A. The impact of collaborative scaffolding in educational video games on the collaborative support skills of middle school students (Doctoral dissertation) . University of Rochester) ( 2016 ).

20. Gillies , R. M. Cooperative Learning: Integrating Theory and Practice: Integrating Theory and Practice . Sage Publications ( 2007 ).

21. Poblete , M. , García Olalla , A . Desarrollo de Competencias y Créditos transferibles. Experiencia multidisciplinar en el contexto universitario . Editorial Mensajero. Bilbao ( 2007 ).

22. Robbins , S. P. , Judge , T. Essentials of organizational behavior . Essex: Pearson ( 2012 ).

23. Levy-Leboyer , C. Evaluación del personal: los métodos a elegir . Ediciones Díaz de Santos ( 1992 ).

24. Prichard , J. S. , Bizo , L. A. , Stratford , R. J. Evaluating the effects of team-skills training on subjective workload . Learning and Instruction , 21 ( 3 ), 429 - 440 ( 2011 ).

25. González-Morales , D., de Antonio, L. M. M. , García , J. L. R. Teaching “ Soft” skills in software engineering . In Global Engineering Education Conference (EDUCON) , 2011 IEEE (pp. 630 - 637 ). IEEE ( 2011 ).

26. Earnest , D. R. , Landis , R. S. The Importance of Applicant Perceptions When Recruiting Employees to Teams . International Journal of Business and Social Science , 5 ( 2 ) ( 2014 ).

27. Mathieu , J. , Maynard , M. T. , Rapp , T. , Gilson , L. Team effectiveness 1997-2007: A review of recent advancements and a glimpse into the future . Journal of Management , 34 ( 3 ), 410 - 476 ( 2008 ).

28. Kozlowski , S. W. , Ilgen , D. R. Enhancing the effectiveness of work groups and teams . Psychological science in the public interest , 7 ( 3 ), 77 - 124 ( 2006 ).

29. LePine , J. A. , Hanson , M. A. , Borman , W. C. , Motowidlo , S. J. Contextual performance and teamwork: Implications for staffing , In G. R. Ferris (Ed.), Research in personnel and human resource management ,vol. 19 , (pp. 53 - 90 ). Elsevier, Amsterdam ( 2000 ).

30. Pérez , P. La formación para enseñar a trabajar en equipo: un análisis experimental ( 2015 ).

31. Gillies , R. M. , Ashman , A. F. Structuring co -operative learning experiences in primary school. Co-operative learning . The social and intellectual outcomes of learning in groups 36-53 ( 2003 ).

32. Salas , E. , Sims , D. E. , Burke , C. S. Is there a “big five” in teamwork? . Small group research, 36 ( 5 ), 555 - 599 ( 2005 ).

33. De Jong, B. A. , Elfring , T. How does trust affect the performance of ongoing teams? The mediating role of reflexivity, monitoring, and effort . Academy of Management Journal , 53 ( 3 ), 535 - 549 ( 2010 ).

34. Ellis , C. A. , Gibbs , S. J. , Rein , G. Groupware: some issues and experiences . Communications of the ACM , 34 ( 1 ), 39 - 58 ( 1991 ).

35. Chen , J. , Qiu , G. , Yuan , L. , Zhang , L. , Lu , G. Assessing teamwork performance in software engineering education: A case in a software engineering undergraduate course . In Software Engineering Conference (APSEC) , 2011 18th Asia Pacific (pp. 17 - 24 ). IEEE ( 2011 ).

36. Chen , B. , Wise , A. F. , Knight , S., Cheng, B. H. Putting temporal analytics into practice: the 5th international workshop on temporality in learning data . In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 488 - 489 ). ACM ( 2016 ).

37. Bovo , A. , Sanchez , S. , Héguy , O. , Duthen , Y. Clustering Moodle data as a tool for profiling students . In e-Learning and e-Technologies in Education (ICEEE) , 2013 Second International Conference on (pp. 121 - 126 ). IEEE ( 2013 ).

38. Fischer , F. , Järvelä , S. Methodological Advances in Research on Learning and Instruction and in the Learning Sciences. Frontline Learning Research 2 ( 4 ), 1 - 6 ( 2014 ).

39. Molenaar , I. , Järvelä , S. Sequential and temporal characteristics of self and socially regulated learning . Metacognition and Learning , 9 ( 2 ), 75 ( 2014 ).

40. LePine , J. A. , Piccolo , R. F. , Jackson , C. L. , Mathieu , J. E. , Saul , J. R. A meta‐analysis of teamwork processes: tests of a multidimensional model and relationships with team effectiveness criteria . Personnel Psychology , 61 ( 2 ), 273 - 307 ( 2008 ).

41. McGrath , J. E. Time , interaction, and performance (TIP) A Theory of Groups . Small group research, 22 ( 2 ), 147 - 174 ( 1991 ).

42. Hackman , J. R. , Morris , C. G. Group Tasks, Group Interaction Process, and Group Performance Effectiveness: A Review and Proposed Integration ( 1978 ).

43. Iglesias-Pradas , S. , Ruiz-de-Azcárate , C. , Agudo-Peregrina , Á. F. Assessing the suitability of student interactions from Moodle data logs as predictors of cross-curricular competencies . Computers in Human Behavior , 47 , 81 - 89 ( 2015 ).

44. Zhao , J. , Jiang , Y. Analyzing the Influencing Factors of Group Learning: A Mixed Approach . International Journal of Modern Education and Computer Science , 1 ( 1 ), 19 ( 2009 ).

45. Prichard , J. S. , Stratford , R. J. , Bizo , L. A. Team-skills training enhances collaborative learning . Learning and instruction , 16 ( 3 ), 256 - 265 ( 2006 ).

46. Bell , B. S. , Kozlowski , S. W. J. , Blawath , S. Team learning: A review and integration . In S. W. J. Kozlowski (Ed.), The Oxford Handbook of Organizational Psychology (vol. 2 , pp. 859 - 909 ). Oxford University Press, Oxford, UK ( 2012 ).

47. Kolbe , M. , Marty , A. , Seelandt , J. , Grande , B. How to debrief teamwork interactions: using circular questions to explore and change team interaction patterns . Advances in Simulation , 1 ( 1 ), 29 ( 2016 ).

48. Marks , M. A. , Mathieu , J. E. , Zaccaro , S. J. A temporally based framework and taxonomy of team processes . Academy of management review , 26 ( 3 ), 356 - 376 ( 2001 ).

49. Rebollo-Catalán , Á., Pérez , R. G. , Buzón-García , O. , Sánchez , R. B. Las comunidades virtuales como potencial pedagógico para el aprendizaje colaborativo a través de las TIC . Enseñanza & Teaching: Revista interuniversitaria de didáctica, ( 30 ), 105 - 126 ( 2012 ).

50. Romero , C. , López , M. I. , Luna , J. M. , Ventura , S. Predicting students' final performance from participation in on-line discussion forums . Computers & Education , 68 , 458 - 472 ( 2013 ).

51. Pargman , D. , Eriksson , E. “ It's not fair!”-making students engage in sustainability ( 2013 ).

52. Aghaee , N. , Hansson , H. Peer Portal: Quality enhancement in thesis writing using self-managed peer review on a mass scale . The International Review of Research in Open and Distributed Learning 14 ( 1 ), 186 - 203 ( 2013 ).

53. Romero , C. , Ventura , S. Data mining in education . Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 3 ( 1 ), 12 - 27 ( 2013 ).

54. Haya , P. A. , Daems , O. , Malzahn , N. , Castellanos , J. , Hoppe , H. U. Analysing content and patterns of interaction for improving the learning design of networked learning environments . British Journal of Educational Technology , 46 ( 2 ), 300 - 316 ( 2015 ).

55. Wu , W. H. , Wu , Y. C. J. , Chen , C. Y. , Kao , H. Y. , Lin , C. H. , Huang , S. H. Review of trends from mobile learning studies: A meta-analysis . Computers & Education , 59 ( 2 ), 817 - 827 ( 2012 ).

56. Shen , W. , Hao , Q. , Li , W. Computer supported collaborative design: Retrospective and perspective . Computers in Industry, 59 ( 9 ), 855 - 862 ( 2008 ).

57. Navimipour , N. J. , Charband , Y. Knowledge sharing mechanisms and techniques in project teams: literature review, classification, and current trends . Computers in Human Behavior , 62 , 730 - 742 ( 2016 ).

58. Hung , J.-L. , Zhang , K. Examining mobile learning trends 2003-2008: a categorical metatrend analysis using text mining techniques . Journal of Computer Higher Education , 24 ( 1 ), 1 - 17 ( 2008 ).