=Paper=
{{Paper
|id=Vol-1925/paper08
|storemode=property
|title=Proposal of a System of Indicators to Assess Teamwork Using Log-based Learning Analytics
|pdfUrl=https://ceur-ws.org/Vol-1925/paper08.pdf
|volume=Vol-1925
|authors=Carmen Ruiz-de-Azcárate,Ángel Hernández-García,Santiago Iglesias-Pradas,Emiliano Acquila-Natale
|dblpUrl=https://dblp.org/rec/conf/lasi-spain/Ruiz-de-Azcarate17
}}
==Proposal of a System of Indicators to Assess Teamwork Using Log-based Learning Analytics==
https://ceur-ws.org/Vol-1925/paper08.pdf
Proposal of a system of indicators to assess teamwork
using log-based learning analytics
Carmen Ruiz-de-Azcárate1, Ángel Hernández-García1, Santiago Iglesias-Pradas1 and
Emiliano Acquila-Natale1
1 Departamento de Ingeniería de Organización, Administración de Empresas y Estadística, Uni-
versidad Politécnica de Madrid, Av. Complutense 30, Despacho A-127, 28040 Madrid (Spain)
carmen.ruizdeazcarate@alumnos.upm.es, [angel.hernandez,
s.iglesias, emiliano.acquila]@upm.es
Abstract. One of the main lines of research in the field of learning analytics fo-
cuses on the identification of adequate data extracted from Learning Management
Systems’ (LMS) databases that may help predicting different behaviors and out-
comes, such as academic achievement or student performance. Prior research has
investigated these indicators at an individual level. The situation is more complex
in collaborative settings involving teamwork, such as project-based learning,
where student assessment considers the group as a single entity, disregard of in-
dividual contributions to the team. Furthermore, most often only the final deliv-
erable is taken into account when assessing teamwork in collaborative learning,
leading to a loss of perspective about the whole process and whether or not team-
work is effectively happening. This research aims to provide a comprehensive
selection of log-based information from LMS databases that could serve as po-
tential indicators to perform learning analytics and assess teamwork in online
learning. The proposal of this novel and theory-grounded framework understands
the multidimensional nature of teamwork, considers different sets of indicators
for each of its dimensions—communication, cooperation, coordination, and mon-
itoring and tracking—and incorporates the temporal dimension of activity data.
This proposal sets the basis for future software development to effectively trans-
form LMS log-based data and provide actionable measures of teamwork using
learning analytics.
Keywords: Learning Analytics, Teamwork, Learning Management Systems,
Logs.
1 Introduction
Online collaboration is at the heart of new networked organizations. Job posts evolve
toward pervasive connectivity embedded in networked organizations that replace tradi-
tional hierarchical structures [1,2]. The origin of this change in organizational models
has three main causes: the relation between collaborative work and productivity [3,4],
flexibility and teamwork structures that maximize the talent of employees.
Copyright © 2017 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.
Organizational initiatives to improve employees’ teamwork skills include hiring of
experts to manage and coordinate teamwork building projects [5]. However, in many
occasions these initiatives do not achieve the expected outcomes due to contextual, in-
dividual or motivational factors [6,7]. Therefore, there is a great interest in investigating
the factors that are characteristic of high-performance teams [8].
The demand for professional workers who have developed their teamwork skills also
affects educational institutions [9]. The number of graduates joining the workforce is a
proxy measure of an institution’s success, which has led to the integration of teamwork
training in most undergraduate and graduate courses [10], primarily through the imple-
mentation of project-based learning approaches. Higher education is also incorporating
the use of Information Technologies such as Learning Management Systems (LMS) as
tools to support learning in general, and group activities in particular, allowing for an-
ywhere-anytime, synchronous and asynchronous participation. LMS keep track of the
contributions from each student–or group/team member–, providing evidences for ef-
fective assessment, as well as information about the learning process, helping instruc-
tors to effectively monitor and support students.
Following similar approaches from prior literature [11], this conceptual research
aims to develop a learning analytics-based classification and to propose a system of
indicators that uses student activity traces in the LMS to provide instructors and teach-
ers with the most relevant information about student teamwork assessment in LMS.
This system may be used to facilitate decision-making about course instruction and
student and group assessment in project-based learning. In order to do so, the study
proposes a framework of learning interactions for the analysis of teamwork in virtual
learning environments. .
2 Conceptualization of teamwork
This study defines teamwork as a collaborative process, details its characteristics, and
builds on Fuks et al’s 3C collaboration model [12], which considers communication,
cooperation and coordination as the main components of collaboration, and further ex-
pands it with a fourth dimension: tracking/monitoring.
2.1 Definition
Teamwork refers to a behavioral pattern between two or more individuals [13,14] who
interact dynamically [15], establish a regular and constant negotiation to reach agree-
ments [16] through knowledge exchanges and problem solving [17], while keeping a
steady pace and coordinating efforts [18] in order to achieve their shared goals [19].
Teamwork is observable when a task is being performed [16,20,21], and thus pre-
sents a behavioral pattern that can be recognized through observation and is different
from other group actions [22]. Teamwork is also stable, extending to other tasks and
contexts [23], even though group member changes may alter the level of success of
outcomes [24]. Finally, and most important, teamwork is a cause and predictor of out-
comes, given an established behavioral pattern or interaction model [19,22,25,26].
2.2 Group teamwork and individual teamwork
Because this study focuses on observable aspects of teamwork, it is necessary to limit
the concept and highlight the differences between group teamwork and the individual
work of team members. When observed closely, both an individual task and teamwork
assignment share some inputs–goal definition and available resources–, process–task
execution– and outputs–delivered result or outcome [27]. Individual and team work
even share the most basic sub-tasks, to the extent that some assignments that are carried
out in teams could easily be done by a single individual.
The main difference between individual and team work is that, when working in
teams, both the final goal and intermediate goals and objectives are shared among the
team members. In other words, their actions are interrelated. Thus, the interdependence
among group members is one of the key elements that make individual and team work
different [28].
This interdependence translates to interactions between members. Member interac-
tions are an essential element of teamwork and lead to observable behaviors because,
in order to achieve the shared goals, the team has to show cooperation and social skills
that are not necessary to perform an individual task [14,29,30].
In sum, teamwork is a multidimensional and interdependent concept. In the context
of this research, teamwork corresponds to the regular communication between two or
more people who coordinate their effort in a period of time during which they cooperate
sharing ideas, knowledge and information, paying attention to the progress of the dif-
ferent tasks in order to achieve a common goal. Therefore, teamwork is the result of
different behaviors: communication, coordination, cooperation and monitoring. These
behaviors are complementary, combine with one another, are observable, include re-
curring activities, and are developed by every team member.
2.3 Dimensions of teamwork
Teamwork is a complex and dynamic concept. The multidimensional and interde-
pendent nature of teamwork implies that the observation of one single dimension or
behavior does not determine that teamwork is happening; on the contrary, all the dif-
ferent behaviors have to be observed in order to confirm that teamwork exists. None-
theless, this does not mean that all the different behaviors occur with the same fre-
quency, intensity or duration.
The first observable dimension when a teamwork task begins is communication, or
interaction between all team members. This interaction is present during the whole pro-
cess, even though with different levels of intensity, and in LMS it becomes manifest in
the form of message exchanges [12] in message boards and chats, with immediate,
lengthy and timely replies [31] between all participant members.
As long as all team members are participating, communication happens in a closed
loop between emitter and receivers [31,32], confirming that if teamwork is happening,
high levels of interaction must occur [12]. High levels of interaction are also related to
cooperation, as they drive an increasing number of contributions of members to the
common task [14,31] in shared workspaces, such as a wiki, glossary or workshop in
LMS. Together with the dimensions of communication and cooperation, teamwork in-
volves coordination. Coordination becomes manifest when every team member com-
municate with each other and cooperate in a synchronous way.
Finally, for teamwork to develop effectively, there must be some kind of control of
performed and pending tasks. This supervision includes monitoring and tracking activ-
ities [33], and constitutes the fourth dimension of teamwork. Monitoring and tracking
activities must be defined from the beginning of the activity, and one of their defining
characteristics is their regularity or consistency [12,34]. Monitoring and tracking re-
quires a timely reading of other members’ contributions [35]. Tracking represents as-
sessment of performed and remaining tasks, as well as of time and resource availability,
and it reflects effort and commitment to the team [33].
In sum, an adequate conceptualization of teamwork must include the following four
dimensions: 1) communication, defined in terms of message exchanges with a structure
of reply and confirmation–active listening–; 2) cooperation, as long as team members
share and exchange information in order to complete a shared goal; 3) coordination that
translates in synchronicity and constant pace during the execution of the different tasks;
and 4) monitoring and tracking, by adequately keeping up to date record of communi-
cations and interactions.
2.4 Temporal dimension
Each and every interaction in LMS generates a record or digital footprint that links the
data to a unique identifier: the user–in this case, a student. Every session login, message,
query, update or click in the LMS has a corresponding record in the database.
Besides the association of each interaction with the user, the fact that in LMS all
records have a timestamp, opens a door to the analysis of time-related factors for the
application of learning analytics [36]. Teamwork is a temporal series of behaviors
[37] that allows to assess learning progress [38,39]. Additionally, and given the
iterative nature of teamwork behaviors [38,39], it is necessary to analyze the
recurrence of these behaviors during the execution of the common task.
3 Interactions in LMS and teamwork dimensions
The previous section has already highlighted the relevance of interactions. Broadly
speaking, an interaction refers to an action that two objects, people or agents execute
reciprocally. When a team of people work together toward a common goal, interactions
refer to every interpersonal observable behavior aiming to synchronize and coordinate
resources and tasks in order to achieve the goal in a limited time [40], during a period
covering two arbitrary time points [41,42].
In the context of LMS, interactions are the basic unit of data in learning analytics
[11], as traces of user activity in LMS are stored in real time as records in the system
database. According to this, collaborative activities–e.g. teamwork–in LMS comprise
groups of records that represent the reciprocal actions–interactions–between team
members in a given period of time.
While there is no existing classification of collaborative interactions in LMS–the
only available classifications offer a study at the individual level [11,43]–, prior re-
search has investigated teamwork interactions in face-to-face learning, using observa-
tion through recording and analysis of conversations. The classifications resulting from
these analyses include meta-analysis of team efficacy [44], necessary behaviors for
teamwork in higher education [45], or organizational approaches focusing on team
structures as predictors of success in project management [19,22,25,26,46]. The main
conclusion of these studies is that effective teams show higher frequency and con-
sistency in their interactions during the execution of the task, with higher levels of in-
teraction during the first days of activity and in the days before a deliverable is due.
Addressing teamwork from a group perspective requires the unit of analysis to be
comprised of groups of interactions that include at least an individual–but not isolated–
interaction of each member, while considering also time and space concurrence [47].
However, it is not enough to group interactions as an aggregate–i.e. the aggregation of
individual contributions and results–; that is, team interactions will be defined by the
group of interactions of all team members that happen in a temporal span [41,48].
The following subsections will detail the proposal of interaction indicators for the
different dimensions of teamwork.
3.1 Communication (Cm)
In online contexts, communication between team members involves message and in-
formation exchanges [12,49]. Therefore, database records informing about message
creation, publication or updating in message boards or chats are potential candidates to
measure communication.
Regarding message exchanges, prior research has noted the positive relation between
message length [50] or number of messages exchanged in a period of time [51,52] and
student outcomes. These records can be considered as part of the communication when
there is reciprocity [53,54]; that is, when there is a quick and timely reply from other
team members. From an individual perspective, reciprocity can be measured by the
average in-reply and out-reply time [19].
Communication has to be persistent: each team member has to show constant impli-
cation in the interactions [54]. The levels of persistence can be measured as the ratio
between actual potential activity time intervals–e.g. real time used and total time avail-
able [19].
From the above, our proposal includes the following observable indicators of com-
munication, at both individual and group levels:
• Individual:
─ Individual message exchanges [12,49]: Ratio between number of messages sent
and total team messages.
─ Individual message length [50]: Average length of sent messages.
─ Individual frequency of messages [51,52]: Number of messages sent by the indi-
vidual by each task-dependent time unit.
─ Individual out-reciprocity [15,54]: Temporal distance of an individual replies’ to
other team members.
─ Individual in-reciprocity [19,35,50]: Temporal distance of other members’ replies
to messages sent by an individual.
─ Individual consistency [51,52]: Time interval between messages sent and interval
variability.
• Group:
─ Team message exchanges [55]: Total number of messages exchanged within the
team.
─ Team message length [50]: Average length of team messages and length variabil-
ity among team members.
─ Team message frequency [56]: Number of team messages by time unit (hour, day,
week, month):
─ Team reciprocity [19,35,50]: Balance between promptness and timeliness be-
tween messages sent and replies received among team members.
─ Team consistency [51,52]: Time interval between team messages and interval
variability.
3.2 Cooperation (Cp)
Cooperation involves sharing knowledge and, the same as communication, is positively
related to learning outcomes [57]. Cooperation in LMS is observable through the col-
lection and combination of all members’ contributions to a shared workspace; in other
words, it reflects the team workload by showing alternating contributions of the differ-
ent group members [12,14]. Production, in terms of knowledge interactions between
members in a shared workspace, can also be seen as a form of cooperation [35]. There-
fore, cooperation includes data related to contributions in a shared workspace–such as
a wiki, workshop or glossary in Moodle–, comprising both original contributions and
content updates.
Cooperation means working together, and it is characterized by: its presence during
the execution of the whole task–consistency [58]–, how early the members start con-
tributing [17,18], timely contributions [59] and on-time delivery of outcomes. Execu-
tion time makes the relation between delays and poor time management evident, both
at an individual and group level [60-62].
In summary, data related to cooperation in an LMS are those that have been gener-
ated through interaction in the shared workspaces that provide support to the creation
of the deliverables, including the time occurred between contributions, and taking into
account start and end dates.
From the above, our proposal includes the following indicators of cooperation:
• Individual:
─ Individual contributions [12,14]: Number of individual contributions.
─ Combination of individual contributions [35]: Ratio between individual workload
and team workload.
─ Individual consistency [58]: Distribution of contributions in the time available to
do the task.
─ Individual earliness [17,18]: Time between the moment an activity starts and the
first contribution.
─ Individual delay [59]: Time between last contribution and deadline to complete
the activity.
─ Individual cooperative interaction [57]: Every interaction recorded in the shared
workspace.
• Group:
─ Team contributions [12,14]: Total contributions made by all team members.
─ Combination of team contributions [35]: Average team workload and variability
across members.
─ Team consistency [58]: Average time separation between contributions of all
members, and variability across members.
─ Team earliness [17,18]: Average distance between every team members’ first
contribution and variability across team members.
─ Team delay [59-62]: Average time distance between the last contribution of the
team and deadline to complete the activity, and variability across members.
─ Team cooperative interaction [35,57]: Ratio between all the interactions of team
members and contributions made in the shared workspace.
3.3 Coordination (Cd)
Teamwork coordination consists on the synchronization of member interactions [14],
which translates to a constant task execution pace and regularity [63]. Concurrency of
interactions is easily observable in LMS through specific information in three record
fields: timestamp, virtual space where the interaction occurs and user id.
In an online context, teamwork coordination offers a link between communication
and cooperation that helps harmonizing and integrating individual efforts to achieve
shared goals [12,33]. As a harmonization tool, message exchanges in earlier stages of
task completion usually serve as an indication of successful task completion [64]. Re-
lationship with outcomes aside, it is during the coordination process where communi-
cation and cooperation occur and may be observed through task completion progress
and message exchanges, as that involves that team members are reaching agreements.
In other words, sudden changes in the frequency of message exchanges and task exe-
cution breaks indicate that the team is experiencing difficulties and an intervention may
be required [31]. Therefore, from a database perspective, coordination becomes evident
if the temporal distance between messages and contributions presents sudden changes.
As an effort-integrating tool, coordination becomes manifest when the work involves
all group members. Global efforts may be detected by observing the time allocated to
available resources [65], time spent in sending messages and time dedicated to each
contribution [33].
Another characteristic that may affect coordination is the fact that teamwork has a
temporal limitation. The existence of delivery dates and deadlines makes it easier to
observe that constant exchanges and contributions help completing the task on time
[53,62,66] or that delays and last-minute delivery are an indication of poorly coordi-
nated teams [60,61]. Along this line of reasoning, time of task execution relative to
available time and the interval between delivery and deadline measure the degree of
coordination and organization of the team [67].
From the above, our proposal includes the following indicators to measure teamwork
coordination at individual and group levels:
• Individual:
─ Quantitative individual synchronicity [38]: Correspondence between individual
active interactions and active interactions of the rest of team members.
─ Spatial individual synchronicity [14]: Concurrency in the same virtual space with
the rest of team members.
─ Temporal individual synchronicity [31,39]: Temporal distance between each
member interactions and the interactions of the rest of team members.
─ Individual communication coordination [33,64]: Ratio between individual time
spent publishing messages and total time spent publishing messages by all team
members.
─ Individual cooperation coordination [33]: Ratio between time spent in individual
contributions and total time spent in contributions by all team members.
─ Individual monitoring coordination [65]: Ratio between individual time spent
reading messages and contributions and overall team spent reading messages and
contributions by all team members.
─ Individual delivery date coordination [67,68]: Temporal distance between the lat-
est reading, messaging and contribution actions and delivery date relative to the
rest of team members.
─ Individual pace [63]: Total individual time spent in a task relative to time spent
in the task by the rest of team members.
• Group:
─ Quantitative team synchronicity [38]: Correspondence between total effort
(global messages and contributions): and the number of team members, including
variability between members.
─ Spatial team synchronicity [14]: Overall concurrence of team members in a given
virtual space.
─ Temporal team synchronicity [31]: Temporal distribution of synchronous inter-
actions between team members during the time available to task execution.
─ Team communication coordination [64]: Average time dedicated to message ex-
changes and variability among members.
─ Team cooperation coordination [60,61]: Average time dedicated to contributions
and variability among members.
─ Team monitoring coordination [65]: Average time dedicated by all team members
to reading activities in the specified period of time available to complete the task.
─ Team delivery date coordination [53,62,66,67]: Temporal distance between the
latest interaction of each team member and delivery date.
─ Team pace [63,67]: Total team time spent in a task relative to time available for
task completion.
3.4 Monitoring and tracking (MT)
Monitoring and tracking refers to the process of observing team actions, detecting
errors and differences of opinion related to the task being performed, which promotes
the generation of suggestions and corrections as feedback for every team member [48].
Therefore, monitoring and tracking has a strong association with both communication
and cooperation interactions. In LMS, monitoring and tracking correspond to what [11]
identify as passive interactions.
As with other dimensions, one characteristic of monitoring and tracking is its regu-
larity. More specifically, monitoring is evident in early stages of the task by observing
when team members access the guidelines and resources, information that is also re-
lated to academic performance [37,69]. Checking resources–access, file downloads,
etc.–is considered a predictor of better results during task execution [70], even though
the required level of resource access is contingent on the difficulty of the task [71], and
changes depend on the number of resources available. This study primarily considers
as monitoring and tracking indicators those that provide information about observa-
tion/reading activities in the beginning of the task–temporal distance to first access–
and also during the execution of the task relative to the number of resources [69].
In relation to communication and cooperation, monitoring and tracking in online
teamwork is a process that includes reviewing and reflecting upon the task being per-
formed, and also upon the changes and agreements made [31]. Reviewing may be no-
ticed by observing passive interaction records, such as time spent in communication–
message board, chat–or cooperative–wiki, workshop, glossary, etc.–spaces. An im-
portant aspect to consider is that the duration of observations/reading must be propor-
tional to the quantity of information that is being accessed [72]: if this duration–differ-
ence between access and leaving–is very short or too long–automatic session logout–,
these records do not provide relevant information about cooperation [73].
Following this discussion, this research proposes the following indicators as poten-
tial candidates to measure monitoring and tracking for teamwork in LMS:
• Individual:
─ Individual reading [31,33,48]: Ratio between number of passive interactions and
active interactions of a team member.
─ Individual message reading [31,33,48]: Ratio between number messages read and
sent/updated by a team member.
─ Individual contribution reading [31,33,48]: Ratio between contributions read and
created/modified by a team member.
─ Individual resource reading [31,33,48]: Ratio between the number of accesses to
resources of a team member and total number of resources available.
─ Individual monitoring consistency [37]: Distribution of passive interactions in a
given period of time.
─ Individual promptness to access the guidelines [69]: Temporal distance between
first access of an individual to the task guidelines and the moment the guidelines
were made available.
─ Individual average message tracking time [70]: Ratio between time spent reading
a message board and total number of messages published by all team members.
─ Individual average contribution tracking time [70]: Ratio between time spent
reading contributions and total number of contributions made by all team mem-
bers.
─ Individual monitoring frequency [37]: Ratio between the number of passive in-
teractions and time available to perform the task.
• Group:
─ Team reading [31,33,48]: Average number of passive interactions of all team
members and variability across members.
─ Team message reading [31,33,48]: Average number of messages read by all team
members and variability across members.
─ Team contribution reading [31,33,48]: Average number of contributions read by
all team members and variability across members.
─ Team resource reading [31,33,48]: Average number of accesses to resources by
all team members and variability across members.
─ Team monitoring consistency [37]: Average temporal distance of all monitoring
interactions in the team and variability across members.
─ Team promptness to access the guidelines [69]: Average temporal distance be-
tween first access to the task guidelines of all team members and the moment the
guidelines were made available, and variability across members.
─ Team average message tracking time [70]: Average time spent reading messages
by all members and variability across members.
─ Team average contribution tracking time [70]: Average time spent reading con-
tributions by all members and variability across members.
─ Team monitoring frequency [37]: Average number of passive interactions relative
to time available to complete the task and variability across team members.
4 Conclusion
4.1 Implications for theory and practice
LMS log-based learning analytics has traditionally investigated the usefulness of data-
base records as a source of meaningful and actionable information for learning analytics
purposes at an individual level–primarily focusing on students–, in the search for vari-
ables that help predicting certain aspects of relevance in learning, such as student suc-
cess, at-risk students, drop-out rates, etc. However, collaborative learning behaviors
have generally been neglected in this kind of studies.
This research provides a novel, theoretically grounded approach to the identification
of information stored in LMS databases that may allow performing learning analytics
and offer further insight of teamwork in collaborative learning processes. To do so, the
study provides a systematic identification of different aspects of teamwork as observa-
ble behaviors that may be registered in LMS databases. The system of indicators cor-
responding to the four dimensions of teamwork presented in this study open a door to
further development in this field. The research also introduces the temporal component,
missing from most log-based learning analytics studies, which is essential to better un-
derstand learning processes.
It should be pointed out that this research is not a purely theoretical exercise. On the
contrary, our intention is to provide a common framework to the study of teamwork in
online collaborative learning contexts. Because of the differences between the formats
of the information stored across different types of LMS, the operationalization of some
of the indicators might differ slightly from one LMS to another, or even be impossible
at this moment because of missing information. Therefore, the proposal developed in
this study might help LMS developers to improve their design regarding information
collection and database design in order to enhance the learning analytics capabilities of
their systems. Furthermore, some of the indicators proposed in this research are not
directly available in the LMS logs, but they can be obtained through data transfor-
mation. In this sense, the research also presents software developers and learning ana-
lytics research teams with some ideas and opportunities for future tool development
and information about approaches that might be required to improve our understanding
and assessment of teamwork in online learning. Finally, it is evident that the true value
of this research will only become evident after empirical validation of the system of
indicators in real settings. Testing the adequacy of the framework, offering detailed
description about the dynamics of teamwork processes in online collaborative learning
and further observing the relation of the indicators with other variables of interest re-
quires a successful implementation and operationalization of the framework proposed
here. In this regard, this study is a starting point for teamwork assessment using learning
analytics approaches.
4.2 Limitations and further research
This proposal focuses on observable actions occurring in LMS, and therefore registered
in the systems’ databases and susceptible to be reduced to quantitative indicators. How-
ever, the proposal excludes additional information about teamwork dynamics from the
interactions that may or may not be registered in the database, such as content and dis-
course information–which refer to the semantic aspect of collaboration–, or to anteced-
ents and specific characteristics of the different team members–e.g. historic record of
interactions in other courses, past grades, personality traits. The proposal further omits
the potential effect of instructional and technological changes–e.g. introduction of a
new learning method or new software, or the effect of feedback from instructors. A
holistic approach that included all these aspects would greatly help to offer further in-
sight about teamwork processes in online education. However, empirical validation of
the indicators proposed in this study–and depuration of those indicators that might not
provide meaningful information–is highly recommended before incorporating these ad-
ditional elements to the analysis.
Acknowledgements
The authors thank the Spanish Ministry of Economy, Industry and Competitiveness for
the support of the SNOLA Network of Excellence (TIN2015-71669-REDT).
References
1. Wastiau, P., Blamire, R., Kearney, C., Quittre, V., Van de Gaer, E., Monseur, C. The use of
ICT in education: a survey of schools in Europe.European Journal of Education, 48(1), 11-
27 (2013).
2. Fundación Telefónica. El trabajo en un mundo de sistemas inteligentes (2015).
3. Hackman, J. R. Why teams don't work. Leader to leader, 1998(7), 24-31 (1998).
4. Pita, C., Pizarro, E. Cómo ser competente. Competencias profesionales demandadas en el
mercado laboral (2013).
5. Ellis, A. P., Bell, B. S., Ployhart, R. E., Hollenbeck, J. R., Ilgen, D. R. An evaluation of
generic teamwork skills training with action teams: effects on cognitive and skill‐based out-
comes. Personnel psychology, 58(3), 641-672 (2005).
6. Aguado, D., Arranz, V., Valera-Rubio, A., Marín-Torres, S. Evaluación de un programa
blended-learning para el desarrollo de la competencia trabajar en equipo. Psicothema, 23(3),
356-361 (2011).
7. Maynard, M. T., Mathieu, J. E., Rapp, T. L., Gilson, L. L. Something(s) old and some-
thing(s) new: Modeling drivers of global virtual team effectiveness. Journal of Organiza-
tional Behavior, 33(3), 342-365 (2012).
8. Bloom, B., Engelhart, M., Furst, E. Taxonomía de los objetivos de la educación. Ateneo,
Buenos Aires (1990).
9. Martínez Leyet, O. L., Puentes Puentes, Ú., Hernández, B., Sahara, M. Las tecnologías de
la información y la comunicación y su integración a la educación universitaria: una mirada
al futuro (2015).
10. Johnson, L., Adams Becker, S., Estrada, V., Freeman, A. NMC Horizon Report: Edición
Educación Superior 2015. The New Media Consortium, Austin, Texas: (2015).
11. Agudo-Peregrina, Á. F., Iglesias-Pradas, S., Conde-González, M. Á., Hernández-García, Á.
Can we predict success from log data in VLEs? Classification of interactions for learning
analytics and their relation with performance in VLE-supported F2F and online learning.
Computers in Human Behavior, 31, 542-550 (2014).
12. Fuks, H., Raposo, A., Gerosa, M. A., Pimentel, M., Lucena, C. J. The 3C collaboration
model. In: Ned Kock (ed.), The Encyclopedia of E-Collaboration, 637-644. IGI Global, Her-
shey PA (2007).
13. Boyatzis, R. E. The competent manager: A model for effective performance. John Wiley &
Sons (1982).
14. Stevens, M. J., Campion, M. A. The knowledge, skill, and ability requirements for team-
work: Implications for human resource management. Journal of Management, 20(2), 503-
530 (1994).
15. Salas, E., Dickinson, T. L., Converse, S. A., Tannenbaum, S. I. Toward an understanding of
team performance and training. In: Swezey RW, Salas E (Eds.), Teams: Their Training and
performance (pp. 3-29). Norwood, NJ: ABLEX (1992).
16. Hobson, C. J., Strupeck, D., Griffin, A., Szostek, J., Rominger, A. S. Teaching MBA stu-
dents’ teamwork and team leadership skills: An empirical evaluation of a classroom educa-
tional program. American Journal of Business Education (Online), 7(3), 191 (2014).
17. Torrelles, C., Coiduras Rodríguez, J., Isus, S., Carrera, F. X., París Mañas, G., Cela, J. M.
Competencia de trabajo en equipo: definición y categorización. Revista de Profesorado,
15(3) (2011).
18. Villa, A., Poblete, M. Aprendizaje basado en competencias. Una propuesta para la evalua-
ción de las competencias genéricas. Mensajero, Bilbao (2007).
19. Loparev, A. The impact of collaborative scaffolding in educational video games on the col-
laborative support skills of middle school students (Doctoral dissertation). University of
Rochester) (2016).
20. Gillies, R. M. Cooperative Learning: Integrating Theory and Practice: Integrating Theory
and Practice. Sage Publications (2007).
21. Poblete, M., García Olalla, A. Desarrollo de Competencias y Créditos transferibles. Expe-
riencia multidisciplinar en el contexto universitario. Editorial Mensajero. Bilbao (2007).
22. Robbins, S. P., Judge, T. Essentials of organizational behavior. Essex: Pearson (2012).
23. Levy-Leboyer, C. Evaluación del personal: los métodos a elegir. Ediciones Díaz de Santos
(1992).
24. Prichard, J. S., Bizo, L. A., Stratford, R. J. Evaluating the effects of team-skills training on
subjective workload. Learning and Instruction, 21(3), 429-440 (2011).
25. González-Morales, D., de Antonio, L. M. M., García, J. L. R. Teaching “Soft” skills in soft-
ware engineering. In Global Engineering Education Conference (EDUCON), 2011 IEEE
(pp. 630-637). IEEE (2011).
26. Earnest, D. R., Landis, R. S. The Importance of Applicant Perceptions When Recruiting
Employees to Teams. International Journal of Business and Social Science, 5(2) (2014).
27. Mathieu, J., Maynard, M. T., Rapp, T., Gilson, L. Team effectiveness 1997-2007: A review
of recent advancements and a glimpse into the future. Journal of Management, 34(3), 410-
476 (2008).
28. Kozlowski, S. W., Ilgen, D. R. Enhancing the effectiveness of work groups and teams. Psy-
chological science in the public interest, 7(3), 77-124 (2006).
29. LePine, J. A., Hanson, M. A., Borman, W. C., Motowidlo, S. J. Contextual performance and
teamwork: Implications for staffing, In G. R. Ferris (Ed.), Research in personnel and human
resource management,vol. 19, (pp.53-90). Elsevier, Amsterdam (2000).
30. Pérez, P. La formación para enseñar a trabajar en equipo: un análisis experimental (2015).
31. Gillies, R. M., Ashman, A. F. Structuring co-operative learning experiences in primary
school. Co-operative learning. The social and intellectual outcomes of learning in groups
36-53 (2003).
32. Salas, E., Sims, D. E., Burke, C. S. Is there a “big five” in teamwork?. Small group research,
36(5), 555-599 (2005).
33. De Jong, B. A., Elfring, T. How does trust affect the performance of ongoing teams? The
mediating role of reflexivity, monitoring, and effort. Academy of Management Journal,
53(3), 535-549 (2010).
34. Ellis, C. A., Gibbs, S. J., Rein, G. Groupware: some issues and experiences. Communica-
tions of the ACM, 34(1), 39-58 (1991).
35. Chen, J., Qiu, G., Yuan, L., Zhang, L., Lu, G. Assessing teamwork performance in software
engineering education: A case in a software engineering undergraduate course. In Software
Engineering Conference (APSEC), 2011 18th Asia Pacific (pp. 17-24). IEEE (2011).
36. Chen, B., Wise, A. F., Knight, S., Cheng, B. H. Putting temporal analytics into practice: the
5th international workshop on temporality in learning data. In Proceedings of the Sixth In-
ternational Conference on Learning Analytics & Knowledge (pp. 488-489). ACM (2016).
37. Bovo, A., Sanchez, S., Héguy, O., Duthen, Y. Clustering Moodle data as a tool for profiling
students. In e-Learning and e-Technologies in Education (ICEEE), 2013 Second Interna-
tional Conference on (pp. 121-126). IEEE (2013).
38. Fischer, F., Järvelä, S. Methodological Advances in Research on Learning and Instruction
and in the Learning Sciences. Frontline Learning Research 2(4), 1-6 (2014).
39. Molenaar, I., Järvelä, S. Sequential and temporal characteristics of self and socially regu-
lated learning. Metacognition and Learning, 9(2), 75 (2014).
40. LePine, J. A., Piccolo, R. F., Jackson, C. L., Mathieu, J. E., Saul, J. R. A meta‐analysis of
teamwork processes: tests of a multidimensional model and relationships with team effec-
tiveness criteria. Personnel Psychology, 61(2), 273-307 (2008).
41. McGrath, J. E. Time, interaction, and performance (TIP) A Theory of Groups. Small group
research, 22(2), 147-174 (1991).
42. Hackman, J. R., Morris, C. G. Group Tasks, Group Interaction Process, and Group Perfor-
mance Effectiveness: A Review and Proposed Integration (1978).
43. Iglesias-Pradas, S., Ruiz-de-Azcárate, C., Agudo-Peregrina, Á. F. Assessing the suitability
of student interactions from Moodle data logs as predictors of cross-curricular competencies.
Computers in Human Behavior, 47, 81-89 (2015).
44. Zhao, J., Jiang, Y. Analyzing the Influencing Factors of Group Learning: A Mixed Ap-
proach. International Journal of Modern Education and Computer Science, 1(1), 19 (2009).
45. Prichard, J. S., Stratford, R. J., Bizo, L. A. Team-skills training enhances collaborative learn-
ing. Learning and instruction, 16(3), 256-265 (2006).
46. Bell, B. S., Kozlowski, S. W. J., Blawath, S. Team learning: A review and integration. In S.
W. J. Kozlowski (Ed.), The Oxford Handbook of Organizational Psychology (vol. 2, pp.859-
909). Oxford University Press, Oxford, UK (2012).
47. Kolbe, M., Marty, A., Seelandt, J., Grande, B. How to debrief teamwork interactions: using
circular questions to explore and change team interaction patterns. Advances in Simulation,
1(1), 29 (2016).
48. Marks, M. A., Mathieu, J. E., Zaccaro, S. J. A temporally based framework and taxonomy
of team processes. Academy of management review, 26(3), 356-376 (2001).
49. Rebollo-Catalán, Á., Pérez, R. G., Buzón-García, O., Sánchez, R. B. Las comunidades vir-
tuales como potencial pedagógico para el aprendizaje colaborativo a través de las TIC. En-
señanza & Teaching: Revista interuniversitaria de didáctica, (30), 105-126 (2012).
50. Romero, C., López, M. I., Luna, J. M., Ventura, S. Predicting students' final performance
from participation in on-line discussion forums. Computers & Education, 68, 458-472
(2013).
51. Pargman, D., Eriksson, E. “It’s not fair!”-making students engage in sustainability (2013).
52. Aghaee, N., Hansson, H. Peer Portal: Quality enhancement in thesis writing using self-man-
aged peer review on a mass scale. The International Review of Research in Open and Dis-
tributed Learning 14(1), 186-203 (2013).
53. Romero, C., Ventura, S. Data mining in education. Wiley Interdisciplinary Reviews: Data
Mining and Knowledge Discovery, 3(1), 12-27 (2013).
54. Haya, P. A., Daems, O., Malzahn, N., Castellanos, J., Hoppe, H. U. Analysing content and
patterns of interaction for improving the learning design of networked learning environ-
ments. British Journal of Educational Technology, 46(2), 300–316 (2015).
55. Wu, W. H., Wu, Y. C. J., Chen, C. Y., Kao, H. Y., Lin, C. H., Huang, S. H. Review of trends
from mobile learning studies: A meta-analysis. Computers & Education, 59(2), 817-827
(2012).
56. Shen, W., Hao, Q., Li, W. Computer supported collaborative design: Retrospective and per-
spective. Computers in Industry, 59(9), 855-862 (2008).
57. Navimipour, N. J., Charband, Y. Knowledge sharing mechanisms and techniques in project
teams: literature review, classification, and current trends. Computers in Human Behavior,
62, 730-742 (2016).
58. Hung, J.-L., Zhang, K. Examining mobile learning trends 2003–2008: a categorical meta-
trend analysis using text mining techniques. Journal of Computer Higher Education, 24(1),
1-17 (2008).
59. Hollenbeck, J. R., Beersma, B., Schouten, M. E. Beyond Team Types and Taxonomies: A
Dimensional Scaling Conceptualization for Team Description. Academy of Management
Review, 37(1), 87-106 (2012).
60. Klassen, R. M., Ang, R. P., Chong, W. H., Krawchuk, L. L., Huan, V. S., Wong, I. Y.F.
Yeo, L. S. Academic Procrastination in Two Settings: Motivation Correlates, Behavioral
Patterns, and Negative Impact of Procrastination in Canada and Singapore. Applied Psy-
chology, 59(3), 361–379 (2010).
61. Levy, Y., Ramim, M. M. A study of online exams procrastination using data analytics tech-
niques. Interdisciplinary Journal of E-Learning and Learning Objects, 8(1), 97-113 (2012).
62. You, J. W. Examining the Effect of Academic Procrastination on Achievement Using LMS
Data in e-Learning. Educational Technology & Society, 18(3), 64-74 (2015).
63. Kozlowski, S. W., Chao, G. T., Chang, C. H., Fernandez, R. Team dynamics: Using “big
data” to advance the science of team effectiveness. Big data at work: The data science rev-
olution and organizational psychology. Routledge Academic, New York, NY (2016).
64. Jiang, S., Williams, A., Schenke, K., Warschauer, M., O'dowd, D. Predicting MOOC per-
formance with week 1 behavior. In Educational Data Mining 2014 (2014).
65. Arnold, K. E., Pistilli, M. D. Course signals at Purdue: Using learning analytics to increase
student success. Proceedings of the 2nd international conference on Learning Analytics and
Knowledge (LAK’12). ACM, New York (2012).
66. Wang, J., Sperling, R. A. Haspel, P. Patterns of procrastination, motivation, and strategy use
across class contexts and students’ abilities. Journal of Psychology and Behavioral Science,
3(2), 61-73 (2015).
67. Baker, R. S., Lindrum, D., Lindrum, M. J., Perkowski, D. Analyzing Early At-Risk Factors
in Higher Education E-Learning Courses. International Educational Data Mining Society
(2015).
68. Yang, Y. F. Engaging students in an online situated language learning environment. Com-
puter Assisted Language Learning, 24(2), 181-198 (2011).
69. Corrigan, O., Glynn, M., McKenna, A., Smeaton, A. F., Smyth, S. Student data: data is
knowledge – putting the knowledge back in the students’ hands. In: 14th European Confer-
ence on e-Learning ECEL-2015, 29-30 Oct 2015, University of Hertfordshire, Hatfield, UK
(2015).
70. Kotsiantis, S., Tselios, N., Filippidi, A., Komis, V. Using learning analytics to identify suc-
cessful learners in a blended learning course. International Journal of Technology Enhanced
Learning, 5(2), 133-150 (2013).
71. Greene, J. A., Moos, D. C., Azevedo, R., Winters, F. I. Exploring differences between gifted
and grade-level students’ use of self-regulatory learning processes with hypermedia. Com-
puters & Education, 50(3), 1069-1083 (2008).
72. Cocea, M., Weibelzahl, S. Can log files analysis estimate learners' level of motivation? In:
Lernen-Wissensentdeckung-Adaptivität, 9-11 October 2006, Hildesheim (pp. 32-35)
(2006).
73. Vermeulen, A. E. Improving Student Engagement Through Visualization of Course Activi-
ties (Doctoral dissertation) (2014).