In computer-supported collaborative learning, teachers do not usually get the information they need for regulating a learning situation during its enactment. To address this need, we propose to provide teachers with monitoring feedback guided by their pedagogical intentions. In particular, those pedagogical intentions captured in the scripts where teachers reflect their learning designs. In order to configure monitoring in alignment with such pedagogical intentions, a model that gathers all the necessary data elements is required. This paper presents this “script-aware monitoring model”, and the teachers' positive opinions about its usefulness and efficiency from three pilot studies where the model was tested.
According to the STELLAR European Network of Excellence in Technology Enhanced
Learning (TEL), one of the main challenges for future teachers and learners is to “make
sense and intelligent use of the data provided by information and communication
technologies (ICTs) in order to facilitate learning” [
Learning Analytics collects available data about learners and their contexts, and
identifies relevant feedback from it. Techniques and methods commonly used for this
purpose include a posteriori social network analysis, discourse and concept analysis, or
learner success prediction, with a strong emphasis on meaningful visualizations [
Our proposal is framed in the field of Computer-Supported Collaborative Learning
(CSCL), in which scripting is a well-known strategy followed by teachers when
designing learning situations [
This paper presents on-going work towards the definition of a script-aware monitoring model that brings together the elements involved in scripting and monitoring.
This document analyzes the required parameters for configuring the monitoring process of a CSCL scenario, indicating which ones are retrievable from the script and which ones must be provided additionally by the teacher. This model has been used in three pilot studies that involved two different teachers in order to validate whether our proposal is useful and efficient for them. This paper also includes preliminary feedback from the participant teachers, who evaluated positively the monitoring results obtained. 2
We have drawn from two trends in the CSCL field for the definition of the script-aware
monitoring model: a) CSCL scripting (since it has dealt with the modeling of learning
scenarios in order to structure the collaborative learning process [
Multiple authors in the field of CSCL agree on the elements that conform a script [
Another trend of research that provides relevant input for our goal comes from
the work on computer-based interaction analysis [
The elements defined in this Common Format are divided into two branches. The context is the general setup of a learning situation (users, roles of the participants, groups, resources). The second branch describes what happens during the learning activity, i.e., the actions carried out by participants, identifying who has done what, and when. “Who” is one user previously identified in the context branch, “what” is type of action among those allowed within the specific learning environment or tool, and “when” is the timestamp of the event.
However, existing proposals in the scripting and interaction analysis fields are not suitable for guiding monitoring on the basis of pedagogical intentions. Scripting proposals cannot be applied for monitoring mainly because they do not consider the users’ actions during the learning process. Regarding interaction analysis, the studied models do not take into account a core scripting concept, namely learning activities. Therefore, we will propose a monitoring model that brings together both scripting and interaction analysis approaches, including the elements shown in Figure 1. The aim of this model is to collect all the elements required to guide a monitoring process on the basis of a CSCL script. Thus, our proposed solution contains elements that model the script (participants, groups, roles, activities and resources), and elements that model the events gathered during the enactment of the script (actions). Figure 2 depicts scripting elements with continuous yellow circles, and monitoring elements with dashed blue circles. Additionally, some attributes of the aforementioned elements need to be specified, as we have detected in several case studies. Those that need to be provided by the teacher have been represented with bold red boxes in Figure 2. The attributes that are relevant for configuring the monitoring process are briefly presented below, together with illustrative examples:
- Group formation policies: the knowledge of how groups are structured simplifies the visualization of relevant interactions and the detection of unexpected eventualities such as isolated students.
- Interactivity type: specifying how students are expected to interact (face-to-face / computer-mediated / blended) helps to guide the selection of monitored actions that may inform about the activity progress.
- Activity social level: individual, group and class-wide activities are interspersed in the learning situation. Knowing the social level of an activity can help us in choosing the way of analyzing the activity events. If an activity is individual, it will be necessary to know how each participant evolves and interacts with the context (use of the ICT tools, attendance to the lectures, etc.). However, if the activity is carried out by groups (or by the whole class), it will be relevant to know if there is evidence of participation and collaboration among group members.
- Activity deadlines: knowing when an activity starts and finishes, limits the data gathering to only the relevant time period. This prevents showing the teacher irrelevant information, and reduces the amount of data to be processed, facilitating the generation of timely results during the enactment of the learning situation.
- Dependences between activities: learning activities within a CSCL scenario are usually related to one another. Sometimes there are time dependences (e.g. when one activity finishes another begins), shared resources (e.g. the output of one activity is the input of another), etc. Identifying in advance these dependences facilitates useful monitoring advice, e.g., when the current state of one activity may have a negative impact on another.
- Expected use of resources: monitoring feedback may be improved by means of additional information about the way resources should be used within the learning activity (e.g. whether one resource is mandatory as opposed to optional, whether participants are expected to use a resource individually or in groups).
- Action timestamp: in order to inform the state of the previous parameters, it is necessary to be able to locate the action in time. Thus, the timestamp indicates whether an action should be taken into account or not, according to the activity deadlines.
- Action type: identifying the nature of the participants’ actions can help us choose the most suitable data sources for monitoring (e.g. checking the attendance to a session may be relevant for a face-to-face activity, but not for an online, asynchronous one).
Although these elements and attributes may seem too general, or even trivial, they are highly relevant during the enactment in order to guide monitoring. For instance, monitored actions can provide evidence to support that, during the period set by the activity deadlines: (a) participants are interacting and collaborating as it was specified, (b) participants are using the resources as it was expected, (c) group formation policies are verified, or (d) the current situation of the activity jeopardizes other activities in terms of sequencing, group formation or reuse of resources. 2.3
In order to evaluate the validity of the model, we have employed it for monitoring three pilot studies, carried out at the University of Valladolid from February to May 2012. All three CSCL situations lasted for 3-4 weeks, interleaving face-to-face with distance activities mediated by several ICT tools. Two of them were carried out with 14 master-level students and an experienced teacher, and the last one with 60 undergraduate students and a non-expert teacher. While a complete evaluation of these studies is in progress, in this section we present the teachers’ opinions collected in semi-structured interviews at the end of the learning situations.
In these pilot studies, teachers were first asked to create their designs using authoring tools (namely Web Instance Collage1 and Pedagogical Pattern Collector2) as well as deployment tools for CSCL scripts (GLUE!-PS3). Then, teachers enriched the scripts filling a form with the additional information required by the model. Finally, during the enactment, teachers received the monitoring results based on the script decisions.
Both teachers confirmed that specifying the aforementioned additional attributes to their scripts required little effort in terms of time and complexity, and that they considered it useful to reflect on those parameters at design-time.
During the enactment, monitoring results showed information to the teachers which they had not realized yet. Teachers argued that, in most cases, monitoring results helped them to confirm that the students were following properly the script (e.g. collaborating as expected or submitting reports on time). Additionally, some unexpected events were detected (e.g. isolated students or groups that were not using mandatory resources) that helped the teacher take regulatory measures.
Teachers highlighted that the monitoring helped them to save time to follow the progress of the CSCL situations. By means of the monitored reports, they could realize at a glance whether there was any potentially critical situation, preventing them from going through all the resources in order to check the progress of the activities. 3
The work presented in this paper is framed within a research proposal where the pedagogical decisions described in a CSCL script guide a monitoring process to better satisfy teachers’ awareness needs. To support this aim, we have studied the factors that influence the monitoring processes, and based on them we have proposed a script-aware monitoring model.
Albeit the concepts presented in this paper, separately, are not new, the main value of our proposal relies on the fact that the model relates concepts commonly applied in existing scripting and monitoring practice. Thus the model we propose is offered to coordinate and align these two existing practices.
This proposal has been put into practice in three higher-education CSCL scenarios. The monitoring results provided during the enactment have demonstrated to be helpful not only for facilitating the regulation tasks but also for saving teachers’ time.
Future work lines include three main threads. First, this model will be the base for the integration of monitoring issues into existing (or new) authoring tools, in order to generate monitorable scripts. Secondly, this work is connected to the effort towards the gathering and integration of monitoring data in distributed learning environments. And finally, it will be necessary to formalize a script-aware monitoring process that supports teachers during enactment of CSCL scenarios. 1 http://pandora.tel.uva.es/wic2/ 2 http://web.lkldev.ioe.ac.uk/PPC/ODC.html 3 http://www.gsic.uva.es/glueps/
This research has been partially funded by the Spanish Ministry of Economy and Competitivity (projects TIN2011-28308-C03-02 and IPT-430000-2010-054) and the Autonomous Government of Castilla y Leo´n (projects VA293A11-2 and VA301B11-2).