=Paper=
{{Paper
|id=None
|storemode=property
|title=Script-Aware Monitoring Model: Using Teachers' Pedagogical Intentions to Guide Learning Analytics
|pdfUrl=https://ceur-ws.org/Vol-894/paper4.pdf
|volume=Vol-894
}}
==Script-Aware Monitoring Model: Using Teachers' Pedagogical Intentions to Guide Learning Analytics==
https://ceur-ws.org/Vol-894/paper4.pdf
Script-Aware Monitoring Model: Using Teachers’
Pedagogical Intentions to Guide Learning Analytics
Marı́a Jesús Rodrı́guez-Triana, Alejandra Martı́nez-Monés, Juan Ignacio
Asensio-Pérez, and Yannis Dimitriadis
GSIC-EMIC, University of Valladolid, Spain
{chus@gsic, amartine@infor, juaase@tel, yannis@tel}.uva.es
http://www.gsic.uva.es
Abstract. In computer-supported collaborative learning, teachers do not usually
get the information they need for regulating a learning situation during its en-
actment. To address this need, we propose to provide teachers with monitoring
feedback guided by their pedagogical intentions. In particular, those pedagogical
intentions captured in the scripts where teachers reflect their learning designs.
In order to configure monitoring in alignment with such pedagogical intentions, a
model that gathers all the necessary data elements is required. This paper presents
this “script-aware monitoring model”, and the teachers’ positive opinions about
its usefulness and efficiency from three pilot studies where the model was tested.
Keywords: CSCL, learning design, scripting, monitoring, learning analytics
1 Introduction
According to the STELLAR European Network of Excellence in Technology Enhanced
Learning (TEL), one of the main challenges for future teachers and learners is to “make
sense and intelligent use of the data provided by information and communication tech-
nologies (ICTs) in order to facilitate learning” [12]. To achieve that aim STELLAR
proposes, among other alternatives, to inform teachers about their students’ progress
and success, and highlights the need of investigating what kind of data teachers require
for monitoring such learning process of their students. Learning Analytics has recently
emerged as a research field aiming to address these and other educational data issues.
Learning Analytics collects available data about learners and their contexts, and
identifies relevant feedback from it. Techniques and methods commonly used for this
purpose include a posteriori social network analysis, discourse and concept analysis, or
learner success prediction, with a strong emphasis on meaningful visualizations [10].
In order to support teachers in monitoring students, we propose an a priori guidance of
the data analysis, using the pedagogical intentions set in their learning designs [8].
Our proposal is framed in the field of Computer-Supported Collaborative Learning
(CSCL), in which scripting is a well-known strategy followed by teachers when design-
ing learning situations [1]. Therefore, we will use the script as a source of information
regarding the pedagogical decisions that will guide the monitoring process.
This paper presents on-going work towards the definition of a script-aware mon-
itoring model that brings together the elements involved in scripting and monitoring.
2 M. J. Rodrı́guez-Triana et al.
This document analyzes the required parameters for configuring the monitoring pro-
cess of a CSCL scenario, indicating which ones are retrievable from the script and
which ones must be provided additionally by the teacher. This model has been used in
three pilot studies that involved two different teachers in order to validate whether our
proposal is useful and efficient for them. This paper also includes preliminary feedback
from the participant teachers, who evaluated positively the monitoring results obtained.
2 Script-Aware Monitoring Model
We have drawn from two trends in the CSCL field for the definition of the script-aware
monitoring model: a) CSCL scripting (since it has dealt with the modeling of learning
scenarios in order to structure the collaborative learning process [9]); and b) the analysis
of computer-mediated interactions (since it has emphasized modeling interactions in
order to process them [11]). After reviewing these two approaches, this section presents
the script-aware monitoring model itself, and a summary of the opinions given by the
participant teachers in three pilot studies where the model was employed.
2.1 Review of Related Modeling Approaches
Multiple authors in the field of CSCL agree on the elements that conform a script [13]
[2] [3] [7] [6]. Scripts can be broken down into five main components (roles, activities,
participants, groups, and resources) that are interrelated by means of three mecha-
nisms (task distribution among groups and roles, group formation, and sequencing of
activities). These concepts can even be made computer-interpretable by means of Edu-
cational Modeling Languages such as IMS Learning Design [5].
Another trend of research that provides relevant input for our goal comes from
the work on computer-based interaction analysis [11]. The need to process interaction
data by automatic means has led researchers to specify the components of interaction.
Among these efforts, it is worth mentioning here the data format defined within the
Kaleidoscope Network of Excellence in TEL, where a number of researchers collabo-
rated to define the so-called “Common Format” [4]. This format had the main objective
of enabling interoperability among learning and analysis tools. It therefore defines a
minimum set of elements that every log event should include in order to be analyzable
by a computer-based interaction analysis tool.
The elements defined in this Common Format are divided into two branches. The
context is the general setup of a learning situation (users, roles of the participants,
groups, resources). The second branch describes what happens during the learning ac-
tivity, i.e., the actions carried out by participants, identifying who has done what, and
when. “Who” is one user previously identified in the context branch, “what” is type
of action among those allowed within the specific learning environment or tool, and
“when” is the timestamp of the event.
However, existing proposals in the scripting and interaction analysis fields are not
suitable for guiding monitoring on the basis of pedagogical intentions. Scripting pro-
posals cannot be applied for monitoring mainly because they do not consider the users’
actions during the learning process. Regarding interaction analysis, the studied models
Script-Aware Monitoring Model 3
do not take into account a core scripting concept, namely learning activities. Therefore,
we will propose a monitoring model that brings together both scripting and interaction
analysis approaches, including the elements shown in Figure 1.
Fig. 1. Relation between the elements that conform scripting and interaction analysis models.
2.2 Monitoring Model Definition
The aim of this model is to collect all the elements required to guide a monitoring pro-
cess on the basis of a CSCL script. Thus, our proposed solution contains elements that
model the script (participants, groups, roles, activities and resources), and elements
that model the events gathered during the enactment of the script (actions). Figure 2 de-
picts scripting elements with continuous yellow circles, and monitoring elements with
dashed blue circles. Additionally, some attributes of the aforementioned elements need
to be specified, as we have detected in several case studies. Those that need to be pro-
vided by the teacher have been represented with bold red boxes in Figure 2. The at-
tributes that are relevant for configuring the monitoring process are briefly presented
below, together with illustrative examples:
- Group formation policies: the knowledge of how groups are structured simplifies
the visualization of relevant interactions and the detection of unexpected eventualities
such as isolated students.
- Interactivity type: specifying how students are expected to interact (face-to-face
/ computer-mediated / blended) helps to guide the selection of monitored actions that
may inform about the activity progress.
- Activity social level: individual, group and class-wide activities are interspersed in
the learning situation. Knowing the social level of an activity can help us in choosing
the way of analyzing the activity events. If an activity is individual, it will be necessary
to know how each participant evolves and interacts with the context (use of the ICT
tools, attendance to the lectures, etc.). However, if the activity is carried out by groups
(or by the whole class), it will be relevant to know if there is evidence of participation
and collaboration among group members.
- Activity deadlines: knowing when an activity starts and finishes, limits the data
gathering to only the relevant time period. This prevents showing the teacher irrelevant
information, and reduces the amount of data to be processed, facilitating the generation
of timely results during the enactment of the learning situation.
- Dependences between activities: learning activities within a CSCL scenario are
usually related to one another. Sometimes there are time dependences (e.g. when one
4 M. J. Rodrı́guez-Triana et al.
activity finishes another begins), shared resources (e.g. the output of one activity is
the input of another), etc. Identifying in advance these dependences facilitates useful
monitoring advice, e.g., when the current state of one activity may have a negative
impact on another.
- Expected use of resources: monitoring feedback may be improved by means of ad-
ditional information about the way resources should be used within the learning activity
(e.g. whether one resource is mandatory as opposed to optional, whether participants are
expected to use a resource individually or in groups).
- Action timestamp: in order to inform the state of the previous parameters, it is
necessary to be able to locate the action in time. Thus, the timestamp indicates whether
an action should be taken into account or not, according to the activity deadlines.
- Action type: identifying the nature of the participants’ actions can help us choose
the most suitable data sources for monitoring (e.g. checking the attendance to a session
may be relevant for a face-to-face activity, but not for an online, asynchronous one).
Fig. 2. Elements and attributes of the script-aware monitoring model. Continuous yellow cir-
cles represent scripting elements, dashed blue circles indicate monitoring elements, and bold red
boxes point to additional information to be specified by the teacher.
Although these elements and attributes may seem too general, or even trivial, they
are highly relevant during the enactment in order to guide monitoring. For instance,
monitored actions can provide evidence to support that, during the period set by the
activity deadlines: (a) participants are interacting and collaborating as it was specified,
(b) participants are using the resources as it was expected, (c) group formation policies
are verified, or (d) the current situation of the activity jeopardizes other activities in
terms of sequencing, group formation or reuse of resources.
2.3 Teachers’ Feedback on the Use of the Model
In order to evaluate the validity of the model, we have employed it for monitoring
three pilot studies, carried out at the University of Valladolid from February to May
2012. All three CSCL situations lasted for 3-4 weeks, interleaving face-to-face with
distance activities mediated by several ICT tools. Two of them were carried out with 14
master-level students and an experienced teacher, and the last one with 60 undergradu-
ate students and a non-expert teacher. While a complete evaluation of these studies is in
Script-Aware Monitoring Model 5
progress, in this section we present the teachers’ opinions collected in semi-structured
interviews at the end of the learning situations.
In these pilot studies, teachers were first asked to create their designs using author-
ing tools (namely Web Instance Collage1 and Pedagogical Pattern Collector2 ) as well
as deployment tools for CSCL scripts (GLUE!-PS3 ). Then, teachers enriched the scripts
filling a form with the additional information required by the model. Finally, during the
enactment, teachers received the monitoring results based on the script decisions.
Both teachers confirmed that specifying the aforementioned additional attributes to
their scripts required little effort in terms of time and complexity, and that they consid-
ered it useful to reflect on those parameters at design-time.
During the enactment, monitoring results showed information to the teachers which
they had not realized yet. Teachers argued that, in most cases, monitoring results helped
them to confirm that the students were following properly the script (e.g. collaborating
as expected or submitting reports on time). Additionally, some unexpected events were
detected (e.g. isolated students or groups that were not using mandatory resources) that
helped the teacher take regulatory measures.
Teachers highlighted that the monitoring helped them to save time to follow the
progress of the CSCL situations. By means of the monitored reports, they could realize
at a glance whether there was any potentially critical situation, preventing them from
going through all the resources in order to check the progress of the activities.
3 Conclusions and Future Work
The work presented in this paper is framed within a research proposal where the ped-
agogical decisions described in a CSCL script guide a monitoring process to better
satisfy teachers’ awareness needs. To support this aim, we have studied the factors that
influence the monitoring processes, and based on them we have proposed a script-aware
monitoring model.
Albeit the concepts presented in this paper, separately, are not new, the main value
of our proposal relies on the fact that the model relates concepts commonly applied
in existing scripting and monitoring practice. Thus the model we propose is offered to
coordinate and align these two existing practices.
This proposal has been put into practice in three higher-education CSCL scenarios.
The monitoring results provided during the enactment have demonstrated to be helpful
not only for facilitating the regulation tasks but also for saving teachers’ time.
Future work lines include three main threads. First, this model will be the base for
the integration of monitoring issues into existing (or new) authoring tools, in order to
generate monitorable scripts. Secondly, this work is connected to the effort towards the
gathering and integration of monitoring data in distributed learning environments. And
finally, it will be necessary to formalize a script-aware monitoring process that supports
teachers during enactment of CSCL scenarios.
1 http://pandora.tel.uva.es/wic2/
2 http://web.lkldev.ioe.ac.uk/PPC/ODC.html
3 http://www.gsic.uva.es/glueps/
6 M. J. Rodrı́guez-Triana et al.
Acknowledgements
This research has been partially funded by the Spanish Ministry of Economy and Com-
petitivity (projects TIN2011-28308-C03-02 and IPT-430000-2010-054) and the Au-
tonomous Government of Castilla y León (projects VA293A11-2 and VA301B11-2).
References
1. Dillenbourg, P., Hong, F.: The mechanics of CSCL macro scripts. International Journal of
Computer-Supported Collaborative Learning 3, 5–23 (2008)
2. Dillenbourg, P., Tchounikine, P.: Flexibility in macro-scripts for CSCL. Journal of Computer
Assisted Learning 23(1), 1–13 (2007)
3. Fischer, F., Kollar, I., Haake, J., Mandl, H.: Perspectives on collaboration scripts. In: Fis-
cher, F., Mandl, H., Haake, J., Kollar, I. (eds.) Scripting computer-supported communication
of knowledge – cognitive, computational, and educational perspectives, pp. 1–10. Springer,
New York (2007)
4. Harrer, A., Martı́nez Monés, A., Dimitracopoulou, A.: Users’ data: collaborative and social
analysis. In: Balacheff, N., Ludvigsen, S., de Jong, T., Lazonder, A., Barnes, S., Montandon,
L. (eds.) Technology-Enhanced Learning. Principles and Products., pp. 175–193. Springer,
UK (2009)
5. IMS Global Learning Consortium: IMS Learning Design Specifications (2003), retrieved
from http://www.imsglobal.org/learningdesign/. Last visit: July 2012
6. Kobbe, L., Weinberger, A., Dillenbourg, P., Harrer, A., Hämäläinen, R., Häkkinen, P.,
Fischer, F.: Specifying computer-supported collaboration scripts. International Journal of
Computer-Supported Collaborative Learning 2, 211–224 (2007)
7. Kollar, I., Fischer, F., Hesse, F.: Collaboration Scripts – A Conceptual Analysis. Educational
Psychology Review 18, 159–185 (2006)
8. Lockyer, L., Dawson, S.: Learning designs and learning analytics. In: Proceedings of the
1st International Conference on Learning Analytics and Knowledge. pp. 153–156. LAK ’11,
ACM, Banff, Alberta, Canada (2011)
9. Miao, Y., Hoeksema, K., Hoppe, H., Harrer, A.: CSCL scripts: modelling features and po-
tential use. In: Proceedings of the 6th International Conference on Computer Support Col-
laborative Learning: the next 10 years! pp. 423–432. CSCL’05, International Society of the
Learning Sciences, Taipei, Taiwan (2005)
10. Siemens, G., Baker, R.: Learning Analytics and Educational Data Mining: Towards Commu-
nication and Collaboration. In: Proceedings of the 2nd International Conference on Learning
Analytics and Knowledge. LAK ’12, Vancouver, British Columbia, Canada (2012)
11. Soller, A., Martı́nez, A., Jermann, P., Muehlenbrock, M.: From Mirroring to Guiding: a re-
view of the state of the art in interaction analysis. International Journal on Artificial Intelli-
gence in Education 15, 261–290 (2005)
12. Sutherland, R., Eagle, S., Joubert, M.: A Vision and Strategy for Technology Enhanced
Learning. Tech. rep., STELLAR - European Network of Excellence in Technology Enhanced
Learning (2012), retrieved from http://www.teleurope.eu/mod/file/download.php?
file_guid=152343. Last visit: July 2012
13. Weinberger, A., Kollar, I., Dimitriadis, Y., Mäkitalo-Siegl, K., Fischer, F.: Computer-
Supported Collaboration Scripts: Perspectives from Educational Psychology and Com-
puter Science. In: Balacheff, N., Ludvigsen, S., Jong, T., Lazonder, A., Barnes, S. (eds.)
Technology-Enhanced Learning. Principles and Products, pp. 155–173. Springer Nether-
lands (2009)