In this workshop paper, we report a study conducted to investigate the use of tracking technologies to measure the teachers' orchestration load when conducting colocated collaborative learning activities. We distinguish the orchestration load experienced by the teachers in the absence and presence of teacher supporting tools, i.e. teacher-facing dashboards. Electrodermal activity (EDA) sensor and other multimodal data including observations, log data and subjective responses to questionnaires have been collected to measure the teachers' orchestration load in authentic collaborative learning scenarios. This workshop paper presents the study context, quantitative and qualitative data collection process undertaken and other considerations in detail.
Two female teachers from a Spanish University participated in the experiments. Teachers had prior
experience in conducting collaborative learning activities and have used dashboard applications to
orchestrate collaboration. Each teacher conducted three collaborative learning activities and students
from the respective classes took part in the study with their informed consent. Each collaborative
learning activity lasted around nine minutes.
2.2
Procedure
Before the classroom trials, to generate appropriate baseline data, teachers were asked to wear the
EDA sensor for two hours for three days and mark the events of those days that were out of the
ordinary working activities. The measurement of two hours per day, was taken during working hours
when teachers conduct work activities outside of the classroom. In this way workload exists, but it is
not affected by the teaching itself and the presence of students and tools used during lessons.
After collecting baseline data, collaborative learning activities were conducted in classroom sessions.
A web-based tool called PyramidApp
(Manathunga & Hernández-Leo, 2018)
. that implements the
Pyramid pattern based on collaborative learning activities was used to design and deploy
collaboration. In the experimental condition, teachers monitored and orchestrated the group
activities using a teacher-facing dashboard; whereas the dashboard was not available in the control
condition. The experimental condition was subdivided into two conditions based on the presence of
certain warnings in the dashboard. For instance, in Dashboard condition I, the dashboard generated
several warnings when; 1) students answers does not contain any keyword that was stated by the
teacher during activity design time, 2) students skipped answer submissions, 3) students require more
time for collaboration, 4) collaborative learning activity reaches the end. In the Dashboard condition
II, the aforementioned warnings were turned off, but all the other features of the dashboard were
available.
2.3
Data collection and analysis
At the beginning of each collaborative learning session we attached the Shimmer3 GSR+ sensor to the
teacher by connecting two electrodes to the wrist and putting arm band that holds the sensor around
the teacher’s arm. The sensor is placed on the non-dominant hand to avoid discomfort to the teacher
and reduce the noise produced by the movement (see Figure 1).
The sensor is mounted before the beginning of the activity and removed right after. Recording begins
as soon as the sensor is removed from the docking station connected to the computer, so that the
signal captured between this moment and the beginning of the activity, is being removed from the
analysis. The same action is applied at the end of the recording. Signal captured between the end of
the activity and connecting the sensor back to the docking station (end of recording) is being removed.
Data transfer from the device was conducted immediately after the activity. Moreover, teacher’s
behaviour during every session was recorded either using a video camera or by a researcher taking
observation notes based on the unique requirements of each classroom session. In the experimental
sessions teacher’s dashboard actions were automatically logged. Teachers’ subjective measurements
of the cognitive load experienced in both control and experimental sessions were also collected using
NASA’s TLX questionnaire
(Hart & Staveland, 1988)
. Stimulated-recall interviews were also conducted
with the teacher to better understand their orchestration requirements and pedagogical
decisionmaking (see Figure 2).
The addition of supporting tools to synchronous collaborative settings could facilitate teachers to
diagnose collaboration
(van Leeuwen, 2015)
. LA dashboards have been seen as a promising tool that
can assist to raise teacher awareness, reflection and sense-making on peer learning activity
engagement and to impact behavior
(van Leeuwen, 2015)
. In this study we have collected qualitative
and quantitative data in different modalities in order to measure orchestration load experienced by
the teachers. A mixed-method approach will be used with the triangulation of quantitative and
qualitative data to warrant results about the three conditions. We will analyse the collected data to
explore how multimodal data can be used as indicators to measure teachers’ orchestration load in
order to propose orchestration load aware design guidelines for teacher-facing dashboards.
ACKNOWLEDGEMENTS
This work has been partially funded by FEDER, the national research agency of the Spanish Ministry
of Science, Innovations and Universities MDM-2015-0502, TIN2017-85179-C3-3-R.
Dillenbourg
,
P.
,
Nussbaum
,
M.
,
Dimitriadis
,
Y.
, &
Roschelle
,
J.
(
2013
).
Design for classroom orchestration
.
Computers & Education
,
69
(
0
),
485
-
492
.
Hart
,
S. G.
, &
Staveland
,
L. E.
(
1988
).
Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research
. Advances in psychology,
52
, (pp.
139
-
183
).
Jivet
,
I.
,
Scheffel
,
M.
,
Specht
,
M.
, &
Drachsler
,
H.
(
2018
).
License to evaluate: Preparing learning analytics dashboards for educational practice
.
Proceedings of the 8th International Conference on Learning Analytics and Knowledge
(pp.
31
-
40
).
Manathunga
,
K.
, &
Hernández-Leo
,
D.
(
2018
).
Authoring and enactment of mobile pyramid-based collaborative learning activities
.
British Journal of Educational Technology
,
49
(
2
),
262
-
275
.
van Leeuwen
,
A.
(
2015
).
Learning analytics to support teachers during synchronous CSCL: Balancing between overview and overload
.
Journal of learning Analytics
,
2
(
2
),
138
-
162
.