-
Data-intensive approaches aimed at understanding and supporting learning, such as
Learning Analytics, Educational Data Mining, Intelligent Tutoring Systems and
Artificial Intelligence in Education, have mostly been focused on the analysis of students’
interactions with particular learning systems and applications
(Khalil & Ebner, 2016;
Roll & Wylie, 2016)
. The student behavioural data that are commonly logged and
analysed mostly correspond to the interactions captured by personal computers or,
more recently, mobile devices. Although mobile and emerging pervasive technologies
have extended capabilities to sense some aspects of the usage context, most student
data used to model students’ behaviours/strategies or to provide automated feedback
are still limited to the interactions between students and learning applications.
However, learning goes beyond students’ interactions with user digital interfaces. Learning
may for example occur while students interact face-to-face with other students or with
their teachers. Alternatively, some learning tasks may require students to interact with
an ecology of non-digital physical tools, to use the physical space indoors and/or
outdoors; or to learn in different ways that cannot be mediated by traditional user
interfaces (e.g. motor and/or audio-visual learning)
(Santos, 2016)
. Multimodal learning
analytics (MMLA) initiatives have been the most robust approach for considering the
complexity of learning tasks
(Blikstein, 2013)
. Multimodal approaches have focused
on methods to integrate data corresponding to alternative dimensions of student
activity besides clickstreams and keystrokes. For example, multimodal learning
analytics have included approaches for automatically analyse speech, handwriting, sketch,
gesture, affective states and neurophysiological signals. However, although there have
been numerous advances in this area, most of the MMLA studies have been
conducted under controlled laboratory conditions
(Blikstein & Worsley, 2016)
. There is still
much work needed to find ways in which these multimodal approaches can solve
challenges in more realistic, mainstream learning scenarios.
This paper raises the question of how learning analytics can be created for physical
learning spaces and learning tasks that include physical activities. This includes the
characteristics of the infrastructure needed, the new features and dimensions of
student data that need to be created. The key overarching questions motivate this
position paper are: How can we envisage the provision of automated support or feedback
to students for tasks where physicality has an important place? How can we sense
student usage of the physical spaces and objects? How can we sense students’
mobility in the learning space? Can we find patterns of learners’ interactions in these
physical settings as we have been doing in computer-mediated settings? If so, what
particular techniques are appropriate for analysing and making sense of the data? Are there
any particular ethical implications or risks in exploring data from physical settings
that were not present with computer-mediated learning systems? The paper is aimed
at motivating our discussion through a series of questions that justify the importance
of designing physical learning analytics innovations. These questions emerged from
recent literature in learning analytics, technology, enhanced learning and
humancomputer interaction, more broadly. We focus our position particularly on
understanding the possible preliminary avenues of research where mobility, proximity and
motion analytics can help us respond questions about or support both learning in
formal and informal educational contexts where the physicality of the space, the task or
the learning may be paramount.
Important?
In this section we discuss a number of learning tasks, modalities and/or educational
activities where physicality of interactions or learning processes can be supported by
learning analytics approaches.
2.1
F-formations in Face-to-face Collaboration.
Learning from others and with others involves physicality to a great extent. When
collaborating face-to-face, people do not only communicate verbally but also through
gestures, postures, presence and other non-verbal cues
(Walther et al., 2005)
. In
addition to these non-verbal communication modes, people also may use the space or
multiple artefacts and objects in the collaborative setting. Kendon (1990) defined that
a key spatial aspect in face-to-face collaboration refers to the physical arrangement
that group members assume around devices or among themselves. These socially and
physically situated arrangements are known as f-formations. F-formations are
concerned with the proximity and body orientation that collaborators feature during
collaborative sessions, which can be indicative of how people position themselves as and
within a group. A recent example of this aspect studied from a learning analytics
perspective was presented by Thompson et al. (2016) who used a computer vision
technique based on video recordings to track collaborators working in a Design Studio.
This study suggests that the mobility trajectories of people in the learning space can
reflect higher order patterns of collaboration. For example, the most engaged
collaborators may show more complex mobility patterns for tasks that require the interaction
of collaborators with multiple devices. By contrast, for tasks that require initial
planning and discussion, mobility patterns can highlight groups that skip this phase and go
straight to hands-on work. Similarly, the first author and colleagues are investigating
mobility data of training nurses around medical beds during simulation labs
(Martinez-Maldonado et al., 2017)
. In this case, the students are tracked using a depth
sensor. The mobility data was wrangled to generate heatmaps of activity around the
patient’s bed. By analysing the heatmaps using time series, some initial visually
assessed patterns emerged and suggested the presence of patterns that can be associated
with distinct types of epistemic approaches to the task. Raca et al. (2014) also
explored how motion data obtained with computer vision algorithms can provide
insights about student’s actions (and those of student’s neighbours) during a lecture.
Some questions that may be followed up in this area include:
What are the kinds of tasks and learning scenarios where various
fformations naturally emerge among collaborators?
How can we measure or evaluate the impact (if it exists) of f-formations on
group performance, learning and collaboration?
How can we capture and integrate other behavioural data while group
members collaborate face-to-face?
How can we link and synchronise mobility data about collaborators with
other activity data that is already being captured (e.g. from the online
learning system, social networks, etc.)?
How can we incorporate contextual information (e.g. aspects of the
learning/cognitive process, epistemic approaches, behavioural cues) to location
data to enhance the sense making process?
2.2
Micromobility in co-present device ecologies.
A second aspect that can be tracked in co-present collaboration corresponds to the
concept of Micromobility. This describes how people orient and tilt objects or devices
towards one another to share information or jointly reflect based on specific data.
Being able to track, analyse and visualise behavioural data linked to this concept can
critical for face-to-face learning or reflection scenarios (the latter, where a group of
students/educators need to make sense of their own data for example). An example of
this approach was presented by Marquardt et al. (2012) who used kinects and
accelerometers to capture information about both f-formations and micromobility.
Although these authors provided collaborators with non-learning related, quite controlled
tasks, they found very distinctive patterns among groups, particularly in the different
ways collaborators interact with objects and share information. This demonstrates that
even small data points captured by the digital devices in use, such as tilting a screen to
allow others to look at the same information, may be indicative of key moments in
collaboration. This is an area that does not seem to have been explored in learning
contexts yet. Some questions that may be followed up in this area include:
Is it possible to distinguish explicit student’s actions and intentions from
implicit micro-actions and micro-interaction data captured from the devices
(e.g. accelerometer data and angle of the device)?
What data processing techniques would be needed to merge and pre-process
these data?
What algorithms and approaches would be needed to classify micro-mobility
actions effectively?
What are the ethical and technical implications of pervasively tracking these
micro data?
2.3
Social interaction, peer communication and networking.
Pentland and colleagues
(Eagle & Pentland, 2006; Kim et al., 2008)
pioneered in the
exploration of using data mining techniques to look for patterns within social
networks in physical environments. For tracking face-to-face interactions at a wider scale
(e.g. within an organisation, at a conference or in public events), they developed the
sociometric badges. These sensors can track basic aspects of social interaction such as
whether two people were talking to each other, levels of voice, and movement. We
are exploring the feasibility of understanding the social networks formed by students
when learning to dance. They are aiming to use mobile technologies and indoor
localisation technologies to understand how students interact with other students, with
different levels of dancing expertise, and how these interactions shape their own
learning paths according to their intrinsic motivations. We envisage that these kinds
of social interaction data can be exploited through social network analysis for
generating understanding in learning environment where collaboration happens not only in
small groups, but also through small and heterogeneous interactions within the
community. Additionally, it may be possible to learn from the more mature area within
learning analytics that have explored patterns within digital social networks. Some
questions that may be followed up in this area include:
Which learning scenarios would benefit from mapping the physical world
social networks that students interact in?
What alternative technological solutions could be used to capture social,
physical interaction data in a sustainable manner?
Can social network analysis techniques be applied to physical social net
analysis? What are the ethical issues of tracking activity from students’
physical social networks?
2.4
Teacher analytics in the classroom.
To a large extent, classrooms still play a critical role for building lifelong skills for
21st Century learners
(O'Flaherty & Phillips, 2015)
. Besides the diversity in
architectural formats, the classroom still basically allows educators to interact with students
and provide feedback in situ. The physicality of the classroom is an aspect in
education that has been quite overlooked by most learning analytics initiatives in all
educational levels. The analysis of mobility of the teacher or the students in the classroom
may provide new insights about things that occur in the classroom such as the
provision of feedback, the communication among students and with the teacher, or the
identification of inactive students. One example of the potential of this type of
analytics was suggested by Martinez-Maldonado et al. (2015) who demonstrated the
usefulness of manually tracking the teacher’s mobility in the classroom in order to
understand the impact of the feedback that the teacher provided to the students working in
small teams. Other approaches have focused on analysing teacher’s actions using
video analysis and other computer vision approaches
(Echeverría et al., 2014)
. More
recently, Prieto et al. (2016) presented a more elaborated approach to collect teaching
analytics automatically using accelerometer data, EEG, audio, video and eye trackers’
data to create, what authors call, ‘orchestration graphs’. These can potentially be
effective indicators of the kinds of learning and teaching processes that occur in
face-toface classrooms. Some questions that may be followed up in this area include:
In which learning scenarios it would be important to know the actions
performed by the teacher (besides small group collaboration classrooms)?
What are the implications of teaching analytics for learning design or for
measuring instructional performance?
What are the ethical implications of using these data for evaluation (of the
teacher)?
What technological innovations would be needed to implement in regular
classrooms to perform teaching analytics at scale?
2.5
Motor learning.
The acquisition of psychomotor or kinaesthetic skills is crucial for many kinds of
tasks associated with both formal and informal learning
(Harrow, 1972)
. Examples
include learning to play a musical instrument, learning a sign language, dancing,
improving handwriting, drawing, training surgical or clinical interventions, improving
the technique in sports, practicing martial arts, etc. Santos (2016) has recently
highlighted both the importance of supporting these types of widely diverse and important
educational tasks and also the potential that data and analytics can offer to leverage
motor learning. This is becoming feasible because of the widespread emergence of
pervasive sensors (e.g. wearable devices); more advanced and less expensive
computer vision devices (e.g. depth/infrared cameras); and more reliable computer vision
algorithms. From a multimodal learning analytics perspective, motor learning has
started to be addressed through action and gesture analysis
(Blikstein & Worsley,
2016)
. Representative examples of this approach include the recognition of human
activity using computer vision
(e.g. [Yilmaz and Shah, 2005])
or identifying gestures
that differentiate experts from novices
(e.g. [Worsley and Blikstein, 2013])
. Key
questions in this area that remain unanswered include:
What motor learning or hybrid learning tasks could be supported using
mobility, proximity or motion analytics?
What particular pedagogical/epistemic stance would be required?
Is motor learning a whole different domain of learning that should be
supported differently by emerging learning analytics, or is it just another
dimension of human activity that can be tackled through multimodal approaches?
What kinds of analytics may be useful for informal education scenarios that
involve the development of motor skills?
2.6
Learning in and from physical spaces.
The areas discussed above are not necessarily comprehensively covering all the
possible learning tasks that can be supported by using mobility, proximity and motion
analytics. Other examples include learning tasks that require field work and that are
more commonly being supported by mobile
(e.g. [Carvalho and Freeman, 2016])
or
augmented reality (e.g. [Muñoz-Cristóbal et al., 2014]) technologies. In these
scenarios, students can be encouraged to explore the physical space, which can be in the
school, in natural areas or in the city, to complete tasks. These may not only require
the student to access information or content online but also make sense of it and
associate it with the physical context where s/he is. Even it would be possible that students
need to access information through embodied interaction modes (e.g. perform tasks or
gain access to information depending on their physical location or proximity). Data
obtained from localisation and usage logs, and the application of learning analytics
techniques, could unveil patterns of the processes that students follow or generate
while learning in the physical space. Some questions that may be followed up in this
area include:
What formal and informal educational tasks invite or require students to
explore and interact with the physical space where the learning activity
unfolds?
What kind of data, besides indoor/outdoor localisation, can be captured in
physical spaces?
What kind of sensemaking can be performed on location data?
What kind of analytics innovations could improve learning in physical
spaces?
What are the ethical implications and risks of exploiting these location data
for learning analytics?
3