=Paper=
{{Paper
|id=Vol-1828/paper-06
|storemode=property
|title=How to Capitalise on Mobility, Proximity and Motion Analytics to Support Formal and
Informal Education?
|pdfUrl=https://ceur-ws.org/Vol-1828/paper-06.pdf
|volume=Vol-1828
|authors=Roberto Martinez-Maldonado,Augusto Dias Pereira Dos Santos,Vanessa Echeverria,Kalina Yacef,Mykola Pechenizkiy
|dblpUrl=https://dblp.org/rec/conf/lak/MaldonadoSEYP17
}}
==How to Capitalise on Mobility, Proximity and Motion Analytics to Support Formal and
Informal Education?==
https://ceur-ws.org/Vol-1828/paper-06.pdf
How to Capitalise on Mobility, Proximity and Motion
Analytics to Support Formal and Informal Education?
Roberto Martinez-Maldonado1, Vanessa Echeverria1, Kalina Yacef2,
Augusto Dias Pereira Dos Santos2, and Mykola Pechenizkiy3
1University of Technology Sydney, Australia
2The University of Sydney, Australia
3Eindhoven University of Technology, The Netherlands
Roberto.Martinez-Maldonado@uts.ed.au
Abstract: Learning Analytics and similar data-intensive approaches aimed at
understanding and/or supporting learning have mostly focused on the analysis
of students’ data automatically captured by personal computers or, more recent-
ly, mobile devices. Thus, most student behavioural data are limited to the inter-
actions between students and particular learning applications. However, learn-
ing can also occur beyond these interface interactions, for instance while stu-
dents interact face-to-face with other students or their teachers. Alternatively,
some learning tasks may require students to interact with non-digital physical
tools, to use the physical space, or to learn in different ways that cannot be me-
diated by traditional user interfaces (e.g. motor and/or audio learning). The key
questions here are: why are we neglecting these kinds of learning activities?
How can we provide automated support or feedback to students during these ac-
tivities? Can we find useful patterns of activity in these physical settings as we
have been doing with computer-mediated settings? This position paper is aimed
at motivating discussion through a series of questions that can justify the im-
portance of designing technological innovations for physical learning settings
where mobility, proximity and motion are tracked, just as digital interactions
have been so far.
Keywords: physical spaces, wearables, indoor localisation, sensors, mobility,
motor learning
In silence and movement you can show the reflection of people.
Marcel Marceau
1 Introduction
Data-intensive approaches aimed at understanding and supporting learning, such as
Learning Analytics, Educational Data Mining, Intelligent Tutoring Systems and Arti-
ficial Intelligence in Education, have mostly been focused on the analysis of students’
interactions with particular learning systems and applications (Khalil & Ebner, 2016;
Roll & Wylie, 2016). The student behavioural data that are commonly logged and
2
analysed mostly correspond to the interactions captured by personal computers or,
more recently, mobile devices. Although mobile and emerging pervasive technologies
have extended capabilities to sense some aspects of the usage context, most student
data used to model students’ behaviours/strategies or to provide automated feedback
are still limited to the interactions between students and learning applications. How-
ever, learning goes beyond students’ interactions with user digital interfaces. Learning
may for example occur while students interact face-to-face with other students or with
their teachers. Alternatively, some learning tasks may require students to interact with
an ecology of non-digital physical tools, to use the physical space indoors and/or out-
doors; or to learn in different ways that cannot be mediated by traditional user inter-
faces (e.g. motor and/or audio-visual learning) (Santos, 2016). Multimodal learning
analytics (MMLA) initiatives have been the most robust approach for considering the
complexity of learning tasks (Blikstein, 2013). Multimodal approaches have focused
on methods to integrate data corresponding to alternative dimensions of student ac-
tivity besides clickstreams and keystrokes. For example, multimodal learning analyt-
ics have included approaches for automatically analyse speech, handwriting, sketch,
gesture, affective states and neurophysiological signals. However, although there have
been numerous advances in this area, most of the MMLA studies have been conduct-
ed under controlled laboratory conditions (Blikstein & Worsley, 2016). There is still
much work needed to find ways in which these multimodal approaches can solve
challenges in more realistic, mainstream learning scenarios.
This paper raises the question of how learning analytics can be created for physical
learning spaces and learning tasks that include physical activities. This includes the
characteristics of the infrastructure needed, the new features and dimensions of stu-
dent data that need to be created. The key overarching questions motivate this posi-
tion paper are: How can we envisage the provision of automated support or feedback
to students for tasks where physicality has an important place? How can we sense
student usage of the physical spaces and objects? How can we sense students’ mobili-
ty in the learning space? Can we find patterns of learners’ interactions in these physi-
cal settings as we have been doing in computer-mediated settings? If so, what particu-
lar techniques are appropriate for analysing and making sense of the data? Are there
any particular ethical implications or risks in exploring data from physical settings
that were not present with computer-mediated learning systems? The paper is aimed
at motivating our discussion through a series of questions that justify the importance
of designing physical learning analytics innovations. These questions emerged from
recent literature in learning analytics, technology, enhanced learning and human-
computer interaction, more broadly. We focus our position particularly on under-
standing the possible preliminary avenues of research where mobility, proximity and
motion analytics can help us respond questions about or support both learning in for-
mal and informal educational contexts where the physicality of the space, the task or
the learning may be paramount.
3
2 Why are Mobility, Proximity and Motion Analytics
Important?
In this section we discuss a number of learning tasks, modalities and/or educational
activities where physicality of interactions or learning processes can be supported by
learning analytics approaches.
2.1 F-formations in Face-to-face Collaboration.
Learning from others and with others involves physicality to a great extent. When
collaborating face-to-face, people do not only communicate verbally but also through
gestures, postures, presence and other non-verbal cues (Walther et al., 2005). In addi-
tion to these non-verbal communication modes, people also may use the space or
multiple artefacts and objects in the collaborative setting. Kendon (1990) defined that
a key spatial aspect in face-to-face collaboration refers to the physical arrangement
that group members assume around devices or among themselves. These socially and
physically situated arrangements are known as f-formations. F-formations are con-
cerned with the proximity and body orientation that collaborators feature during col-
laborative sessions, which can be indicative of how people position themselves as and
within a group. A recent example of this aspect studied from a learning analytics per-
spective was presented by Thompson et al. (2016) who used a computer vision tech-
nique based on video recordings to track collaborators working in a Design Studio.
This study suggests that the mobility trajectories of people in the learning space can
reflect higher order patterns of collaboration. For example, the most engaged collabo-
rators may show more complex mobility patterns for tasks that require the interaction
of collaborators with multiple devices. By contrast, for tasks that require initial plan-
ning and discussion, mobility patterns can highlight groups that skip this phase and go
straight to hands-on work. Similarly, the first author and colleagues are investigating
mobility data of training nurses around medical beds during simulation labs
(Martinez-Maldonado et al., 2017). In this case, the students are tracked using a depth
sensor. The mobility data was wrangled to generate heatmaps of activity around the
patient’s bed. By analysing the heatmaps using time series, some initial visually as-
sessed patterns emerged and suggested the presence of patterns that can be associated
with distinct types of epistemic approaches to the task. Raca et al. (2014) also ex-
plored how motion data obtained with computer vision algorithms can provide in-
sights about student’s actions (and those of student’s neighbours) during a lecture.
Some questions that may be followed up in this area include:
What are the kinds of tasks and learning scenarios where various f-
formations naturally emerge among collaborators?
How can we measure or evaluate the impact (if it exists) of f-formations on
group performance, learning and collaboration?
How can we capture and integrate other behavioural data while group mem-
bers collaborate face-to-face?
How can we link and synchronise mobility data about collaborators with
other activity data that is already being captured (e.g. from the online learn-
ing system, social networks, etc.)?
4
How can we incorporate contextual information (e.g. aspects of the learn-
ing/cognitive process, epistemic approaches, behavioural cues) to location
data to enhance the sense making process?
2.2 Micromobility in co-present device ecologies.
A second aspect that can be tracked in co-present collaboration corresponds to the
concept of Micromobility. This describes how people orient and tilt objects or devices
towards one another to share information or jointly reflect based on specific data.
Being able to track, analyse and visualise behavioural data linked to this concept can
critical for face-to-face learning or reflection scenarios (the latter, where a group of
students/educators need to make sense of their own data for example). An example of
this approach was presented by Marquardt et al. (2012) who used kinects and accel-
erometers to capture information about both f-formations and micromobility. Alt-
hough these authors provided collaborators with non-learning related, quite controlled
tasks, they found very distinctive patterns among groups, particularly in the different
ways collaborators interact with objects and share information. This demonstrates that
even small data points captured by the digital devices in use, such as tilting a screen to
allow others to look at the same information, may be indicative of key moments in
collaboration. This is an area that does not seem to have been explored in learning
contexts yet. Some questions that may be followed up in this area include:
Is it possible to distinguish explicit student’s actions and intentions from im-
plicit micro-actions and micro-interaction data captured from the devices
(e.g. accelerometer data and angle of the device)?
What data processing techniques would be needed to merge and pre-process
these data?
What algorithms and approaches would be needed to classify micro-mobility
actions effectively?
What are the ethical and technical implications of pervasively tracking these
micro data?
2.3 Social interaction, peer communication and networking.
Pentland and colleagues (Eagle & Pentland, 2006; Kim et al., 2008) pioneered in the
exploration of using data mining techniques to look for patterns within social net-
works in physical environments. For tracking face-to-face interactions at a wider scale
(e.g. within an organisation, at a conference or in public events), they developed the
sociometric badges. These sensors can track basic aspects of social interaction such as
whether two people were talking to each other, levels of voice, and movement. We
are exploring the feasibility of understanding the social networks formed by students
when learning to dance. They are aiming to use mobile technologies and indoor local-
isation technologies to understand how students interact with other students, with
different levels of dancing expertise, and how these interactions shape their own
learning paths according to their intrinsic motivations. We envisage that these kinds
of social interaction data can be exploited through social network analysis for generat-
ing understanding in learning environment where collaboration happens not only in
small groups, but also through small and heterogeneous interactions within the com-
5
munity. Additionally, it may be possible to learn from the more mature area within
learning analytics that have explored patterns within digital social networks. Some
questions that may be followed up in this area include:
Which learning scenarios would benefit from mapping the physical world
social networks that students interact in?
What alternative technological solutions could be used to capture social,
physical interaction data in a sustainable manner?
Can social network analysis techniques be applied to physical social net
analysis? What are the ethical issues of tracking activity from students’ phys-
ical social networks?
2.4 Teacher analytics in the classroom.
To a large extent, classrooms still play a critical role for building lifelong skills for
21st Century learners (O'Flaherty & Phillips, 2015). Besides the diversity in architec-
tural formats, the classroom still basically allows educators to interact with students
and provide feedback in situ. The physicality of the classroom is an aspect in educa-
tion that has been quite overlooked by most learning analytics initiatives in all educa-
tional levels. The analysis of mobility of the teacher or the students in the classroom
may provide new insights about things that occur in the classroom such as the provi-
sion of feedback, the communication among students and with the teacher, or the
identification of inactive students. One example of the potential of this type of analyt-
ics was suggested by Martinez-Maldonado et al. (2015) who demonstrated the useful-
ness of manually tracking the teacher’s mobility in the classroom in order to under-
stand the impact of the feedback that the teacher provided to the students working in
small teams. Other approaches have focused on analysing teacher’s actions using
video analysis and other computer vision approaches (Echeverría et al., 2014). More
recently, Prieto et al. (2016) presented a more elaborated approach to collect teaching
analytics automatically using accelerometer data, EEG, audio, video and eye trackers’
data to create, what authors call, ‘orchestration graphs’. These can potentially be ef-
fective indicators of the kinds of learning and teaching processes that occur in face-to-
face classrooms. Some questions that may be followed up in this area include:
In which learning scenarios it would be important to know the actions per-
formed by the teacher (besides small group collaboration classrooms)?
What are the implications of teaching analytics for learning design or for
measuring instructional performance?
What are the ethical implications of using these data for evaluation (of the
teacher)?
What technological innovations would be needed to implement in regular
classrooms to perform teaching analytics at scale?
2.5 Motor learning.
The acquisition of psychomotor or kinaesthetic skills is crucial for many kinds of
tasks associated with both formal and informal learning (Harrow, 1972). Examples
include learning to play a musical instrument, learning a sign language, dancing, im-
proving handwriting, drawing, training surgical or clinical interventions, improving
6
the technique in sports, practicing martial arts, etc. Santos (2016) has recently high-
lighted both the importance of supporting these types of widely diverse and important
educational tasks and also the potential that data and analytics can offer to leverage
motor learning. This is becoming feasible because of the widespread emergence of
pervasive sensors (e.g. wearable devices); more advanced and less expensive comput-
er vision devices (e.g. depth/infrared cameras); and more reliable computer vision
algorithms. From a multimodal learning analytics perspective, motor learning has
started to be addressed through action and gesture analysis (Blikstein & Worsley,
2016). Representative examples of this approach include the recognition of human
activity using computer vision (e.g. [Yilmaz and Shah, 2005]) or identifying gestures
that differentiate experts from novices (e.g. [Worsley and Blikstein, 2013]). Key
questions in this area that remain unanswered include:
What motor learning or hybrid learning tasks could be supported using mo-
bility, proximity or motion analytics?
What particular pedagogical/epistemic stance would be required?
Is motor learning a whole different domain of learning that should be sup-
ported differently by emerging learning analytics, or is it just another dimen-
sion of human activity that can be tackled through multimodal approaches?
What kinds of analytics may be useful for informal education scenarios that
involve the development of motor skills?
2.6 Learning in and from physical spaces.
The areas discussed above are not necessarily comprehensively covering all the pos-
sible learning tasks that can be supported by using mobility, proximity and motion
analytics. Other examples include learning tasks that require field work and that are
more commonly being supported by mobile (e.g. [Carvalho and Freeman, 2016]) or
augmented reality (e.g. [Muñoz-Cristóbal et al., 2014]) technologies. In these scenari-
os, students can be encouraged to explore the physical space, which can be in the
school, in natural areas or in the city, to complete tasks. These may not only require
the student to access information or content online but also make sense of it and asso-
ciate it with the physical context where s/he is. Even it would be possible that students
need to access information through embodied interaction modes (e.g. perform tasks or
gain access to information depending on their physical location or proximity). Data
obtained from localisation and usage logs, and the application of learning analytics
techniques, could unveil patterns of the processes that students follow or generate
while learning in the physical space. Some questions that may be followed up in this
area include:
What formal and informal educational tasks invite or require students to ex-
plore and interact with the physical space where the learning activity un-
folds?
What kind of data, besides indoor/outdoor localisation, can be captured in
physical spaces?
What kind of sensemaking can be performed on location data?
What kind of analytics innovations could improve learning in physical spac-
es?
7
What are the ethical implications and risks of exploiting these location data
for learning analytics?
3 Conclusions
This position paper aims at starting a discussion about the current approaches and the
future potential of learning analytics for supporting learning across physical spaces.
The learning analytics field and related fields have paid much attention to cognitive or
intellectual domains. There has also been a strong interest in supporting the affective
domain (Rogaten et al., 2016). It is now time to start supporting psychomotor skills
and/or the physicality aspects of a traditional intellectual domian, which are crucial
for the full development of a life-long learner. The lack of interest in this domain may
be affected by the regular pedagogies and the curricula which may not explicitly in-
clude this into the learning tasks. This is the reason why we need to also look at (the
so-called) informal learning activities, which have an important role in complement-
ing the more ‘thinking-oriented’ formal education. Nonetheless, the paper highlights
some examples of learning analytics innovations that are tackling this domain. The
questions posed for each area aim to trigger discussion and motivate formal studies to
support psychomotor learning through Mobility, Proximity and Motion Analytics.
Current and future work by authors is aiming to illustrate the feasibility and potential
of performing this kind of analytics through three case studies in three different con-
texts, including: i) health simulation labs; ii) a dance education studio; and iii) regular
small-group collaboration classrooms.
References
1. Blikstein, P. (2013). Multimodal learning analytics. In Proceedings of the Proceedings of
the Third International Conference on Learning Analytics and Knowledge, (pp. 102-106).
Leuven, Belgium. 2460316: ACM.
2. Blikstein, P., & Worsley, M. (2016). Multimodal Learning Analytics and Education Data
Mining: using computational technologies to measure complex learning tasks. Journal of
Learning Analytics, 3(2), 220-238.
3. Carvalho, L., & Freeman, C. G. (2016). CmyView: Walking together apart. Paper present-
ed at the Proceedings of the 10th International Conference on Networked Learning 2016,
(pp. 313-321). Unknown.
4. Eagle, N., & Pentland, A. (2006). Reality mining: sensing complex social systems. Perso-
nal and Ubiquitous Computing, 10(4), 255-268.
5. Echeverría, V., Avendaño, A., Chiluiza, K., Vásquez, A., & Ochoa, X. (2014). Presenta-
tion skills estimation based on video and kinect data analysis. Paper presented at the Pro-
ceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and
Grand Challenge, (pp. 53-60). ACM.
6. Harrow, A. J. (1972). A taxonomy of the psychomotor domain: A guide for developing be-
havioral objectives: Addison-Wesley Longman Ltd.
7. Kendon, A. (1990). Spatial organization in social encounters: The F-formation system.
Conducting interaction: Patterns of behavior in focused encounters, 209-238.
8
8. Khalil, M., & Ebner, M. (2016). What is Learning Analytics about? A Survey of Different
Methods Used in 2013-2015. arXiv preprint arXiv:1606.02878.
9. Kim, T., Chang, A., Holland, L., & Pentland, A. S. (2008). Meeting mediator: enhancing
group collaborationusing sociometric feedback. In Proceedings of the International Con-
ference on Computer Supported Cooperative Work 2008 (CSCW 2008), (pp. 457-466). San
Diego, CA, USA. ACM.
10. Marquardt, N., Hinckley, K., & Greenberg, S. (2012). Cross-device interaction via micro-
mobility and f-formations. In Proceedings of the 25th ACM Symposium on User Interface
Software and Technology, (pp. 13-22). Cambridge, Massachusetts, USA. 2380121: ACM.
11. Martinez-Maldonado, R., Clayphan, A., Yacef, K., & Kay, J. (2015). MTFeedback:
providing notifications to enhance teacher awareness of small group work in the class-
room. IEEE Transactions on Learning Technologies, 8(2), 187-200.
12. Martinez-Maldonado, R., Power, T., Hayes, C., Abdipranoto, A., Vo, T., Axisa, C., &
Buckingham-Shum, S. (2017). Analytics Meet Patient Manikins: Challenges in an Authen-
tic Small-Group Healthcare Simulation Classroom. In Proceedings of the International
Conference on Learning Analytics and Knowledge, (LAK 2017), Vancouver, Canada.
13. Muñoz-Cristóbal, J. A., Prieto, L. P., Asensio-Pérez, J. I., Martínez-Monés, A., Jorrín-
Abellán, I. M., & Dimitriadis, Y. (2014). Deploying learning designs across physical and
web spaces: Making pervasive learning affordable for teachers. Pervasive and Mobile
Computing, 14(Special Issue on Pervasive Education), 31–46.
14. O'Flaherty, J., & Phillips, C. (2015). The use of flipped classrooms in higher education: A
scoping review. The Internet and Higher Education, 25, 85-95.
15. Prieto, L. P., Sharma, K., Dillenbourg, P., & Jesús, M. (2016). Teaching analytics: to-
wards automatic extraction of orchestration graphs using wearable sensors. Paper pre-
sented at the Proceedings of the Sixth International Conference on Learning Analytics &
Knowledge, (pp. 148-157). ACM.
16. Raca, M., Tormey, R., & Dillenbourg, P. (2014). Sleepers' lag-study on motion and atten-
tion. Paper presented at the Proceedings of the Fourth International Conference on Learn-
ing Analytics And Knowledge, (pp. 36-43). ACM.
17. Rogaten, J., Rienties, B., Whitelock, D., Cross, S., & Littlejohn, A. (2016). A multi-level
longitudinal analysis of 80,000 online learners: Affective-Behaviour-Cognition models of
learning gains.
18. Roll, I., & Wylie, R. (2016). Evolution and Revolution in Artificial Intelligence in Educa-
tion. International Journal of Artificial Intelligence in Education, 26(2), 582-599.
19. Santos, O. C. (2016). Training the Body: The Potential of AIED to Support Personalized
Motor Skills Learning. International Journal of Artificial Intelligence in Education, 26(2),
730-755.
20. Thompson, K., Howard, S., Yang, J., & Ma, J. (2016). Mining video data: tracking learn-
ers for orchestration and design. In Proceedings of the Australian Society for Computers in
Learning in Tertiary Education (ASCILITE' 17), Adelaide, Australia.
21. Walther, J. B., Loh, T., & Granka, L. (2005). Let Me Count the Ways. Journal of Lan-
guage and Social Psychology, 24(1), 36-65.
22. Worsley, M., & Blikstein, P. (2013). Towards the development of multimodal action based
assessment. Paper presented at the Proceedings of the third international conference on
learning analytics and knowledge, (pp. 94-101). ACM.
23. Yilmaz, A., & Shah, M. (2005). Actions sketch: A novel action representation. Paper pre-
sented at the Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Comput-
er Society Conference on, (pp. 984-989). IEEE.