A recent trend in the field of learning analytics is to use sensor data about learners to support self-regulated learning. Combining personal, sensor based data with log data derived from a learning environment is a very promising approach, but also poses big challenges for the design of learner models and learner interaction methods, for the interpretation techniques of such data, and on applicable learning scenarios with their ethical and privacy demands. This paper provides a brief review of the emerging field of sensory aided learning analytics, and presents first results towards modeling and developing solutions for sensor-based adaptive learning in different learning contexts.
Learning analytics is becoming a multi-faceted field. One recent direction that research in learning analytics has taken is to shift the focus away from the traditional perspectives of institutions and instructors towards more user-centric views and methods in which learning analytics has the purpose of supporting adaptive and self-regulated learning. Another recent trend in educational data mining and learning analytics that goes along with the widespread availability and use of sensor technology, increasingly also integrated into smart wearable devices, is to investigate the extent to which this sensor data about learners can be used to support learning processes. Here, research is needed on applicable methods for learning analytics, on technical requirements and design options for systems providing learner support via feedback, on algorithms for recommendation or adaptiveness, and on interpretation methods and principles of personal sensor data in a learning context. In educational psychology, factors such as learner’s motivation, time management skills and metacognitive skills have been investigated in various studies. Some of these have found correlates for these factors, thus linking them to measurable variables like clicks, postings, messages, views, writes, likes and other types of learner behavior in online learning environment. In the psychology and medical literature, there is also evidence on the correlation between certain types of data that can be collected with sensors (e.g. heart rate, or skin conductance) and higher-level states of persons (e.g., anxiety). In the field of learning analytics, one typical goal is to use records of learner behavior and state (either Sensor Data for Learning Support 29 online or offline) and to feed this into learning analytic algorithms in order to derive an educational meaning or decide whether to adapt a learning technology to the user. However, relating a specific learner’s behavior and state, represented via complex interleaved concepts such as emotion, cognition, motivation or meta-cognition (in addition to user actions) has not yet been thoroughly investigated by either discipline - especially not with the perspective of feeding this information back to the user in order to support his selfregulatory processes. This paper addresses this research gap. While we are currently far from solving the problems stated above, the first goal of this present work is to review some of the pertinent literature and provide a categorization of sensors by learning domain (section 2). Based on the argument that a suitable technical framework for learning analytics based on data collected by multiple sensors is currently lacking, section 3 of this paper presents a first prototype of a sensor learning device and discusses some findings from a pilot study conducted with this device. This section also contains possible industrial and university based usage scenarios for a sensor based learning analytics framework (and the corresponding technical device). We conclude section 3 with discussions on ethical and privacy aspects of such scenarios. 2
Learning analytics allows learning data to provide a more accurate description of the learning context, learning endeavor, and might result in a better learning experience design. The observable behavior of students can be utilized to achieve a better learning environment. In education, sensor data refer to observable data in online learning environment. The quantifiable data such as log-in duration, log-in and -out timestamp, number of views, duration of views, frequency of log-in, and clicking point and qualitative data such as text analysis, social interaction analysis and learning path detection have been used to interpret a learning state. The approach to relate the measurable data from the online learning environment with theoretical background requires sophisticated interpretation [JKY14]. For example, an overall log-in time to an online learning environment could be interpreted as the total studying time, which is used as an indicator to explain learning performance. In the literature [PAI01] [DG05] [KKP09], data have been paired up to describe and suggest a better online learning environment design. Examples are: login frequency and course satisfaction, login frequency and attendance rate, participation frequency and learning outcome.
Learning analytics considers recorded (log) data from online learning environments that can be read and processed by machines [Pr16]. This approach limits the learning environment to an online learning environment, and there still exist opportunities for further enhancement by providing rationales between measured data and educational theories. Research on sensors provides new chances for learning environment design, since modern sensors are affordable and provide elaborate physiological data; studies on sensors are well advanced to support learners’ learning activities [Ma16]. Sensors can detect extrinsic contexts, e.g. position, time and environmental values, whereas an intrinsic context is personal to a learner, e.g. motivation or cognition [Th12]. With sensor data available, the obtainable data is not limited to offline settings, but also includes a learner’s state and condition during online learning. This implies that face-to-face and online learning environments can benefit by including learners’ physiological data which are detected by hardware sensors. In the research related to sensors, gaze awareness tools, EEG, eye tracking and accelerometer have been utilized to facilitate learning and teaching [Pr16]. Sensors for learning support can be categorized by learning domains which were introduced by Bloom and colleagues [Bl56] and thoroughly described in [Sc15]. In Tab. 1, we focus on sensors pertinent for our learning domain.
Sensors While compass, GPS, gyroscopes and inertial sensors may be utilized for the detection of psychomotor activities, accelerometers are most widely utilized to monitor physical movement, analyzes the position of the activity and a specific behaviors [AS02] [BK06] [GLJ09] [Hi11] [He06]. Heart-rate monitors are used to analyze the vital state of a person related to sports, health and everyday activity [Pe05] [SF12] [Va10]. For emotional detection, galvanic skin conductance (electrodermal activity) sensors have been utilized for monitoring health and learning situations [Ar09] [Ca13]. Data from air pollutants sensors, humistors or thermometers may also be valuable to detect environmental values like quality of the air of the confined space, humidity level and the actual temperature in a learning space. 3
The review in the previous section shows that, while sensor data is increasingly being used in educational technology and several Learning Analytics methods include feedback to the learner (as opposed to feedback to the teacher) as a central mechanism, approaches Sensor Data for Learning Support 31 that combine these two aspects are surprisingly rare. Currently, little evidence concerning the benefits (and possible drawbacks and problems) of using sensor collected data to provide users with adaptive feedback on their learning processes is available, and integrated technical solutions that are capable of handling multiple sensors while at the same time allowing students to interact and explore the (pre-processed and analyzed) sensor data are scarce.
In the following of this section, we propose a technical approach for collecting and processing learner data, followed by a spectrum of scenarios for making use of this data collection approach effectively in various educational settings, covering formal as well as informal ones. We conclude this section with some remarks on privacy and ethical issues that the scenarios raise. 3.1
As part of a feasibility study, we assembled a prototype of a sensor device with commercially available sensors, which can prospectively indicate learning domains (cognitive, affective and psychomotor). Among the sensors listed in Tab.1, we have chosen the three sensors electrodermal activity sensor (EDA), heart rate sensor (HR) and an accelerometer as shown in Abb. 1.
Abb. 1: Experiment setting for emotional intelligence with EDA sensor (prototype 1) The electrodermal activity is measured via a simple resistive voltage divider, which determines changes in conductance of the skin. Changes in the signal (voltage) correspond to changes in skin conductance, which might indicate changes in stress level, or indicate the emotional state of a learner. An optical sensor was selected for HR, as it is non-invasive to the learning process. Criteria for the choice of an acceleration sensor are low energy consumption, small size and low costs. Wearability was not a design concern of the actual prototype, further versions will be implemented as suitable wearable devices. For this experiment, the focus was on the collection of sensor data, and the relationship between sensor data and the learning domain.
To explore the prototype in educational context, emotional intelligence questionnaires (measuring affective domain in a learning context) were provided to the students. During the experiments, EDA, HR and acceleration data were measured. Participants’ writing hand (active hand) was wired with an accelerometer and their inactive hand was wired with the EDA and the HR sensors. Even though the HR and the accelerometer provided interesting data to review, the focus of this analysis was on the relationship between the EDA and the affective learning domain (emotional intelligence), as the emotion in learning poses an important indicator for self-regulation in motivation and metacognition, and it is strongly linked to predict academic success [Pe02]. For the detection of the emotion, the EDA sensor was chosen based on previous literature research [Ca13] [Mc12] [La93].
Abb. 2: EDA signal of participant with highest emotional intelligence Data from a total of 13 participants were collected for analysis. The mean response time for each question was 8.6 seconds among all participants and the lowest emotional intelligence observed was 61%, whereas the highest quotient was 88%. Due to the insufficient sample size (N = 13), these results cannot be generalized, yet it was observable that for each question, regardless of the response scale, the EDA signal shows a peak as shown in Abb. 2 and 3.
Abb. 3: EDA signal of participant with lowest emotional intelligence From the observation of the participants’ data, the participants with a higher emotional intelligence show less fluctuation in the EDA signal between the beginning and the end of the experiment (Abb. 2). However, this may be due to the cognitive effort of a participant during the experiment [Bo92] [VV96]. Also, the emotional state of frustration [LN04], unpleasantness [Se09] or undefined factors might have an affect. A further study with a Sensor Data for Learning Support 33 sample size of N=50 and above should be followed to investigate the relationship between the EDA data and the learning state. 3.2
The potential of a sensor device like the prototype presented in 3.1 emerges when analyzing possible applications in different teaching styles: blended learning courses in universities, pure online courses, informal learning platforms and gamification applications. We consider these teaching styles in industrial education contexts and university, addressing students in secondary school and higher education as target groups.
In May 2016 a focus group consisting of 12 participants (9 students, 2 scientific assistants, 1 professor) led by the HTW discussed their learning habits in detail, without specifically discussing e-learning environments. The discussion revealed several support interests: helping with the learning content, managing time, logging of learning activities, evaluating development and performance, proposing of better learning techniques, reducing distraction, providing real time feedback and health state.
For the application in blended learning courses, students of Computer Science at HumboldtUniversity were asked to explore the potential of a sensory aid by defining personas as typical users and their usage scenarios. The task was assigned to them as two exercise sessions of 1.5 hours each in summer 2016. Several needs result from the defined personas, like optimization of their learning times and concentration phases, formative feedback, and regular summative feedback, reduction of distractions, learning on the run, and including handwriting and speech into the digital learning settings. The scenarios combined explicit preference settings, data tracked by the online learning environment and physiological sensor data. For example, one application focused on reducing distractions, defines active phases and breaks, blocks apps and websites, unblocks them during breaks, and if it recognizes a lack of concentration during an active phase, it intervenes with differently presented content to catch the user’s attention again.
A pure online learning application are e-learning courses for distance learning universities. The missing personal contact enforces the need for automated assessment and feedback, allowing the users to optimize their learning. Established learning management systems like Canvas, Open edX or Moodle are providing visual dashboards helping with self awareness [Ve13]. They base on performance monitoring within the system and interaction tracked by the system or the respective plugins, but lack in analyzing parameters beyond the system. Including data about the physical environment and physiological data of the learners is an open research question. The difficulty here is to determine the proxy variables for observation of the learning process - is it the pulse rate, room temperature, or the combination of noise and eye-movements that makes a difference. An approach addressing this issue could be a dashboard finding autonomously correlations between the sensed data, tracked interaction data and the performance of the user. Adequate assessment like these statements could be desirable, e.g. “Sustained and unchanging low level activity lowers concentration.”, “A short rest, or a change in activity, every 15 minutes or so restores performance almost to the original level.” [Bi03] Another ongoing research topic opens the visualization form itself, presenting the data adequately for the user [Ve13]. Informal teaching like tutorials and news blogs, as often applied for internal training in bigger companies or simply personal extension studies, might benefit from sensory data for personalization and context awareness. To provide an example, we consider a sanitary retail employee learning through an online training magazine about hygiene and a new product. The magazine presents its contents to the user in a very personalized way, by generating an adaptive learning path, fitting the user’s learning situation, knowledge and emotions. This way the learning outcome could be enhanced, similar as sales volume of online shops grows with personalized offers. Context awareness for the above situation could allow training the installation of the new product in augmented or virtual reality at home. In the retail situation, the magazine could offer context aware informations to answer customers questions quickly.
Gamification approaches for learning are bringing even more possibilities to use sensors, supporting the fun factor and therefore addressing the affective learning domain more than other learning forms. For example, secondary school children could learn about brain functionality through a game challenging dexterity, speed and teamplay. Physiological sensors can be used for intentional game control, but also for adapting to the feelings of the user, adjusting the difficulty of the gameplay or complexity level of the learned content. In serious games as applied by professionals in emergency for the training of dangerous situations, even feelings themselves could be part of the learning content and adequate reaction could be tought. One example: “Biohazard: Hot zone, is a game aimed to help emergency first responders deal with toxic spills in public locations. In the game, users work in teams, responding to a gas attack in a suburban shopping mall. The aim of the game is to help people prepare for potentially catastrophic situations.” [SJB07].
Some analogies appear between the applications. The needs identified by HTW Berlin and Humboldt-University are similar, differences could be explained by the observation methods and questions asked to the students. Common interests in blended and informal online learning are personalized and adaptive learning paths to support motivation and optimize the learning outcome. Correlating learning performance with mental states like uncertainty, boredom, concentration or frustration and adequate adaptive reactions of learning systems is a great chance for enhancing the learning experience, across multiple teaching styles and subjects. The input of informations about the mental states is most likely to be achieved utilizing personal sensor devices for learners. 3.3
Learner-centric analysis of educational data retrieved from various learning environments aims at improving learning, and at providing a better learning experience. Combining activity data from a learning environment with physiological data obtained from wearable sensors leads to rich data sets, with new chances for the adaption of the learning environment and personalized learning support. On the other hand, analyzing a learner’s physiological data poses a big challenge, and many open questions, to learning analytics: what Sensor Data for Learning Support 35 are the ethical, legal and social applications of learning (analytics) applications, that might process, store, analyze, or visualize personal data? Legal implications are obvious: any learning application must be compliant with (national) data privacy legislation. An even more critical issue is the acceptance of a learning support system by the learner: recording and usage of sensor data must be completely transparent to the user, under the control of the learner, and private data with no relevance to the learning process must not be analyzed! Research on ELSI - ethical, legal, social implications of emerging life sciences - goes back to bioethics, and to the Human Genome Project (HGP) [TBM97]. Beyond the HGP, ELSI guidelines have been formulated for many projects and applications which collect, process and share personal data, an example being ELSI guidelines for biomedical data at the European Bioinformatics Institute.
To meet the demands of an ELSI compliant learning application, a technical concept to provide data privacy is essential. Data privacy is a central concept for learning analytics [PS14] but recently, privacy has also been regarded as a limiting factor for the adaption of learning analytics [DG16]. With advances in sensor technology, and with the availability of sensor-based applications in everyday life, ethical and privacy issues have to resolved [Bo04]. This not only applies to popular health or fitness apps, but also to learning applications using sensor data.
A learning analytics system, which uses a sensor device as presented in 3.1, needs a technical data privacy concept on different layers: on the layer of the learning application / learning environment, on the layer of a learning analytics engine (backend), providing services for learner support, and on the layer of a sensor device.
For the scenarios with a sensor device, a data privacy concept should address different topcis: locality of sensor data, user interaction, a technical concept for exchange and storage of learners’ data, a non-technical concept for learning scenarios and applications, and transparency to a user.
Locality of sensor data means that sensor data are filtered, processed and stored within the smart monitor - only data relevant for learner support are transmitted to a learning analytics application. Example: heart rate sensors provide very detailed information about a learner’s health, whereas just a pulse rate might be needed as an indicator for the actual learning state. The process of filtering data, and maintaining data locality, must be transparent, and should be controlled by the learner himself. This implies the need for local interaction with the SmartMonitor, as part of the interaction and usability concept. Transparency also implies visualizing data (which are kept local) on a sensor device. An emerging standard for the exchange and storage of educational data is the xAPI (Experience API), which evolved from the TinCan project [KR16]. xAPI was designed for better interoperability between different educational systems, which allows to link sensor data to a system for self-regulated learning [MCL15]. xAPI “recipes”, which can help with the design of xAPI systems can be found in [Ba15]. From a data privacy view, personal data lockers can be implemented as an extension to learner record stores, defined in xAPI.
Personal data lockers transfer control over personal learner data from an analytics system to the user (learner), forming the basis of a technical data privacy concept. Finally, a non-technical data privacy concept for the above mentioned learning applications and scenarios addresses legal and ethical issues. This includes transparent definition, configuration and enforcement of data ownership. Also, the results of data analyses, in form of feedback, recommendation, or adaption of the learning environment, must be easy to comprehend for learners. Both data privacy and transparency form the basis of a learning application or tool which is acceptable by learners. 4
In this paper, we addressed an area that the Learning Analytics research community is currently starting to investigate: the use of data collected by various sensors in order to provide effective learning support. While several studies on the applicability of specific sensors as predictors for certain cognitive or affective states have been conducted, limited research on the use of integrated data coming from multiple sensors available. Also, there is a lack of research on how to provide learners with an overview of this multiple sensor data and analysis results so that they can regulate their learning processes aided by this information. Furthermore, design solutions that respect privacy while providing efficient learning support need further investigations.
The technical design solution (shown in early prototype stage) and the use case scenarios presented in this paper will be further explored as part of a recently started research project funded by the German Ministry of Education and Research. In this project, we are currently eliciting requirements and designing use case scenarios for sensor based learning analytics technologies by taking into account educational, content-related, methodological, technical and ethical perspectives. Based on these, we will then design, implement and evaluate learning analytics methods for supporting self-directed learning in different sensor based environments. Three company partners will then implement and test the methods in the different scenarios sketched in section 3 of this paper: while NEOCOSMO will focus on professional education in the hygiene sector, SGM will investigate e-learning applications for the higher education sector, and Promotion Software is going to develop educational games. All of these scenarios (plus additional university application areas) will serve as test beds for empirically validating the acceptance and efficiency of the Learning Analytics methods and the technology used to implement them in both professional and university settings.
This research was funded by the BMBF (German Ministry of Education and Research) under the grant “LISA - Learning Analytics fu¨r sensorbasiertes adaptives Lernen”. Sensor Data for Learning Support 37 [Ar09]
Arroyo, Ivon; Cooper, David G; Burleson, Winslow; Woolf, Beverly Park; Muldner, Kasia; Christopherson, Robert: Emotion Sensors Go To School. Artificial Intelligence in Education: Building Learning Systems That Care: From Knowledge Representation To Affective Modelling, 200:17–24, 2009. [AS02] [Ba15] [Bi03] [BK06] [Bl56] [Bo92] [Bo04] [Ca13] [DG05] [DG16] [GLJ09] [He06] [Hi11]
Amaratunga, Kevin; Sudarshan, Raghunathan: Innovations in Virtual and Remote Laboratories A VIRTUAL LABORATORY FOR REAL-TIME MONITORING OF CIVIL. Engineering Education, pp. 1–6, 2002.
Bakharia, Aneesha; Kitto, Kirsty; Pardo, Abelardo; Dawson, Shane: Recipe for Success Lessons Learnt from Using xAPI within the Connected Learning Analytics Toolkit. Submitted to LAK 2016, (October), 2015.
Biggs, John: Teaching for Quality Learning at University Assessing for learning quality: II . Practice. Teaching for Quality Learning at University, (JANUARY 2003):1–36, 2003. Baca, Arnold; Kornfeind, Philipp: Rapid feedback systems for elite sports training. IEEE Pervasive Computing, 5(4):70–76, 2006. [KR16] [La93] [LN04] [Ma16] [Mc12] [PAI01] [Pe02] [Pe05] [Pr16] [PS14] [Sc15] [JKY14] Jo, Il-Hyun; Kim, Dongho; Yoon, Meehyun: Analyzing the log patterns of adult learners in LMS using learning analytics. In: Learning Analytics and Knowledge Conference 2014, LAK ’14, Indianapolis, IN, USA, March 24-28, 2014. pp. 183–187, 2014. [KKP09] Kang, Min-Seok; Kim, Jin-Il; Park, Inn-Woo: The Examination of the Variables related to the Students’ e-learning Participation that Have an Effect on Learning Achievement in elearning Environment of Cyber University. Journal of Internet Computing and Services, 10(5):135–143, 2009.
Kevan, Jonathan M.; Ryan, Paul R.: Experience API: Flexible, Decentralized and Activity-Centric Data Collection. Technology, Knowledge and Learning, 21(1):143–149, 2016.
Lang, P J; Greenwald, M K; Bradley, M M; Hamm, a O: Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology, 30(3):261–273, 1993. Lisetti, Christine Laetitia; Nasoz, Fatma: Using Noninvasive Wereable Computers to Te recognize human emotions from Physiological signals. EURASIP Journal on Advances in Signal Processing, pp. 1672–1687, 2004.
Maldonado, Roberto Mart´ınez; Schneider, Bertrand; Charleer, Sven; Shum, Simon Buckingham; Klerkx, Joris; Duval, Erik: Interactive surfaces and learning analytics: data, orchestration aspects, pedagogical uses and challenges. In: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, LAK 2016, Edinburgh, United Kingdom, April 25-29, 2016. pp. 124–133, 2016.
McDuff, Daniel; Karlson, Amy; Kapoor, Ashish; Roseway, Asta; Czerwinski, Mary: AffectAura: An Intelligent System for Emotional Memory. Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems - CHI ’12, p. 849, 2012. [MCL15] Manso Va´zquez, Mario; Caeiro Rodr´ıguez, Manuel; Llamas Nistal, Mart´ın: Development of a xAPI application profile for self-regulated learning requirements for capturing SRL related data. IEEE Global Engineering Education Conference, EDUCON, 2015April(March):358–365, 2015.
Piccoli, Gabriele; Ahmad, Rami; Ives, Blake: Web-Based Virtual Learning Environments: A Research Framework and a Preliminary Assessment of Effectiveness in Basic IT Skills Training. Source: MIS Quarterly, 2521440140(4):401–426, 2001.
Pekrun, Reinhard; Goetz, Thomas; Titz, Wolfram; Perry, Raymond P.: Academic Emotions in Students’ Self-Regulated Learning and Achievement: A Program of Qualitative and Quantitative Research. Educational Psychologist, 37(2):91–105, 2002.
Pentland, Alex: Healthwear : Medical Technology Becomes Wearable. In (Bushko, Renata G., ed.): Future of Intelligent and Extelligent Health Environment, pp. 55 – 65. IOS Press, 2005.
Prieto, Luis Pablo; Sharma, Kshitij; Dillenbourg, Pierre; Jesu´s, Mar´ıa: Teaching analytics: towards automatic extraction of orchestration graphs using wearable sensors. In: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, LAK 2016, Edinburgh, United Kingdom, April 25-29, 2016. pp. 148–157, 2016. Pardo, Abelardo; Siemens, George: Ethical and privacy principles for learning analytics. BJET, 45(3):438–450, 2014. [Se09] [SF12] [SJB07] [Th12] [Va10] [Ve13] [VV96] Sequeira, Henrique; Hot, Pascal; Silvert, Laetitia; Delplanque, Sylvain: Electrical autonomic correlates of emotion. International Journal of Psychophysiology, 71(1):50–56, jan 2009.
Serbedzija, Nikola; Fairclough, Stephen: Reflective pervasive systems. ACM Transactions on Autonomous and Adaptive Systems, 7(1):1–19, 2012.