Am I meeting my lifelong life-wide goals? Towards answers from Scrutable Sca olded Open Learner Models (SOLMs).

Abstract

There are increasingly diverse and large sources of data about our long-term and life-wide learning. We propose a new way for a long term personal learner model to support such learning by helping people answer the question: Am I meeting my lifelong life-wide goals? This is fundamental to self-regulated learning. This paper outlines an approach to creating Scrutable Sca olded Open Learner Models (SOLMs) to make this possible. We argue the need for three core aspects: 1) goal models; 2) scrutability; and 3) sca olding. The core contribution of this position paper is the conceptual model for SOLMs.

INTRODUCTION

People increasingly carry, wear, use and are tracked by technology. This can provide rich data that could play a role in modelling their long term learning across life-wide contexts. At present, it is quite di cult to harness that data. This is because we do not yet have convenient ways to bring that relevant data together and manage it e ectively.

We present a way to tackle this with the learner’s long term goals as the foundation for transforming diverse sources of personal data into a useful form as a Scrutable Sca olded Open Learner Model (SOLM). This builds on the decades of work on Open Learner Models (OLMs) [ 5, 6 ] where the learner is able to usefully interact with a representation of their knowledge state. We also build on the idea of an independent learner model [ 6 ], one that exists outside any single application.

PeCrmopisysiroignhttohmealkde digital or hard copies of all or part of this work for personal or classroom use is granbtyedthweithaouutthfoeer(psr)o.vUidsedetphaetrcmopititeesdaruenndoetmrtahdeeoCr Cdi-sBtriYbuted folricperonsteorCcroemamtievreciCaloamdvmanotangse.aonrdg/tlhicatecnospeiess/bbeya/r4th.0is/ notice and the full citation on the rst page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speci c permission and/or a fee. Request permissions from [e-mail].

Conference’17, July 2017, Washington, DC, USA © 2019 Association for Computing Machinery.

ACM ISBN 978-x-xxxx-xxxx-x/YY/MM. . . $15.00 https://doi.org/10.1145/nnnnnnn.nnnnnnn

We envisage the SOLM as a long term user model that is framed in terms of the user’s long term goals. It aggregates diverse collections of data, linking them to the goal model components. It then interprets that data to model the learner’s progress. The user can then scrutinise this to monitor their progress on each goal and subgoal to see if the model indicates they are meeting the targets. This is a starting point for deeper re ection to decide if they need to nd ways to do better and, when that is the case, to plan how to do that. They should also be able to scrutinise the model to decide whether it is accurate enough to be trusted. 2

DEFINING GOAL MODELS AND

SCAFFOLDING CASE STUDIES

The goal model needs to represent the key aspects needed to describe each of the user’s goals and it needs to link this to evidence for reasoning about the goal. We build on previous work which de ned a systematic way to model long term goals [ 3 ]. This draws on goal setting theory [ 17 ] in terms of the following elements: speci c target, measurement tools, measured progress, relevant subgoals, and timescale to achieve this target. It requires input from the learner for some key aspects. This requires that the learner uses an interface to the SOLM to consider and score the intrinsic elements from goal setting theory. These are important for achievability: importance of the goal, perceived di culty of achieving it, commitment to the goal, self-e cacy and self-satisfaction. It also has extrinsic components: reminders, feedback, rewards and social involvement.

We will illustrate the ideas in terms of two case studies. The rst concerns learning to program, similar to classic AIED research involving formal skills. But we extend it to address the long-term goals of a person who wants to be an excellent software engineer. This involves learning that spans decades and includes formal education at school, university and targeted professional development courses as well as personal re ection on long term work.

An example for one sub-goal for learning to program is shown in Table 1. This the speci c target maps to a Knowledge Component in the learner model. The tools and measures of progress de ne the sources of evidence that will be made available to the model to determine the learner’s progress. Similarly, the subgoal to read the text could be tracked. The timescale is important for the feedback on progress at the interfaces to the SOLM. This learner’s intrinsic factors all indicate they are highly committed and con dent of success, even though they consider it a di cult goal to achieve. The extrinsic elements require the SOLM to deliver daily summaries on progress and to share the progress with a partner.

Component Value for this user speci c target learn how to write loops in C measurement tools LMS programming exercises measured progress scores on exercises relevant subgoals read online textbook chapter 3 timescale 2 weeks importance very high perceived di culty very high commitment very high self-e cacy very high self-satisfaction very high reminders none feedback daily summary rewards none social share with partner Table 1: Model for sub-goal within learning to program The second is for a life-wide goal, an important form of learning that has had little attention in AIED. This is a goal to “square the curve”1. This refers to a curve that shows a measure of wellness on the y-axis, against the person’s age on the x-axis. Typically, healthy, happy people over the age of 40 can anticipate that they will stay that way for some years but will gradually decay as they approach death. The goal to square-the-curve means that the person aims for healthy aging, so that the healthy level is maintained for as long as possible and to minimise the time on the downward period with steadily failing health, frailty and falling quality of life. The person strives to keep the line as high as possible for as long as possible and then rapidly decline to death.

Table 2 has an example of a model that is important for squaring the curve; this is a physical activity goal where the user is currently far below their new target and moderately motivated. In this case, a smart-watch can provide data about the person’s activity for this goal. 1https://www.news-press.com/story/life/wellness/2015/06/08/ tness-beyondsquaring-curve/28548311/

We have created interfaces that enable users to provide the information needed to populate such goal models [ 3 ]. Since most people do not know why some of these aspects are important, this interface provided sca olding to explain this. Similarly, we built an interface to a long term learner model for a set of physical activity goals [ 21 ] and while users could readily understand the model well enough to gain valuable insights, a goal re ection sca old enabled them to gain additional insights by considering factors they had not realised are important (e.g., Although all participants had long term Fitbit data, and all were able to gain insights from exploring their model, many had not thought of targets. Once prompted to consider a target, all judged whether they had met it. They needed a sca old prompt to realise they were less active on weekends.) In more formal learning, Azevedo and colleagues demonstrated bene ts of sca olding to support meta-cognitive processes [ 2 ] in a more formal STEM context.

Both these goals, software-engineering-excellence and squaringthe-curve are very long term and each involves sub-goals. For software-engineering-excellence, these may involve a sub-goal of learning how to read and write loops, as part of a larger semesterlong sub-goal of doing well in Programming 101. This, in turn is a part of a goal to graduate in a software engineering degree, and then to go on to identify and achieve new sub-goals over decades, such as learning new languages and group work skills. To squarethe-curve, which covers many aspects of life, sub-goals are needed for aspects such as tness (with its sub-goals for strength, balance), weight, nutrition and a sense of well-being and self-actualisation. 3

SCAFFOLDING SCRUTABILITY

Although our conceptual approach is not restricted to any particular implementation, we plan to use the Personis user modelling system [ 14 ] to create SOLMs. We now list the high level processes involved in building a Personis learner model. In parentheses after each process, we outline the aspects that a user may wish to scrutinise.

(1) De ne what is to be modelled, the ontology problem of de ning the components of the model (scrutinise the de nition of each component and the explanation of what it means). (2) Accrete the data that can serve as evidence to reason about the components (explore the raw data in the model, including its time-stamp, source-description and value). (3) Internal inference reasons from one state of the model to create new evidence about components e.g., rule-based, Bayesian inference, sequence mining ....(explore this inferred data, its time-stamp, the description of the inference process and value). (4) Resolve the value of each component by interpreting its evidence (understand the description of the process used to resolve the value - potentially select between available resolvers to make use of di erent interpretations and standards). (5) run scheduled processes to do steps 2-4 where these may activate programs outside the model e.g., start processes to collected new data from a source, send an alert to the learner to visit their model to review their progress (review these processes to understand what actions are driven by the user Am I meeting my lifelong life-wide goals? Towards answers from Scrutable Sca olded Open Learner Models (SOLMs). model - and the learner may want to alter these, ebaling new prompts and disabling others) [ 16 ].

A learner may want to scrutinise any or all of these aspects of their model for several reasons: to understand an unexpected aspect they see in the OLM interface (e.g., it indicates they have not been reaching their goal target when they thought they were); to check whether they can trust their seeming progress on the OLM (e.g., there may be missing or incorrect data coming into the model); or to satisfy curiosity and interest in understanding their data and inferred model and the way that it works.

For a long term, rich and complex learner model with many data sources for a goal, it is challenging for a user to be aware that they can scrutinise all these aspects. Based on the experience of scrutably personalised interfaces it is possible to create interfaces that enable people to readily scrutinise some aspects of the ve elements above but they bene t from sca olding [ 14 ]. 4

DISCUSSION

we now discuss the challenges and opportunities for creating and harnessings SOLMs.

Build it and they will come - or will they?

We have made an underlying assumption that people will want a long-term model of their learning with an OLM interface. We brie y summarise some of the evidence supporting this assumption.

In a study where people were asked to imagine sensors had collected various forms of long term data about them, most participants indicated that they would want to have their data in a personal store [ 4 ] and the most common reason was just in case they later found a use for it. Similarly, work on life-logging reviewed in [ 12 ] highlights the potential for rich multi-media streams of sensor data. The small number of people committed to life-logging and the larger Quanti ed Self community represent people who already want to capture their own long term data, even though there is limited support for aggregating it and using it in meaningful ways. Our SOLMs could ll this gap, with long term user models that are structured around personal goals.

The substantial body of work by Susan Bull and colleagues points to the extensive use of OLMs by students [ 6, 7 ]. There is also strong evidence that OLMs actually resulted in improved learning [ 18, 19 ]. These results indicate that OLMs are used by learners, albeit in the case where they are embedded in the actual learning tool.

We also have some evidence that people may actually scrutinise their data and model in the context of a eld study of SASY, a personalised teaching system with scrutability of both the learner model and the way it was used for personalisation [ 8, 14 ]. When students used SASY for homework, 77% of the 105 students used at least one scrutiny tool and 13 students used at least 9, mostly around their domain knowledge. Almost all use (96%) was either on nishing a module or just before starting one. This makes sense since it is a logical stopping point and a time to re ect. The SASY results indicate that many learners did use such a scrutiny interface in an authentic setting of a homework tutorial - even in a context of time pressure, as most students used it close to the homework deadline.

Both the case of OLMs and SASY had the interface onto the model available to the learner. Our SOLM interface could also be initiated by an application so that it is low e ort for the learner to have access to it. Based on the SASY experience, a learning application could start the SOLM at the start and end of each main stage of the learning, such as completing a module, quiz or other activity.

Twenty- rst century skills in self-regulated learning and personalised sca olding all ages from children to elders

Within education research, there is wide recognition of the importance of metacognitive skills such as self-monitoring, re ection and planning as part of self-regulated learning [ 10 ]. Learners need to learn what these skills mean and why they are valuable. One part of this is to provide explicit instructions on how to do them in a new context. We can build upon work on such sca olding meta-cognitive skills [ 2 ] in a science context.

We see two levels of sca olding, each personalised. First, there is a generic level of sca olding for setting goals. This would prompt the learner to consider each of the factors shown in the tables above. Since there are quite a number of factors and learners may not appreciate why they matter, the sca olding should provide personalised teaching about them. The SOLM would model the learner’s progress on goal setting and use this to personalise the guidance.

The second level of learning and associated sca olding is domainspeci c. We illustrate this in terms of a physical activity goal, where people bene ted from advice on considering suitable targets and prompts to check for di erences between weekdays and weekends [ 21 ]. This, too, should be personalised, so that learner’s a prompted to consider just the aspects that are relevant to them.

Trust and uncertainty

In a comprehensive review of ways to depict uncertainty in OLMs, Epp and Bull [ 9 ] identify the diverse sources on uncertainty: accuracy, precision, completeness, lineage, judgement, validity, as well as currency (timeliness), statistical variance and consistency. Any learner model with have incomplete information about the learner. So a SOLM should sca old the learner to scrutinise their model to understand just what evidence it has and how it uses it. So, for example, a learner may consider some evidence sources to be unreliable for them or they may consider that some important sources of evidence are missing from the model. In either case, the tuning of the resolver process could account for these.

SOLMs as a uni ed and independent Personal

Informatics platform

With the huge amount of available learning software, the SOLM would o er a straightforward way to add the bene ts of an OLM without changing the application. So long as the data from the application can be made available to the SOLM, the learner could then view their model. This could be based on data from the multiple tools that may be used in formal course, including the Learner Management System and other specialised learning tools.

For broader Personal Informatics data, such as that collected by worn trackers for physical activity and sleep, a current barrier is that people

nd it too hard to bring their data together so that they can explore long term data [ 20 ].

Where will the SOLM be stored?

We rst built user models that resided in the user’s own le space at the only machine available to our users [ 13 ]. With the emergence of the web, we created a user model server [ 15 ], then a distributed version with active rules that could interact with ubicomp applications [ 1 ] and a light-weight mobile version [ 11 ]. Other researchers explored similar approaches. We now can move to a personal cloud that based on tools like Dropbox, Box and Google Drive. This can serve as a repository for the learner’s raw learning data and user models, with the user installing SOLM apps and plug-ins. 5

CONCLUSIONS

We have presented a conceptual model for Scrutable Sca olded Open Learner Models (SOLMs) to support life-long and life-wide learner modelling. We have argued the need for three core aspects: 1) goal models; 2) scrutability so that the learner is always in control of their learner model, the processes to construct and interpret it, and all the ways it is used; and 3) sca olding interfaces to guide the learner both to provide input for the goal model and to scrutinise their model. We have illustrated this in terms of two quite di erent case studies. We have shown how we propose to build the infrastructure to support the learner modelling from diverse sources of evidence. We have highlighted the need for scrutability, particularly so that the learner can decide on the accuracy of the model, especially over the long term. We have also described the sca olding interface to help learners formulate, review and track their long-term and life-long goals and then self-monitor their progress, re ect on the barriers to progress and plan on ways to improve progress.

Acknowledgements

We thank the anonymous reviewers for their valuable feedback on the earlier version of the paper. This has informed a richer discussion of the challenges and opportunities.

References

[1] Mark Assad , David J. Carmichael , Judy Kay, and Bob Kummerfeld . 2007 . PersonisAD: distributed, active, scrutable model framework for context-aware services . In Proceedings of Pervasive 07, 5th International Conference on Pervasive Computing (LNCS) , Vol. 4480 . Springer, 55 - 72 . https://doi.org/10.1007/978-3- 540 -72037- 9 _ 4

[2] Roger Azevedo and Allyson F Hadwin . 2005 . Sca olding self-regulated learning and metacognition-Implications for the design of computer-based sca olds . Instructional Science 33 , 5 ( 2005 ), 367 - 379 .

[3] Debjanee Barua , Judy Kay, Bob Kummerfeld, and Cécile Paris. 2014 . Modelling long term goals . In Intn'l Conf on User Modeling, Adaptation, and Personalization . Springer, 1 - 12 .

[4] Debjanee Barua , Judy Kay, and Cécile Paris. 2013 . Viewing and controlling personal sensor data: what do users want? . In International Conference on Persuasive Technology . Springer, 15 - 26 .

[5] Susan Bull and Judy Kay . 2007 . Student models that invite the learner in: The SMILI:-) Open Learner Modelling framework . International Journal of Arti cial Intelligence in Education 17 , 2 ( 2007 ), 89 - 120 .

[6] Susan Bull and Judy Kay . 2016 . SMILI: a Framework for Interfaces to Learning Data in Open Learner Models, Learning Analytics and Related Fields . International Journal of Arti cial Intelligence in Education 26 , 1 ( 2016 ), 293 - 331 .

[7] Susan Bull and Andrew Mabbott . 2006 . 20000 inspections of a domainindependent open learner model with individual and comparison views . In International conference on intelligent tutoring systems . Springer, 422 - 432 .

[8] Marek Czarkowski and Judy Kay . 2006 . Giving learners a real sense of control over adaptivity, even if they are not quite ready for it yet . In Advances in Webbased Education: Personalized Learning Environments , S. Chen and G. Magoulas (Eds.). IDEA Information Science Publishing , Hershey, PA, USA, 93 - 125 . http: //www.idea-group.com/books/details.asp?id= 5248

[9] Carrie Demmans Epp and Susan Bull . 2015 . Uncertainty representation in visualizations of learning analytics for learners: current approaches and opportunities . IEEE Transactions on Learning Technologies 8 , 3 ( 2015 ), 242 - 260 .

[10] John H Flavell . 1979 . Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry . American psychologist 34 , 10 ( 1979 ), 906 .

[11] Simon Gerber , Michael Fry, Judy Kay, Bob Kummerfeld, Glen Pink, and Rainer Wasinger . 2010 . PersonisJ: mobile, client-side user modelling . In UMAP 2010, LNCS 6075 . Springer-Verlag Berlin Heidelberg, 111 - 122 . http://www.springerlink. com/content/4m271r8322303152/

[12] Cathal Gurrin , Alan F Smeaton, Aiden R Doherty , et al. 2014 . Lifelogging: Personal big data . Foundations and Trends® in information retrieval 8 , 1 ( 2014 ), 1 - 125 .

[13] Judy Kay . 1994 . The um toolkit for cooperative user modelling . User Modeling and User-Adapted Interaction 4 , 3 ( 1994 ), 149 - 196 .

[14] Judy Kay and Bob Kummerfeld . 2013 . Creating Personalized Systems That People Can Scrutinize and Control: Drivers, Principles and Experience . ACM Trans. Interact. Intell. Syst. 2 , 4 , Article 24 ( Jan . 2013 ), 42 pages.

[15] J. Kay , B. Kummerfeld , and P. Lauder . 2002 . Personis: a server for user models . In Proceedings of AH 2002 , 2nd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems (LNCS), P. De Bra, P. Brusilovsky , and R. Conejo (Eds.) , Vol. 2347 . Springer, 203 - 212 .

[16] Bob Kummerfeld and Judy Kay . 2017 . User modeling for the internet of things . In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization. ACM , 367 - 368 .

[17] Edwin A Locke and Gary P Latham . 2002 . Building a practically useful theory of goal setting and task motivation: A 35-year odyssey . American psychologist 57 , 9 ( 2002 ), 705 .

[18] Yanjin Long and Vincent Aleven . 2013 . Supporting studentsâĂŹ self-regulated learning with an open learner model in a linear equation tutor . In Arti cial intelligence in education . Springer, 219 - 228 .

[19] A. Mitrovic and B. Martin . 2007 . Evaluating the E ect of Open Student Models on Self-Assessment . International Journal of Arti cial Intelligence in Education 17 , 2 ( 2007 ), 121 - 144 .

[20] Lie Ming Tang and Judy Kay . 2017 . Harnessing Long Term Physical Activity Data&Mdash;How Long-term Trackers Use Data and How an Adherence-based Interface Supports New Insights . Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1 , 2 , Article 26 ( June 2017 ), 28 pages. https://doi.org/10.1145/3090091

[21] Lie Ming Tang and Judy Kay . 2018 . Sca olding for an OLM for Long-Term Physical Activity Goals . In Int'l Conf on User Modeling, Adaptation and Personalization. ACM , 147 - 156 .