One of the major challenges in higher education is developing self-regulation skills for lifelong learning. We address this challenge within the myPAL project, in medical education context, utilising the vast amount of student assessment and feedback data collected throughout the programme. The underlying principle of myPAL is Quanti ed Self { the use of personal data to enable students to become lifelong learners. myPAL is facilitating this with learning analytics combined with interactive nudges. This paper reviews the state of the art in Quanti ed Self analytics tools to identify what approaches can be adopted in myPAL and what gaps require further research. The paper contributes to awareness and re ection in technology-enhanced learning by: (i) identifying requirements for intelligent personal adaptive learning systems that foster self-regulation (using myPAL as an example); (ii) analysing the state of the art in text analytics and visualisation related to Quantied Self for self-regulated learning; and (iii) identifying open issues and suggesting possible ways to address them.
A major goal for educational institutions is to prepare lifelong learners who
{ through continuous professional practice { grow as professionals throughout
their university degree and beyond. At the heart of this is self-regulation: a
cyclic process underpinned by re ection to identify areas of strengths and
weaknesses, set personal learning goals, develop strategies to attain these goals, and
optimise learning and performance [
However, simply exposing students to the workplace will not on its own equip them with self-regulation skills. The key challenge is supporting students to fully engage with the work-based experience and the feedback they gather, using it to re ect on their performance and improve their professional development planning. Moreover, there remains the longstanding challenge of connecting formal subject-based education and informal work-based learning. Educational institutions have tried to tackle these challenges by providing students with access to workplace tutors/mentors to link experience to professional development. But one-to-one mentoring is neither sustainable, nor cost-e ective, nor scalable.
The myPAL project addresses these challenges within the context of medical education by tapping into the ubiquity of digital devices and the availability of a vast amount of student assessment and feedback data collected throughout the programme [40]. The project is a strategic technology enhanced learning initiative at the Leeds Institute of Medical Education (LIME3), funded by the Higher Education Funding Council for England. The project develops a personalised adaptive learning companion co-designed with students and educators that o ers `learning support at your ngertips' to foster the development of selfregulated learning skills. It will adapt to an individual's learning preferences and will provide continuous intelligent feedback { for example, a student may have reviewed materials and performed well in some areas, but have a `blind spot' and performed less well in others.
The underlying principle of myPAL is the use of personal data, namely student assessment and feedback data collected during a range of medical education activities, to enable students to become lifelong learners. This aligns with the Quanti ed Self approach4, where data collected about a person's life is used to facilitate taking control of their lifestyle by fostering self-awareness and selfmanagement. Self-knowledge through data o ers disruptive innovation in health, well-being, green living, energy consumption, and is now entering the educational domain5. In the context of myPAL, the `lifestyle' data is about the student's curriculum engagement and professional development. Through innovative use of learning analytics (including text analytics and visualisation) combined with interactive nudges, myPAL will support students to develop self-regulation skills. The students will be able to: (i) contextualise their assessment within the overall medical education curriculum; (ii) develop a holistic awareness of where they are in the medical curriculum and indicate their strengths and weaknesses; (iii) recognise, actively seek, and interpret feedback to personalise their learning by setting goals and identifying learning activities.
In this paper, we will review the state of the art in Quanti ed Self analytics
tools to identify what approaches can be adopted in myPAL and what gaps
require further research. By doing so, the paper explores outstanding challenges to
awareness and re ection in technology enhanced learning, such as relatively little
attention paid to lifelong professional development, underexplored heterogeneity
3 http://medhealth.leeds.ac.uk/info/800/leeds_institute_of_medical_
education
4 http://quanti edself.com/
5
https://www.forbes.com/sites/ryancraig/2016/01/14/2016-the-year-of-thequanti ed-student/
of data sources, and lack of theoretical foundations when designing and
building learning analytics [
In the next section we outline the context and motivation behind myPAL. In Section 3 we present the realisation of Quanti ed Self for self-regulated learning in myPAL. Section 4 lists the requirements of myPAL. Then Section 5 gives an overview of the state-of-the-art methods in text analytics and visualisation. Section 6 discusses the relationships between the myPAL requirements and the state of the art, and gives recommendations for future work. Finally, we provide conclusions and future work in Section 7. 2
In this section we rst provide the context of the myPAL project and then outline the pedagogical underpinning of our work. 2.1
The context of this work is a 5-year undergraduate programme leading to the degree of MBChB (Bachelor of Medicine and Bachelor of Surgery). A successful completion of the degree allows students to provisionally register with the General Medical Council and start supervised practice of medicine. In the UK as a requirement for unsupervised practice a further Foundation Year programme has to be undertaken.
As part of the MBChB degree a set of professional values and core themes is
integrated throughout the programme's ve years (a so-called `spiral' curriculum
[
Placements and clinical settings are an integral part of the degree. As they move through the degree years students increasingly spend time outside of traditional academic settings6. The students' progress is measured using an `entrustability' scale (from Observe to Supervise, Initiate and then Peer Teach), expressing higher level of attainment (and responsibility) in clinical settings7.
The design and delivery of the MBChB curriculum for the University is carried out by the Leeds Institute of Medical Education. Technology-enhanced learning is used extensively throughout the curriculum. Within the Institute,
7 https://www.medicine.leeds.ac.uk/mbchb/assessment/Expectations/
ExpectationsGuide(poster).pdf the Technology in Medical Education (TIME8) team has been responsible for developing and deploying digital resources to students. This has been done in close collaboration with clinicians, academics, students, patients and carers to ensure quality and relevance. We implement a Bring-Your-Own-Device mobileenabled paradigm to deliver TEL content.
Our primary focus now is on nding methods that further enrich that experience by enabling a more personalised and adaptive learning experience for students in medical education. 2.2
We put self-regulated learning at the core of the myPAL project. Through selfregulated learning students \actively research what they do to learn and how well their goals are achieved by variations in their approaches to learning" [43]. Crucially, self-regulated learning is an iterative process where the learner goes through phases of surveying resources, setting goals, carrying out tasks, evaluating results and making changes. Therefore, temporal traces are necessary to monitor and analyse this process [42]. In myPAL, Quanti ed Self tools enable carrying out learning analytics on such data at various levels of granularity.
The core part of myPAL is re ection on work practice to contextualise
practice within the medical curriculum, identify strengths and weaknesses, identify
learning opportunities, and devise a professional development plan. To articulate
and critique their growing understanding of practice, students need to engage
in `learning conversations' that are usually with their workplace tutors [
The timing of re ection activities is very important [
Guidance and supervision are key to re ective practice [
We envisage that this will impact the interaction with the learner, e.g. provide nudges to unleash the learner creativity (combined with nudges for re ection). 3
We adapt the work in [
Stages. The learner goes through three stages (cf. three rounded boxes at the top of Figure 3) which are captured with multiple sources and types of data in myPAL. Firstly, Experiences of the learner which encompass not just the actions, but also the resources available to the learner and their environment. The traces of these experiences available to us with myPAL are any data relating to the work placement (e.g. location, date) and assessment carried out on placement (e.g. assessment focus, assessor name). Secondly, Re ective process is captured predominantly with text data. This includes feedback (e.g. assessor feedback on placement) and reactions to feedback (e.g. student comments on feedback), as well as evaluation and feedback forms which form a broader context around the placement. Finally, Outcomes refer to any aspects of student performance. This can be summative (e.g exam results) or formative (e.g. number of assessments undertaken at placement, on-placement assessment scores). As self-regulated learning is iterative, these stages are repeated.
Quanti ed Self tools. In order to trigger and support the learner in their
re ective practice, we implement three stages of Quanti ed Self analytics tools
(cf. square box in the lower half of Figure 3). Firstly, Tracking involves initial
ingestion of archival data and then continuous monitoring and updating of student
actions throughout the three stages outlined above. In myPAL the initial stage
has been completed and continuous monitoring and updating established. All log
and input data are collected. The next (optional) stage is Analytics. This refers
to applying data mining methods, such as clustering or classi cation, to both
quantitative and qualitative data (cf. [
At his point in the myPAL project we have completed the rst stage (Tracking ) of Quanti ed Self tools development. Our primary goal in this paper is to survey the state of the art in analytics and visualisation for Quanti ed Self tools (cf. bolded items in the square box in Figure 3). In the following section we list the requirements of the myPAL project for the capabilities of these tools. 4
Following the above, the primary focus of our survey of the state of the art in analytics and visualisation (in bold in Figure 3) methods is to identify techniques that will ensure that these requirements for myPAL are met. R1: Provide analytics for multiple sources of quantitative and qualitative data. (a) Enable computational processing of both quantitative and qualitative data.
Using computational methods we are able to process larger and more complicated datasets, which might to lead to identifying patterns that will be useful to the learners. In myPAL, quantitative data includes log and assessment data, while qualitative data includes various types of text like feedback or comments. (b) Integrate multiple sources of data encompassing various aspects of the learning process. In order to build a holistic view of the student, we need to integrate data coming from various sources (e.g. nal exam results, onplacement assessment throughout the year, usage of learning resources) and relate them to each other.
R2: Develop reliable proxies of learner characteristics, including from short text. (a) Automatically extract relevant data from text (i.e. feature engineering).
This includes both characterising the text (e.g. in terms of writing quality) and characterising the learner (e.g. level of re ection). The characterisation of the learner such as classi cation needs to be openly presented to the learner to ensure scrutability. (b) Include methods for reliable extraction of information from short texts.
Much of the text data in myPAL is created on-the- y during work placements when time is scarce. This results in relatively short length of text { the average length of recorded assessor feedback on placement is only 15 tokens (which roughly correspond to words). The distribution of text length in our data is highly skewed towards fewer tokens.
R3: Provide interactive visualisations of temporal patterns and text at various levels of granularity (a) Provide interactive ways to explore and notice patterns in data over time.
Since rstly, self-regulation necessarily involves iterations of goal setting, evaluation, and goal re-setting, and secondly, students are expected to complete on-placement assessments throughout the year, visualising temporal data plays a key role in myPAL. A well-designed visualisation will enable students to notice patterns or `unpatterns' and help them re ect and act on those. It is also a source of feedback, which does not require additional resources from educators like describing the performance of the student in a given time period. (b) Provide interactive visualisations at di erent levels of granularity. Students should have the freedom to choose the level of detail in a visualisation: from a broad overview (e.g. a whole academic year), through the ability to lter to particular assessments (e.g. where performance was particularly low), to the ability to inspect individual assessment details (e.g. feedback on a particular placement). (c) Include methods for aggregating and visualising text data in a meaningful way. While most visualistions focus on numeric or categorical data, we need a method of visualising text data such as assessor feedback. This includes meaningful aggregation of text data, such as summarisation which would allow for an overview of on-placement feedback throughout the year (e.g. identify strengths or weaknesses that were frequently mentioned).
R4: Provide a combination of analytics and visualisations through the use of nudges that enables the student to understand, re ect, and make changes based on their data.
With these requirements in mind, in the following section we turn to the state-of-the-art methods that will enable us to realise them in myPAL. 5
To address the requirements for myPAL presented in the previous section we make use of two types of Quanti ed Self tools: text analytics and visualisation (in bold in Figure 3). In this section we identi ed the key determinations in these two areas that need to be made when designing and developing Quanti ed Self analytics tools for self-regulated learning with myPAL. 5.1
While analytics methods that are part of Quanti ed Self tools (cf. middle of
square box in Figure 3) are used with all types of data, there is a considerable
gap in research of text analytics methods. Winne [42] points out that \learning
analytics for SRL may bene t by blending counts and other quantitative
descriptions [. . . ] with semantic, syntactic, and rhetorical features [. . . ]". Qualitative
data such as text might be particularly useful when trying to gauge re ection
in learners [
Characterising the text. There is considerable literature on describing and
quantifying learner-produced writing. This work largely draws from research in
natural language processing (NLP), e.g. research on readability [
In the context of myPAL, we can implement metrics of text quality in order to identify feedback or comments that might be insu cient and prompt the student to recall the circumstances { in some cases the most valuable feedback had been given to the student verbally without being recorded in the app. Identifying particularly low-quality instances might be the rst step in such a scenario.
Charactersing the learner. One of the most challenging tasks for
researchers analysing text is nding the appropriate proxies (or signals) in the
text that point to some high level concepts which characterise the learners.
Examples include: comprehension of science concepts [
In myPAL, we want to focus on the re ective process (cf. middle rounded box in Figure 3). We will evaluate existing approaches and their suitability to work with our data. We might also consider developing new feature engineering methods for other learner characteristics (e.g. con dence).
Text length. Texts in the learning analytics domain vary widely in length
{ from documents numbering several paragraphs to very short text snippets.
The longer text types mostly include academic essays [
In myPAL, most text data is short (5-50 tokens). We will focus on dictionarybased methods, since we most probably do not have enough context for syntaxbased or discourse methods.
Text data in context. While most approaches in text analytics only look
at texts independently, there are also examples of considering the wider context
in which texts are produced. In particular, this is the case for MOOCs, where
both temporal and social contexts are present and text can be analysed as it
relates to those aspects [
In myPAL, we look particularly at the temporal aspect with the longitudanal data we have readily available. However, with the use of graph-based methods we can also explore the text content with relation to actors (students, assessors) and locations (placements).
Overall, considerable advances have been made in terms of characterising both learner-produced text and learners themselves through their writing. However, the depth of analysis largely depends on the type of text. Longer texts, such as essays, are the subject of re ective writing analytics research. For the myPAL project, we need to investigate the signals of awareness and re ection in much shorter texts (e.g. comments), as well as develop methods to meaningfully aggregate these snippets and relate them to other sources of data. 5.2
Visualisation is the nal stage in Quanti ed Self tools (cf. Figure 3 in the
middle). Many visualisation methods already address the topics of awareness and
re ection (37% of reporting systems reviewed in [
Types of visualisations. There is a wide range of visualisation types. [
In myPAL context, we will make use of all three types of charts with a focus on combining comparison charts and timelines. For example, how the number of on-placement assessments undertaken by the student changes over the year compared to the rest of the cohort.
Instruction vs. guiding. [
In myPAL, we will take the guiding approach through the use of nudges (cf. R4 in Section 4). As we take the co-design approach to development, we will have a good idea of general student impressions about nudges before we implement them. By design nudges are a guiding mechanism, however to ensure that the students retain autonomy in their use of myPAL a range of opt-out procedures will be implemented throughout the system.
Interaction and gami cation. In a review of student-facing reporting
systems [
In myPAL, we will focus on interactive visualisations that address all three steps of the Visual Information Seeking Mantra (overview, ltering, details). As stated in requirement R3c (cf. Section 4) we will also develop methods of visualising text data at these three levels, where the detailed view presents the individual texts while overview presents a broad summary. While we will not speci cally look at gami cation (this is outside of the scope of the current project), we might include some gami cation elements in visualisations if it proves to help with student engagement.
Overall, state-of-the-art methods in visualisation tend to (i) focus on comparisons, (ii) include some form of recommendation or guidance, and (iii) increasingly include interactive or gami ed elements. In order to meet myPAL requirements we need visualisations that include a broader learner path (i.e. timelines), the right choice architecture to guide learners, and an interactive visualisations that cover all three aspects of the visual information seeking (overview, ltering, details).
Based on the state of the art in text analytics and visualisation methods outlined in the previous section, we now relate these to the requirements of the myPAL project (cf. Section 4).
R1: Enable handling multiple sources of quantitative and
qualitative data. Using quantitative data (e.g. logs) can only tell us what interaction
took place when, however analysing the text associated with that interaction (e.g.
text of a comment) can provide further context and potentially explain the
motivation behind the action of the learner. Automatically produced quantitative
data can be more readily used for awareness, whereas intentionally produced
qualitative data like text can be used to measure { and potentially trigger {
re ection [
R2: Provide reliable proxies for learner characteristics, even with
short text. One of the biggest challenges when developing Quanti ed Self
analytics tools is bridging the gap between low level data from logs and high level
concepts characterising the learner like motivation or engagement. So far
quantitative data from logs has received the most attention when designing such
proxies (e.g. proxies for productivity or initiative using log data in [
R3: Provide interactive visualisations of progress and text at various levels of granularity. In order to support a learner in their self-regulation process they need to be confronted with their Quanti ed Self. Innovative and interactive visualisations can aid in that. A learner needs to have full choice of visualisations of their learning path at various granularities (overview, zoom and lter, details). Text data needs to be available not just at the details stage, but also aggregated at higher levels.
visualisation. The goal of using Quanti ed Self analytics tools is to enable the learners to be (i) better aware of, and (ii) better able to re ect on their learning. This can be achieved by using nudges, i.e. system interventions that give the students the choice to carry out an action that will help them. Being able to carry out sophisticated text analytics and developing innovative visualisation methods is not enough, if the learner does not actually process and re ect on the visualisation. 7
In this paper we presented the myPAL project which aims to provide personalised and adaptive learning to medical students. We will achieve this by applying Quanti ed Self tools (focusing on text analytics and interactive visualisations) to self-regulated learning. We listed what capabilities we require from Quanti ed Self tools and how these are addressed (or not) in the state-of-the-art methods in text analytics and visualisation. We identi ed key determinations that need to be made when designing and developing Quanti ed Self tools for self-regulated learning and how they relate to our design and implementation of myPAL.
We are now establishing the co-design framework. Students will not only participate in focus groups, but will also work with us in a co-design team as part of an iterative approach to explore their views and ideas about myPAL. Students will be involved in piloting the system and will regularly provide feedback and ideas for further developments.
In parallel, we will evaluate and develop methods for text analytics and interactive visualisations that meet the myPAL requirements. The rst stage of the text analytics research will determine to what extent short text snippets that are available to use can be used as proxies for awareness and re ection. The rst stage of visualisation research will create an overview- lter-details dashboard of the number and quality of on-placement assessments of clinical skills undertaken by students in Year 1.
The myPAL project is funded by the University of Leeds and the Higher Education Funding Council for England (HEFCE). We are grateful to the members of the Technology-Enhanced Learning team at the Leeds Institute of Medical Education who have developed the current version of the myPAL system which provides the backbone for realising the Quanti ed Self vision presented here. 38. Shum, S.B., Sandor, A., Goldsmith, R., Wang, X., Bass, R., McWilliams, M.: Re ecting on re ective writing analytics: Assessment challenges and iterative evaluation of a prototype tool. In: Proceedings of the sixth international conference on learning analytics & knowledge. pp. 213{222. ACM (2016) 39. Ullmann, T.D.: Re ective writing analytics-empirically determined keywords of written re ection. In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference (2017) 40. Van Labeke, N., Kirby, J., Roberts, T.E.: Personalised and adaptive mentoring in medical education{the mypal project. In: First International Workshop on Intelligent Mentoring Systems (IMS 2016) Proceedings (2016) 41. Wen, M., Yang, D., Rose, C.P.: Linguistic re ections of student engagement in massive open online courses. In: ICWSM (2014) 42. Winne, P.: Learning analytics for self-regulated learning. In: Lang, C., Siemens, G., WIse, A., Gasevic, D. (eds.) Handbook of Learning Analytics, pp. 241{249.
Society for Learning Analytics Research (2017) 43. Winne, P.H.: Bootstrapping learners self-regulated learning. Psychological Test and Assessment Modeling 52(4), 472{490 (2010)