=Paper=
{{Paper
|id=Vol-2193/paper5
|storemode=property
|title=How do Students Want their Workplace-based Feedback Visualised in order to Support Self-regulated Learning? Initial Results & Reflections from a Co-Design Study in Medical Education
|pdfUrl=https://ceur-ws.org/Vol-2193/paper5.pdf
|volume=Vol-2193
|authors=Tamsin Treasure-Jones, Rosie Dent-Spargo, Sisira Dharmaratne
|dblpUrl=https://dblp.org/rec/conf/ectel/Treasure-JonesD18
}}
==How do Students Want their Workplace-based Feedback Visualised in order to Support Self-regulated Learning? Initial Results & Reflections from a Co-Design Study in Medical Education==
https://ceur-ws.org/Vol-2193/paper5.pdf
How do students want their workplace-based feedback
visualized in order to support self-regulated learning?
Initial results & reflections from a co-design study in medical
education
Tamsin Treasure-Jones, Rosie Dent-Spargo and Sisira Dharmaratne
Leeds Institute of Medial Education, University of Leeds, Leeds, UK
t.treasure-jones@leeds.ac.uk
Abstract. Developing good self-regulated learning (SRL) skills is highly im-
portant for medical students, not only to help them to navigate and succeed at
their current study, but to support their continuing professional development and
lifelong learning once they enter the workplace. A key component of SRL is the
ability to reflect on feedback and to use this to spot gaps in knowledge/skills,
identify learning opportunities and plan new learning goals and activities. Tech-
nology can help by providing students with tools that scaffold their development
of these skills. This paper reports on the co-design of myPAL, a student-facing
learning analytics system. Within co-design workshops, we worked with students
to improve myPAL. These hands-on, creative workshops involved students in
discussion of their current and desired use of feedback, practical interface/visu-
alisation design and prototype use and adaptation. Using this participative ap-
proach we have identified one key visualisation and a set of functions/features
that students want to be available to help them to review and act on their feed-
back. In this paper we report and reflect on the co-design approach that has been
used, including the observed benefits of taking such an approach as well as its
limitations. We also outline the further work that is planned to develop & evalu-
ate the required improvements to myPAL.
Keywords: co-design, self-regulated learning, visualisations, workplace.
1 Background
Developing good self-regulated learning (SRL) skills is highly important for medical
students, not only to help them to succeed at their current study, but to support their
continuing professional development & lifelong learning in the workplace.
From a social cognitive perspective, feedback assumes a significant role in SRL and
serves the purpose of minimizing the gap between current performance of learners and
expected goals (Zimmerman, 2013; Hattie and Timperley, 2007). The cyclical process
of SRL highlights the ability to reflect on feedback and to use this to spot gaps in
2
knowledge/skills, identify learning opportunities and plan new learning goals and ac-
tivities as a key component (Panadero, 2017; Pintrich, 2000). Medical degree students
are provided with a wealth of feedback, which they are encouraged to use in this way
(Cho et al., 2017; Sandars and Cleary, 2011). One particularly interesting source of
feedback for SRL is workplace-based assessment (WBA) feedback, since this is pro-
vided (and potentially used) within the context in which students must be most self-
reliant. In these placements students are provided with minimal imposed structure to
their learning, since the learning opportunities presented will vary from day-to-day de-
pending on individual clinical settings and the presenting patients. Medical students
spend a significant proportion of their study in these workplace settings, since it allows
them to observe and practice skills, make links between their classroom-based learning
& practice and improve their understanding of the healthcare settings & cultures in
which they will eventually work. However, this rich workplace-based learning relies
on the students reflecting on their own & others’ practice and using these reflections to
plan their learning and thus improve their understanding and practice. Typically medi-
cal schools and placement organisations will provide some structure and guidance to
help students to make the most of these placement experiences.
For example, in the medical school in
which the research was undertaken, stu-
dents are provided with a list of clinical
tasks/procedures with a guidance on the
level of entrustability at which they should
perform them. Level of entrustability is a
scale starting from observation of a skill
performed by healthcare staff and moving
through supervised performance, unsuper-
vised performance to teaching peers. Stu-
dents are required to initiate & undertake
formative workplace-based assessments
(WBAs) to collect feedback from a clini-
Figure 1 Example Completed WBA cian (see Figure 1) on their performance of
these tasks. As part of this process the stu-
dent is also asked to reflect on the feedback
and complete a comment section which could include a learning action plan. The feed-
back is completed on a WBA app on the student’s phone and sent into their university
portfolio system. These WBAs provide the potential for the student to engage in mean-
ingful SRL, since initiation and identification of follow-on actions are all in their hands.
As such, the number, frequency and timing of engagement with WBA and quality of
students’ reflective logs become useful measures of SLR behavior.
However, students are in the process of transitioning from highly structured, school
learning environments to less structured, self-directed higher education and placement
learning environments, and therefore many have not yet fully developed the self-regu-
lated learning skills required to make the most of this feedback (Hughes and Smail,
3
2014; Juma et al., 2016). Analysis, of the relationship between WBA activities & sub-
sequent academic achievement, has identified that students with poor patterns of en-
gagement with WBAs are at greater risk of failing (Hallam and Fuller, 2017).
2 myPAL – Technology support for SRL
One possible solution to this problem is to provide the students with additional support
for self-regulated learning within the technology tools they are already using. myPAL
(myPAL, 2017) is a student-facing learning analytics system that has been developed
by one medical school in order to bring students’ learning data (assessment data, in-
cluding WBA feedback and, in the future, resource usage) into one system so that the
students can more easily review, reflect and act on this feedback. This is a live system
already in use by all 5 years of students (approx. 1,500) on the medical degree. How-
ever, this is also a system under continuous development and improvement, as we ex-
plore ways in which it can be enhanced to provide better support for self-regulated
learning. We are guided by a behavior change approach (Michie et al., 2011) and ex-
ploring two ways in which the system can ‘nudge’ students towards engaging in SRL
activity. One branch of this work (Piotrkowicz, A. et al., 2017; Piotrkowicz, A et al.,
2018) is exploring how we can use data analytics (text analytics, temporal analytics and
process mining) to identify patterns in student data and use these as the basis for
prompts (questions or suggestions) made to students within the system. In parallel, (the
work reported in this paper), we are also exploring how we can improve the data visu-
alisations in myPAL so that the interface facilitates and encourages SRL activity. In
behavior change terms this can be seen as an example of environmental restructuring.
These two branches of work are intertwined and inform each other, but in this paper we
will be focusing on the visualisation co-design work.
3 Co-designing myPAL visualisations
3.1 Objectives
The aims of our co-design work are:
To better understand our students’ current engagement with workplace-based
learning & assessment, including their use of myPAL & their SRL activities
To create designs for improvements to the myPAL visualisations, that have the
potential to provide support for SRL activities
In this paper we aim to provide readers with practical ideas & approaches that they can
apply in their own work. To this end, we describe in some detail the co-design process
that we followed, reflecting on its benefits and limitations, as well as highlighting some
open questions. We also identify initial results of what medical students want in terms
of technology support for workplace-based SRL, as well as discussing some of the ob-
stacles they encounter within this context. Future work will involve implementing the
improvements in the live system and evaluating their impact.
4
3.2 Approach
We adopted a participative, co-design approach in which we worked with students &
tutors to understand current workplace-based learning (WBL) behavior and co-design
new visualisations for myPAL. Co-design involves stakeholders in the design process
itself in order to help address some of the issues that have led to failures in technology
adoption. It recognizes that stakeholders bring a deep understanding of their context,
their needs and the opportunities that can then be explored with the developers and it
values their ability to create solutions themselves (Sanders and Stappers, 2008; Mor
and Winters, 2007). It is an approach that we have found to be useful in a range of
healthcare TEL research and development projects (Treasure-Jones and Joynes, 2017)
Participants & Procedure: Our work was based in a UK medical school. It involved
two phases of co-design workshops, the second building iteratively on the first. Each
workshop lasted between 60-90 minutes. Phase 1 workshops took place between No-
vember 2017 and January 2018. Phase 2 workshops all took place in February 2018.
Participants: Students, academic tutors (staff based at the University) and clinical tu-
tors (staff based in the placement settings) were invited to join the co-design workshops.
Invites were emailed to all medical students, lead academic tutors from each year and
clinical tutors who had engaged with continuing professional development activities
around education. Table 1 shows the participation across these different stakeholder
groups.
Y1 Y2 Y3 Y4 Y5 Clinical Academic Total
Tutors Tutors
Phase 1 3 7 8 7 1 7 6 39
Phase 2 2 7 4 5 1 3 8 30
In Phase 1 separate workshops were held with each group (except for Y4 and Y5
which were combined) for logistical reasons and to help preserve their unique view-
points. In Phase 2 Y1 and Y2 students were invited to one combined workshop and Y3,
4 and 5 were invited to another. This decision was taken because their perspectives and
experience had been judged to be similar following the Phase 1 discussions.
Participants were free to join one or both phases of the work. Sixteen (70%) of the
twenty three students from phase 1 returned to take part in phase 2 and three new stu-
dent participants joined. Two (29%) of the seven clinical tutors from phase 1 returned
to take part in phase 2 and one new tutor joined. Three (50%) of the six academic tutors
from phase 1 retuned to take part in phase 2 and five new tutors joined. Therefore across
both phases of the co-design work we had 48 participants involved, as well as 4 mem-
bers of the research team.
Phase 1 Co-design Workshops – Procedure. The co-design workshops were designed
to be welcoming and informal. The research staff facilitating the workshops were not
part of the teaching and assessment team for the medical degree and they encouraged
5
participants to be very open in their ideas and feedback. To help set the appropriate
mood and get people working together each workshop started with an ice breaker game.
In phase 1 the focus then moved onto three key questions 1:
1. How do you currently use myPAL? (discussion)
2. How do you approach your workplace-based assessments? (discussion)
3. What do you want to find out from myPAL? (hands-on activity)
The researchers posed these opening questions and facilitated the resulting discus-
sion. Questions 1 and 2 were posed to help us to understand participants’ current expe-
riences and behavior and also to encourage them to think about the context in which
the myPAL tool would be used, before moving onto question 3 and attempting to design
an interface.
In designing the hands-on part of this workshop (question 3), we faced some im-
portant challenges. We set out to design a format that would support participants, with
no assumed knowledge of app design, visualisation methods and self-regulated learn-
ing, to design complex interfaces to support students in planning their own learning, all
within a timeframe of around 45 minutes. In the workshop, we used a visual metaphor
of a bowl filled with snippets of student data to exemplify myPAL’s role as a data
aggregator and its potential to combine data sources to create visualisations and demon-
strate patterns. Despite priming participants in this way, we realised that simply pre-
senting participants with a blank sheet of paper could be potentially off-putting. So we
created a collection of common visualisation types (x and y axes, badge boards, time-
lines, etc.) taken from a well-known typography (Shneiderman, 1996), printed and lam-
inated them on A3 paper so that participants could adapt them in accordance with their
own ideas. In addition, we also provided blank sheets of A3 paper, pens and craft ma-
terials. The researchers were not merely observers in this process, we participated in
conversations, idea-generation and drawing, often working through ideas verbally with
participants to get them to the stage where they could be drawn on paper. In total 50
visualisations were created in the phase 1 co-design workshops. The full visualisation
dataset has been made available (Dent-Spargo et al., 2018).
Phase 1 – Decision-making Procedure. After each phase of co-design (creating ideas),
decisions need to be made about which ideas/designs will be taken forward into the
next design iteration. This decision-making was undertaken by the myPAL manage-
ment team. This interdisciplinary team includes medical educators & researchers, TEL
researchers, developers, administrators and IT specialists. Together they bring a wide
range of perspectives and collectively they are tasked with identifying the most prom-
ising work to take forward in terms of valuable educational support, development fea-
sibility & research value.
At the end of phase 1 this team undertook a review and analysis of the 50 created
visualisations with the aim of identifying a small number these to be implemented as
1 For tutors these were slightly rephrased – e.g. for academic tutors question 2 became “How
would you ideally like students to approach their workplace-based assessments?” and for all
tutors question 3 became “What would you like students to find out from myPAL?”
6
prototypes and taken into phase 2 of the co-design. In order to do this, the team tagged
each visualisation by the questions they felt it was answering. A list of eight questions
was created that the team judged to cover all 50 visualisations. The team then calculated
which questions were most frequently “asked” (Dent-Spargo et al., 2018) and then fo-
cused on the visualisations that covered the 3 most popular questions. As an output of
this simple analysis and the resulting discussion (about perceived educational value,
usability and ease of development) three participant-designed visualisations were cho-
sen to take into phase 2.
Phase 2 Co-design Procedure. The 3 prototypes that were taken into phase 2 were
created as physical paper prototypes and clickable electronic paper prototypes 2. This
meant that we had on the table the physical paper prototypes themselves, which the
participants could easily annotate, amend or re-order. However, the participants could
also explore the user journey and functionality more interactively by using the elec-
tronic paper prototype which was installed on iPads provided to them at the workshop.
The reasoning behind this approach was that we wanted to ensure that the message we
conveyed was that these were work in progress: designs that participants could still
easily adapt or even reject. We felt that if we had gone straight to wireframes then we
would have given the impression that the designs were more fixed and further devel-
oped than was in fact the case. We did not want to lock-in decisions too early in the
process.
The phase 2 workshops started with an ice breaking activity & then moved into an
exploration of each of the 3 prototypes in turn. Each prototype was briefly presented,
then the participants were given the chance to explore it on the iPad before providing
their comments and further ideas. This discussion was structured around 3 questions:
1. Do you feel you understand what this is showing you?
2. How would you use this?
3. If this were someone else’s data, what prompting questions would you ask
them as they looked at this?
These questions were chosen to help us to identify whether the visualisation was intui-
tively understandable, whether the students envisaged using this for SRL activities and
potential places were nudges/prompts could be added to the interface. Whilst these
opening questions were phrased positively, it was also made clear to the participants
that they could provide negative feedback as well, which they did. Participants were
also encouraged to make changes on the paper versions of the prototypes.
Phase 2 Decision-making Procedure. A series of meetings was held with the myPAL
management team to review the feedback collected during the phase 2 co-design work-
shops. The researchers created and shared a summary of this data with the myPAL
management team. This summary identified which visualisation was most positively
received across the workshops (this was unanimous) and included a list of changes and
additional features/functions which the participants suggested for this visualisation.
The myPAL management team collectively reviewed this and prioritized the changes
2
These clickable electronic paper prototypes were created using marvel https://marvelapp.com
7
to take into the next stage of co-design. This prioritization again was considered from
an educational, research and practical development perspective. Throughout we fol-
lowed a consensus decision-making process, with decisions agreed, by the group, fol-
lowing consideration and discussion of these different perspectives. The prototype will
now be developed into a working visualisation within the development environment of
the myPAL system and this visualisation will be explored and used by participants in a
third stage of co-design workshops planned for Autumn 2018.
3.3 Data collection
The discussions in all of the co-design workshops were audio-recorded and transcribed.
Additionally the artefacts created in the workshops (drawings, notes, annotated visual-
isations and researchers’ fieldwork notes) were all digitized (photographed or scanned)
so that they could also be included in to the dataset for later analysis.
3.4 Two stage analysis – for decision-making and for deeper understanding
As has been described earlier, each phase of co-design is followed by a decision-
making phase. This decision-making phase involves analyzing the data gathered, but
this analysis is undertaken as part of the agile development cycle. As such it has to be
completed fairly quickly in order to let development work move forwards. It is therefore
done at a high-level, without waiting for the completion of the detailed, qualitative
analysis of all the rich data gathered. It relies on the multiple perspectives, brought by
the members of the interdisciplinary myPAL management team, in order to pick out
key features and messages from the data and use this to take collective decisions about
which designs to take to the next development iteration.
However, the rich data we are gathering during these co-design workshops also pro-
vides us with the opportunity to undertake a much more detailed qualitative analysis to
gain a deeper understanding of the students’ approaches to WBAs, their SRL activity
and the opportunities and challenges in using TEL to support this. This more detailed
analysis will take several months to complete. All the data has been uploaded to NVivo3
(a qualitative data analysis tool) and the authors of this paper are in the process of un-
dertaking a thematic analysis (Braun and Clarke, 2006) of this data. Some initial reflec-
tions are included in the later section of this paper, however the full analysis will be
reported in future papers and ideally will also inform later iterations of the myPAL
development. We also anticipate that this work will identify other opportunities within
the curriculum and the placements to support SRL. We will not be restricting this to
technology interventions alone.
3
NVivo - https://www.qsrinternational.com/nvivo/home
8
4 Results and discussion
4.1 Initial results – desired visualisations, functionality and use cases
Phase 1: The most popular questions that students wanted the visualisations to answer
(based on the myPAL management team’s analysis) were:
How am I doing? (high-level overview)
How am I progressing?
What do I need to do?
There was less interest in, and some concerns about, views that would compare their
progress to that of their peers. Comparisons to course expectations were less conten-
tious. The sequential stepping through these 3 most popular questions can be seen as
helpful for SRL, as students did appear to want to identify and fill gaps in their
knowledge. However, this was perhaps more focused on filling required curriculum/as-
sessment gaps, rather than knowledge/skills gaps identified opportunistically through
engagement with workplace-based activities.
The three prototypes chosen to take through to phase 2 are shown in Figs 2, 3 & 4.
Figure 2 Prototype 1 WBA Skills Overview
Prototype 1 shows an overview of the students’ completion of WBAs, with the option
to drill-down for more information on each. Prototype 2 shows a timeline view of the
students’ WBA, with the ability to filter on a range of characteristics. Prototype 3 shows
a visualisation that fore-fronts their action planning.
9
Figure 3 Prototype 2 - Timeline view Figure 4 Prototype 3 - Action plan view
Phase 2: Based on an initial analysis of the transcripts and on the reports from the
researchers involved in the co-design workshops, it was clear that Prototype 1 was per-
ceived by both students and tutors to have the greatest potential to support students’
reflection and action on their WBA feedback. It was considered to be easily understood
and to provide a clear view that allowed students to compare their current progress
against expectations and drill-down to see the feedback. Students reported that they
would use it to plan their activities on placements.
Prototype 2 received the least favourable reaction. It was considered to be difficult
to interpret. Some of the filtering options were not considered to be helpful or mean-
ingful (e.g. the location of the assessment). The students could not easily envisage a
way in which they would use this visualisation to support their learning.
Prototype 3 received mixed comments. It was considered to be useful to be able to
fore-front the action planning. However, it was felt that this would be most useful if
this calendar view also included all coursework deadlines and other personal tasks – all
planning in one place. Yet, at the same time, all the students indicated that they already
used other systems (apps, calendars) to keep track of their other tasks and that they
would be very unlikely to move all their task management into this interface.
Therefore Protoype 1 was chosen as the visualisation to be developed and taken into
the next phase of co-design. However, some changes will be made, based on the sug-
gestions raised in the co-design workshops and the decision of the myPAL management
team. These changes include providing filtering and flagging (student suggestions) and
providing more prominent display of the feedback and action planning associated with
the assessment and a temporal view of progress (tutor suggestions). So in fact the
amended Prototype 1 will incorporate the aspects of the other prototypes that were
deemed to be useful (meaningful filters, temporal progress and action plan view), whilst
keeping its fore-grounding of an overview by skill completion.
10
4.2 Reflections on the desired support: congruence with theory
The most popular questions students were seeking answers to through visualizations
largely coincided with the four main functions for formative feedback which are also
compatible with the cyclical process of SLR (Hattie and Timperley, 2007; Panadero,
2017; Pintrich, 2000). The four main functions which coincided with students expec-
tations are a) ‘how did I do ?’, b) ‘where am I going ? (goals)’, c) ‘How am I going ?
(progress towards these goals)’, and ‘Where should I go next ?’. The observed con-
gruence supports the potential for transferability of our findings to other contexts
where feedback from formative WBAs could be digitally collected and processed.
Providing visualisations to answer these questions involves some level of digital pro-
cessing of the basic feedback data, and can be seen as therefore providing ‘digital
scaffolding’ of the SRL process. The technology would be doing some of the analyti-
cal or reflective processing that traditionally would be undertaken by the learner
themselves in SRL. Therefore we recommend that future work explores the impact of
providing such scaffolding and considers when it would be beneficial to fade it out in
order to pass greater responsibility and autonomy back to the student.
4.3 Reflections on the co-design process
Providing participants with the opportunity and support to create solutions:
Overall we found the co-design process to be rich and dynamic, allowing participants
enough structure to feel supported whilst also enabling them to be creative. Many ses-
sions began with a participant claiming they were not creative and could not draw but
almost without exception, every member of the workshop was able to contribute
something of value in a medium they had not used before. There were however chal-
lenges in this approach - participants often suggested ideas that would be difficult to
implement either from a development perspective or because they involved data not
currently collected by the medical school. This challenge means that managing expec-
tations is a particularly important part of the co-design process. Our approach was to
let all ideas be created (none were rejected in the workshops themselves) but to make
participants aware of the fact that it would not be possible to implement everything.
Engagement and feelings of ownership: As we have found in previous co-design
work (Kämäräinen et al., 2017), we observed that over time many participants devel-
oped a strong sense of engagement in the process and a feeling of ownership of the
developing solution. One of the tutors commented that: “I feel really invested in it and
I would really like to know what comes of it” (Clinical Tutor, P2 Workshop). A year 3
student commented that “I feel I’ve actually contributed quite a lot as a student <…>
that’s quite rewarding to have, as a student, knowing that the app is going to include
things that I and the majority of the other students would like”. Interestingly another
year 3 student felt engaged but brought a different (more critical) perspective “I was
quite enthusiastic about the whole idea by the end of the two meetings <…> In terms
of my contribution I think was more of a sceptic <…> I tried to bring a bit of reflection
and realism to the situation.” Both of these quotes are from a video that the students
made in their own time to contribute to a reflective presentation of the myPAL project
(Dent-Spargo, 2018).
11
The high retention rate between phases 1 and 2 of the co-design (70% of students,
29% of clinical tutors and 50% of academic tutors) also indicates that participants were
engaged and committed to the project. This is particularly notable given that all partic-
ipants (students and tutors) have very heavy workloads and it is therefore difficult to
find the time to attend the sessions.
One open question for the co-design approach is whether this feeling of ownership
in the solution only extends to those who actively participate in the co-design or
whether their fellow students (who did not join the workshops) also regard the final app
as a student-designed solution. Another open question is whether the feeling of owner-
ship and engagement persists even if student designs are dropped at the decision stage.
We are very aware of the fact that in this participatory study, the students do not in fact
participate in the decision-making itself.
In terms of developing tools that are suitable for the wider user group there is poten-
tially also a danger that the students who take part in co-design may not be a representa-
tive group. Additionally their growing feelings of ownership of the solution might also
make it more difficult for them to provide critical perspective.
Lag between development and research-oriented analysis cycles: As described ear-
lier, we undertook an initial high-level analysis of our rich data after each co-design
phase and used this analysis to inform the development decision-making process. This
decision was taken since the agile development cycle could not reasonably wait for a
detailed qualitative analysis to be undertaken. This approach does introduce a risk that
the later qualitative analysis will highlight something that was missed in the decision-
making analysis, and thus could led to inefficient development if subsequent changes
had to be made. By involving multiple perspectives from the interdisciplinary team in
the collective high-level decision-making process we aim to minimize the chance of
this happening. However, this is something we will be monitoring in order to assess
whether this two-stage analysis is a reasonable compromise or whether it leads to wrong
decisions being made and developments having to be backtracked.
4.4 Reflections on students’ views on workplace-based learning
We are undertaking a thematic analysis of the rich data collected during the co-de-
sign workshops. Our aim is to identify insights into the nature of workplace-based
learning, potential limitations in the WBA and feedback process and to potentially iden-
tify further opportunities for support for SRL both within and outside myPAL.
A preliminary thematic analysis has been conducted on one of the richest workshop
transcripts (P1Y4 - Phase 1, Year 4), which exemplified some of the important obser-
vations made across workshops. We share some of these observations below, but note
that this is an ongoing piece of work.
Students are rarely able to formally plan their own workplace-based learning as their
ability to complete assessments depends on many factors such as the type of placement,
the schedules of assessors and the patients available. Students saw workplace-based
learning as a ‘… dynamic learning process…’ where learning was ‘…fluid and on the
spot …’
12
They considered the chances for formal WBA within the workplace to be ‘…very
opportunistic…’. For example, situations where a suitable clinical encounter coincided
with a clinician who was ‘… nice… willing to do it … and isn’t super stressed…’ were
‘…hard to find’. Students also found and agreed that student feedback can be the least
priority under certain circumstances, especially “…when there’s a life on the line…”
Yet, within such natural constraints of the workplace students also found that they still
have ‘… lots of opportunity to volunteer…’ and complete WBA activities.
However, not all of these encounters ensured usefully recorded feedback. It was not
uncommon to receive less specific & non-descriptive feedback such as ‘practice more’
or ‘improve confidence’ which were ‘… not really helpful’. It was felt that in some
cases student performance &feedback may be ‘… difficult to encapsulate … in an offi-
cial, written down way’ for the purpose of recording and reviewing. Potentially the
WBA app provided by this medical school can help by providing a structure to the
feedback and (by imposing minimum character limits) a way to promote longer/richer
feedback. However, students felt this didn’t necessarily work given the ‘…pressures on
doctors’. The option within the WBA app to record feedback verbally was not known
by all, and sometimes was impractical as the ‘… ward was just too noisy…’
It was interesting to note however, that specific and constructive feedback was val-
ued and students did report acting on it to ‘… try and rectify…’ their deficiencies ‘…
quite soon’. This appears to happen because these students make concrete plans for
future learning based on useful feedback.
This initial insights suggest that there is potential for technological interventions to
support SRL within these real workplace settings, but that technology is likely to be
only one part of the solution.
5 Further work
The thematic qualitative analysis of the rich data from the co-design workshops is
ongoing. We expect to finish this and have results ready during the summer and there
is still the potential for lessons from this analysis to have some influence on the phase
3 co-design work. We recognize that the numbers of students involved is relatively
small and so we also plan to design a survey, informed by our qualitative analysis, to
collect responses from a much larger set of students about their workplace-based learn-
ing approach & use of TEL in order to check and verify the qualitative findings.
The visualisation design that has arisen from these two phases of co-design work
now needs to be implemented in the development system and tested/evaluated within a
third phase of co-design. The development work is already underway and the next co-
design workshops and activities (which will involve using the visualisation on live data
and exploring use cases) are planned for Autumn 2018. These workshops should give
us the first indications of whether and how this visualisation can support SRL activities
connected to these WBAs. Future work will roll this into the live system and evaluate
whether it leads to actual changes in SRL practice. Our co-design work has provided
us with some indication of current practice, against which we can compare. We can
also explore using proxy measures for SRL (such as the data on WBA engagement that
is described in Section 1) to measure the impact of the new visualisations. Another
13
research question, we would be keen to explore in our future work, is whether the own-
ership of the solution, felt by the students who participated in the co-design, extends to
the wider student group and whether that supports adoption.
6 Conclusions
There is a clear need to provide some medical students with additional support to help
them develop better self-regulated learning behaviour, particularly in the context of
workplace-based learning. This intensive co-design study has resulted in the design of
a visualisation and functionality that students and tutors believe could provide addi-
tional help. The students placed greatest value on visualisations that could give them
an overview of how they were doing, show their progress and signpost what they still
needed to do. The preferred visualisation presented an overview focused on skills com-
pletion with the option of drilling down to get more detail on feedback, filtering for
(and focusing) on certain activities and flagging skills. The co-design approach that was
used was successful in supporting both creative activities and feelings of ownership and
engagement with the solution. It also created a very rich dataset, which is already yield-
ing more detailed insights into students’ workplace-based learning behaviour. There is
further work to be done on testing the designed visualisation to see if it does lead to
changes in behaviour and further analysis of the rich dataset to help us identify further
opportunities to support SRL.
7 Acknowledgements
This work would not have been possible without the support of the whole myPAL team
(past and present) and all our co-design participants. We would like to thank them for
their support, enthusiasm and creative & critical engagement.
References
Braun, V. and Clarke, V. 2006. Using thematic analysis in psychology. Qualitative Research in
Psychology. 3(2), pp.77-101.
Cho, K.K., Marjadi, B., Langendyk, V. and Hu, W. 2017. The self-regulated learning of medical
students in the clinical environment - a scoping review. BMC Med Educ. 17(1), p112.
Dent-Spargo, R. 2018. myPAL HEFCE ePoster. [Online]. [Accessed 28th April]. Available from:
https://api.ltb.io/show/BEDBW
Dent-Spargo, R., Dimitrova, V., Gatrell, R., Hallam, J., Harkin, P., Hennessy, S., Piotrkowicz,
A. and Treasure-Jones, T. 2018. myPAL Co-design Phase 1 Visualisations. [Online].
Available from: https://doi.org/10.5518/404
Hallam, J. and Fuller, R. 2017. Exploring and improving a programme of Workplace Based
Assessments (WBA) using personalised ‘nudges’. [Online]. [Accessed 28th April].
Available from: https://amee.org/getattachment/Conferences/AMEE-Past-
Conferences/AMEE-2017/AMEE-2017-Abstract-Book.pdf p308
14
Hattie, J. and Timperley, H. 2007. The power of feedback. Review of educational research. 77(1),
pp.81-112.
Hughes, G. and Smail, O. 2014. Which aspects of university life are most and least helpful in the
transition to HE? A qualitative snapshot of student perceptions. Journal of Further and
Higher Education. 39(4), pp.466-480.
Juma, F.Z., Abas, T. and Banu, S. 2016. The transition from high school to university: a medical
student's perspective. Adv Med Educ Pract. 7, pp.517-518.
Kämäräinen, P., Treasure-Jones, T., Atwell, G. and Dewey, R. 2017. Changing and Sustaining
Practices in the Healthcare and Construction Sectors? - Reflections on the Achievements
of the Learning Layers Pilots. [Online]. [Accessed 28th April]. Available from:
http://results.learning-layers.eu/scenarios/changing-sustaining/
Michie, S., van Stralen, M.M. and West, R. 2011. The behaviour change wheel: a new method
for characterising and designing behaviour change interventions. Implement Sci. 6(42).
Mor, Y. and Winters, N. 2007. Design approaches in technology-enhanced learning. Interactive
Learning Environments. 15(1), pp.61-75.
myPAL. 2017. myPAL project website. [Online]. [Accessed 28/4/2018]. Available from:
http://mypalinfo.leeds.ac.uk/
Panadero, E. 2017. A Review of Self-regulated Learning: Six Models and Four Directions for
Research. Front Psychol. 8, p422.
Pintrich, P.R. 2000. An Achievement Goal Theory Perspective on Issues in Motivation
Terminology, Theory, and Research. Contemp Educ Psychol. 25(1), pp.92-104.
Piotrkowicz, A., Dimitrova, D., Treasure-Jones, T., Roberts, T. and myPAL. 2018. The Concept
of the 'Quantified Self' and Its Use in Developing Lifelong Learners. In: American
Educational Research Association (AERA) Annual Meeting, New York, USA.
Piotrkowicz, A., Dimitrova, V., Treasure-Jones, T., Smithies, A., Harkin, P., Kirby, J. and
Roberts, T. 2017. Quantied Self Analytics Tools for Self-regulated Learning with myPAL.
[Online]. [Accessed 28th April]. Available from:
http://eprints.whiterose.ac.uk/121100/1/final-quantified-analytics.pdf
Sandars, J. and Cleary, T.J. 2011. Self-regulation theory: applications to medical education:
AMEE Guide No. 58. Med Teach. 33(11), pp.875-886.
Sanders, E.B.N. and Stappers, P.J. 2008. Co-creation and the new landscapes of design.
CoDesign. 4(1), pp.5-18.
Shneiderman, B. 1996. The Eyes Have It: A Task by Data Type Taxonomy for Information
Visualizations. In: Proceedings of the 1996 IEEE Symposium on Visual Languages.
834354: IEEE Computer Society, pp.336-343.
Treasure-Jones, T. and Joynes, V. 2017. Co-design of technology-enhanced learning resources.
Clin Teach. 14, pp.1-6.
Zimmerman, B.J. 2013. From cognitive modelling to self-regulation: a social cognitive career
path Educ Psychol. 48, pp.135-147.