=Paper=
{{Paper
|id=Vol-1738/IWTA_2016_paper8
|storemode=property
|title=Improving Personalized Feedback at the Workplace with a Learning Analytics enhanced E-portfolio
|pdfUrl=https://ceur-ws.org/Vol-1738/IWTA_2016_paper8.pdf
|volume=Vol-1738
|authors=Marieke van der Schaaf,Geraldine Clarebout
|dblpUrl=https://dblp.org/rec/conf/ectel/SchaafC16
}}
==Improving Personalized Feedback at the Workplace with a Learning Analytics enhanced E-portfolio==
https://ceur-ws.org/Vol-1738/IWTA_2016_paper8.pdf
Improving Personalized Feedback at the Workplace
with a Learning Analytics enhanced E-portfolio
M. van der Schaaf G. Clarebout
Utrecht University, The Netherlands Maastricht University, The Netherlands
3508 TC Utrecht, The Netherlands
+31 (0)30 2534944
m.f.vanderschaaf@uu.nl
independently executable within a time frame, observable
ABSTRACT and measurable in their process and outcome, and,
therefore, suitable for entrustment decisions. This is a
During workplace based learning, e.g. clinical practice or
promising route that is now being explored and
during an internship, there is an urgent need for solutions implemented in several countries across the globe (e.g.
to restore and to guarantee the quality of feedback for
USA, Canada, Australia, Singapore, The Netherlands).
(becoming) professionals. In continuing education at the
workplace the use of Electronic portfolios (EPs) is a So far the implementation of E-portfolios in workplace-
crucial means for acquiring the requisite professional based learning is often ineffective; its quality (in terms of
knowledge and skills. Although EPs provide a useful validity and reliability) is generally low and moreover the
approach to view each trainee’s progress, often only impact on learning is limited (Van Schaik, Plan, &
limited use is made of the rich contextual learning O’Sullivan, 2013). This seems especially the case when
assessment data to support responsive adaptation for the E-portfolios are not tailored to show what really
more efficient and rewarding training and hence to happened in the workplace (Van der Schaaf, Stokking, &
provide personalized feedback. This contribution Verloop, 2008). Part of this failure may be attributed to a
advocates that EPs enhanced with a Learning Analytics wish to translate competencies, designed as rather
engine, may increase the quality and efficiency of theoretical descriptions of professionals, into items in a
workplace-based feedback and assessment. This portfolio for assessment. Furthermore, potential data
contribution addresses this by outlining an approach that about trainees’ behaviour in the workplace are often
is applied in a European 7th framework project, called underused, because the management of the data is too
WATCHME (www.project-watchme.eu). The aim of the complex for the trainees and their supervisors. This paper
contribution is to provide insight in underlying rationales addresses this by outlining an iterative development
to improve workplace-based feedback and assessment approach that is applied in a European 7th framework
and how this is applied in an EP environment that is project, called WATCHME (www.project-watchme.eu).
enhanced with Learning Analytics. The project uses an E-portfolio system that is enhanced
with a Learning Analytics (LA) engine to provide
Keywords personalized (just-in-time) assessment and feedback. LA
Learning analytics; workplace-based learning; include the measurement, collection, analysis and
competencies; electronic portfolios. reporting of data about learners and their contexts, for
purposes of understanding and optimising learning and
1. INTRODUCTION the environments in which it occurs (Clow, 2013;
Feedback at the workplace is crucial for trainees to Ferguson, 2012; Siemens & Long, 2011).
become professionals. Paralleling the movement towards
The design approach for the LA engine that drives the E-
alternative assessments of students (Boud, 1990;
portfolio is of a cyclical nature based on ongoing
Birenbaum 1996), (becoming) professionals are
refinement and improvement of the engine during
increasingly assessed using competence-based
successive phases of collection, analysis and visualising
assessment instruments, such as portfolios. A portfolio
information (Baker & Yacef, 2008; Elias, 2011). Though
contains selected evidence of trainees’ learning
LA are driven by a computerised processing of large
processes, their performances and products in various
amounts of data, the analytical process is a ”single
contexts, accompanied by supervisors’ comments and
amalgam of human and machine processing which is
reflections (Wolf & Dietz, 1998). Depending on its
instantiated through an interface that both drives and is
content and mode of presentation an electronic portfolio
driven by the whole system, human and machine” (Dron
(E-portfolio) can do justice to the fact that professional
& Anderson, 2009, p. 369). Student Models will be used
practice is complex and context dependent.
as a means of analysis, the results of which will lead to
In this paper we use Entrustable Professional Activities two types of feedback: Just-in Time feedback messages
(EPAs) to describe units of professional practice that and visualization of both individual and aggregated data.
underlie workplace-based feedback and assessment In order to provide meaningful just-in-time information,
(Gilhooly, Schumacher, West & Jones, 2014; Jones, the Student Model should represent the actual internal
Rosenberg, Gilhooly, & Carraccio, 2011; Ten Cate, state of each trainee as well as their actual learning
2013). EPAs are tasks or responsibilities entrusted to be context. For this, it must be able to interpret the contents
executed by an unsupervised learner once sufficient of the E-portfolio. The Student Model should also contain
specific competence has been obtained. EPAs are enough pedagogical knowledge in order to be able to
ELECTRONIC PORTFOLIOS FOR WORKPLACE-BASED FEEDBACK 2
translate the internal state and context into meaningful perform at several EPAs at the workplace and on a
messages and information for visualization. Key in second level reveals their performance on the underlying
enhancing E-portfolios with LA is that data about competencies. The personalized feedback aims to give
trainees’ workplace performances are linked to trainees insight into their obtained score compared with
assessment and feedback scores. This requires the the expected norm (they can infer at what entrustment
alignment of a statistical model with a substantive theory, level they are), it provides them the chance to evaluate
operationalized in EPA descriptions, regarding expertise and monitor the own process (trainees need to reflect
development in the profession. To this end, an iterative upon their performance) and the exemplar performance
development approach, using various cycles will be (example feedback) gives suggestion upon how to close
applied. the gap between the expected norm and the actual
performance. Hence, the feedback is based upon the three
The aim of this contribution is to develop a design for principles of effective feedback and uses exemplar
personalized feedback in a LA-driven E-portfolio. The performance (Sadler, 1989; 2010).
central question is: How can a LA-enhanced E-portfolio
improve feedback at the workplace to enhance 3. Student Model
(becoming) professionals’ development? Decisions on entrustability (or proficiency) levels for
EPAs are made on the basis of a set of workplace-based
2. Personalized Feedback assessments, not using strict addition of scores but using
High quality feedback is essential to stimulate rich, partly narrative, information. This means that a crisp
(becoming) professionals’ EPA development. Feedback rule-based approach is not feasible whereas a
can be conceptualised as information provided by an
probabilistic approach is able to deal with the
agent regarding aspects of one’s performance or uncertainties in this type of decision making. The
understanding. For feedback to be effective certain underlying Student Model needs to be able to advice on
conditions must hold; the feedback must be given timely (at least):
and adequately, it needs to be of high quality, and
learners should be able to act upon the feedback (Gibbs & 1. Prediction of entrustability: What is, at this moment,
Simpson, 2004). Furthermore, there is a large body of probably the current level of
research to show that good feedback leads to achieve entrustability/proficiency for a trainee in a given
aimed performances (Nicol & Macfarlane-Dick, 2006). EPA? This can be expressed as a probability
At least three conditions should be fulfilled for feedback distribution over the levels x for that EPA given the
to be effective: 1) it gives insight into obtained current evidence:
performances compared to an expected norm, 2) it gives 𝑃(𝑙𝑒𝑣𝑒𝑙 𝑥 𝑓𝑜𝑟 𝐸𝑃𝐴 | 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 𝑒𝑣𝑖𝑑𝑒𝑛𝑐𝑒 𝑖𝑛 𝑝𝑜𝑟𝑡𝑓𝑜𝑙𝑖𝑜)
the ability to evaluate and monitor the own process and 3) If feasible, a Value-of-Information analysis can be
it gives suggestions to fill the gap between the expected performed to identify the unknown variables that
norm and the actual performance (Sadler, 1989; 2010). would provide the most information to answer.
Hence, helpful feedback states what aimed performances 2. Selection of feedback: What is the best feedback to
are and how current performance is related to the select for a given trainee at a given moment?
performances aimed at. Further, it provides action points 3. Selection of topic of interest: What EPA, task or
on how to close the gap between current and aimed competency is at the moment the most of interest for
performance. Furthermore, effective feedback enhances trainee/supervisor?
learning when it provides answers to the following
question: Where am I going? How am I going? and
Where to next? (Hattie & Timperley, 2007). It is thus
important that trainees get acquainted with the goals and
‘criteria’ of an EPA, infer how they performed and know
how to enhance their performance.
Trainees can only achieve development goals when they
understand those goals and can assess their progress
(Sadler, 1989). One approach that is particularly powerful
in clarifying goals and standards has been to provide
trainees with rubrics (Dekker-Groen, Van der Schaaf &
Stokking, 2012). Rubrics can be effective because they
make explicit what is required of trainees’ performance,
they define a valid standard against which trainees can
compare their work and hence, may enhance trainees’
further learning.
This contribution focuses on providing trainees
personalized feedback on the process of becoming a
professional. The feedback module is based on EPAs that
go with rubrics that describe entrustability or proficiency
levels. It consists of a personalized feedback module
(JIT) and a visualization module (VIZ). This JIT and
visualization uses Student Models to depict how trainees’
ELECTRONIC PORTFOLIOS FOR WORKPLACE-BASED FEEDBACK 3
that describes the educational context is specific for each
trainee and needs to be re-constructed frequently, since
4. Designing LA-enhanced Electronic the actual educational context changes continuously.
Given the high levels of uncertainty in the educational
Portfolios domain, probabilistic approaches are appropriate and
The design of a LA-enhanced E-portfolio in our project graphic models such as Bayesian networks support the
demanded interdepended phases in which the involved modular structure most appropriately. Before SMs can be
educational and technical partners have to answer developed, different questions have to be answered
specific questions. amongst the users, e.g.: When do users require feedback?
Phase 1. Development of EPAs and assessment How do users perceive feedback? What timing of
instruments. Phase 1 started the development cycle by feedback is useful?
defining the competencies needed and types of evidence Phase 3. Development of initial Personalized
(e.g. products and performances) that should go in the E- Feedback Module. This phase addressed the
portfolios for valid workplace-based assessment. Users development of initial Personalized feedback module that
(experts and trainees) are consulted to generate markers produces, on the basis of information retrieved from
for progress within the professional domain and Student Models, feedback to trainee and supervisors.
consensus will be sought to arrive at generalizable Also visualization modules (VIZ) are developed that will,
weighted markers that will be suitable to translate to on the basis of information retrieved from Student
Learner Analytics input, i.e. the “Student Models” in Models and portfolio data, produce informative graphical
phase 3. Main questions to be answered are: what representations of aggregated and individual data, see
competencies need to be assessed and what types of Figure 2. The detailed designs of JIT and VIZ demand
evidence (e.g. product, performance, processes) should input from the users on questions like: What kind of
go in the E-portfolios? In previous studies, in which we feedback do they prefer, with what graphically display?
used a Delphi technique (Linstone & Turoff 1975), What are the time constraints for giving and receiving
stakeholders successfully developed EPAs for the feedback? What kinds of devices are available when
professional fields of medical education, veterinary assessment is performed and received? The personalized
education and teacher education. See Figure 1 for an feedback module will be accessible from the E-portfolio,
example of teacher education. representing the output of the underlying SMs. The SM is
Phase 2. Development of Student Models. Phase 2 took a back-end service in itself and is not available for user
the output of phase 1 and technical considerations, such interaction in the display, but the JIT and VIZ that are
as scalability, into account. Educational mining tools and driven by SM are. See Figures 2a-2c. These figures show
techniques are selected that will be deployed to learn, a possible example of personalized feedback and EPAs
update and store the Student Models. Student Models attained. The personalized feedback is dynamic and
(SMs) are statistical models that predict trainees’ continually receives input from new incoming portfolio
progress based on existing data. They translate the data. The final display knows several layers providing
portfolio and assessment data into the progress state of extra detailed information when one clicks on a certain
the trainee. As a consequence SMs will predict the actual graph, message etc. in the display.
state of performance of each trainee within their actual
workplace based learning context. The part of the SM
EPA 1. Sets learning goals for the whole curriculum and specific lessons
Assessment and The teacher does/does not formulate (self formulated) learning goals in connection with specific
evaluation criteria subject content
The teacher does/does not make use of SMART (specific, measurable, acceptable, realistic and
time related) formulated learning goals.
The teacher does/does not take into consideration the starting situation of students when
formulating learning goals.
Proficiency levels The teacher takes over the learning goals or course material from others. He/she incidentally
considers the starting situation of the students and the connection with specific subject content.
The teacher does not check if the learning goals are SMART formulated. (starting)
The teacher regularly checks if the learning goals of others or the course material connect to
specific subject content and the starting situation of the students. The teacher checks if the set
learning goals are SMART formulated. (sufficient)
The teacher formulates his/her own learning goals, which usually connect to the specific subject
content and the starting situation of the students. These learning goals are partially SMART
formulated. (good)
The teacher formulates his/her own coherent learning goals, which connect to the specific
subject content and the investigated starting situation of the students. The learning goals are
SMART formulated. (Excellent)
Assessment forms Lesson plans/series of lessons and student placement evaluation form.
Assessor Institute and internship supervisor.
Figure 1. Rubrics in Teacher Education
Figures 2a-2c. Personalized feedback at Entrustment level
5. Rationale of Personalized Feedback standards. This is visualized in the overviews with scores
The personalized feedback module that we developed in on EPAs and competencies (see Figures 3 and 4).
the project is inspired by Nicol and MacFarlane-Dick’s 2. Facilitates the development of self-assessment
seven principles of good feedback practice (2006) that (reflection) in learning. Our design allows for close
facilitate self-regulation. These principles were translated monitoring of trainees’ progress by visualizing trainees’
in the design as follows. Good feedback: performance on the EPAs by means of graphs and figures
1. Helps clarify what good performance is . For as well as narrative feedback. In this way it provides an
professional development at the workplace the learning overview of students’ strengths and points for further
goals should be crystal clear in order to stimulate learning development, which can be used for self-assessment and
and above that should stimulate (learn) trainees to clarify peer assessment and discussion about trainees’ portfolio.
own goals (Sadler, 1989). It is well known that often Further, compiling the portfolio (selecting materials as
mismatches occur between supervisors’ and trainees’ input for the portfolio) already demands trainees’
interpretation of assessment criteria and standards, reflection.
especially when it comes down to complex tasks at the
workplace that can be tacit and culture related. An
approach that we provided is the development of EPAs
connected in rubrics (see Figure 1). Rubrics have proven
to be very helpful in clarifying goals and standards and
stimulating trainees in goal clarification and goal setting,
for instance by involving trainees in the assessment and
stimulating discussion and reflection about criteria and
ELECTRONIC PORTFOLIOS FOR WORKPLACE-BASED FEEDBACK 5
Figure 3. Spider chart view of scores on the EPAs Figure 4. Spider chart view of scores on competencies
Figure 5. Timeline view of trainees’ performance on EPAs (called tasks in this example).
ELECTRONIC PORTFOLIOS FOR WORKPLACE-BASED FEEDBACK 6
3. Delivers high quality information to trainees about Example of Aggregated Feedback
their learning. Trainees need detailed information of Feedback Type message (Level 1)
high level to monitor and correct their own performance Improvement There is room for improvement for
and to take action to improve. In the preliminary this EPA. Please click on the
personalized feedback module this is enhanced by: (a) message to see how you can improve
linking the feedback to predefined EPAs that includes your performance.
criteria and standards; (b) ensuring timely feedback by Positive You have recently received good
means of underlying SMs that feed into the system; (c) scores for this EPA. Please click
giving trainees advise on their learning and showing here to see how you can improve
(prioritizing) needs for improvement; (d) regulating the more.
amount of feedback by giving trainees the option to click Trend You currently have a trend of
further if they want more detailed information; (e) decreasing scores for this EPA.
allowing to upload information in the portfolio system
Supervisor Your supervisor added few
anytime anywhere, which makes the feedback system up
improvement comments on this EPA.
to date. See Figure 5 for examples of types of feedback.
Cohort Compared to your cohort, you
4. Encourages teacher and peer dialogue around received better scores than your
learning. The system allows for supervisor and peer
peers on this EPA.
dialogues about progress and possible improvement.
Gaps You have less assessments than your
Such dialogues are important to make sure that trainees
peers on this EPA.
understand the feedback, can value and verify it and
know how to act on it (Van der Schaaf et al., 2008). The Feedback Type Some examples of Detailed
E-portfolio environment allows for interaction between Feedback message (Level 2)
supervisors, trainees and peers and has the possibility that Improvement You are level 2 on your Physical
several stakeholders upload documents, so that for Examination Competency. To
instance peer feedback can be used as ‘evidence’ for a achieve the next level your
trainee’s performance. examination and research should be
reasonably complete and technically
5. Encourages positive motivational beliefs and self-
esteem. Positive motivational beliefs and self-esteem are adequate. Overview of the
prerequisite for learning and improved performance. It is examination and consistency are
known that both benefit most when trainees receive many reasonably developed.
low-stakes assessment tasks, with immediate feedback Trend You were level 3 on your Physical
for improvement (if needed), rather than receiving few Examination Competency and you
high-stakes summative assessment tasks. The E-portfolio dropped on level 2 during your last
allows the trainee to select and rewrite own pieces of assessment. To achieve the next
work/documents (drafts and resubmissions) and level your examination and research
formative feedback in de long run. The SM instantly should be reasonably complete and
updates when new information comes in. technically adequate. Overview of
6. Provides opportunities to close the gap between the examination and consistency are
current and desired performance . Feedback in the EP reasonably developed.
should support trainees to take the next steps to improve Supervisor "You are performing well, but you
their performance. This demands engagement for further can take more notes during the
improvement and can be stimulated by providing examination process." (13/05/2015)
feedback on work in progress, provide feedback in
several stages in which feedback (Gibbs, 2004). The E- Figure 6. Examples of detailed feedback messages for
portfolio allows this. each feedback type
7. Provides information to teachers that can be used
to help shape the teaching. Not only trainees need to be 6. Discussion
informed about their progress and options for The aim of this contribution was to elucidate how
improvements, this also counts for the supervisors. They personalized feedback based upon Learning Analytics
need to be informed with detailed and quality information could be used in an E-portfolio environment. The E-
about their trainees in order to guide them at the portfolio offers learners (students, trainees, professionals)
workplace. This especially counts for professional and their supervisors an environment to monitor and
education in which trainees have many supervisors for provide evidence of their learning and competency
several internships. These supervisors often do not know development. The progress of the user can be closely
what feedback a trainee received from previous monitored by choosing from amongst several display
supervisors and how trainees’ longitudinal progress looks modes, such as radar, line and bar charts, which are
like. The preliminary personalized feedback design feeds automatically generated by the system. The scores (on the
into this by a specific portfolio entry for supervisors with different competencies) used for these visualizations are
long term information about the trainee and the digital abstracted form the assessment tools inserted in the
option for trainees to ask for supervisor feedback. portfolio. Other overviews are also displayed, for
example numerical overviews of the total inserted forms
and an overview of the progress, based on all activities,
forms and procedures linked to it. The developed LA-
tools will be open source.
ELECTRONIC PORTFOLIOS FOR WORKPLACE-BASED FEEDBACK 7
7. ACKNOWLEDGMENTS [8] Ferguson, R. (2012). Learning analytics: drivers,
This study was conducted within the framework of developments and challenges. International Journal
“Workplace-Based e-Assessment Technology for of Technology Enhanced Learning, 4(5), 304-317.
competency-Based Higher Multi-Professional Education” [9] Gibbs, G., & Simpson, C. (2004). Conditions under
(WATCHME) project supported by the European which assessment supports trainees’ learning.
Commission 7th Framework Programme (grant Learning and Teaching in Higher Education, 1, 3-31.
agreement No. 619349).
[10] Hattie, J., & Timperley, H. (2007). The power of
8. References feedback. Review of educational research, 77(1), 81-
112.
[1] Baker, R. S. J. D., & Yacef, K. (2009). The state of
educational data mining in 2009: A review and [11] Nicol, D. J., & Macfarlane‐ Dick, D. (2006).
future visions. Journal of Educational Data Mining, Formative assessment and self‐ regulated learning: a
1(1), 3-17. model and seven principles of good feedback
practice. Studies in Higher Education, 31, 199-218.
[2] Birenbaum, M. (1994). Toward adaptive assessment
– the students’ angle. Studies in Educational [12] Sadler, D. R. (1989). Formative assessment and the
Evaluation, 20, 239-255. design of instructional systems. Instructional
Science, 18, 119-144.
[3] Boud, D. (1990). Assessment and the promotion of
academic values. Studies in Higher Education, 15, [13] Sadler, D. R. (2010). Beyond feedback: Developing
101-111. trainee capability in complex appraisal. Assessment
& Evaluation in Higher Education, 35, 535-550.
[4] Clow, D. (2013). An overview of learning analytics.
Teaching in Higher Education, 18(6), 683 - 695. [14] Siemens, G., & Long, P. (2011). Penetrating the fog:
doi:10.1080/13562517.2013.827653 Analytics in learning and education. Educause
Review, 46(5), 30-32.
[5] Dekker-Groen, A., Van der Schaaf, M., & Stokking,
K. (2012). Performance standards for teachers [15] Van Schaik, S., Plant, J., & O’Sullivan (2013).
supporting nursing students’ reflection skills Promoting self-directed learning through portfolios
development. Journal of Nursing Education and in undergraduate medical education: The mentors’
Practice, 2, 1, 9-19. doi: 10.5430/jnep.v2n1p9. perspective. Medical Teacher, 35, 139-144. doi:
10.3109/0142159x.2012.733832.
[6] Dron, J., & Anderson, T. (2009). How the crowd can
teach. In S. Hatzipanagos & S. Warburton (Eds.), [16] Van der Schaaf, M., Stokking, K., & Verloop, N.
Handbook of research on social software and (2008). Developing and validating a design for
developing community ontologies (pp. 1–17). teacher portfolio assessment. Assessment &
Hershey, PA: IGI Global Information Science. Evaluation in Higher Education, 33(3), 245-262.
Retrieved from www.igi- doi: 10.1080/02602930701292522.
global.com/downloads/excerpts/33011.pdf. [17] Wolf, K., & Dietz, M. (1998). Teaching portfolios:
[7] Elias, T. (2011). Learning analytics: definitions, purposes and possibilities. Teacher Education
processes and potential. Creative Commons Quarterly, 25, 9-22.
Attribution 3.0.