=Paper=
{{Paper
|id=Vol-1850/TEFA2016_paper_1-4
|storemode=property
|title=Viewbrics: Formative Assessment of Complex Skills with Video-Enhanced Rubrics in Dutch Secondary Education
|pdfUrl=https://ceur-ws.org/Vol-1850/TEFA2016_paper_1-4.pdf
|volume=Vol-1850
|authors=Ellen Rusman,Rob Nadolski,Jo Boon,Kevin Ackermans
}}
==Viewbrics: Formative Assessment of Complex Skills with Video-Enhanced Rubrics in Dutch Secondary Education==
https://ceur-ws.org/Vol-1850/TEFA2016_paper_1-4.pdf
Viewbrics: Formative Assessment of Complex Skills with
Video-Enhanced Rubrics (VER) in Dutch Secondary
Education
Ellen Rusman, Rob Nadolski, Jo Boon and Kevin Ackermans
Welten-Institute, Open Universiteit, Heerlen, The Netherlands
{Ellen.Rusman, R o b . N a d o l s k i , Jo.Boon, Kevin.Ackermans}@ou.nl
Abstract. To learn complex skills, like collaboration, learners need to acquire a
concrete and consistent mental model of what it means to master this skill. If
learners know their current mastery level and know their targeted mastery level,
they can better determine their subsequent learning activities. Rubrics support
learners in judging their skill performance as they provide textual descriptions
mastery levels with performance indicators for all constituent
subskills. However, text-based rubrics have a limited capacity to support the
formation of mental models with contextualized, time-related and observable
behavioral aspects of a complex skill. This paper outlines the design of a study
that intends to investigate the effect of rubrics with video modelling examples
compared to text-based rubrics on skills acquisition and feedback provisioning.
The hypothesis is that video-enhanced rubrics, compared to text based rubrics,
will improve mental model formation of a complex skill and improve the
feedback quality a learner receives (from e.g. teachers, peers) while practicing a
skill, hence positively effecting final mastery of a skill.
Keywords: video; rubrics; (formative) assessment; complex skills; feedback;
mental model; technology-enhanced assessment; 21st century skills
1 Introduction
Co mplex generic skills, such as collaborating, presenting and being information
literate, are gaining increased attention in Dutch secondary education. These skills
and others, although not new in their nature and disposition, are even at both national
and international policy level renamed and their importance further emphasized under
the label st 21st century skills are generic co mp lex skills (and
associated knowledge and attitudes) which are generally seen as necessary in order to
live and wo rk in the future and to contribute to a knowledge society. Complex skills
consist of constituent subskills which concertation require high cognitive effort and
concentration [1][2], and prolonged repetitive practice in order to master them.
Co mplex generic skills are not specific for a do main, occupation or type of task, but
important for all kinds of work, education and life in general. These skills are
applicable in a broad range of situations and many subject domains [3].
Many primary and secondary schools are struggling with how to teach and evaluate
these complex generic skills in their daily educational
practice [4]. Although these skills are seen as important by both policy makers as well
as teachers and school managers, few countries have developed teaching
methodologies and assessment policies for them. Project-based education, wherein
mu ltip le do mains (like chemistry, physics and biology) are co mbined in the project, is
often used to address these skills It is then assumed that when learners need to use
these generic skills during the execution of a project, they will auto matically acquire
these skills. Ho wever, as in many cases no specific instructional support is given
towards acquisition, it is h ighly unlikely that students will actually master such
skills during project-based education. Dutch teachers are aware of the importance of
acquiring co mplex generic skills by their pupils and they do in fact pay attention to
them in their educational practice, however not very exp licitly, structurally and on a
regular basis [5]. What is needed is to structurally support the learning process
towards pupi co mplex skill mastery. One of the instruments to support a skills
acquisition is an analytic assessment rubric [6].
An analytic assessment rubric is a textual description of the different mastery
levels of a skill (represented by learn ing objectives) by means of (a set of)
performance ind icators [6]. Such indicators specify aspects of variation in the
complexity of a skill (e.g. presenting for a small, homogeneous group compared to a
more co mplex presentation for a large heterogeneous group), constituent sub-skills
and related performance levels [7]. An analytic rubric is a suitable instrument to
structure timely and informat ive feedback while practicing a skill, but
also to clarify and make expectations about the strived -for mastery level(s) of a skill
clear in advance to the learner. This helps learners at the start and during their
learning activ ities to envisage the targeted mastery level of the skill [8]. The target
mastery level of the skill depends on the educational level, e.g. a rubric for a skill in
lower secondary education will be different compared to one in upper secondary
education. By using analytic assessment rubrics actors (e.g. learners (amongst each
other), teachers, experts) can communicate about the requirements , expectations of
tery levels [9] and in this way mon itor skills progress and eventually adjust
the teaching-learning process [10]. This enables learners, while p racticing a skill, to
pay extra attention to the aspects of a skill
Currently the targeted mastery level of a skill is often expressed by means of a
text-based (analytic) rubric. However, many aspects of this targeted behavior refer to
motoric activ ities, time-consecutive operations and processes that are hardly captured
in text ( e.g. body posture or use of voice during a presentation ). In addit ion, the
context in wh ich the skill is practiced is important for its learning, as it imp lies and
n
with practical activit ies, operations and behavior in the physical world [11].
Therefore, text -based rubrics only have a restricted capacity to clarify the targeted
mastery level of a skill and to assess shown behaviour [12]. However, these
restrictions can be overcome with video-enhanced rubrics (VER), which will be
developed and studied within the Viewbrics-project (www.v iewbrics.nl). A video-
enhanced rubric (VER) is the synthesis of video modelling examples and a text-based
analytic rubric in a dig ital fo rmative assessment format. Supposedly, video-enhanced
rubrics can foster learning fro m observation of (good/bad) video modelling examples
[11] [12][13], thus supporting mental model fo rmation, when co mbined with textual
performance indicators. For examp le, looking at positive effects of video-modelling
examples only, Van Gog and colleagues found an increased performance of task
execution when a v ideo-modelling example of an expert was shown[16] and De Grez
and colleagues found comparable results while learning presentation skills [17].
Furthermore, text supposedly leaves more space for personal interpretation of the
performance ind icators of a co mplex skill than video, wh ich probably negatively
influences mental model format ion of a skill and is detrimental for feedback
consistency of assessors. When teacher trainees compare their own performance with
video- ng self-
reflection than without these examples and had an improved insight in their
performance compared to the targeted mastery level of a co mplex skill [18]).
Overarching research on mastering complex skills [1] indicates that both modelling
examples, variety of application context as frequent feedback positively influence the
learning process aquisition. Finally, video modelling examp les also capture
- -
, wh ich can be used during practicing a skill as
well as while providing feedback.
The idea of developing and testing the effect o f v ideo -enhanced rubrics with
modelling examp les also grew fro m experiences in daily educational p ractice in a
previous project (PREATY). Here students indicated that, although they had rubrics
to support them, they were still not yet sure of what was exactly expected fro m them
when they mastered a complex skill. This was shown in the kind of questions they
asked,
in an insufficient ab ility to underpin their assessment of peers performances while
-modelling examp les of
presentations in combination with the rubrics, students seemed to gain in fact more
insight in what exactly was expected from them.
To summarize, we expect that the use of video modelling examp les (illustrating
(moving images, sound), combined with text -based rubrics, would better foster a
more concrete and consistent mental model of a co mplex skill amongst students and
teachers than solely using text-based rubrics, leading to imp roved quality of feedback
given by teachers or peers during skills train ing, and su bsequently resulting in mo re
effective o r efficient skill mastery. These expectations are not only grounded in
combin ing several (previously uncomb ined) research perspectives, but was also
inspired by previous practical rics to assess each
other within the PREATY-project.
2 The Viewbrics project - Participatory research and design
(R&D)
The Viewbrics project (www.viewbrics.nl) is a three year research and development
project, wh ich started September 2015. In this project we study whether a combined
use of video-modelling examples with text -based rubrics indeed more positively
effects the mental model format ion of, the feedback on and the mastery of several
complex skills by students in Dutch lower secondary education when compared with
text-based rubrics. Th is will be done for three co mplex t ransversal skills: 1)
presenting, 2) information literacy, 3) co llaborating. These skills were chosen while
they vary in the way they are pract iced, e.g. in the nu mber of actors providin g
feedback and timing of feedback mo ments (e.g. feedback on a presentation will be
given after a performance of this skill, whereas feedback on co llaboration will be
given while pupils still co llaborate), as well as because they are often implicit ly
offered and practiced in co mbination with other learn ing tasks in Dutch secondary
education, however not yet in a structured and explicit manner, re ceiv ing separate
attention [5], p.103).
In the project a cyclic, participatory research and design (R&D) approach is chosen,
in which teachers (fro m d ifferent disciplines), students and researchers collaborate in
a core team. The team is supplemented with additional design expertise, like e.g.
script-writing, mu ltimedia, programming and interface experts, when needed. For the
validation of the three rubrics and the video-modelling examples a b roader group of
teachers and students will be involved (e.g. by means of a MOOC on format ive
assessment).
The core team designs and develops rubrics with video-modelling examp les for
presenting, collaborating and informat ion literacy skills, based on general as well as
specific mult imedia design principles as well as scientific literature of and existing
text-based rubrics for these complex skills. Intermediate results are discussed in joint
focus group meetings. Furthermore, a dig ital 360-degree feedback and assessment
instrument is developed, based upon the final version of the validated rubrics , wh ich
secondary schools can use to provide formative feedback on students practicing
complex skills mastering the skill. Th is will be
accompanied with instruction and workshops for teachers and students on formative
assessment, feedback and the use of assessment rubrics. For research purposes several
instruments will be developed by the core team: 1) design guidelines (grounded in
theory) for video-enhanced rubrics with modelling examp les 2) an analyt ic
framework for feedback quality, which will be used to compare the feedback quality
between different research conditions, 3) questionnaires, interviews and a coding
scheme for analy zing teachers as well as students experiences with the use of v ideo-
enhanced rubrics.
The cyclic, part icipatory R&D process is structured in four phases, three phases
dedicated to design-oriented research in which different (g roups of) actors
(researchers, teachers, students) are involved in developing and testing (with broader
stakeholder groups) different versions of the (theory-and practice informed) v ideo-
enhanced rubrics with the video-
scenes acted by pupils => scenes acted by actors) and the digital 360 degree feedback
instrument. In these phases also two lab experiments are foreseen. The first
experiment will study three alternative designs of a VER and the second experiment
will look at effects of alternatively a text -based rubric, v ideo-modelling and a VER on
mental model fo rmation and on the working memo ry of a student. The fourth phase
is dedicated to an empirical study with the validated video-enhanced rubrics.
A quasi-experimental design will then study the effectiveness of the video-enhanced
rubrics (VER) , where the
experimental group receives VER and the control group only text-based rubrics. This
study will take p lace in three secondary schools, where generic skills will be
implemented in different domains (e.g. Science, linguistics).
Table1 provides a schematic overview of the research design.
Table 1
Research design of empirical study
!"#$%&$"#'("!"#$%&'&#$()) * +, ! )))))) +- +. /0) 1&$2)34))
56"7#88)%&99:7:$')8";##<8=
)("#%(*!"#$%&'&#$()))))) * +, +-) +. /0) 1&$2)34))
56"7#88)%&99:7:$')8";##<8=)
)))))))))))))))) ) )
))))) )))))))))))))))))
)
"#$###### >#7?8;#@)#$)9#716'&A:)688:881:$'B)@7#A&%&$CD7:":&A&$C)9::%E6"?)6$%)F8:)#9)7FE7&"8)
6$%)%&C&'6<).34)%:C7::)688:881:$')&$8'7F1:$')#
%&))0)))# ! 1:68F7:)#9)@:79#716$":)<:A:<)#9)8'F%:$'8)#$)';:)8?&<<8)@7:8:$'&$CB)"#<<6E#76'&$C)
6$%)&$9#716'&#$)<&':76"G)EG)1:6$8)#9)':H'!E68:%)7FE7&"8)6$%)"#$'7#<)A67&6E<:8)
5:2C2)C:$%:7B)6C:=)5@7:!':8'=)
#####! $# ! F8:)#9)7FE7&"8)>&';)A&%:#!:$;6$":%)7FE7&"8)>;&<:)@76"'&"&$C)';:)8?&<<8))
'()*+,*()'*## 5&$)';:)"#$'7#<)"#$%&'&#$)';:)861:)':H'!E68:%)@67')#9)';:)7FE7&"8)&8)F8:%B)
;#>:A:7)>&';#F')A&%:#)1#%:<<&$C):H61@<:8)=)
#####%-# #$### I:68F7:)#9)9::%E6"?)JF6<&'G)5"#$8&8':$"GB)"#$"7:':$:88)6$%)68)@:7":&A:%=)7:":&A:%)
>;&<:)@76"'&"&$C)6)8?&<<)
%.#$### ! 1:68F7:)#9)@:79#716$":)<:A:<)#9)8'F%:$'8)#$)';:)8?&<<8)@7:8:$'&$CB)"#<<6E#76'&$C)
6$%)&$9#716'&#$)<&':76"G)EG)1:6$8)#9)':H'!E68:%)7FE7&"8)6$%)"#$'7#<)A67&6E<:8)
5:2C2)C:$%:7B)6C:=)5@#8'!':8'=)
Both groups receive instruction in advance about providing feedback, the use of
rubrics and the digital assessment instrument for assessing own or peers
performances. In both groups the feedback is provided through the digital assessment
instrument by one or more peers and their teacher. The in itial performance level on
the studied complex skills is measured in both groups, as well as the performance
level at the end of the study by means of the developed (text-based part of the)
assessment rubrics by two raters. Feedback effects in both groups are measured in
terms of:
1) Consistency of feedback: -,peer-and
teacher/expert feedback) assessment of the same performance. The
assumption is that if the mental model of a skills is more consistent between
actors, than the assessment similarity is higher. We expect a higher
consistency of feedback in the video-enhanced rubric condition compared to
the text-based rubric condition.
2) Concreteness of feedback: number of times that in the free space for
feedback and in additional interviews is referred to concrete behavioral
aspects and the practical applicat ion of a skill. We expect more concrete
references in the video-enhanced rubric condition co mpared to the text-based
rubric condition, due to a richer mental model.
3) (Perceived) feedback quality: feedback quality of the textual feedback
provided by actors as measured through indicators in an analysis framework
as well as in 10 samp led interviews (with 5 students and 5 teachers) per
condition.
The initial and final skill mastery levels of students participating in this study will be
measured by using the text-based analytic rubrics, so that we can determine whether
the final mastery of the skills differ between both research groups.
Next to the (co-)constructed scientific and practical knowledge developed in this
project, the project also has several practical outcomes, relevant for (Dutch) lower
secondary education. It delivers validated and re-usable video-enhanced rubrics for
three (very different) co mplex skills, that are expected to be more effective for
learning a skill than the currently used rubrics. Additionally, a digital 360 degree
assessment instrument is developed for three skills that can be easily customized by
teachers. Furthermore, guidelines and instructions for conducting formative
assessment and providing feedback, use of (video-enhanced) rubrics and the digital
360 degree assessment instrument are generated, which will be used in workshops and
an online masterclass for teachers training. Similar workshops will be designed for
students. All final outcomes will become available through the project website.
3 Conclusion and future research
The research and development activities of the Viewbrics project are currently in the
first phase of the project, where the rubrics of three co mp lex skills (collaborating,
presenting and information literacy) and the accompanying video -modelling examples
are developed (grounded in theory as well as practice) in a video-enhanced rubric
(VER) and tested with stakeholders. Rubrics as well as the v ideo-modelling examples
will be adapted and further developed and tested in the near future, finally resulting in
the test of effectiveness of a VER in the last phase of this study.
Although there is research available on formative assessment, the use of rubrics,
modelling examp les and the use of mult imedia for learn ing respectively, research on
the combination of these concepts to learn complex skills is rare. Moreover, Dutch
secondary education is in the process of a transformat ion, where generic co mp lex
skills will receive more emphasis and are integrated with learn ing and applying
domain-specific knowledge. Video-enhanced rubrics may beco me one of the
instruments providing teachers with so me structure to deal with this change in their
daily educational practice.
Concluding, the results of this study are expected to fill a gap in current scientific
research as well as in (Dutch) educational practice. Future research and development
results will reveal whether v ideo-enhanced rubrics (VER: the synthesis of video
modelling examp les and a text-based analytic rubric in a digital format ive assessment
format) will indeed have the expected positive impact on mental model format ion,
feedback quality and final mastery of a complex skill.
Acknowledgement
We would like to gratefu lly acknowledge the contribution of the project, a three
year research and development project that is funded by the practice - oriented
research programme of the Netherlands Initiative for Education Research (NRO), part
of The Netherlands Organisation for Scientific Research (NWO).
References
[1]
New In T. L. Good
(Ed.), 21st century education: A reference handbook , Thousand Oaks, CA :
Sage., 2008, pp. 244 253.
[2] D. Galligan, Frank; Maskery, Co lin; Spence, Jon; Ho we, David; Barry, Tim;
Ruston, Andy; Crawford, Advanced PE for Edexcel. Oxford: Heinemann
Educational Publishers, 2000.
[3]
Canberra, 2010.
[4] E. Rusman, A. Mart ínez-monés, J. Boon, M. J. Rodríguez-t riana, and S.
Villagrá- -
Enhanced Formative Assessment ( TEFA ) of 21 st Century Skills in the
Co mputer Assisted Assessment. Research into E -Assessment.
Proceedings of the International Computer Assisted Assessment (CAA)
Conference, 2014, pp. 1 14.
[5]
[6] C. Sluijsmans, Do min ique; Joosten-ten Brin ke, Désirée; Van der Vleuten,
[7] J. J. G. Van Merriënboer, Training Complex Cognitive Skills. Educational
Technology, 1997.
[8]
Assess. Eval. High. Educ., Aug. 2009.
[9]
Educ. Res. Rev., vol. 9, pp. 129
144, Jun. 2013.
[10] K. Schildkamp , M. Heit ink, F. Van Der Kleij, I. Hoogland, A. Dijkstra, and
- Een
[11]
Educ. Technol. Soc., vol. 14, no. 2, pp.
201 212, 2011.
[12]
understanding of assess Teach. High. Educ., no.
March 2015, pp. 37 41, 2007.
[13]
Trends Neurosci. Educ., vol. 2, no.
1, pp. 27 32, 2013.
[14]
4W Weten Wat Werkt en Waarom, vol. 1, no. 1, pp.
17 22, 2012.
[15] -
Cognitive and Social- Educ. Psychol. Rev.,
vol. 22, no. 2, pp. 155 174, 2010.
[16]
Comput. Educ., vol.
72, pp. 323 327, 2014.
[17]
observational learning and practice- based learning on the development of
High. Educ. Res. Dev., vol. 33,
no. 2, pp. 256 271, 2013.
[18]
self- Teach. Teach. Educ., vol. 36, pp.
189 197, 2013.