To learn complex skills, like collaboration, learners need to acquire a concrete and consistent mental model of what it means to master this skill. If learners know their current mastery level and know their targeted mastery level, they can better determine their subsequent learning activities. Rubrics support learners in judging their skill performance as they provide textual descriptions mastery levels with performance indicators for all constituent subskills. However, text-based rubrics have a limited capacity to support the formation of mental models with contextualized, time-related and observable behavioral aspects of a complex skill. This paper outlines the design of a study that intends to investigate the effect of rubrics with video modelling examples compared to text-based rubrics on skills acquisition and feedback provisioning. The hypothesis is that video-enhanced rubrics, compared to text based rubrics, will improve mental model formation of a complex skill and improve the feedback quality a learner receives (from e.g. teachers, peers) while practicing a skill, hence positively effecting final mastery of a skill.
Co mple x generic skills, such as collaborating, presenting and being information literate, are gaining increased attention in Dut ch secondary education. These skills and others , although not new in their nature and disposition, are even at both national and international policy level rena med and their importance further e mphasized under the label st 21st century skills are generic co mp le x skills (and associated knowledge and attitudes ) which are generally seen as necessary in order to live and wo rk in the future and to contribute to a knowledge society . Comple x skills consist of constituent subskills which concertation require high cognitive effort and concentration [1][2], and prolonged repetitive practice in order to master them. Co mple x generic skills are not specific for a do main, occupation or type of task, but important for all kinds of work, education and life in general. These skills are applicable in a broad range of situations and many subject domains [3].
Many primary and s econdary schools are struggling with how to teach and evaluate
these comple x generic skills in their daily educational
practice [
An analytic assessment rubric is a te xtual description of the different mastery levels of a skill (represented by learn ing objectives ) by means of (a set of) performance ind icators [6]. Such indicators specify aspects of variation in the comple xity of a skill (e.g. presenting for a small, homogeneous group compared to a more co mple x presentation for a large heterogeneous group), constituent sub-skills and related performance levels [7]. An analytic rubric is a suitable instrument to structure timely and informat ive feedback while practicing a skill, but also to clarify and ma ke e xpectations about the strived -for mastery level(s) of a skill clear in advance to the learner. This helps learners at the start and during their learning activ ities to envisage the targeted mastery level of the skill [8]. The target mastery level of the skill depends on the educational level, e.g. a rubric for a skill in lower secondary education will be diffe rent compared to one in upper secondary education. By using analytic assessment rubrics actors (e.g. learners (a mongst each other), teachers, experts) can communicate about the requirements , expectations of tery levels [9] and in this way mon itor skills progress and eventually adjust the teaching-learning process [10]. This enables learners, while p racticing a skill, to pay extra attention to the aspects of a skill
Currently the targeted mastery level of a skill is often e xpressed by means of a text-based (analytic) rubric. However, many aspects of this targeted behavior refer to motoric activ ities, time-consecutive operations and processes that are hardly captured in te xt ( e.g. body posture or use of voice during a presentation ). In addit ion, the context in wh ich the skill is practiced is important for its learning, as it imp lies and n with practica l activit ies, operations and behavior in the physical world [11]. Therefore, te xt -based rubrics only have a restricted capacity to clarify the targeted mastery leve l of a skill and to assess shown behaviour [12]. However, these restrictions can be overcome with video-enhanced rubrics (VER), which will be developed and studied within the Viewbrics-project (www.v iewbrics.nl). A videoenhanced rubric (VER) is the synthesis of video mode lling e xa mples and a te xt-based analytic rubric in a dig ital fo rmative assessment format. Supposedly, video-enhanced rubrics can foster learning fro m observation of (good/bad) video modelling e xa mples [11] [12][13], thus supporting mental model fo rmation, when co mbined with te xtual performance indicators. For e xa mp le, looking at positive effects of video-modelling e xa mples only, Van Gog and colleagues found an increased performance of task e xecution when a v ideo-modelling e xa mple of an e xpert was shown[16] and De Grez and colleagues found comparable results while learning presentation skills [17].
Furthermore, te xt supposedly leaves more space for personal interpretation of the performance ind icators of a co mple x skill than video, wh ich probably negatively influences mental model format ion of a skill and is detrimental for feedback consistency of assessors. When teacher trainees compare their own performance with video- ng selfreflection than without these e xa mples and had an improved insight in their performance compared to the targeted mastery level of a co mple x skill [18]). Overarching research on mastering comple x skills [1] indicates that both modelling e xa mples, variety of application context as frequent feedback positively influence the learning process aquisition. Finally, video modelling e xa mp les also capture - , wh ich can be used during practicing a skill as well as while providing feedback.
The idea of developing and testing the effect o f v ideo -enhanced rubrics with modelling e xa mp les a lso grew fro m e xpe riences in daily educational p ractice in a previous project (PREATY). Here students indicated that, although they had rubrics to support them, they were still not yet sure of what was exactly e xpected fro m them when they mastered a comple x skill. This was shown in the kind of questions they asked, in an insufficient ab ility to underpin their assessment of peers performances while -modelling e xa mp les of presentations in combination with the rubrics, students seemed to gain in fact more insight in what exactly was expected from them.
To summarize, we e xpect that the use of video modelling e xa mp les (illustrating (moving images, sound), combined with te xt -based rubrics, would better foster a more concrete and consistent mental mode l of a co mple x skill a mongst students and teachers than solely using te xt-based rubrics, leading to imp roved quality of feedback given by teachers or peers during skills train ing, and su bsequently resulting in mo re effective o r e fficient skill mastery. These e xpectations are not only grounded in combin ing several (previously uncomb ined) research perspectives, but was a lso inspired by previous practical rics to assess each other within the PREATY-project.
The Viewbrics project (www.vie wbrics.nl) is a three year research and development project, wh ich started September 2015. In this project we study whether a combined use of video-modelling e xa mples with text -based rubrics indeed more positively effects the mental model format ion of, the feedback on and the mastery of several comple x skills by students in Dutch lower secondary education when compared with text-based rubrics. Th is will be done for three co mple x t ransversal skills: 1) presenting, 2) information literacy, 3) co llaborating. These skills were chosen while they vary in the way they are pract iced, e.g. in the nu mber of actors providin g feedback and timing of feedback mo ments (e.g. feedback on a presentation will be given after a performance of this skill, whereas feedback on co llaboration will be given while pupils still co llaborate), as we ll as because they are often implicit ly offered and practiced in co mbination with other learn ing tasks in Dutch secondary education, however not yet in a structured and e xplicit manner, re ceiv ing separate attention [5], p.103).
In the project a cyclic, partic ipatory research and design (R&D) approach is chosen, in which teachers (fro m d ifferent disciplines), students and researchers collaborate in a core team. The team is supplemented with additional design expertise, like e.g. script-writing, mu ltimedia, progra mming and interface e xperts, when needed. For the validation of the three rubrics and the video-modelling e xa mples a b roader group of teachers and students will be involved (e.g. by means of a MOOC on format ive assessment).
The core team designs and develops rubrics with video-modelling e xa mp les for presenting, collaborating and informat ion literacy skills, based on general as we ll as specific mult imedia design principles as well as scientific literature of and e xisting text-based rubrics for these comple x skills. Intermediate results are discussed in joint focus group meetings . Furthermore, a dig ital 360-degree feedback and assessment instrument is developed, based upon the final version of the validated rubrics , wh ich secondary schools can use to provide formative feedback on students practicing comple x skills mastering the skill. Th is will be accompanied with instruction and workshops for teachers and students on formative assessment, feedback and the use of assessment rubrics. For research purposes several instruments will be developed by the core team: 1) design guidelines (grounded in theory) for video-enhanced rubrics with modelling e xa mp les 2) an analyt ic fra me work for feedback quality, which will be used to compare the feedback quality between diffe rent research conditions, 3) questionnaires, intervie ws and a coding scheme for analy zing teachers as we ll as students experiences with the use of v ideoenhanced rubrics.
The cyclic , part icipatory R&D process is structured in four phases, three phases dedicated to design-oriented research in which different (g roups of) actors (researchers, teachers, students) are involved in developing and testing (with broader stakeholder groups) different versions of the (theory-and practice informed) v ideoenhanced rubrics with the videoscenes acted by pupils => scenes acted by actors) and the digital 360 degree feedback instrument. In these phases also two lab e xpe riments are foreseen. The first e xperiment will study three alternative designs of a VER and the second experiment will look at e ffects of a lternatively a te xt -based rubric, v ideo-modelling and a VER on mental model fo rmation and on the working me mo ry of a student. The fourth phase is dedicated to an empirical study with the validated video-enhanced rubrics . A quasi-e xperimental design will then study the effectiveness of the video-enhanced rubrics (VER) , where the e xperimental group receives VER and the control group only text-based rubrics . This study will ta ke p lace in three secondary schools, where generic skills will be implemented in different domains (e.g. Science, linguistics). Both groups receive instruction in advance about providing feedback, the use of rubrics and the digita l assessment instrument for assessing own or peers performances . In both groups the feedback is provided through the digital assessment instrument by one or more peers and their teacher. The in itia l performance level on the studied comple x skills is measured in both groups, as well as the performance level at the end of the study by means of the developed (text-based part of the) assessment rubrics by two raters. Feedback effects in both groups are measured in terms of: 1)
Consistency of feedback: -,peer-and teacher/expert feedback) assessment of the same performance. The assumption is that if the mental model of a skills is more consistent betwe en actors, than the assessment similarity is higher. We expect a higher consistency of feedback in the video-enhanced rubric condition compared to the text-based rubric condition. 2) Concreteness of feedback: number of times that in the free space for feedback and in additional intervie ws is referred to concrete behavioral aspects and the practical applicat ion of a skill. We e xpect more concrete references in the video-enhanced rubric condition co mpared to the te xt-based rubric condition, due to a richer mental model. 3) (Perceived) feedback quality: feedback quality of the te xtual feedback provided by actors as measured through indicators in an analysis frame work as well as in 10 samp led interviews (with 5 students and 5 teachers) per condition.
The initia l and final skill mastery levels of students participating in this study will be measured by using the text-based analytic rubrics, so that we can determine whether the final mastery of the skills differ between both research groups.
Ne xt to the (co-)constructed scientific and practical knowledge developed in this project, the project a lso has several practical outcomes, re levant for (Dutch) lower secondary education. It delivers validated and re-usable video-enhanced rubrics for three (very different) co mple x skills, that are expected to be more effective for learning a skill than the currently used rubrics. Additionally, a digital 360 degree assessment instrument is developed for three skills that can be easily customized by teachers. Furthermore, guide lines and instructions for conducting formative assessment and providing feedback, use of (video-enhanced) rubrics and the digital 360 degree assessment instrument are generated, which will be used in workshops and an online masterclass for teachers training. Simila r workshops will be designed for students. All final outcomes will become available through the project website. 3
The research and development activities of the Vie wbrics project are currently in the first phase of the project, where the rubrics of three co mp le x skills (collaborating, presenting and information lite racy) and the accompanying video -modelling e xa mples are developed (grounded in theory as well as practice ) in a video-enhanced rubric (VER) and tested with stakeholders. Rubrics as well as the v ideo-modelling e xa mples will be adapted and further developed and tested in the near future, finally resulting in the test of effectiveness of a VER in the last phase of this study.
Although there is research available on formative assessment, the use of rubrics , modelling e xa mp les and the use of mult imedia for learn ing respectively, research on the combination of these concepts to learn comple x skills is rare. Moreover, Dutch secondary education is in the process of a transformat ion, where generic co mp le x skills will receive more e mphasis and are integrated with learn ing and applying domain-specific knowledge. Video-enhanced rubrics may beco me one of the instruments providing teachers with so me structure to deal with this change in their daily educational practice.
Concluding, the results of this study are e xpected to fill a gap in current scientific research as well as in (Dutch) educational practice. Future research and development results will reveal whether v ideo-enhanced rubrics (VER: the synthesis of video modelling e xa mp les and a te xt-based analytic rubric in a digita l format ive assessment format) will indeed have the expected positive impact on mental model format ion, feedback quality and final masteryof acomplex skill.
We would like to gratefu lly acknowledge the contribution of the project, a three year research and development project that is funded by the practice - oriented research programme of the Netherlands Initiative for Education Research (NRO), part of The Netherlands Organisation for Scientific Research (NWO). J. J. G. Van Merriënboer, Training Complex Cognitive Sk ills. Educational Technology, 1997. 17 22, 2012. 72, pp. 323 327, 2014.