=Paper=
{{Paper
|id=Vol-1147/paper1
|storemode=property
|title=Towards the Use of New Methods for Formative e-Assessment of 21st
Century Skills in Schools
|pdfUrl=https://ceur-ws.org/Vol-1147/paper1.pdf
|volume=Vol-1147
}}
==Towards the Use of New Methods for Formative e-Assessment of 21st
Century Skills in Schools==
Towards the Use of New Methods for Formative e-Assessment of 21st Century Skills in Schools Ellen Rusman1, Jo Boon1 , Alejandra Martínez-Monés2, María Jesús Rodríguez- Triana2, Simeon Retalis3 1 Open University of the Netherlands, CELSTEC, Valkenburgerweg 177, 6419 AT Heerlen, The Netherlands {ellen.rusman, jo.boon}ou.nl 2 Universidad de Valladolid, GSIC-EMIC, Paseo de Belén, 12, 47014 Valladolid, Spain amartine@infor.uva.es; chus@gsic.uva.es 3 University of Piraeus, Department of Digital Systems, 80 Karaoli & Dimitriou 185 34 Piraeus, Greece etal@unipi.gr Abstract. Schools are in the process of reorienting their curricula to the devel- opment of 21st century skills and competences; however, their assessment methods have not yet been updated. The PREATY project aims to make teach- ers in primary and secondary schools aware of the use and benefits of modern e-assessment methods and tools. This paper presents the preliminary work de- veloped in the project. Keywords: Formative assessment, e-assessment, 21st century skills, key com- petences, primary and secondary school, teacher training 1 Introduction There has been a recent trend across the EU towards competence-based teaching and learning [5]. Nowadays, school curricula pay special attention to the development of 21st century skills (e.g. creativity, critical thinking, collaboration, or problem solv- ing skills) and key competencies that are seen as needed for global citizens and work- ers. However, in many countries, the assessment system has not changed and is still primarily focused on summative assessment of static knowledge and end-results, without attention on the learning process and the acquisition of competences and skills. Though there are alternative assessment methods -more process and formative ori- entated such as portfolios or rubrics-, they are not widespread and they are often seen as additional activities rather than an integral part of the curriculum. Also ICT tech- nologies enable alternative assessment forms, also called e-assessment [6]. A number of e-assessment tools are already available, and have proved their usefulness to meet assessment needs and promote learning. However, teachers, especially in primary and secondary education, have not adopted them yet. Therefore, there is a need to change the current practices and to support teachers with integrating modern assessment ap- proaches and tools for learning as part of the curriculum. Aware of this need, the PREATY (PRoposing modern E-assessment Approaches and Tools to Young and experienced in-service teachers) project aims to equip teach- ers in primary and secondary schools with e-assessment strategies and tools to evalu- ate a number of key 21st century skills and competencies. Opposite to the traditional test and output-focussed perspective on assessment, the project pursues to promote assessment for learning, therefore focussing mainly on the formative assessment of these skills and competences. This paper presents initial work carried out in the project. Section 2 describes our perspective on formative assessment and the potential benefits and requirements of supportive technology. Section 3 presents the modern e-assessment instruments that will be studied, with a description of their main characteristics and supporting tools. Section 4 points out the challenges of implementing ‘assessment for learning’ and formative assessment in schools. Finally, Section 5 concludes with the approach and further steps taken in the PREATY project to introduce formative e-assessment of 21st century skills in schools. 2 Formative e-assessment: Characteristics and Potential Surplus Value of Technology The purpose of formative assessment is to support teaching and learning processes by providing developmental feedback (on an item, topic and/or assessment level by means of different sources, such as peers, experts or teachers) to a learner (and their teacher) on his/her current understanding or skills [ 3][5][1][2] during a period of in- struction [11][15]. Formative assessment differs from summative assessment in that it is a continuing process of feedback; therefore the information on performances is gathered continuously, mirrored against a set of predefined criteria or good practices, and is used to shape improvements and promote an individual's learning, rather than serve as a final formal summary of learners’ achievement [ 8][17]. The responsibility for learning is with both learners as teachers and eventually their peers [ 2][9]. They determine (jointly) where a learner is going, where (s)he is now and how a learner can get where (s)he wants [20]. Formative assessment strategies address and integrate both role as well as reflective questioning aspects. Technology can offer different functions that potentially can facilitate and enhance formative assessment aspects [6][10]. It can improve access to assessments by differ- ent actors (e.g. also by peers, experts and teachers) at anytime, anyplace and any- where, enabling learners to measure their understanding when and how often they want and allow them more control of their learning. Feedback times can be short- ened and this can help to change misconceptions rapidly, or feedback may be given from different perspectives, within a group or adopted to a learner. Thus, technology can affect feedback quality and the motivation of a learner. Also, technology can track, trace store, process and visualize learners’ results as well as actions, which makes them visible and available for various learning purposes, such as individual or group reflection or to evaluate and visualize a learners progress. Technology can also affect teacher efficiency, as teachers can be supported with various tools helping to reduce assessment time, thus saving time that can be spent otherwise. Also, as technology enables rapid updating and combining material, it can also contribute to more varied and authentic assessment designs. We further describe three e- assessment instruments and their potential for formative assessment. 3 Identification of Available Formative e-assessment Approaches and Tools As a first step towards the introduction of formative e-assessment in primary and secondary schools, PREATY makes an inventory of innovative e-assessment instru- ments, starting with: e-Portfolio, learning analytics and enriched rubrics. These in- struments may help teachers to have a better understanding of students’ performance and offer valuable feedback to students and parents. They can support formative as- sessment methods that have proven to be effective for learning [17], such as self- or peer-assessment, instructional dialogues, questioning and reflective lessons. This section describes the main characteristics of the mentioned e-assessment instruments. The first instrument examined is an ePortfolio. They are an electronic version of traditional portfolios, meaning that the work samples inside them will be of digital nature; coming from different sources and tools [18]. The ePortfolio facilitates the capabilities of the learning portfolio to show the process and results of a learning path, proving the quality and level of achievement of the targeted competences [4]. The ePortfolio should include basic storing capabilities, with interoperability support, as well the management of permissions and mechanisms to provide feedback on the shared artefacts. Finally, different views of the work samples should be possible, in order to highlight the competences of interest in each case. A second e-assessment instrument of interest is Learning Analytics (LA), which can be defined as the measurement, collection, analysis, and reporting of data about learn- ers and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs [16]. LA allows people who are responsible for the teaching and learning process to identify possible learning risks, and make the neces- sary changes, depending on the situations encountered. Many LA systems and ap- proaches are being developed, like the ones based on Social Network Analysis. In spite of the recent interest in these techniques, still a major challenge is to adapt the output and visualizations to the teacher and learner’s needs. Thirdly, learning analytics can be combined with rubrics in the so-called enriched rubrics [12]. These instruments have the form of a traditional rubric, but (at least part of) the criteria are assessed by indicators automatically computed from the interaction of the students with the system. One of these systems is LAe-R [12], a Moodle plugin that allows a teacher to create an enriched rubric (ER) from scratch, or create a new one from a template or edit an existing one. When creating an ER, the teacher adds the assessment criteria with their corresponding descriptions and the grading levels (see Error! Reference source not found.). LAe-R allows a teacher to add types of criteria that are associated to learning analysis indicators: collaboration, grades to assignments, and study of learning resources. Fig. 1. Screenshot of how a teacher can specify an assessment criterion in LAe-R The three instruments and their electronic applications and tools each in their own way support the continuing process of feedback and help to shape students and teach- ers understanding of it, both at the individual and the group level. An ePortfolio can be used in formative and summative assessment procedures; they are very useful for formative feedback because they typically allow to demonstrate and monitor both the learning products and the –process. In principle it encourages students to take respon- sibility for their own learning and for the sharing of its results. The online portfolio tools often provide additional tools for sharing, analysis and support. LA systems can cover a wide range of applications, from the small-scale awareness tools, with a long tradition in educational technology, to large-scale academic analyt- ics. Our focus in PREATY is at the small-scale level, which is closer to our target users. Several tools exist at this level, that based on the data collected from the learn- ing systems make the processes visible, hereby allowing interventions and reflection. Many of these tools are research prototypes, though. Our goal in PREATY regarding these systems is to analyze what is needed to make them operational in the target school contexts. Finally, most of the software tools supporting rubrics allow for cooperation on the formulation and modification of rubrics, both between teachers and between teachers and students. Tools that support enriched rubrics add the possibility to automate part of the assessment, thus making it more efficient. In spite of the potential benefits of the reviewed instruments and associated tools, their uptake by the target group is very small. Next section goes into some explana- tions for this lack of interest. 4 Challenges and Approaches Towards Implementing Formative e- assessment in Schools Different bottlenecks can be mentioned that prevent easy implementation of formative assessment in schools and the technological support of it. As the authors in [17] men- tion, based on an extensive review study, still a large amount of conceptual confusion exists on the definition of formative assessment, and scientific research about the conditions of effectiveness of formative assessment is still in an early phase of devel- opment. Implementing formative e-assessment in schools also requires some essential pre- paratory steps. In the first place more information and communication about forma- tive assessment and possible approaches herein is necessary, as they are not yet widely known and accepted. The tools mentioned in section 3 are seen as powerful, although only partially applicable for the target groups of the PREATY project. Suc- cess – and acceptance on the longer term by teachers- depends on some other crucial conditions. Reflection about formative assessment and the goals one can achieve be- longs to a school culture where teachers – and management- are willing to think about innovation and come loose of a culture where they alone are in total control [17]. Looking at the conceptualization of [20], described earlier in this paper, it is also clear that different target groups are involved in the implementation process, not only teachers, but also learners and their peers. Thus, implementation of formative assess- ment in schools requires a willingness to completely (re)think (about) assessment. A second issue at stake in the PREATY project is the introduction and training re- lated to the technological tools and the formative e-assessment aspects. Regarding the training about technological aspects it is important to focus on acceptance by giving attention on how to present the objective of technology; are tools presented as ‘doing things better’ or as ‘ doing better things’ [14]. These distinctions are important as they include implicit views on the kind of enhancement that is at stake. The importance of training is also stressed by Whitelock [19].She argues that four components are im- portant to embed modern methods of assessment: (1) tool development; (2) staff train- ing; (3) rethinking the assessment tasks; and (4) learning from the assessment tasks. Teacher professionalization in the field of formative assessment and the related technological tools implies the training of complex skills. Trainings therefore will have to include [13]: 1. information and training for in service teachers on formative assessment – enhanc- ing: assessment for learning, learn how to reflect, how to use criteria, how to han- dle feedback, how to work in teams etc. 2. information and training on technological tools; as very few tools are available training could be used to gather information about desirability and requirements for tools to be developed in this field. From the project results thus far it is clear that very few tools in this field are available for the target groups of the project (pri- mary/secondary in-service teachers). Most tools are developed for use in higher education; this seems especially the case for the learning analytics tools, as they use mainly online activities within learning environments. Implementation and use of these environments is only starting in the context of secondary schools and is almost inexistent in primary schools. This might take the project to a training design focussing on sketching some alterna- tive scenarios for formative assessment based on what is available from higher educa- tional tools and from there going to the formulation of requirements for formative assessment tools within primary and secondary education. 5 Conclusions The adoption of formative assessment in schools requires a change of mind, from a focus on summative testing, end results and teacher responsibility towards a process and learning oriented perspective, with a joint responsibility of learners, teachers and peers. It is important to learn to see formative assessment as a process, rather than a particular kind of assessment [9], with different formative assessment strategies that can support that process. Next to teacher training material and training strategies that help to adopt and develop this mind-set, the availability of effective (for learning) formative e-assessment methods and tools can help to facilitate adoption within schools. The vision of the PREATY project is to advance in these directions, by set- ting up a set of teacher training workshops implementing these principles. We aim to explore the ways in which the e-assessment tools identified in the project can (or can- not) be adopted by primary and secondary school teachers to assist them with apply- ing these principles in practice. Acknowledgements We would like to gratefully acknowledge the contribution of the PREATY Project, that is funded by the European Commission's Lifelong Learning Programme, Project Number 526965-LLP-1-2012-1-GR-COMENIUS-CMP, which has part-funded this work. References 1. Black, P. J., Wiliam, D.: Inside the Black Box: Raising standards through classroom as- sessment. Phi Delta Kappa, 80, 139-48 (1998) 2. Black, P.J., Wiliam, D.: Developing the theory of formative assessment. Educational As- sessment, Evaluation and Accountability (formerly the Journal of Personal Evaluation in Education), 21 (1), 5-31 (2009) 3. Busuttil-Reynaud, G., Winkley, J. : JISC e-Assessment Glossary, Bristol (2006) 4. Butler, P.: A Review Of The Literature On Portfolios And Electronic Portfolios, Massey University College of Education, New Zealand and eCDF ePortfolio Project Steering Committee (2006) 5. Gordon, J., Halasz, G., Krawczyk, M., Leney, T., Michel, A., Pepper, D., et al.: Key com- petences in Europe: Opening doors for lifelong learners across the school curriculum and teacher education. Warsaw (2009) 6. JISC: Effective Practice with e-Assessment. An overview of technologies, policies and practice in further and higher education. Bristol (2007) 7. Looney, J. : Making it Happen: Formative Assessment and Educational Technologies. Promethean Education Strategy Group: Thinking Deeper Research Paper No.1. – Part 3. Online available at: http://www.prometheanworld.com/rx_content/files/PDF/ Mak- ingitHappenFormativeAssessmentEducationalTechnologies-169721.pdf (2010) 8. Looney, J., Siemens, G.: Assessment Competency: Knowing What You Know and Learn- ing Analytics. It is Time for a Breakthrough. Promethean Thinking Deeper Research Pa- per, No.3 (2011) 9. McManus, S.: Attributes of effective formative assessment. Washington DC: CCSSO (2008) 10. Pachler, N., Daly, C., Mor, Y., Mellar, H.: Formative e-assessment: practitioner cases. Computers & Education, 54, 715-721 (2010) 11. Pepper, D.: Literature Review: Assessment for key competences. KeyCoNet Report (2012) 12. Petropoulou, O., Vassilikopoulou, M., Retalis, S.: Enriched assessment rubrics: a new me- dium for enabling teachers to easily assess student's performance when participating in complex interactive learning scenarios, Operational Research, Volume 11 (2), 171-186 (2011) 13. Pinock, N. Brandt, W.C.: Connecting Formative Assessment Research to practice. Learn- ing Point Associates. Online available at: http://www.learningpt.org/pdfs/FormativeAssessment.pdf (2009) 14. Reilly, R..: Guest Editorial-Web Based Instruction: Doing Things Better and Doing Better Things. IEEEE Transactions on Education, 48 (4), 565-566 (2005) 15. Ridgway, J., McCusker, S., Pead, D.: Literature Review of E-assessment, Bristol (2004) 16. Siemens, G., Baker, R.: Learning analytics and educational data mining: towards commu- nication and collaboration. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12), Simon Buckingham Shum, Dragan Gasevic, and Rebecca Ferguson (Eds.). ACM, New York, NY, USA, 252-254 (2012) 17. Sluijsmans, J.T., Brinke, D., Vleuten, van der, C.: Toetsen met leerwaarde. Samenvatting van de reviewstudie naar de effectieve kenmerken van formatief toetsen. NWO-PROO (2013) 18. Subrahmanyam, V.V., Kanjilal, U., Sharma, V.K., Dudeja, A.K.: ePortfolio: An eWay of showcasing the artifacts. In: IEEE International Conference on Technology Enhanced Education (ICTEE), pp. 1-4. IEEE Press, Kerala, India (2012) 19. Whitelock, D.: Activating Assessment for Learning: are we on the way with Web 2.0?’ In M.J.W. Lee & McLoughlin (Eds.). Web 2.0-Based-E-Learning: Applying Social Informat- ics for Tertiary Teaching. IGI Global, 219-342 (2010) 20. Wiliam, D., Thompson, M.: Integrating assessment with instruction: What will it take to make it work? In C.A. Dwyer (Ed.), The future of assessment: Shaping teaching and learn- ing (pp. 53–82). Mahwah, NJ: Erlbaum (2007)