=Paper= {{Paper |id=Vol-1147/paper1 |storemode=property |title=Towards the Use of New Methods for Formative e-Assessment of 21st Century Skills in Schools |pdfUrl=https://ceur-ws.org/Vol-1147/paper1.pdf |volume=Vol-1147 }} ==Towards the Use of New Methods for Formative e-Assessment of 21st Century Skills in Schools== https://ceur-ws.org/Vol-1147/paper1.pdf
            Towards the Use of New Methods for Formative
             e-Assessment of 21st Century Skills in Schools

        Ellen Rusman1, Jo Boon1 , Alejandra Martínez-Monés2, María Jesús Rodríguez-
                                  Triana2, Simeon Retalis3
    1
        Open University of the Netherlands, CELSTEC, Valkenburgerweg 177, 6419 AT Heerlen,
                                           The Netherlands
                               {ellen.rusman, jo.boon}ou.nl
        2
          Universidad de Valladolid, GSIC-EMIC, Paseo de Belén, 12, 47014 Valladolid, Spain
                      amartine@infor.uva.es; chus@gsic.uva.es
           3
             University of Piraeus, Department of Digital Systems, 80 Karaoli & Dimitriou
                                        185 34 Piraeus, Greece
                                         etal@unipi.gr



            Abstract. Schools are in the process of reorienting their curricula to the devel-
            opment of 21st century skills and competences; however, their assessment
            methods have not yet been updated. The PREATY project aims to make teach-
            ers in primary and secondary schools aware of the use and benefits of modern
            e-assessment methods and tools. This paper presents the preliminary work de-
            veloped in the project.

            Keywords: Formative assessment, e-assessment, 21st century skills, key com-
            petences, primary and secondary school, teacher training


1           Introduction

   There has been a recent trend across the EU towards competence-based teaching
and learning [‎5]. Nowadays, school curricula pay special attention to the development
of 21st century skills (e.g. creativity, critical thinking, collaboration, or problem solv-
ing skills) and key competencies that are seen as needed for global citizens and work-
ers. However, in many countries, the assessment system has not changed and is still
primarily focused on summative assessment of static knowledge and end-results,
without attention on the learning process and the acquisition of competences and
skills.
   Though there are alternative assessment methods -more process and formative ori-
entated such as portfolios or rubrics-, they are not widespread and they are often seen
as additional activities rather than an integral part of the curriculum. Also ICT tech-
nologies enable alternative assessment forms, also called e-assessment [‎6]. A number
of e-assessment tools are already available, and have proved their usefulness to meet
assessment needs and promote learning. However, teachers, especially in primary and
secondary education, have not adopted them yet. Therefore, there is a need to change
the current practices and to support teachers with integrating modern assessment ap-
proaches and tools for learning as part of the curriculum.
   Aware of this need, the PREATY (PRoposing modern E-assessment Approaches
and Tools to Young and experienced in-service teachers) project aims to equip teach-
ers in primary and secondary schools with e-assessment strategies and tools to evalu-
ate a number of key 21st century skills and competencies. Opposite to the traditional
test and output-focussed perspective on assessment, the project pursues to promote
assessment for learning, therefore focussing mainly on the formative assessment of
these skills and competences.
   This paper presents initial work carried out in the project. Section 2 describes our
perspective on formative assessment and the potential benefits and requirements of
supportive technology. Section 3 presents the modern e-assessment instruments that
will be studied, with a description of their main characteristics and supporting tools.
Section 4 points out the challenges of implementing ‘assessment for learning’ and
formative assessment in schools. Finally, Section 5 concludes with the approach and
further steps taken in the PREATY project to introduce formative e-assessment of
21st century skills in schools.


2      Formative e-assessment: Characteristics and Potential Surplus Value of
       Technology

The purpose of formative assessment is to support teaching and learning processes by
providing developmental feedback (on an item, topic and/or assessment level by
means of different sources, such as peers, experts or teachers) to a learner (and their
teacher) on his/her current understanding or skills [ ‎3][‎5][‎1][‎2] during a period of in-
struction [‎11][‎15]. Formative assessment differs from summative assessment in that it
is a continuing process of feedback; therefore the information on performances is
gathered continuously, mirrored against a set of predefined criteria or good practices,
and is used to shape improvements and promote an individual's learning, rather than
serve as a final formal summary of learners’ achievement [ ‎8][‎17]. The responsibility
for learning is with both learners as teachers and eventually their peers [ ‎2][‎9]. They
determine (jointly) where a learner is going, where (s)he is now and how a learner can
get where (s)he wants [20]. Formative assessment strategies address and integrate
both role as well as reflective questioning aspects.
   Technology can offer different functions that potentially can facilitate and enhance
formative assessment aspects [‎6][‎10]. It can improve access to assessments by differ-
ent actors (e.g. also by peers, experts and teachers) at anytime, anyplace and any-
where, enabling learners to measure their understanding when and how often they
want and allow them more control of their learning. Feedback times can be short-
ened and this can help to change misconceptions rapidly, or feedback may be given
from different perspectives, within a group or adopted to a learner. Thus, technology
can affect feedback quality and the motivation of a learner. Also, technology can
track, trace store, process and visualize learners’ results as well as actions, which
makes them visible and available for various learning purposes, such as individual or
group reflection or to evaluate and visualize a learners progress. Technology can
also affect teacher efficiency, as teachers can be supported with various tools helping
to reduce assessment time, thus saving time that can be spent otherwise. Also, as
technology enables rapid updating and combining material, it can also contribute to
more varied and authentic assessment designs. We further describe three e-
assessment instruments and their potential for formative assessment.


3      Identification of Available Formative e-assessment Approaches and Tools

As a first step towards the introduction of formative e-assessment in primary and
secondary schools, PREATY makes an inventory of innovative e-assessment instru-
ments, starting with: e-Portfolio, learning analytics and enriched rubrics. These in-
struments may help teachers to have a better understanding of students’ performance
and offer valuable feedback to students and parents. They can support formative as-
sessment methods that have proven to be effective for learning [‎17], such as self- or
peer-assessment, instructional dialogues, questioning and reflective lessons. This
section describes the main characteristics of the mentioned e-assessment instruments.
   The first instrument examined is an ePortfolio. They are an electronic version of
traditional portfolios, meaning that the work samples inside them will be of digital
nature; coming from different sources and tools [‎18]. The ePortfolio facilitates the
capabilities of the learning portfolio to show the process and results of a learning path,
proving the quality and level of achievement of the targeted competences [‎4]. The
ePortfolio should include basic storing capabilities, with interoperability support, as
well the management of permissions and mechanisms to provide feedback on the
shared artefacts. Finally, different views of the work samples should be possible, in
order to highlight the competences of interest in each case.
A second e-assessment instrument of interest is Learning Analytics (LA), which can
be defined as the measurement, collection, analysis, and reporting of data about learn-
ers and their contexts, for purposes of understanding and optimizing learning and the
environments in which it occurs [‎16]. LA allows people who are responsible for the
teaching and learning process to identify possible learning risks, and make the neces-
sary changes, depending on the situations encountered. Many LA systems and ap-
proaches are being developed, like the ones based on Social Network Analysis. In
spite of the recent interest in these techniques, still a major challenge is to adapt the
output and visualizations to the teacher and learner’s needs.
   Thirdly, learning analytics can be combined with rubrics in the so-called enriched
rubrics [‎12]. These instruments have the form of a traditional rubric, but (at least part
of) the criteria are assessed by indicators automatically computed from the interaction
of the students with the system. One of these systems is LAe-R [12], a Moodle plugin
that allows a teacher to create an enriched rubric (ER) from scratch, or create a new
one from a template or edit an existing one. When creating an ER, the teacher adds
the assessment criteria with their corresponding descriptions and the grading levels
(see Error! Reference source not found.). LAe-R allows a teacher to add types of
criteria that are associated to learning analysis indicators: collaboration, grades to
assignments, and study of learning resources.




      Fig. 1. Screenshot of how a teacher can specify an assessment criterion in LAe-R

   The three instruments and their electronic applications and tools each in their own
way support the continuing process of feedback and help to shape students and teach-
ers understanding of it, both at the individual and the group level. An ePortfolio can
be used in formative and summative assessment procedures; they are very useful for
formative feedback because they typically allow to demonstrate and monitor both the
learning products and the –process. In principle it encourages students to take respon-
sibility for their own learning and for the sharing of its results. The online portfolio
tools often provide additional tools for sharing, analysis and support.
   LA systems can cover a wide range of applications, from the small-scale awareness
tools, with a long tradition in educational technology, to large-scale academic analyt-
ics. Our focus in PREATY is at the small-scale level, which is closer to our target
users. Several tools exist at this level, that based on the data collected from the learn-
ing systems make the processes visible, hereby allowing interventions and reflection.
Many of these tools are research prototypes, though. Our goal in PREATY regarding
these systems is to analyze what is needed to make them operational in the target
school contexts.
   Finally, most of the software tools supporting rubrics allow for cooperation on the
formulation and modification of rubrics, both between teachers and between teachers
and students. Tools that support enriched rubrics add the possibility to automate part
of the assessment, thus making it more efficient.
   In spite of the potential benefits of the reviewed instruments and associated tools,
their uptake by the target group is very small. Next section goes into some explana-
tions for this lack of interest.
4      Challenges and Approaches Towards Implementing Formative e-
       assessment in Schools

Different bottlenecks can be mentioned that prevent easy implementation of formative
assessment in schools and the technological support of it. As the authors in [17] men-
tion, based on an extensive review study, still a large amount of conceptual confusion
exists on the definition of formative assessment, and scientific research about the
conditions of effectiveness of formative assessment is still in an early phase of devel-
opment.
   Implementing formative e-assessment in schools also requires some essential pre-
paratory steps. In the first place more information and communication about forma-
tive assessment and possible approaches herein is necessary, as they are not yet
widely known and accepted. The tools mentioned in section 3 are seen as powerful,
although only partially applicable for the target groups of the PREATY project. Suc-
cess – and acceptance on the longer term by teachers- depends on some other crucial
conditions. Reflection about formative assessment and the goals one can achieve be-
longs to a school culture where teachers – and management- are willing to think about
innovation and come loose of a culture where they alone are in total control [‎17].
Looking at the conceptualization of [20], described earlier in this paper, it is also clear
that different target groups are involved in the implementation process, not only
teachers, but also learners and their peers. Thus, implementation of formative assess-
ment in schools requires a willingness to completely (re)think (about) assessment.
   A second issue at stake in the PREATY project is the introduction and training re-
lated to the technological tools and the formative e-assessment aspects. Regarding the
training about technological aspects it is important to focus on acceptance by giving
attention on how to present the objective of technology; are tools presented as ‘doing
things better’ or as ‘ doing better things’ [‎14]. These distinctions are important as they
include implicit views on the kind of enhancement that is at stake. The importance of
training is also stressed by Whitelock [19].She argues that four components are im-
portant to embed modern methods of assessment: (1) tool development; (2) staff train-
ing; (3) rethinking the assessment tasks; and (4) learning from the assessment tasks.
   Teacher professionalization in the field of formative assessment and the related
technological tools implies the training of complex skills. Trainings therefore will
have to include [‎13]:

1. information and training for in service teachers on formative assessment – enhanc-
   ing: assessment for learning, learn how to reflect, how to use criteria, how to han-
   dle feedback, how to work in teams etc.
2. information and training on technological tools; as very few tools are available
   training could be used to gather information about desirability and requirements for
   tools to be developed in this field. From the project results thus far it is clear that
   very few tools in this field are available for the target groups of the project (pri-
   mary/secondary in-service teachers). Most tools are developed for use in higher
   education; this seems especially the case for the learning analytics tools, as they
    use mainly online activities within learning environments. Implementation and use
    of these environments is only starting in the context of secondary schools and is
    almost inexistent in primary schools.

This might take the project to a training design focussing on sketching some alterna-
tive scenarios for formative assessment based on what is available from higher educa-
tional tools and from there going to the formulation of requirements for formative
assessment tools within primary and secondary education.


5       Conclusions

The adoption of formative assessment in schools requires a change of mind, from a
focus on summative testing, end results and teacher responsibility towards a process
and learning oriented perspective, with a joint responsibility of learners, teachers and
peers. It is important to learn to see formative assessment as a process, rather than a
particular kind of assessment [‎9], with different formative assessment strategies that
can support that process. Next to teacher training material and training strategies that
help to adopt and develop this mind-set, the availability of effective (for learning)
formative e-assessment methods and tools can help to facilitate adoption within
schools. The vision of the PREATY project is to advance in these directions, by set-
ting up a set of teacher training workshops implementing these principles. We aim to
explore the ways in which the e-assessment tools identified in the project can (or can-
not) be adopted by primary and secondary school teachers to assist them with apply-
ing these principles in practice.

Acknowledgements
We would like to gratefully acknowledge the contribution of the PREATY Project,
that is funded by the European Commission's Lifelong Learning Programme, Project
Number 526965-LLP-1-2012-1-GR-COMENIUS-CMP, which has part-funded this
work.


References

 1. Black, P. J., Wiliam, D.: Inside the Black Box: Raising standards through classroom as-
    sessment. Phi Delta Kappa, 80, 139-48 (1998)
 2. Black, P.J., Wiliam, D.: Developing the theory of formative assessment. Educational As-
    sessment, Evaluation and Accountability (formerly the Journal of Personal Evaluation in
    Education), 21 (1), 5-31 (2009)
 3. Busuttil-Reynaud, G., Winkley, J. : JISC e-Assessment Glossary, Bristol (2006)
 4. Butler, P.: A Review Of The Literature On Portfolios And Electronic Portfolios, Massey
    University College of Education, New Zealand and eCDF ePortfolio Project Steering
    Committee (2006)
 5. Gordon, J., Halasz, G., Krawczyk, M., Leney, T., Michel, A., Pepper, D., et al.: Key com-
    petences in Europe: Opening doors for lifelong learners across the school curriculum and
    teacher education. Warsaw (2009)
  6. JISC: Effective Practice with e-Assessment. An overview of technologies, policies and
     practice in further and higher education. Bristol (2007)
  7. Looney, J. : Making it Happen: Formative Assessment and Educational Technologies.
     Promethean Education Strategy Group: Thinking Deeper Research Paper No.1. – Part 3.
     Online available at: http://www.prometheanworld.com/rx_content/files/PDF/ Mak-
     ingitHappenFormativeAssessmentEducationalTechnologies-169721.pdf (2010)
  8. Looney, J., Siemens, G.: Assessment Competency: Knowing What You Know and Learn-
     ing Analytics. It is Time for a Breakthrough. Promethean Thinking Deeper Research Pa-
     per, No.3 (2011)
  9. McManus, S.: Attributes of effective formative assessment. Washington DC: CCSSO
     (2008)
 10. Pachler, N., Daly, C., Mor, Y., Mellar, H.: Formative e-assessment: practitioner cases.
     Computers & Education, 54, 715-721 (2010)
11. Pepper, D.: Literature Review: Assessment for key competences. KeyCoNet Report (2012)
 12. Petropoulou, O., Vassilikopoulou, M., Retalis, S.: Enriched assessment rubrics: a new me-
     dium for enabling teachers to easily assess student's performance when participating in
     complex interactive learning scenarios, Operational Research, Volume 11 (2), 171-186
     (2011)
 13. Pinock, N. Brandt, W.C.: Connecting Formative Assessment Research to practice. Learn-
     ing Point Associates. Online available at:
     http://www.learningpt.org/pdfs/FormativeAssessment.pdf (2009)
 14. Reilly, R..: Guest Editorial-Web Based Instruction: Doing Things Better and Doing Better
     Things. IEEEE Transactions on Education, 48 (4), 565-566 (2005)
 15. Ridgway, J., McCusker, S., Pead, D.: Literature Review of E-assessment, Bristol (2004)
 16. Siemens, G., Baker, R.: Learning analytics and educational data mining: towards commu-
     nication and collaboration. In Proceedings of the 2nd International Conference on Learning
     Analytics and Knowledge (LAK '12), Simon Buckingham Shum, Dragan Gasevic, and
     Rebecca Ferguson (Eds.). ACM, New York, NY, USA, 252-254 (2012)
 17. Sluijsmans, J.T., Brinke, D., Vleuten, van der, C.: Toetsen met leerwaarde. Samenvatting
     van de reviewstudie naar de effectieve kenmerken van formatief toetsen. NWO-PROO
     (2013)
 18. Subrahmanyam, V.V., Kanjilal, U., Sharma, V.K., Dudeja, A.K.: ePortfolio: An eWay of
     showcasing the artifacts. In: IEEE International Conference on Technology Enhanced
     Education (ICTEE), pp. 1-4. IEEE Press, Kerala, India (2012)
 19. Whitelock, D.: Activating Assessment for Learning: are we on the way with Web 2.0?’ In
     M.J.W. Lee & McLoughlin (Eds.). Web 2.0-Based-E-Learning: Applying Social Informat-
     ics for Tertiary Teaching. IGI Global, 219-342 (2010)
 20. Wiliam, D., Thompson, M.: Integrating assessment with instruction: What will it take to
     make it work? In C.A. Dwyer (Ed.), The future of assessment: Shaping teaching and learn-
     ing (pp. 53–82). Mahwah, NJ: Erlbaum (2007)