The LearnWeb Formative Assessment Extension: Supporting Awareness and Reflection in Blended Courses

Abstract

Blended solutions provide a means to orchestrate various types of activities and to schedule interactions at different times, nonetheless it is difficult to maintain a general overview of the class. In this paper, we build on the LearnWeb Design Framework to design and implement a Formative Assessment extension that supports the monitoring of the learning process in order to increase awareness and support reflection in a specific learning scenario. The extension offers a common basis for the various stakeholders (researchers, teachers and students) to collaboratively reflect on and design effective learning activities.

Keywords

• Awareness
• formative assessment
• co-design
• reflection
• TEL

-

Very often University courses entail large numbers of students which makes it difficult to design and carry out learning tasks, as well as to assess learning outcomes [ 13 ].

The use of technology makes it possible to record the tracks of student activity and to provide the teacher with dedicated analytics to improve awareness [ 2-3,12 ]. Learning analytics techniques are a valuable tool to support formative assessment practices that are based on two main pillars: (1) the collection of evidence concerning students’ progress towards learning outcomes; (2) the teachers’ and students’ reflections on the feedback of this information in order to enhance teaching and learning [ 11 ].

In this paper we propose a formative assessment strategy based on visualisation techniques to support teachers’ awareness and reflection in University learning contexts that integrate technology enhanced learning activities in the curriculum. 2

The LearnWeb Formative Assessment Extension as a Means to Support Awareness and Reflection LearnWeb1 is an online learning environment, which allows users to share and collaboratively work on user-generated resources either uploaded from the desktop, or collected from the web [ 5-6, 9 ]. In order to make the knowledge processes explicit both for teachers and for students granting them more agency in learning activities, the LearnWeb system has been designed in keeping with the Learning by Design approach [ 8,10,14 ]. The LearnWeb Design Framework (Fig. 1) has been demonstrated to be effective in supporting reflection and collaboration in the co-design of courses in the past [ 7-10 ]. Now we want to enhance the framework by providing tools that allow teachers to evaluate students’ work throughout their learning pathway. In order to provide the teacher with evidence of each student’s (or group’s) involvement in the various knowledge processes when carrying out the learning tasks, we need to explicitly associate and display the logged data that corresponds to each activity (see Table 1). In this way, the teacher can refer back to the original framework and course design, and monitor the students’ performance in line with the expected learning goals.

In order to address the needs of different scenarios the LearnWeb Formative Assessment extension has been designed from three main perspectives: (i) a course perspective, where the teacher has an overview of a specific course and can make comparisons between/among classes, (ii) a class perspective, where the teacher can moni1 http://learnweb.l3s.uni-hannover.de tor and compare the activities of small groups within the same class, and (iii) a personal perspective where the teacher can visualise information about a specific user. Each phase on the user interface is located in a different tab so as to visualise the data relating to each phase in a specific context (see Fig. 2 (1)). The visualisation of each of these perspectives takes into account the activities carried out by students aggregated with four groups related to the four LearnWeb Design Framework features as described in Table 1 (see Fig. 2 (2)), thus enabling teachers to analyse the factors involved in the various learning tasks. A specific learning scenario is described in the following section so as to provide a preliminary evaluation of the teacher’s feedback concerning the usefulness of the visualisation of data to support teaching strategies and practice. For the implementation of the Formative Assessment extension, we carried out frequency analysis and built the charts using the PrimeFaces3 library.

In agreement with the teacher, we started with the visualisation of the data collected in previous years so as to provide a diachronic overview for the teacher who will be able to compare the performances of different courses and reflect on the course design 2 http://wordwanderer.org/ 3 http://primefaces.org/ of future editions [ 4 ]. The objective is to evaluate the work carried out in the past and improve future teaching/learning experiences by personalising and adapting [ 11 ]. 2.2

The LearnWeb Formative Assessment Extension in Practice During the Academic Year 2011-2012 we carried out a study at the University of Pavia, in Italy that involved 284 first-year medical students divided into five classes: GolgiA (85 students), GolgiB (71), GolgiC (44), HarveyD (50) and HarveyE (34).

The syllabus was based on English linguistics and focused on text-based studies of (bio)-medical English. Students were expected to learn about multimodal theory and how to carry out multimodal text analysis, that is, the study of printed, website, digital and film texts in English and the ways in which these texts are used in different medical and biomedical contexts [ 1 ]. For their project work, students were required to carry out research in groups of 8-12 members on the topic “health/biomedical/scientific education through entertainment of young children and teenagers” and create a corpus of at least 50 websites. The work entailed the annotation of their search trajectories (failures and successes) and the use of the LearnWeb options to communicate with their group members and exchange information and comments.

At the time of the described scenario, the Formative Assessment Extension was not yet available, and the teacher had to explore the work done by students by browsing through the various groups. Using the current version of the system, it is now possible to obtain a rapid overview of student contributions with various levels of detail. (1) Course Perspective – the interface provides two search fields where the teacher can select two classes to be compared and choose different types of graphs to visualise the data. Fig. 3 presents a comparison between the HarveyE and the GolgiA class as regards the first pedagogical phase (Search and Exploration). Students in HarveyE searched and added more resources than the students in GolgiA. Since the project work task was the same for all groups (i.e. build a corpus of at least 50 websites), the teacher might want to intervene and invite 1th0e0GolgiA groups to speed up the work.

The comparison is about the type of activities carried out by students in the two classes and can be between courses of the past, or courses functioning at the same time. In the first case (past courses), the teacher can see whether the current class is performing better or worse than the previous class. Consequently, the teacher can reflect on how to improve the course design or introduce better explanatory strategies for students by using notifications to communicate with them and give support. In the second case (current courses), the system visualises the actual value at a specific time so that the teacher is constantly up to date on how the students are working in the platform. For example, if one class is performing at a slower rate than another in the same course, the teacher can send a notification and a request to the group leader to speed up the work. This strategy can be useful when the teacher wants to stimulate competition.

(2) Class Perspective – the teacher can choose a specific class in the course and select two sub-groups to be compared. In the GolgiA class for example, the results of the comparison between the activities of the Euronics group and the Children`s genes group in the Search and Exploration phase are very similar. Whilst in the Annotation and Description phase (Fig. 4) we notice that a larger number of tags were added by the members of the Euronics group, the students in the Children`s genes group were more active in commenting and editing resources.

The results show how the two groups use a different strategy to conceptualise and categorise contents, thus helping the teacher to understand the learning behaviour and evaluate the group work accordingly. The teacher might decide to discuss the findings with students and reflect on their behaviour during the course or the final exam. (3) Personal Perspective – The window provides two search fields where the teacher can select a specific class and a specific student in that class. The resulting graph shows the number of activities carried out by the student throughout the course (see Fig. 5). This information can help the teacher to better understand the performance of each student and provide personalised feedback, for example according to: (i) the specific role (e.g. group leaders can/should carry out additional activities compared to the other group members), (ii) the given task (e.g. the teacher can encourage the slower students to carry their weight). Such visualisations are useful to detect an increase or decrease in student participation in learning activities and can encourage the teacher to restructure some tasks or to adopt different pedagogical strategies if deemed necessary. Prompt intervention can be a crucial factor in determining the success or failure of a course. Using the Formative Assessment extension, the teacher is made aware of the dynamics that are taking place in the course and can speedily intervene in order to raise interest when it appears to be waning. As matter of fact, the teacher in Pavia evaluated the prototype of the system and confirmed its potential: “it has practical applications which can save time and allow for the constant realignment of the teaching strategies with the learning goals”. 3

Conclusions and Future Work The LearnWeb Formative Assessment extension is designed to offer a common basis for various stakeholders: for teachers to reflect on the teaching practices and refine their pedagogical strategies; for students to keep track of their personal progress and measure their performance in comparison with their peers; for researchers to realise what functionalities work better to support specific learning tasks and improve the system.

While some components of the technical approach are already available, others are under development. For the moment we focused on developing tools to support teach102 er awareness and to facilitate monitoring and mentoring activities. In the future, we will develop the assignment and recommendation component that includes the user interface for the learner. Another step will be the addition of a temporal dimension in order to give a diachronic visualisation of group interactions. The impact of the feedback provided through the proposed extension on student learning pathways will be investigated in future projects both in Italy and in Brazil. An extended study will be carried out with the aim of analysing how feedback is perceived by students and the impact it has on moulding the next learning stages. 4

References

1. Baldry , A. , Thibault , Paul J. Multimodal transcription and text analysis. A multimodal toolkit and coursebook . London and New York: Equinox ( 2006 ).

2. Charleer , S. , Santos , J. L. , Klerkx , J. , Duval , E.: LARAe: Learning Analytics Reflection & Awareness environment . In Milos Kravcik et al. (Eds.). In: Proceedings of the ARTEL 2014 Workshop , vol. 1238 , pp. 85 - 87 .

3. Govaerts Sten , Verbert Katrien, Klerkx Joris, Erik Duval ( 2010 ). Visualizing Activities for Self-reflection and Awareness. Advances in Web-Based Learning (ICWL 2010 ).

4. Greller , W. , Drachsler , H.: Translating Learning into Numbers: A Generic Framework for Learning Analytics . Educational Technology & Society , 15 ( 3 ), 42 - 57 ( 2012 ).

5. Marenzi , I. , Nejdl , W.: I search therefore I learn. Supporting active and collaborative learning in language teaching . In: A. Okada , T. Connolly , P. Scott (eds.): Collaborative Learning 2.0: Open Educational Resources . IGI Global , pp. 103 - 125 ( 2012 ).

6. Marenzi , I. , Zerr , S. : Multiliteracies and active learning in CLIL: the development of LearnWeb2.0. IEEE Transactions on Learning Technologies (TLT) . Los Alamitos, CA, USA, 5 ( 4 ), 336 - 348 ( 2012 ).

7. Marenzi , I.: A multiliteracies approach for supporting language learning courses at university level . In: J. Jovanovic & R. Chiong (eds.): Technological and Social Environments for Interactive Learning . Santa Rosa, CA: Informing Science Press, 249 - 276 ( 2013 ).

8. Marenzi , I. , Kantz , D. : ESP course design - a multiliteracies approach . In: Reshaping Learning: Transforming Education through the Fusion of Learning and Technology , Proceedings of the 13th ICALT Conference ( 2013 ).

9. Marenzi , I. : Interactive and collaborative supports for CLIL: towards a formal model based on digital literacy . In: R. Kupetz & C. Becker (eds.): Content and Language Integrated Learning (CLIL) by Interaction . Frankfurt am Main: Peter Lang , 75 - 99 ( 2014 ).

10. Marenzi , I. : Multiliteracies and e-learning2.0. Foreign language pedagogy content- and learner-oriented , Vol. 28 . Frankfurt am Main: Peter Lang ( 2014 ).

11. Taibi , D. , Kantz , D. and Fulantelli , G.: Supporting formative assessment in content and language integrated learning: the MWSWeb platform . In: IJTEL , 6,( 4 ), pp. 361 - 379 ( 2014 ).

12. Vozniuk , A. , Triana , M. J. R. , Holzer , A. , Govaerts , S. , Sandoz , D. , Gillet , D. : Contextual Learning Analytics Apps to Create Awareness in Blended Inquiry Learning . In: Proceedings of the 14th ITHET ( 2015 ).

13. Wang , T. H. : What strategies are effective for formative assessment in an e-learning environment? In: Journal of Computer Assisted Learning , 23 ( 3 ), 171 - 186 ( 2007 ).

14. Yelland N. , Cope B. , Kalantzis M. : Learning by Design: creating pedagogical frameworks for knowledge building in the twenty-first century . Asia-Pacific Journal of Teacher Education , 36 ( 3 ), 197 - 213 ( 2008 ).