Proceedings of the 5th Workshop on Awareness and Reflection in Please refer to these proceedings as
Milos Kravcik, Alexander Mikroyannidis, Viktoria Pammer, Michael Prilla, Thomas Ullmann (Eds.): Proceedings of the 5th Workshop on Awareness and Reflection in Technology Enhanced Learning. In conjunction with the 10th European Conference on Technology Enhanced Learning: Design for Teaching and Learning in a Networked World, Toledo, Spain, September 15, 2015.
Available online at http://ceur-ws.org/. c 2015 for the individual papers by the papers’ authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.
The front-cover was created by Harriett Cornish (The Open University, KMi).
Addresses of the editors:
Thomas Daniel Ullmann Institute of Educational Technology The Open University Walton Hall Milton Keynes MK7 6AA
Alexander Mikroyannidis Knowledge Media Institute The Open University Walton Hall Milton Keynes MK7 6AA
The #ARTEL15 workshop accepted 3 full papers, and 7 short paper. The accepted papers discuss awareness and reflection in formal education, outside formal education, and methods and analytics of awareness and reflection research.
Three papers focus on technology enhanced awareness and reflection in formal eduction. The full paper ’Annotations as reflection amplifiers in online learning - an exploratory study’ of Dominique Verpoorten, Wim Westera, and Marcus Specht discusses an annotation tool, and experiment to investigate the effects of three conditions: No annotations, free-note annotations, and structured question-based annotations. The paper provides insight into annotations as reflection amplifiers.
The short paper ’Formal concept analysis for modelling students in a technology-enhanced learning setting’ of Michael A. Bedek, Michael Kickmeier-Rust, and Dietrich Albert presents the Formal Concept Analysis (FCA) framework for visualising a domain with concept lattices. FCA aims at facilitating student reflection upon their learned and still-to-learn concepts via an open learning modelling approach.
The short paper ’Learning to look - purpose and design of an awareness-raising online course in veterinary sciences; of Sophie Tasnier, Valeria Busoni, Christian Hanzen, Jeff Van de Poel, Geraldine Bolen, Catherine Delguste, Nadine Antoine, Veronique Delvaux, Tania Art, and Dominique Verpoorten discusses an approach to improve veterinary student visual awareness in use of clinical images.
Four papers discuss technology enhanced reflection and awareness outside formal educa
The full paper ’Prompting users to facilitate support needs in collaborative reflection of Oliver Blunk, and Michael Prilla outlines a concept of how to support collaborative reflection with the help of prompts. They show an initial implementation of a prototype and an approach how to evaluate the concept.
The short paper ’A course concept for enhancing reflective learning - bringing research project results into the field’ of Nils Faltin, Margret Jung describes the development of a course for learning time management for professionals. The course is based on reflective learning and practice, in a mix of coaching and computer-support.
The short paper ’Feeler: supporting awareness and reflection about learning through EEG data’ of Eva Durall, and Teemu Leinonen presents a prototyp which aims at supporting awareness and reflection about learning experiences through student’s EEG data. The short paper ’Mood in the city - data-driven reflection on mood in relation to public spaces’ of Viktoria Pammer discusses potential benefits for stakeholders, as well as system design issues, for reflection on urban development issues with mood as entry point to reflection.
Three papers elaborate on methods and analytics of awareness and reflection research in technology enhanced learning.
The full paper ’Keywords of written reflection - a comparison between reflective and descriptive datasets’ of Thomas Daniel Ullmann investigates a method to derive reflection keywords by contrasting two datasets, one of reflective sentences and another of descriptive sentences. The reflection keywords are discussed in the context of a model for reflective writing.
The short paper ’The LearnWeb formative assessment extension: Supporting awareness and reflection in blended courses’ of Alana Morais, Ivana Marenzi, and Deirdre Kantz presents work about the development of visualisations summarising activity of users, groups, and classes of a collaborative sharing and working platform.
The short paper ’Deploying learning analytics for awareness and reflection in online scientific experimentation’ of Alexander Mikroyannidis, Aitor Gomez-Goiri, John Domingue, Christos Tranoris, Daan Pareit, Jono Vanhie-Van Gerwen, and Johann M. Marquez-Barja introduces a European initiative for online learning and experimentation (FORGE), especially its aim to use of learning analytics to support awareness and reflection for learners and educators.
The official workshop webpage can be found at http://teleurope.eu/artel15 The 5th Workshop on Awareness and Reflection in Technology Enhanced Learning (ARTEL 2015) is part of a successful series of workshops.
4th Workshop on Awareness and Reflection in Technology Enhanced Learning (AR
TEL14). Workshop homepage: http://teleurope.eu/artel14. Proceedings: http://ceur-ws.org/Vol-1238/. 3rd Workshop on Awareness and Reflection in Technology Enhanced Learning (ARTEL13). Workshop homepage: http://teleurope.eu/artel13. Proceedings: http://ceur-ws.org/Vol-1103/. 2nd Workshop on Awareness and Reflection in Technology Enhanced Learning (ARTEL12). Workshop homepage: http://www.teleurope.eu/artel12. Proceedings: http://ceur-ws.org/Vol-931/. 1st European Workshop on Awareness and Reflection in Learning Networks (ARNets11). Workshop homepage: http://teleurope.eu/arnets11. Proceedings: http://ceur-ws.org/Vol-790/ Augmenting the Learning Experience with Collaboratice Reflection (ALECR11). Workshop homepage: http://www.i-maginary.it/ectel2011/index. html 1st Workshop on Awareness and Reflection in Personal Learning Environments (ARPLE11). Workshop homepage: http://teleurope.eu/arple11. Proceedings: http://journal.webscience.org/view/events/The_PLE_ Conference_2011/paper.html#group_Proceedings_of_the_1st_ Workshop_on_Awareness_and_Reflection_in_Personal_Learning_ Environments To stay updated about future events, to share your research, or simple to participate with other researchers, consider joining the group about Awareness and Reflection in Technology Enhanced Learning: http://teleurope.eu/artel We especially would like to thank the members of the programme committee for their invaluable work in scoping and promoting the workshop and quality assuring the contributions with their peer reviews.
September 2015
Milos Kravcik, Alexander Mikroyannidis,
Viktoria Pammer,
Michael Prilla, Thomas Ullmann
Milos Kravcik, RWTH Aachen University, Germany Alexander Mikroyannidis, The Open University, United Kingdom Viktoria Pammer, Graz University of Technology, Austria Michael Prilla, University of Bochum, Germany Thomas Ullmann, The Open University, United Kingdom
Philippe Dessus, Universite Pierre-Mendes-France, France Ines Di Loreto, UTT - Universite de Technologie de Troyes, France Eva Durall, Aalto University, Finland Nils Faltin, IMC AG, Germany Rebecca Ferguson, The Open University, UK Angela Fessl, Know-Center, Austria Michael, Kickmeier-Rust, Graz University of Technology, Austria Kinshuk, Athabasca University, Canada Simon Knight, The Open University, UK Effie Law, University of Leicester, UK Paul Lefrere, The Open University, UK Elvira Popescu, University of Craiova, Romania Carsten Ullrich, DFKI, Germany Dominique Verpoorten, University of Liege, Belgium Riina Vuorikari, European Commission JRC IPTS, Spain
Supporting projects http://www.mirror-project.eu http://learning-layers.eu http://www.laceproject.eu/ http://wespot-project.eu http://employid.eu http://www.boost-project.eu
Summary of the contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . Awareness and reflection workshop series . . . . . . . . . . . . . . . . . . . . Organisation committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Program committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Supporting projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Dominique Verpoorten, Wim Westera, Marcus Specht
Michael A. Bedek, Michael Kickmeier-Rust, Dietrich Albert
Sophie Tasnier, Valeria Busoni, Christian Hanzen, Jeff Van de Poel, Geraldine Bolen, Catherine Delguste, Nadine Antoine, Veronique Delvaux, Tania Art, Dominique Verpoorten
Oliver Blunk, Michael Prilla
Nils Faltin, Margret Jung
Eva Durall, Teemu Leinonen
Viktoria Pammer
Thomas Daniel Ullmann 3 3 4 6 7 8 11 27 35 43 59 67 75 83
Alana Morais, Ivana Marenzi, Deirdre Kantz
Alexander Mikroyannidis, Aitor Gomez-Goiri, John Domingue, Christos Tranoris, Daan Pareit, Jono Vanhie-Van Gerwen, Johann M. Marquez-Barja 97 105 Annotations as reflection amplifiers in online learning – An exploratory study Dominique Verpoorten1, Wim Westera2, Marcus Specht2
{dvepoorten}@ulg.ac.be
{wim, spe}@ou.nl Abstract. In a controlled experiment on the effects of frequent and local digital annotations, 137 volunteers covered an online course at 3 conditions: no/free/question-based electronic annotations. Results show no difference in performance between groups. However, analyses conducted within treatments suggest positive impacts on performance when annotation rates are taken into consideration, and coupled with other reflective enactments.
Keywords: annotations, reflection amplifiers, students set the test, widgets, split screen learning 1
Introduction
Note-taking, either when listening to lectures or reading texts, is a “totem” of teaching
and learning. It seems that for centuries tutors have been expecting that students do
take notes and that tutees consider note-taking as a natural activity in a scholarly life
[
While the effects of note-taking are well documented for paper-based practice [
Beyond their variety, the new alleys of research (for an extended view on recent
work, see [
Boch and Piolat [
Reflection Amplifiers
In this study, the annotations are conceived as “reflection amplifiers”. Following the
definition by Verpoorten, Westera, and Specht [
The concise reflection they call for further characterises RAs. As support to
condensed reflective processes, RAs operate though miniature Web applications
(sometimes called “widgets”) performing a single task, displaying a very clear and
appropriate graphical style, and providing a single interaction point for direct provision of a
given kind of data [
Hypotheses In a comparative study an online course was delivered at 3 conditions: without annotation tool, with annotation tool and free-style notes, with annotation tool and question-based notes (i.e. expressed in the format of a question on the read material). The study investigated the effects of the digital annotations – conceived as multiple short episodes of analytical reflection – upon the enhancement of the quality of learning. Two hypotheses guided the experiment.
Hypothesis 1. “The availability of an annotation tool and the assignment to use it for frequent and local notes induce higher marks at the test and an increased study time”. Short but repeated efforts of reflection are predicted beneficial to the content internal12 ization because they are seen as a way to stay analytically engaged with the supplied learning material. It is also speculated that such a reflective approach to learning has a price with regard to time spent on the material.
Hypothesis 2. “The question-based annotation strategy induces higher marks at the test than the spontaneous way of annotating”. The study includes a concern for annotation methods by challenging conventional practice of note-taking “as a student” with a different mode wherein the learner is invited to reflect “as an instructor” (details in section “The annotation methods”). 2 2.1
Method
Independent Variables The intervention variables were the provision of an embedded annotation tool and the exposure to a strategy for frequent and local annotations. 2.2
Dependant Variables The dependent variable was the subjects’ reflective engagement with the content, broken down into seven tangible indices: • Index 1: mark at the final test (FinalTest). This index designated the score obtained at the final test taken after the study session. It measured learners’ achievement through 16 multiple-choice questions assessing knowledge and comprehension; • Index 2: time spent in the course (TimeSpent). This index, measured as the number of “active ten-minute periods” in the course, was an estimate. One active period was counted each time that at least one click occurred in a time span of ten min. Longer periods were left out in an attempt to correct for the time students would spend in activities foreign to the study while still being logged into the course; • Index 3: learning efficiency (LearnEff). It is fair to say that the speed of learning is an important achievement (many performance tests, e.g. IQ tests, use time as one of the main indicators). In order to incorporate this temporal dimension in the measures, the marks at the final test were related to the time spent in the course: slow learners got a lower score per unit of time than fast learners. Low-efficiency students did not necessarily receive lower marks, but they needed more time to reach their mark; • Index 4: number of page views (NumberPages). The browsing behaviour, and in this case the action of re-visiting pages, was considered as an index of reflective engagement because it assumed a meta-learning decision about the need of rereading the material; • Index 5: quantity of annotations (NumberAnnot). • Index 6: total number of characters for the annotations (CharactInAnnot); • Index 7: number of visits (VisitDash) to the Learning Dashboard (see section “Apparatus”).
The indices FinalTest, TimeSpent, LearnEff, and NumberPages were common to the 3 conditions. NumberAnnot, CharactInAnnot, and VisitDash were premised upon the annotation tool and therefore only offered in Treatments 2 and 3.
A post-test questionnaire allowed measuring the effects of the intervention on the following additional variables: sense of control and opinion over the annotation experience. 3
Apparatus
The Online Course
The learning material of the experiment was the four-hour online course “Seks en de
evolutie” (Sex and the evolution), a course signed and offered in Dutch by the Open
Universiteit on the eLearning platform Moodle [
The tool The digital annotation tool was a comment box displayed on each page (Fig.1). It kept record of all annotations produced by the learner on this very page. The annotation tool unfolded through a click by the learner. Consistently with the length of the reading material and the action requested from learners (frequent but short notes), the surface of the tool was intentionally not large and its function deliberately restrained to the basic typing. As for pedagogy, the annotation tool was offered to promote analytical scrutiny and internalization of the learning material’s meaning by making it possible for learners to capture, within the study task, the gist of what has been read.
In order to prevent effects of fragmentation and to also support the function of
“annotations as products” [
Results
An alpha level of .05 was used for all statistical tests.
Background Questionnaire. To ensure equivalence between treatments, statistical
tests were performed on the data collected in the background questionnaire. The
procedure exhibited an even distribution in the 3 conditions regarding:
• meta-cognitive capacities, measured with a shortened version [
N = 137) = .36, p = .83; • self-reported familiarity with eLearning, measured with a 3-point Likert scale, χ2(2, N = 137) = 3.94, p = .13; • demographics: age F(2, 134) = .4, p = .92, ηp2 = .07 (X = 39, SD = 11), sex χ2(2) = .73, p = .69 (56% female, 44% male), and education level χ2(2, N = 137) = 4.8, p = .09 (75% of the sample ticked the category “Higher education”). Indices. An ANOVA procedure (Table 1) exhibited no significant difference between conditions regarding mean marks obtained at the final test, F(2, 134) = .44, p = .64, ηp2 = .007. Significant differences emerged between conditions with regard to the: • total time spent on the course, F(2, 134) = 3.49, p = .03, ηp2= .05; • number of page views, F(2, 134) = 5.29, p = .006, ηp2 = .07; • learning efficiency (mark at the test/time spent in the course), F(2, 134) = 4.76, p = .01, ηp2 = .01.
Additional Fisher contrast tests disclosed that the differences were significant only against the control group and not between the treatments. This lack of observable divergence made it reasonable and beneficial to statistical power and clarity to redefine the treatment conditions as one single group (N = 103) for the following analyses. 4.2
Measures Within Treatment Group Amount of Reflective Enactments. No correlation was found between the mark at the test and the absolute number of annotations (Index 5), characters (Index 6), page views (Index 4) and dashboard views (Index 7).
Rates of Reflective Enactments. Beyond the mere amount of reflective actions (NumberAnnot, CharactInAnnot, NumberPages, Dashvisits), the rates at which these enactments occur while studying might be an important aspect of the reflective activity. For this reason, “reflection rates” were calculated to express the displayed reflection per unit of time (minute) for different indices. These rates were obtained for each individual by dividing the quantity of reflective enactments (the different indices) by the individual time spent in the course (Index: TimeSpent). Based on these ratios, post-hoc splits were applied: subjects were categorized against the mean of the group as either high/low annotators (HA/LA via Index 5), high/low producers of annotation characters (HC/LC via Index 6), high/low browsers (HB/LB via Index 4) and high/low visitors of the learning dashboard (HD/LD via Index 7). For instance, participant 45 took 87 annotations (against an average of 43 for the whole group), produced 13958 characters (against an average of 4792), visited a content page 56 times (against an average of 78), and paid 2 visits to the dashboard (against an average of 3). According to the ratios obtained by dividing these indices by the study time (410 minutes for participant 45 against an average 328 minutes), participant 45 was labelled: HAHCLBLD (High Annotator – High producer of Characters – Low Browser – Low Dashboarder). It was assumed that this fourfold “learning DNA” captured different facets of the participant’s reflective engagement with the learning material. Assigning such a multivariate reflective engagement profile to the 103 participants revealed some new insights.
One-index Learning DNA and Mark at the Test. Table 2 displays the performance of high and low groups for each reflective enactments. When treated independently from each others, the indices deliver significant differences for index 4 (annotations), t(101) = 2.146, p = 0.034, d = 0.35 and for index 5 (characters in annotations), t(101) = 2.76, p = 0.007, d = 0.35. N Score SD
Low 59 5,8 1,81
High
Low
High 44 6,5 1,7 61 5,7 1,8 42 6,7 1,4 1,7
High 32 6,3 1,9
Low 53 6,1 1,5
High 50 6,1 2 Two-index Learning DNA and Mark at the Test. Twofold combinations of reflective rates, for instance HA+HB (high annotation rate + high browsing rate) versus HA+LB (high annotation rate + low browsing rate) exhibited significant differences at the omnibus ANOVA, F(3, 99) = 3.19, p = .027, ηp2 = .088. Table 3 shows the data for significant cases. Post hoc comparisons using the Fisher LSD test located significant mean differences between HA+HB and LA+HB (p = 0,002), and HA+HB and LA+LB (p = 0,022), but not between HA+HB and HA+LB (p = 0,082). Three-index Learning DNA and Mark at the Test.The attempts made with a profile combining 3 reflection rates gave a significant mark advantage to the most reflective profile (HA+HB+HD) onto all other combinations. However, the creation of such additional combinations induced more numerous groups (Table 4) and quickly created a problem of statistical power that hampered significance tests.
SD
Qualitative Results The explored qualitative aspects - overall satisfaction, sense of control, perceived intensity of reflection, - were self-reported on 5-point Likert scales in the evaluation questionnaire.
Sense of Control. Mann-Whitney test on the sense of control of the high versus low annotation rates (HA/LA) did not disclose significant differences, U = 1225, p = .61, r = .4. But when the browsing rate was added in the profile, the highly engaged people (HA+HB) reported a significantly higher level of control (Mdn = 4) compared to HA+LB (Mdn = 3), LA+HB (Mdn = 3), LA+LB (Mdn = 3), χ²(3, N = 103) = 7.69, p = .04.
Stimulation of Reflection by the Annotation Process. When asked about the effect of taking frequent annotations, 71.2% of the sample answered that reflection increased, 24.6% that it was not influenced and 4.2% that it diminished. 5
Discussion
“Put simply, reflection is about maximising deep and minimising surface approaches
to learning.” [30, p. 3]. As a strategy to promote deep learning, this study asked
learners to use annotations as “reflection amplifiers”, i.e. brief and repeated reflection
affordances, interspersed in the learning material and activated, through the support of a
dedicated widget, in support to the first-order learning task at hand. These
stop-andthink episodes could be seen as a tentative instantiation of “split screen learning” [
RAs do expand time on task without delivering benefit for learning achievement: the control group gets the same mark while using less time (see Table 1). Yet, the first hypothesis is not confirmed. From a strict performance-oriented viewpoint, frequent and local annotations are counter-productive.
No noticeable performance appeared between distinct uses of the annotation tool: free annotation versus question-based annotation (see section Indices). Yet, the second hypothesis is not confirmed.
This study delivers however one unexpected and intriguing pattern when the analysis operates 2 shifts of focus (see section 4.2): a) from one single reflective enactment taken in isolation (writing annotations) to multiple reflective enactments (producing a longer annotation, navigating amongst course pages, visiting one’s learning dashboard), and b) from the mere amount of these reflective enactments to their rates of use. In this case, results provide insights about ways students balance and combine the primary activity (studying the course) and the secondary reflective activities (annotations, page re-visits, dashboard views). Here, a different pattern emerges: students who write more personal annotations per unit of time than the average get a higher mark. Combinations of this reflection rate with other reflective enactments (page views, dashboard views) bring extra benefits to performance. The qualitative data also seems to be affected by combined reflective rates: a significant effect on student’s sense of control is obtained only from high aggregated rates of reflective enactments (see section “Sense of Control”).
On this basis, it can be advanced that the dynamics of reflective commitment to a study task encompasses and interweaves several reflective enactments performed at a certain rhythm. It is possible that the reflective passivity of some students might be counteracted by inviting them not only to deploy more reflective actions on the material but also to accelerate their frequency. 6
Further Work Four main issues raised in this study call for further research.
RAs and Performance. Although performance tests are not the only way to measure
learning, it remains a legitimate and largely-practised way to assess mastery of
content. In this perspective, final scores should reasonably be expected to reflect benefits
resulting from using RAs. It has not happened here, at least in comparison with the
control group. This lack of benefits from annotations contrasts with other studies in
the field [
RAs and Intellectual Dynamics. On a more fundamental level, the study findings,
and especially those related to the effects of combined reflective enactment rates
(annotations, revision of annotations, page re-visits), highlight the intellectual dynamics
at work in a deep approach of study material. Further investigation is required to
establish whether it could be a characteristic of high achievers and a hallmark of
intellectual life in general to operate an “active study”, defined as an ongoing crisscrossing
between a primary learning activity and secondary reflective or meta-cognitive
enactments. This periodic and persistent to-and-fro mental move offers a very different
aspect of reflection than the one conveyed by portfolios or learning diaries wherein
the experience and the reflection thereon are temporally distinct. If metaphors can be
invoked, their complementarity presents as Lego blocks while the real-time dynamics
of learning would look more after pinballs. Practical ways to evidence and sustain this
interplay between cognitive and reflective landscapes [
RAs and Personalized Learning. An important issue for future research on annotations is also tied to the selection of relevant frames to ascertain their effects. More research needs to be undertaken to see if prompts used to amplify reflective appraisal of the study material can be related to ownership of learning and sharper feelings that learning is or becomes a “personal matter”. In such an approach, personalised learnopportunity to cooperate however, in hindsight I feel some aspects could have been handled differently'. Another example is the sentence, 'My reaction to her and the situation surprised me as I became quiet agitated and in hindsight was probably just as unaccommodating as she was'. The two sentences also contain the keyword 'hindsight', which is used to express a retrospective understanding of a situation. 6.2 Personal It is notable that first person singular pronouns, such as 'I', 'me', 'my', and 'myself', are keywords overused in the reflective dataset. This may indicate that reflective writings are frequently written from the first person perspective, which is also in line with the category 'Personal' of the reflection model as outlined above. The third person singular pronoun 'he' is more associated with a descriptive text. The category 'Personal' is also about own beliefs. The keyword 'believes' may be indicative for expressing beliefs. The following sentence is an example of this: 'Reflecting about conflicts I had in the past, I believe that I could have handled some of them better'. Another example of this is the sentence, 'When the time came to allocate work for the plan I believe the team dynamics were developed enough to assess accurately everybody's strengths and apportioned work accordingly'. The modal verb 'would' can also refer to beliefs. An example is the sentence, 'I felt that as team leader I would have control in the group and I would have more say in the way our team was run, little did I know then'. Another example is the sentence, 'Knowing about the tradition, I would definitely have acted differently, hopefully being in the position to speak at least a bit of the language'. 6.3 Feelings The words 'feel' and 'felt' are at top of the list, and they occur relatively more often in the dataset of reflective sentences. This is in line the category 'Feelings' of the reflection model. Expressing feelings is often mentioned as part of reflective writing. 6.4 Critical stance Several words can be associated with the category 'Critical stance', for example, the keywords: 'more', 'better', 'if', 'but', 'never', 'could', and 'although'.
The word 'more' could relate to the critical thought of a writer that something is lacking and that more of something would be better, or it could relate to the realisation that there is now more of something that was previously not there. For example, the sentence, 'I should be more aware about the power issues and how they silence patients', expresses the first sense, which is the realisation that something is still lacking. The sentence, 'I noticed more discussions taking place after the first couple of sessions, and I felt our group was more established as we began to get to grips with what the vignette would entail and felt comfortable with each other', refers to the second meaning—the realisation of a change.
The word 'better' could refer to the critical awareness of the writer that something is now better, as expressed in the sentence, 'I hadn't really thought of it like this before 90 but by empathising with Mary's situation I better appreciate the importance of the patient's perspective'. It could also express that something should have been better, as in the sentence, 'I might have better explored Jim's internal thoughts and his wondering about what he finds in the cave'.
The conjunction 'if' could express a premise followed by a conclusion as part of reasoning about something. For example, the sentence, 'It would have been helpful if I had shared my concerns about the group with the LSA to start with'. If he had shared his concerns (premise), then it would have been helpful (conclusion).
The conjunction 'but' could express a contrasting thought. An example is the sentence, 'Looking at it from Marissa's point of view, she may have known that I was on the wrong track, but she probably would not have been able to do anything about it because I am a doctor'. Another example is the sentence, 'I did not do all that well in the exam so maybe I need to prepare differently - but I really don't know how to do it'.
A writer could flag with the adverb 'never' a realisation that something was never experienced or that something never happened before. The following sentence is indicative for this: 'In the topic I found it most interesting about the lack of invariance problem as I have never realised the fact before although the point is reasonably understandable'. Another example is the following sentence: 'I had never experienced those same feelings of disconnect in real time though, never felt as though the person talking was somehow not me, until last Tuesday'.
The verb 'could' might indicate the awareness of a possibility or alternative. The sentence, 'On the whole our group worked well but could have been improved by more openness and discussion about issues affecting the group, such as social loafing', shows that the writer describes a realisation that there is a real possibility for improvement. Another example is the sentence, 'Reflecting about conflicts I had in the past, I believe that I could have handled some of them better'.
The word 'although' could be used in a contrasting way. An example is the following sentence: 'Although throughout my training to date, I have dutifully reflected on various clinical situations and considered learning objectives within the practice portfolio; I can not say that I had actually fully taken on the implications of what it is to be a truly reflective practitioner'. The writer describes the contrast between the perception of reflection in previous trainings and the current perception. Another example is the sentence, 'Applying the learning cycle proved to be a useful tool, although I was very sceptic at the beginning'. This sentence describes a contrast in perception. The sceptical few dissolved over time. 6.5 Perspective The keyword list from Table 1 does not contain a keyword that allows us to establish a strong link between a keyword and the category 'Perspective'. 6.6 Outcome The 'Outcome' category of the model contains a retrospective dimension that entails, for example, the description of lessons learned, but also a prospective dimension directed to statements about what to do in future. The keyword list contains the word 'learnt'. This word could express that something was learned. An example of this outcome facet is the sentence, 'I have learnt that when I am requesting something different, I must explain my needs fully, and communicate the message more effectively'. Another example is the sentence, 'I have also learnt from participating in this group that I was, in this case, one of the more dominant group members, and felt confident in expressing my views and ideas'. Another keyword is the noun 'future'. This keyword could be used to express future intentions. An example is the sentence, 'Although this situation didn't have a satisfactory outcome I hope to have learned from the experience and aim to use my new insight to develop my future practice'. Another example is the sentence, 'This makes you realise that you could come across this within the profession and luckily from this activity I am now aware of this and I can now make the most of any opportunities that a rise to enable me to take this into account and maybe build up my confidence so in the future I can maybe go onto stand up for what I believe in and also get my opinions noticed if I feel this necessary'. 6.7 Summary Overall, this discussion showed that for several keywords, the log-likelihood statistic can derive words that are in line with the categories of the chosen model of reflection.
The keywords listed in Table 1 can be seen as good candidate words for the
construction of dictionaries. With the shown approach, we can form a set of words, such
as a dictionary, that can be used to automatically summarise texts with regard to the
frequency of occurrence for each category. However, Table 1 also shows that words,
although they are found frequently in one dataset, cannot be used to distinguish
completely between reflective and descriptive use. For example, the keyword 'i' is found
700 times in the reflective dataset, but 'i' is also used 105 times in the dataset of
descriptive sentences. A definite classification of text passages based on single words is
also not the aim of dictionary-based approaches. One of the use cases there is that the
dictionaries are used to predict important outcomes. This is a quantitative indicator that
can be used to corroborate the findings of a study (for example, see [
Conclusion and outlook This paper demonstrated the application of the keyword method on a dataset of reflective and descriptive sentences. The log-likelihood statistics was used to determine words with high 'keyness' in either the dataset of reflective sentences or that of descriptive sentences. The words derived with the described method represent words that occur with unusual relative frequency in the datasets. In the discussion of the results, several of these keywords were assigned to categories of a model of reflective writing. This step illustrated that the investigated keywords can be associated with the reflection 92 model categories. This supports the applicability of the keyword method to derive words important for reflection.
An extension of the shown approach is to investigate the 'keyness' of the categories of the reflection model. For example, a comparison of a dataset that describes the outcomes of reflective writings with a reference dataset would allow us to derive keywords of reflection outcomes by adding an in-depth study of keywords for this category.
Furthermore, reflection dictionaries can be combined with rule-based systems [
As outlined, the automated detection of reflection in writings relies on patterns of reflection, because these patterns can be codified and used for the automated analysis of writings. The study showed that reflective writings contain such patterns at the word level, because there are words that occur significantly more often in the dataset of reflective sentences than in the dataset of descriptive sentences. 8
The LearnWeb Formative Assessment Extension: Supporting Awareness and Reflection in Blended Courses
Alana Morais1, Ivana Marenzi2, Deirdre Kantz3 1 3 University of Pavia, Centro Linguistico (Faculty of Medicine), 27100 Pavia, Italy dkantz@unipv.it Abstract. Blended solutions provide a means to orchestrate various types of activities and to schedule interactions at different times, nonetheless it is difficult to maintain a general overview of the class. In this paper, we build on the LearnWeb Design Framework to design and implement a Formative Assessment extension that supports the monitoring of the learning process in order to increase awareness and support reflection in a specific learning scenario. The extension offers a common basis for the various stakeholders (researchers, teachers and students) to collaboratively reflect on and design effective learning activities.
Keywords: Awareness, formative assessment, co-design, reflection, TEL.
1
Very often University courses entail large numbers of students which makes it
difficult to design and carry out learning tasks, as well as to assess learning outcomes [
The use of technology makes it possible to record the tracks of student activity and
to provide the teacher with dedicated analytics to improve awareness [
In this paper we propose a formative assessment strategy based on visualisation techniques to support teachers’ awareness and reflection in University learning contexts that integrate technology enhanced learning activities in the curriculum.
The LearnWeb Formative Assessment Extension as a Means
to Support Awareness and Reflection
LearnWeb1 is an online learning environment, which allows users to share and
collaboratively work on user-generated resources either uploaded from the desktop, or
collected from the web [
Fig. 1. Web2.0 features to the LearnWeb Design framework (multi-tier model)
Design of the LearnWeb Formative Assessment Extension In order to provide the teacher with evidence of each student’s (or group’s) involvement in the various knowledge processes when carrying out the learning tasks, we need to explicitly associate and display the logged data that corresponds to each activity (see Table 1). In this way, the teacher can refer back to the original framework and course design, and monitor the students’ performance in line with the expected learning goals.
In order to address the needs of different scenarios the LearnWeb Formative Assessment extension has been designed from three main perspectives: (i) a course perspective, where the teacher has an overview of a specific course and can make comparisons between/among classes, (ii) a class perspective, where the teacher can moni1 http://learnweb.l3s.uni-hannover.de tor and compare the activities of small groups within the same class, and (iii) a personal perspective where the teacher can visualise information about a specific user.
Logged activities Searching, Download, Open resource, Add resource, Delete resource, Create group, Group joining, Group leaving Tagging resource, Ratings resource, Edit resource, Comments, Deleting comments Text from comments visualised with WordWanderer 2
Grouping resources and presentation functionality Each phase on the user interface is located in a different tab so as to visualise the data relating to each phase in a specific context (see Fig. 2 (1)). The visualisation of each of these perspectives takes into account the activities carried out by students aggregated with four groups related to the four LearnWeb Design Framework features as described in Table 1 (see Fig. 2 (2)), thus enabling teachers to analyse the factors involved in the various learning tasks. A specific learning scenario is described in the following section so as to provide a preliminary evaluation of the teacher’s feedback concerning the usefulness of the visualisation of data to support teaching strategies and practice. For the implementation of the Formative Assessment extension, we carried out frequency analysis and built the charts using the PrimeFaces3 library.
In agreement with the teacher, we started with the visualisation of the data collected
in previous years so as to provide a diachronic overview for the teacher who will be
able to compare the performances of different courses and reflect on the course design
2 http://wordwanderer.org/
3 http://primefaces.org/
of future editions [
The LearnWeb Formative Assessment Extension in Practice During the Academic Year 2011-2012 we carried out a study at the University of Pavia, in Italy that involved 284 first-year medical students divided into five classes: GolgiA (85 students), GolgiB (71), GolgiC (44), HarveyD (50) and HarveyE (34).
The syllabus was based on English linguistics and focused on text-based studies of
(bio)-medical English. Students were expected to learn about multimodal theory and
how to carry out multimodal text analysis, that is, the study of printed, website, digital
and film texts in English and the ways in which these texts are used in different
medical and biomedical contexts [
At the time of the described scenario, the Formative Assessment Extension was not yet available, and the teacher had to explore the work done by students by browsing through the various groups. Using the current version of the system, it is now possible to obtain a rapid overview of student contributions with various levels of detail.
Fig. 3. Course perspective (Comparison between classes) – Search and Exploration (1) Course Perspective – the interface provides two search fields where the teacher can select two classes to be compared and choose different types of graphs to visualise the data. Fig. 3 presents a comparison between the HarveyE and the GolgiA class as regards the first pedagogical phase (Search and Exploration). Students in HarveyE searched and added more resources than the students in GolgiA. Since the project work task was the same for all groups (i.e. build a corpus of at least 50 websites), the teacher might want to intervene and invite the GolgiA groups to speed up the work.
100
The comparison is about the type of activities carried out by students in the two classes and can be between courses of the past, or courses functioning at the same time. In the first case (past courses), the teacher can see whether the current class is performing better or worse than the previous class. Consequently, the teacher can reflect on how to improve the course design or introduce better explanatory strategies for students by using notifications to communicate with them and give support. In the second case (current courses), the system visualises the actual value at a specific time so that the teacher is constantly up to date on how the students are working in the platform. For example, if one class is performing at a slower rate than another in the same course, the teacher can send a notification and a request to the group leader to speed up the work. This strategy can be useful when the teacher wants to stimulate competition.
(2) Class Perspective – the teacher can choose a specific class in the course and select two sub-groups to be compared. In the GolgiA class for example, the results of the comparison between the activities of the Euronics group and the Children`s genes group in the Search and Exploration phase are very similar. Whilst in the Annotation and Description phase (Fig. 4) we notice that a larger number of tags were added by the members of the Euronics group, the students in the Children`s genes group were more active in commenting and editing resources.
The results show how the two groups use a different strategy to conceptualise and categorise contents, thus helping the teacher to understand the learning behaviour and evaluate the group work accordingly. The teacher might decide to discuss the findings with students and reflect on their behaviour during the course or the final exam.
Fig. 4. Class perspective (group overview) – Annotation and Description (3) Personal Perspective – The window provides two search fields where the teacher can select a specific class and a specific student in that class. The resulting graph shows the number of activities carried out by the student throughout the course (see Fig. 5). This information can help the teacher to better understand the performance of each student and provide personalised feedback, for example according to: (i) the specific role (e.g. group leaders can/should carry out additional activities compared to the other group members), (ii) the given task (e.g. the teacher can encourage the slower students to carry their weight).
Fig. 5(a) shows an example of this analysis, where the teacher investigates the profile of a student (anonymized in the picture for privacy reasons). The student mainly searched and added resources; strangely no traces for Open resources are logged. This could mean that the student added materials to the group only relying on the title, without checking the content, and the teacher might want to investigate this student’s behaviour further. Fig. 5(b) shows the traces of the same user as regards the pedagogical phase of Annotation and Description. He mainly commented resources, but he also used ratings and tags to annotate resources. According to the tasks assigned in the course, the teacher can judge whether the student is behaving as expected, and decide to send him specific feedback or additional directions.
Fig. 5. Personal perspective (User Analysis) - (a) Search and Exploration, (b) Annotation and Description Such visualisations are useful to detect an increase or decrease in student participation in learning activities and can encourage the teacher to restructure some tasks or to adopt different pedagogical strategies if deemed necessary. Prompt intervention can be a crucial factor in determining the success or failure of a course. Using the Formative Assessment extension, the teacher is made aware of the dynamics that are taking place in the course and can speedily intervene in order to raise interest when it appears to be waning. As matter of fact, the teacher in Pavia evaluated the prototype of the system and confirmed its potential: “it has practical applications which can save time and allow for the constant realignment of the teaching strategies with the learning goals”. 3
Conclusions and Future Work The LearnWeb Formative Assessment extension is designed to offer a common basis for various stakeholders: for teachers to reflect on the teaching practices and refine their pedagogical strategies; for students to keep track of their personal progress and measure their performance in comparison with their peers; for researchers to realise what functionalities work better to support specific learning tasks and improve the system.
While some components of the technical approach are already available, others are under development. For the moment we focused on developing tools to support teacher awareness and to facilitate monitoring and mentoring activities. In the future, we will develop the assignment and recommendation component that includes the user interface for the learner. Another step will be the addition of a temporal dimension in order to give a diachronic visualisation of group interactions. The impact of the feedback provided through the proposed extension on student learning pathways will be investigated in future projects both in Italy and in Brazil. An extended study will be carried out with the aim of analysing how feedback is perceived by students and the impact it has on moulding the next learning stages. 4
106
●
109
110