In a fast-changing world, there is an increasingly felt need to bring what we teach and how we teach it into the 21st Century. Learning@Europe is an attempt in this direction: a shared online virtual world where students from different European countries meet to play and learn about European history. Chat-based discussions of study material, research homework to prepare in collaboration with remote peers on online forums, team games and a cultural competition are the main ingredients of this innovative experience, already tested by over 6000 high-school students and teachers from 18 European countries. This paper focuses on a particular Learning@Europe activity - chat-based cultural discussions about history - and analyzes the elements that are essential to its success. Basing on evaluation data and our 3-years experience, we describe strategies deal with the different elements to be taken into account: Technology; Content; Interaction Design; and - most important of all - Social Behavior.
In a fast-changing world, there is an increasingly felt need to bring what we teach and how we teach it into the 21st Century. The global economy needs people familiar with collaborative technologies, who know how to work in team also with partners in remote locations, proficiently write and speak in a second language, and be aware of the endless possibilities offered by Internet-based resources. How can our schools teach, develop, and assess these sorts of skills?
Learning@Europe, a collaborative edutainment experience based on 3D online virtual worlds, is an example of how collaborative technologies – and relevant educational activities involving their use – can be effectively integrated in school practices, offering students and teachers opportunities to develop a wide range of skills while at the same time learning in depth about curricular topics.
Learning@Europe has involved since school year 2004-05 over 5000 high-school students and 300 teachers from 18 European countries. While teachers’ feedback (measured through surveys) has been consistently positive over the 3 years, we are continuously finding ways to improve the overall experience and increase its educational impact. Designing or re-designing any part of the educational experience is an extremely complex process, in that several different aspects must taken into account: technology, contents, interaction design, organizational issues and social behaviour of participants. Information from users’ feedback and evaluation data must be taken into account, and it is essential to consider how changes to one specific activity may impact on the rest of the experience.
After shortly introducing the experience and a few data on participants and monitoring tools, we describe the different elements contributing to the impact of one particular activity of Learning@Europe: the chat-based cultural discussions. 1.1
Learning@Europe (L@E) is a joint project of Politecnico di Milano (Hypermedia Open center) and Accenture International Foundation, meant to foster in European youths a deeper understanding of their country’s and other countries’ history, and to promote through cross-cultural dialogue an increased awareness of their national and European identity.
L@E is structured upon a 6-8 weeks experience: four classes of high school students from different European countries take part in a cultural competition (2 classes against 2). Students meet together on a shared online 3D environment accessible over the Internet, for four times: four cooperative sessions distributed across two months, each session lasting about one hour. 2 students per class are represented in the 3D world by “avatars” (i.e. graphical human-shaped representations of users, see Fig. 1), whereas 2 more students per class access a parallel chat panel without 3D graphics (the “2D chat”), entirely devoted to score-awarding question-answering. Meetings are devoted to cooperative activities, such as quiz-based discussions and games.
Students meet their peers from different geographical areas, discuss with them about European modern history, and play with them in the virtual world, divided in two teams; all games are based on cultural riddles requiring an accurate knowledge of the subject matter: competition is a powerful motivator, adding excitement to the experience. Besides, students are encouraged to work together with their remote teammembers, strengthening ties with them. They interact via chat in the 3D world and asynchronously through online forums during the intervals between sessions.
Two online tutors from L@E staff (the Guide in the 3D world and the Helper in the 2D chat), also represented as avatars, coordinate each session, stimulate discussions with the help of “boards” (pop-up windows showing text and images, activated from hotspots in the 3D world), ask cultural quizzes, referee the games and assign scores.
The language of all interactions, interfaces, and study materials is English.
To achieve in-depth comprehension of the cultural themes proposed, further educational activities are performed in the intervals between a session in the virtual world and the following: students are asked to study a set of contents - in the format of interviews to international experts- and to prepare a homework in collaboration with their team members, by doing some research; for example, they may have to relate a significant historical issue to its present consequences and analyze the traces left in their local context (e.g. the influence of languages or religion in the formation of national states); comparing their research with the works of the other students, they shall gain an interesting picture of how differently the same process is regarded from different European perspectives.
Fig. 2 shows a schema of the virtual educational experience's structure: Session 1 introduces students to each other and to the cultural topic of the experience. They meet remote peers, show in turn pictures of their class and country (sent in advance for upload in the 3D), discuss the preliminary material (read before the session), and play the first game. This provides the motivation to study the materials to be discussed in Session 2, and to prepare the first collaborative team assignment.
In Session 2, students present their team work and discuss the history of the countries involved; they also play a game based on the contents (a Treasure Hunt). At the end they are assigned materials on a specific European issue for Session 3 and the title of the research homework to be prepared for Session 4.
In Session 3, a specific European issue is dealt with in depth, e.g. the role of religion or of languages in the formation of nation-states. Again, discussion and games take place. Afterwards students have another couple of weeks to complete the assignment (i.e. connecting a relevant European subject to their local context) researching on the Internet and in the libraries, interviewing locals and collaborating with remote team partners on the team forum. One week before the last session students submit their homework and read the works of the other schools.
Finally, in Session 4 students present their homework, comparing their ideas and discussing with foreign peers. The final scores are also announced, and the “crowning” of the winning team concludes the experience.
The following data are based on surveys to participants, described in detail in the next section.
54.9% of the teachers participating in L@E 2004-05 (N=51), 67% of those participating in L@E 2005-06 (N=200), and 58.4% in L@E 2006-07 (N=83) are at least 40 years old. 68-75% are women. Surprisingly, teachers of English as a second language are more numerous than the teachers of humanities (history, literature, economics).
20% of the teachers in L@E 2004-05 (N=50), 28.2% of those in L@E 2005-06 (N= 206) and 19.2% of those in L@E 2006-07 (N=83) use computers for professional or personal purposes less than 3 hours a week .
Students are aged between 14 and 19, with the majority between 15 and 17. Girls are slightly more than boys (61.7% females in the two most recent years).
45.2% of students in L@E 2004-05 (N=585) and 38.9% in L@E 2005-06 (N=1436) use computers less than 3 hours a week. Although over 70% of participants in L@E 2006-07 use computers almost every day, 59.1% do not use computers at school more than once a week. At home they use computers for listening to music (72-78%), playing games (60.5-69.1%), doing schoolwork (60-70.4%), writing email (56-70%). Only 24.7% of students write in forums, whereas 45-62.6% use chat systems.
In order to evaluate an edutainment experience involving multiple actors, tools,
and tasks, against a complex set of educational goals, a rich set of monitoring tools
was implemented to collect both quantitative and qualitative data from different
sources, allowing to triangulate findings from multiple sources [
A wide-spectrum set of data collection methods has been employed before, during and after the project in order to evaluate the overall experience, its educational impact, the “user satisfaction” and the organization's effectiveness: • Surveys to teachers and students before, during and after the experience (the surveys collected in L@E 2004-05, 2005-06, and 2006-07 cover in average 75-90% of teachers and 40-50% of students). • Focus groups and meta-surveys with teachers about the surveys results, held at the end of school years 2004-05, 2005-06. In the focus group, 12 teachers from 5 of the 6 countries participating in the first L@E year discussed the surveys results and provided insights about the most relevant outcomes; at the end of L@E 2005-06 all 160 teachers were sent the survey results: of the 101 who answered, all except 2 reported that results in their class either reflected survey outcomes or were better. • Sessions monitoring. Guide and Helper wrote a short debriefing and completed an online questionnaire immediately after every session. In addition, about 12 hours of online sessions of L@E 2004-05, more than 20 hours of L@E 2005-06 and most sessions of L@E 2006-07 were captured from the Guide's monitor and recorded. • Chat transcripts. The chat logs of all sessions since the first edition have been recorded, and those of L@E 2004-05 have been analyzed through coding techniques. • On-field observation. Observers went into some of the Italian school to videotape class interaction during seven sessions; tapes provide evidence of the students' engagement and enthusiasm. • Forum posts. All messages posted on the forum have been recorded in a database since the project began. The forum moderators wrote a weekly report about their forums’ activity. A detailed qualitative analysis of one forum was performed. • Student-produced artefacts. All the students' works have been collected. • Expert reviews. International leading experts of interactive educational technologies (Thomas C. Reeves, Michael Orey, University of Georgia) and of online communities (Jennifer Preece, University of Maryland) provided independent reviews of the results, assessing the quality of the educational format.
While qualitative data such as forum posts, chat logs and students' works are very
information-rich, analysis of such extensive amounts of data is a long and difficult
process. The aggregated data obtained through online surveys offered a first set of
results guiding our research: we then sought details and explanations for relevant,
surprising, or questionable findings through ad-hoc analysis of qualitative data [
Results over the 3 years are encouraging: 88-93.5% of teachers rated the global educational effectiveness of the experience “good”, “very good”, or “excellent”, and 90% of teachers reported improvements in their students’ understanding of history, technical skills, use of new learning methods, English, and group work. The majority of teachers rate their students’ improvements “good” or “very good”.
We shall now focus on one particular feature of Learning@Europe, especially meant to foster cross-cultural dialogue and understanding of history: the Cultural Discussions via chat.
Learning@Europe participants are expected to learn about the history of European countries. Contents are presented in the study materials, and also partly in the participants’ background knowledge about their own country: one of the reasons for putting in contact students from different European countries is to offer them a chance to learn from each other, by discussing, exchanging views, and interacting.
This is why a significant part of cooperative online sessions is dedicated to cultural discussions.
The cultural discussions have the following goals: • verifying the students' knowledge of the contents • stimulating exchange of opinions and multiple perspectives on the topic • encouraging deeper reflection and understanding of materials • (for discussions about assignments): offering a show-case of the students' works; encouraging them to share relevant background knowledge and reflections • offering a richer picture of an issue through multiple contributions from participants of different nationalities.
To achieve these goals, it is necessary to engage students, motivate them to prepare adequately and ensure their active participation. 2.2
This section illustrates how the above requirements were translated into design, and how evaluation results forced a reflection on the ways various design elements may impact on the user experience.
A substantial part of content-related learning benefits is achieved offline, while classes study materials and prepare the research homework. Cultural discussions stimulate students to study, help them focus on the most relevant concepts, reason on them, and learn from comparison with the others' views.
The Guide (in the 3D world) and the Helper (in the 2D chat) stimulate discussion and test learning by asking students a series of questions on the materials, at different levels of complexity (see also Tab. 2).
Cultural discussions generate engagement through the cultural competition: points are awarded for correct answers, active participation in the discussion, and good homework presentation.
Homework discussion explicitly encourages students to apply general historical concepts to their local context, compare their findings with foreign peers, and discuss them, analyzing different perspectives.
In general, cultural discussions both via chat and in the 3D world went well: they were rated “good” or “very good” by 70-80% of participants in L@E 2005-06 (N=109) and by 81% of teachers who have completed so far an experience in L@E 2006-07 (N=47). Also, the guides reported that in 78-80% of sessions most students were able to correctly answer questions (Fig. 4).
Fig. 3. Percentages of sessions in which the Guides reported elements of cultural quality (L@E 2004-05 and 2005-06).
In addition to learning about history, students also showed substantially improved motivation and significant attitude changes with respect to History (e.g. “Because of this program I like history”), technology as a cognitive tool (e.g. “It’s good idea to use computer in school”), English (“I feel the need to use English”), team work (e.g. “working in group you get better results”), their own country (e.g. “I am a bigger patriot than I thought”), other countries (e.g. “people are very friendly”) and Europe: “now I feel that I'm a part of European culture and history.”
However, the quality of cultural interaction could be improved. Fig. 5 shows a number of negative (or not very positive) aspects that occurred in a significant number of sessions: not all students were always engaged; some were not prepared; they rarely discussed spontaneously of the contents, and even more rarely contributed personal reflection, critical thinking, or relevant topics outside those explained in the materials.
What was responsible for limiting the depth of cultural discussions? What could be done to improve them? 3
Actors and design elements operate mainly in four areas, all interrelated: contents, technology, interaction design, and social behavior.
Let's analyze in which sense they influence the cultural discussion.
Difficulties of students in visualizing the boards1 may force the guide to paste the boards' content into the chat: this is an annoying, but manageable problem - except when it creates a disadvantage for the cultural competition (i.e. those who cannot see the boards have a smaller chance to be the first at answering).
Interface problems also have been observed: boards cover the chat, so that users cannot simultaneously read the board and keep an eye on the chat discussion, with the risk of missing bits of conversation.
However irritating, these issues can hardly compromise the success of a cultural discussion. The situation is different when, for example, nobody can visualize the boards; when one or more users have connection problems and keep on disappearing and reappearing in the discussion, interrupting it to ask what happened in the meantime; when one user experiencing a serious malfunctioning (e.g. being unable to access the 3D environment) keeps the guide busy in trying to fix the problem, to the detriment of the cultural discussion - which can hardly flourish without the guide's lead.
Unfortunately, modifying the interface is not sufficient to prevent this kind of problems. They are mainly related to client machines. Refining preliminary technical tests and troubleshooting strategies, and training guides to offer quick technical support for recurring problems, may help limiting the damage; however, these eventualities can not be completely eliminated.
The cultural discussion is centred on the contents at the core of the edutainment experience. Their quality, their level of complexity, their interest2, and the way they are presented are crucial for the success of related discussions.
The quality of contents is especially critical in educational contexts. With quality contents and little else, good teachers can still obtain satisfactory learning results. 11 Non--empty cache, pop-up blockers, and low bandwidth happened to cause this kind of problem. 2 Interest is intended in terms of perceived relevance in the eyes of the learners. Conversely, if teachers perceive the contents as inaccurate, imprecise, incomplete, irrelevant, and maybe also full of grammar mistakes, they will hardly motivate students to work on them, or see any value in related activities.
The difficulty level of contents is also crucial. Auxiliary materials3 and other resources may help supporting comprehension, especially when there may be significant differences in the background knowledge of participants.
The complexity level of testing materials is even more crucial. For example, if questions are too difficult for a particular group of students, they will feel frustrated for not knowing the answer, or try and guess without reasoning on the question. If questions are too easy, they will feel uninvolved, disappointed, and even underestimated.
The study and testing materials should be difficult enough to be challenging, but still interesting and understandable. Creating versions at different levels of complexity for different user groups is a possible strategy.
If questions are confusing and ambiguous in the way they are formulated (e.g. with unclear criteria to distinguish partially correct answers from correct and wrong ones), or if the answer cannot be found in the study materials, students feel cheated on, and protest vigorously.
If boards - which should serve as content-reminders, not as content-holders - present too long texts, nobody will read them through. The same if the content is hardly readable, or poorly structured, or unclear and unappealing.
The content structure and the way it is presented must be adequate to the practical situation of use.
For example, boards should contain very short texts, even cryptic ones - so that the guide can stimulate students to disambiguate them - and large pictures4, which can be viewed in colour on the computer screen. On the contrary, the study materials should be as clear and detailed as possible, since they are printed out and read on paper, and ought to have few or no pictures, given the poor quality of school printers and the cost of paper for teachers and of ink cartridges for schools.
Finally, certain topics are more likely than others to stimulate the students' interest. Sensational events, reference to topical issues, matters that have a perceivable impact on the students' life, will probably motivate more students to express their opinion than issues which appear totally abstract and unrelated to any present situation. The challenge is to find and highlight in every topic its relevance and its connections with the interests or the direct experience of the learners. 3 L@E provides a set of maps and chronologies of the most relevant historical characters, battles and events in the history of each country mentioned in the interviews. 4 Large in terms of dimensions, not of file size: they must not take long to download.
Interaction design can influence cultural discussions in terms of the time allotted to this activity within the session: while it is difficult to keep students actively engaged on cultural matters for too long a time, it is also difficult to reach the desired depth level of a discussion in a very short time.
How long should cultural discussions last, then? Time varies depending on several variables, including the students' preparation, their motivation and attention level, their age, the complexity and the interest of the topic. Constraints related to the length of a period in different European school schedules (45 to 60 minutes) also play a relevant role. As usual, all elements of design are tightly interrelated.
Since conditions may vary across participants, contents, and time, it is good practice to allow some degrees of flexibility in the way activities are structured, while providing limits and criteria so that goals are nonetheless achieved.
For example, in the session structure Guides can vary the length of discussions between 15 and 20 minutes depending on the time spent to complete previous activities, and choose among a large set of questions those most suitable for the level of the discussion and the direction it has taken - provided that they cover all the main session topics.
Moreover, the order of the cultural discussion with respect to other activities within the session may have an influence on its quality: it has been decided that discussions always take place before games, so that the excitement of the game does not hinder the students' attention for the cultural topics.
On the other hand, Guides often observed that students' expectations about the game scheduled to follow the discussion had a distracting influence. Only the fact that they could gain points by correctly answering questions motivated them to pay attention.
Even the place of the discussion within the entire structure of the experience is important: if students do not have sufficient time to read the study materials, or if they
Discussion aid Quick question Check question Full-answer question Board with image Board with quote Board question
with have mistaken expectations about the session, based on previous experience, this may compromise the success of the discussion. For example, guides reported that in the discussion of students' works in the final session, instead of presenting their works and commenting the other team's work as they were expected to do, students kept on asking “next question please”: they wanted the guide to continue the cultural competition as in the previous sessions, through a series of points-awarding questions.
The role planned for the Guide, the way to use the boards and move around the 3D world, and the types of questions, shape the way a discussion is conducted.
Observation of experience through time is the best source of information on how to refine the design of human-intensive interaction activities with unstructured goals (such as ``learning'', or ``improving critical thinking''), especially in unusual, innovative contexts.
At the time when the first cultural discussion in SEE5 took place, there was little research available on how to design educational cross-cultural discussions among high-school students of different nationalities in a shared 3D virtual environment. In time, through direct experience in the role of Guide and on-field observation, a number of strategies were devised and discussion-aids were designed in order to support the guide in this activity. 5 SEE – Shrine Educational Experience was the first collaborative experience based on virtual 3D environments by the HOC Laboratory of Politecnico di Milano, developed in 2001-02.
Since discussion is an intrinsically collaborative activity, its success greatly depends on the behavior of each actor involved.
Figure 10 shows the organizational aspects and issues related to social behavior that may affect the quality of a cultural discussion.
The students' level of preparation and their willingness to discuss are crucial: if teachers do not give the content materials to students, or if students did not study them, or if they do not listen to the guide and are not interested in discussing the cultural content as much as in chatting with each other, the discussion will hardly be a success.
Often, lack of discipline is a consequence of the teacher's absence. The teacher is responsible for maintaining discipline in the class6, since inappropriate behavior by one user may set a negative mood in all participants.
Students' willingness to collaborate is also not to be taken for granted: although collaboration with team partners (in the team whisper chat) is expressly required when answering questions in the 2D chat, students may choose not to collaborate: each class sends its own answer without consulting team mates. Cross-question strategies were tried in order to force collaboration. For example, the Helper asked a class in each team to provide a factual answer to the question, while the partner class had to comment on the answer. This did not always work. On one hand, it requires a number of organization activities which are awkward to perform via chat, with respect to face-to-face situations; on the other hand, students protested when they had to comment an answer they did not agree with, and seemed unable to try finding an 6 Following a few negative experiences, “maintaining discipline” is now explicitly stated among teacher's responsibilities in the agreement schools sign before joining a L@E experience. agreement. We are still looking for an effective strategy to make students discuss with team partners and produce a common answer to a question via chat.
Moreover, if participants have problems expressing themselves in English, interaction will be poor.
Also a guide with poor moderating skills, who bores or irritates the students rather than stimulating them, may have a very negative impact on the cultural discussion. On the other side, the guide's ability to start a lively conversation and keep the students involved in the topic and in the competition is critical for the success of the cultural discussion.
The way the guides are trained is also critical. Experienced guides manage discussions more easily. Motivated guides, well instructed about the goals of the experience, produce better results than confused guides, or guides who approach the session with a negative attitude.
The same is true for students: while a few unprepared, unmotivated, undisciplined students can make it impossible to have a satisfactory discussion, a few involved, prepared, and motivated students can bring invaluable contributions in terms of ideas, insights, and attitude.
Many are the elements influencing the success of a chat-based cultural discussion such as those taking place in Learning@Europe. Factors range from interface issues to the content structure, from the order of activities in a sequence to the willingness of participants to contribute actively to the discussion.
This paper describes the factors that, through experience, we have found most relevant in determining the success or failure of cultural discussions. Through time, we have improved contents, adjusted design elements, defined strategies and refined procedures to deal with many of the technical, interactivity and organizational problems that may occur during a session.
However, issues related to human and social behaviour can only in part be controlled. We can provide clear instructions, incentives, deterrents, motivate students and teachers to behave according to plans. However, nothing can force learners to be interested and learn, if they choose not to; at the same time, the best design, contents, technological platform and interaction sequence are nothing but tools, which talented teachers and motivated students can transform in the most exciting learning experience, far above the designers’ expectations. The human factor is at the same time the greatest liability and the greatest asset of a learning activity.