Game@School. Teaching through Gaming and Mobile-based Tutoring Systems Annalisa Terracina Sapienza Università di Roma, Italy Abstract. Keywords: Educational Games, Virtual Learning Environ- ment, Intelligent Pedagogical Agent 1 Introduction and motivation Over the last ten years, the way in which education and training is delivered has changed considerably with the advent of new technologies. The term “digital native” coined by [7] in 2001, perfectly describes the nowadays students that continuously deal with technology. Thus, technology should be a prominent part of the learning process and should be intended as a support for teachers and learners. One new methodology relying on present advanced technologies that holds considerable promise for helping to engage learners is Games-Based Learn- ing (GBL). Games are well suited for developing students’ scenario competence, which can be defined as the ability to imagine, enact and reflect upon game- specific choices and their consequences, as Hangoj does in [2]. Unfortunately, gaming and schooling have developed into two distinct “knowledge traditions” that often rely on opposing validity criteria for determining what counts and what does not count as relevant knowledge. To avoid that dichotomy, Game Based Learning should integrate different aspects that are related to the knowl- edge itself, to pedagogical aspects, to scenario-based and every day practise. Following [3] we adopted the idea that there is a complex translation involved in using games for educational purposes and thus we can see Game Based Learning as a dynamic interplay of four knowledge practice, as depicted in 1. The model is descriptive and assumes no a priori hierarchy between these four forms of knowledge when teaching and playing games. However, in practice teachers and students tend to value certain forms of knowledge, especially disciplinary and school-only knowledge forms, as being more valid or serious than other forms of knowledge. Actually, there are many efforts in the field of Serious Games, discussing the potential of games to provide new methods for supporting learning. At this stage of the current research we can say that, respect to the past: – the importance and meaning of pedagogical foundation in educational design is more clear to researchers [4] – the infrastructure of schools has developed a lot during the last decade – we are moving toward a new generation of educational use of games [6] 46 Fig. 1. Game based learning as an interplay of knowledge practice In the described scenario has maturated the idea to develop a game in order to teach STEM (Science, Technology, Engineering and Mathematics) subjects, in particular physics. In this context, we followed the idea to integrate the most up-to-date technologies in new teaching trends, namely Virtual Learning En- vironments (VLEs, [10]) and Intelligent Pedagogical Agent (IPAs), as deeply investigated in [18]. In particular, the developed Serious Game offers: – A 3D virtual environment that the learner can explore and interact with. The student can collect objects, solve quizzes and setup experiments, all related to the subjects of study. – An Intelligent Pedagogical Agent (IPA) that guides the learner throughout the game. The learners can ask questions to the IPA and in such way, per- sonalize the learning path. In fact, students can express in free text and ask whatever they want to the IPA, that will provide the best answer to the posed question. – The IPA is emotionally intelligent, it performs a sentiment analysis on the student sentence and assigns a tag (positive, negative or neutral) to the sentence. Basing on the received tag the IPA behaves in different ways to establish an emotionally safe relationship. – A shared knowledge forum in which students can share and discuss notes about the learning objectives of the game. – An assessment tool in which the teacher can follow students engagement and involvement during the game. 2 State of the art The work in [9] presents a systematic literature review of empirical evidence about the positive impacts and outcomes of games. Using the same search terms used in that work, on a smaller number of databases,, the current update to the systematic review of Boyle found many more papers reporting empirical evidence of the positive outcomes of playing games (512) than the previous review (129). This illustrates the increased interest in the positive impacts that digital games 47 had during the five-year period from 2009 to 2014 compared to the previous five-year period. Regarding emotional learning, [1] presents an approach to modelling the user affect that is designed to assess a variety of emotional states during interactions. Knowing the details of a users emotional reaction can enhance a system capa- bility to interact with the user effectively. Instead of reducing the uncertainty in emotion recognition by constraining the task and the granularity of the model, the proposed approach explicitly encodes and processes this uncertainty by re- lying on probabilistic reasoning. The authors discuss their model in the context of the interaction with pedagogical agents designed to improve the effectiveness of computer-based educational games. [22] presents a pedagogical agent capable of active affective support, guided by the logic that integrates the learners cognitive and affective states. The au- thors developed an algorithm for feature tracking, which utilizes a combination of common image processing techniques, such as thresholding, integral projections, contour-tracing and Haar object classification. The experimental results show a range of preferences associated with pedagogical agents and affective communi- cation. According to the authors, affective interaction is individually driven, and they suggest that in task-oriented environments affective communication carries less importance for certain learners. [19], presents a system that embodies the idea of virtual humans that act and interact like humans, bringing social elements in the interaction. A couple of twins that are virtual teachers in the Museum of Science in Boston is de- signed to engage visitors and raise their awareness and knowledge of science. The twins have some aspects that were built in advance, and some that oper- ate in real time as the user interacts with them. The aspects built in advance include the character bodies, animations, textual content, and spoken output. The speech recognition, natural language understanding, and dialogue manage- ment decisions of what to say are computed in real time, as is the scheduling and rendering of spoken and gestural outputs. Speech recognition, natural lan- guage understanding, and dialogue policies also make use of knowledge sources constructed in advance, using supervised machine learning. In [5], the system allows the player to express himself in natural language. The system processes users’ input sentences and returns the best answer among a set of possible stored answers. The communication is implemented through an NLP (Natural Language Processing) algorithm based on an ad hoc text retrieval problem solver and on a Naive Bayes text classifier with an inner product-based threshold criterion. The algorithm implemented in the system is a variation of a text retrieval algorithm. The system has been used in a quest game in which the player can ask questions in free text to the Non Player Character to arrive to the solution. We start from the result found by Mori to implement our dialogue system. 3 Research questions Previous studies and literature appraisals suggest that educational games [9] impact on learning as well as other behavioural and affective outcomes, but also 48 point to the lack of empirical evidence to support several claims. Contrasting opinions suggest that gamification fosters shallow learning. But ultimately the question of retention and how gamification supports or undermines long-term learning still remains open and further enquiry is needed [8]. That considerations lead to some open issues, like: RQ1: Does Educational Game support learning? RQ2: Does Educational Game undermine long-term learning? RQ3: Does the use of Intelligent Pedagogical Agents foster the idea of per- sonalized mentors? In order to give my contribution to the above mentioned research questions, the aim of my research is to develop a brand-new educational game and to experiment it in the context of formal learning. 4 Advancement The boundary of the research described so far are quite wide and cover different fields of research: – Pedagogy and psychology issues • emotion and learning • gaming and learning • role-play game • classroom practice – Artificial Intelligent and Tutoring System • Intelligent Pedagogical Agent – Virtual Learning Environment • 3D Virtual World • Teacher’s workplace • Game engine – Game Based Learning • Educational game The aim of that research is not to give an original contribution on each of the listed research fields but, mainly, to integrate all the aspects in an original and not yet existing educational game. First of all, as stated by [18], currently it does not exist a Serious Game that integrates Virtual Learning Environment and Intelligent Pedagogical Agent and thus this is a very innovative point of this research project. Secondly, the effort in designing e-learning scenario or se- rious game are more focused on kids or at University level, as it can be inferred, for example, reading the proceedings of the 8th European Conference on Game Based Learning. The high school domain is quite never considered while there is a big interest in teaching STEM subjects in a different way also in high school. Thirdly, the majority of the e-learning systems are intended for distance teach- ing or used una tantum (like game) while the aim of the project is to change the classroom practice using the developed game as a teaching tool like it could be an e-book. Finally the majority of the existing Serious Game are not spe- cially designed but are re-adaptation of existing games, or even worst, commer- cial games are used to teach STEM subjects, i.e. teaching physics with portal 49 2 (http://www.thinkwithportals.com/). Thus, we can say that the outcome of the project will be a Serious Game that integrates new teaching trends and paradigms making use of the most advanced technologies. From an engineering point of view the challenges are: 1. System architecture design 2. Design and development of a Virtual Learning Environment 3. Design and development of an Intelligent Pedagogical Agent 4. Integration of VLE and IPA 5. Natural Language Processing 6. Emotion recognition From a pedagogy point of view the challenges are: 1. Integrate emotional learning in the game practice 2. Adopt a role-playing game as a model of collaboration and cooperation 3. Change classroom practice One of the game component, the IPA, has been tested in three different schools between December 2015 and February 2016, involving roughly 50 stu- dents. In order to evaluate the use of the IPA, we developed a stand alone Android application that make use of the IPA as interlocutor while studying specific subjects. The app came with two different learning paths developed by field experts and selected by teachers as possible insight. We decided to test the IPA, before the entire game was finished, for several reasons. First of all the IPA is an important component of the game and we wished to have feedback on its usage and liking before finishing the game. Secondly the AI algorithm behind the IPA needed to have a real validation to fine tuning its threshold. Finally the IPA was easy to integrate in a stand-alone application. In this context we tested appreciation, usability and learnability of the Android app in three exper- imental sessions. The first run of test validation occurred the 19 November 2015 (11 students, all males, aged 17-18), the second run occurred the 20 of January 2016 (16 students, two female, aged 17-18). After a preliminary evaluation of the first two test runs, we refined the AI algorithm behind the IPA. The third run occurred the 8 February 2016 (13 students, 8 males and 5 females, aged 17-18). In all three runs, students were able to play with the application roughly 40 minutes (a bit less of a standard lesson time); we just asked them to play with the application asking question to the IPA about the subject of study. After that they compiled a questionnaire that has been structured in three main sections: liking, usability and learnability. Appreciation. The first set of questions aimed at understanding if students like the idea of a personalized learning path and the possibility to ask free questions guided by a virtual agent: 75 % of students like this possibility. The aspect of the empathetic guide was felt as well: 68% of students liked the idea to have an empathic guide and would like to chat also about personal matters. 68% of students enjoyed the app and would recommend it to others, even if just 56% of them perceived it as a game. 50% declared the app is more effective for STEM subject while the other 50% per cent said both STEM and humanistic. 50 Usability. It emerged that 68% finds the interface and the touch screen function clear. Just 50 % of the students used the side menu and the majority of them declare they wish to add other functionalities. Most of the interviewed disliked the scientist avatar (70%) while liked the young girl (72%). Learnability. Tests revealed that 78% of students thought avatar explanations are clear and declared that provided answers are in line with asked questions. 64% declared tips were useful. We asked how many questions they asked to the avatar before quitting the game: 46% gave up after few questions, 34% ended the game after roughly 10 questions, 15% around 20 and just 1 student declared to ask more then 20 questions. Those results were obtained in the first two runs. That is mainly because very often pertinent questions received no answers due to the very high threshold settings of the AI algorithm. However, after AI algorithm threshold calibration, in the third test run we obtained that 38% ended the game after roughly 20 questions while 30% gave up after few questions. So we had an increment of roughly 20% in content exploration. The game will be fully tested in November 2016 with 100 students, however the first test runs have already proved that the developed Serious Game is engag- ing for students and that the IPA can really help students in their personalized learning path. References 1. C. Conati. Probabilistic assessment of users emotions in educational games. Jour- nal of Applied Artificial Intelligence, 2002. 2. H. Ketamo, K. Kiili, S. Arnab and I. Dunwell Integrating games into the classroom: towards new teachership. New pedagogical approach in Game Enhanced Learning: Curriculum Integration, 2013 3. T. Hanghj Game-Based Teaching: Practices, Roles, and Pedagogies. New pedagog- ical approach in Game Enhanced Learning: Curriculum Integration., 2013 4. I. Dunwell, S. De Freitas, S. Jarvis Digital Games and Learning., 2011 5. D. Mori, R. Berta, A. De Gloria, V. Fiore, and L. Magnani An Easy to Author Dialogue Management System for Serious Games. J. Comput. Cult. Herit., 2013 6. S. Egenfeldt-Nielsen Third generation educational use of computer games. Journal of Educational Media and Hypermedia, 2007 7. M. Prensky. Digital native, Digital Immigrant On the Horizon, 2001. 8. L. de Marcos, E. Garcia-Lopez, A. Garcia-Cabot. On the effectiveness of game-like and social approaches in learning: Comparing educational gaming, gamification & social networking. Computer & Education, 2016. 9. E. Boyle,T. Hainey,T.M. Connolly,G. Gray,J. Earp,M. Ott,T. Lim,M. Ninaus, C. Ribeiro and J. Pereira. An update to the systematic literature review of empir- ical evidence of the impacts and outcomes of computer games and serious games Computer & Education, 2016. 10. P. Dillenbourg, D. Schneider, and P. Synteta. Virtual learning environments. In 3rd Hellenic Conference Information & Communication Technologies in Education, 2002. 11. H. Gardner. Frames of mind: The theory of multiple intelligences. 12. H. Gardner. Intelligence reframed: Multiple intelligences for the 21st century. 13. H. Ketamo, K. Kiili, S. Arnab, and I. Dunwell. Integrating games into the class- room: towards new teachership. 51 14. M. Lease. Natural language processing for information retrieval: the time is ripe (again). In Proc. ACM PhD Workshop in CIKM, 2007. 15. C. D. Manning, P. Raghavan, and H. Schtze. Introduction to Information Retrieval. Cambridge University Press, 2008. 16. D. Mori, R. Berta, A. De Gloria, V. Fiore, and L. Magnani. An easy to author dialogue management system for serious games. J. Comput. Cult. Herit., 6(2), 2013. 17. B. Scheucher, P. H. Bailey, C. Gütl, and J. V. Harward. Collaborative virtual 3d environment for internet-accessible physics experiments. iJOE, 5(S1):65–71, 2009. 18. M. Soliman and C. Guetl. Intelligent pedagogical agents in immersive virtual learning environments: A review. In Proc. MIPRO 2010, IEEE. 19. W. Swartout, R. Artstein, E. Forbell, S. Foutz, H. C. Lane, B. Lange, J. F. Morie, A. S. Rizzo, and D. Traum. Virtual humans for learning. AI Magazine, 2013. 20. A. Terracina and M. Mecella. Building an emotional ipa through empirical design with high-school students. In 9th European Conference on Game Based Learning. 21. B. P. Woolf, H. C. Lane, V. K. Chaudhri, and J. L. Kolodner. AI Grand Challenges for Education. AI Magazine, June 2013. 22. K. Zakharov, A. Mitrovic, and L. Johnston. Towards emotionally-intelligent ped- agogical agents. In Intelligent Tutoring Systems, 2008. 52