The study presents the initial stage of the development of a scale for measuring the factors that affect and shape the learning experience when playing serious games. A comprehensive literature review revealed a number of studies on educational games. A draft questionnaire was developed on the basis of their results.
Perceived realism. Although a game's realism is the sum of many technical details, it can also be viewed as a subjective feature, because individuals perceive it differently. Studies indicated that realism plays an important role acting as a facilitator of the learning experience (Dalgarno & Lee 2010; Harrington, 2012).
Perceived ease of use. Perceived ease of use refers to the degree to which a person believes that the use of a tool is free of effort (Davis, Bagozzi, & Warshaw, 1989) and it is included in many studies regarding diverse ICT tools (e.g., Lee, Wong, & Fung, 2010).
Usability. The usability of any system has to be specified considering who are the intended users, the tasks they will perform, and in the context the system in which it will be used (Brooke, 1996).
Perceived audio and visual adequacy.
Comprehensiveness of the learning material Perceived narratives' adequacy. The narrative is the story that is told by the game and acted out by the player. Thus, the narrative provides the player a background, motivating him/her to become emotionally involved (Sweetser, 2006). Perceived feedback's adequacy. Feedback allows the player to determine the gap between the current knowledge level and the knowledge required for completing the task (Jsselsteijn, De Kort, Poels, Jurgelionis, & Bellotti, 2007). The feedback can be direct or indirect (e.g., score, health, and the player's status).
Perceived goals' clarity. Clear goals (overall and intermediate), as well as other factors such as feedback and the player's control over the game, allows the player to concentrate on the game (IJsselsteijn et al., 2007).
Perceived adequacy of the learning material.
Immersion. In short, immersion can be defined as the feeling of submersion in a fictional environment (Qin, Rau, & Salvendy, 2009). Moreover, there is a tendency to use immersion as a synonym of engagement (e.g., Jennett et al., 2008), or presence (e.g., Ermi & Mäyrä, 2005). In this study, we embrace the view of Jennett and colleagues (2008) who asserted that in evaluating the gaming experience, measuring immersion is more important than presence.
Enjoyment. Enjoyment arises when one's capabilities are equal to the opportunities for action that this person is given during the process of performing an intrinsically rewarding activity (Csikszentmihalyi, 1988; Moneta & Csikszentmihalyi, 1996). Enjoyment is considered a key factor when designing and evaluating a game's effectiveness (Sweetser & Wyeth, 2005). Studies have shown that enjoyment and, in general, positive feelings towards a game or application contribute to knowledge acquisition (Faiola, Newlon, Pfaff, & Smyslova, 2013).
Competence. The cognitive evaluation theory (CET) (Ryan & Deci, 2000), proposes that events and conditions that enhance a person’s sense of autonomy and competence support intrinsic motivation. Competence is defined as the feeling of being effective in overcoming challenges (Deci, 1975). Thus, it can be enhanced by opportunities to acquire new skills, or by providing positive feedback (Ryan, Rigby, Przybylski, 2006).
Incentives for learning Motivation. Motivation is sensitive to the way the content is presented and it is inherently volatile (Linnenbrink & Pintrich, 2002). Research has indicated that motivation and the learning outcomes are closely connected (Benbunan-Fich & Hiltz, 2003; McLellan 2004).
Relevance to personal interests. Engagement refers to a player’s level of involvement in a game (Brockmyer, Fox, Curtiss, McBroom, Burkhart, & Pidruzny, 2009). One of the key factors, together with attention and immersion (already included as a factor), that can explain engagement, is relevance to personal interests (Chen, Kolko, Cuddihy, & Medina, 2011). Thus, in this study relevance to personal interests refers to the extent one found the learning material interesting and closely related to his/her interests so that he/she got engaged.
Perceived usefulness. The usefulness of an ICT tool refers to the extent to which a person believes that by using this tool his/her performance will be enhanced. As a result, the person is motivated and the course of the learning process is positively affec ted (Sun, Tsai, Finger, Chen, & Yeh, 2008).
Impact on knowledge acquisition Perceived knowledge improvement. Educational and serious games try to convey knowledge through the gaming experience, therefore their effectiveness of raising knowledge is an important criterion in their evaluation (Fu, Wu, & Ho, 2007). Questions coming from the above studies contributed to the development of a questionnaire which included seventy-six items examining a total of sixteen factors, presented in a five-point Likert-type scale (worded "Strongly Agree", "Agree", "Neutral", "Disagree" and "Strongly Disagree"). We also decided to include seventeen open-ended questions. Scores were to be obtained by allocating numerical values to responses: "Strongly Agree" scored 5, "Agree" scored 4; "Neutral" scored 3; "Disagree" scored 2 and "Strongly Disagree" scored 1. It has to be noted that some of the original questions were slightly rephrased in order to fit the study's needs. The questionnaire was translated into Greek and completed through google docs. The questionnaire's factors are presented in Table 1, while the questions are presented in the Appendix.
The study's target group was university students studying at the Department of Audio and Visual Arts (AVARTS), Ionian University in Corfu, Greece and at the Department of Primary Education (PRIMEDU), University of the Aegean in Rhodes, Greece. Students coming from the AVARTS department are specialized in ICT, while students from the PRIMEDU are specialized in education. A total of 110 students participated in the study. They were asked to play Triseum's Variant: Limits and ARTé: Mecenas. Triseum is a company that grew out of the LIVE Lab at Texas A&M University. Variant: Limits deals with calculus at high school and college level. ARTé: Mecenas deals with Art history and Art Appreciation. Students played both games for at least an hour (after finishing the tutorial). Immediately after playing each game, they completed the questionnaire (twice if they played both applications). The study was conducted in two phases (1st phase: Fall 2017 and 2nd phase: Spring 2018). At the 1st phase, students played and evaluated the games at their homes and at the 2nd phase they did the same in a lab. This procedure was followed on purpose, as one of our further research objectives is to discover if there are any differences due to the playing conditions (free or controlled). All in all, it was found that both applications scored low in immersion, perceived realism, motivation, and perceived relevance to personal interests (Table 2). On the other hand, both applications received high scores in perceived audiovisual adequacy, perceived feedback's adequacy, and perceived usability. These results provided a basic understanding on which factors are the influential ones when individuals play serious games. They also provided some ideas on how these factors interplay with each other.
Perceived sound effects' adequacy (max = 20) Perceived visual effects' adequacy (max = 15) Perceived feedback's adequacy (max = 20) Perceived usability (max = 30) Perceived ease of use (max = 30) Perceived goals' clarity (max = 20) Enjoyment (max = 30) Perceived usefulness (max = 20) Perceived knowledge improvement (max = 25) Perceived narrative's adequacy (max = 25) Perceived competence (max = 20) Perceived adequacy of the learning material (max = 25) Immersion (max = 30) Perceived realism (max = 20) Motivation (max = 30) Perceived relevance to personal interests (max = 20)
For examining the user's experience when playing ARTeMecenas and VariantLimits a questionnaire was developed in which sixteen factors were included belonging to five categories. Generally speaking, ARTeMecenas received more positive views than VariantLimits. That is because ARTeMecenas had low scores in four factors, while VariantLimits had in nine. Moreover, ARTeMecenas had medium scores in five factors, while VariantLimits had in three. Finally, ARTeMecenas had high scores in seven factors, while VariantLimits had in four. Both applications scored low in immersion, perceived realism, motivation, and perceived relevance to personal interests. It is almost certain that both groups of students had different personal interests than what the learning material was about in both applications. Thus, it is easy to explain the low scores in this factor. On the other hand, both applications failed to motivate students and immerse them to the games while the applications' realism was also rated low. We support the view that these three factors are closely connected. That is because an application's realism contributes to immersion, which, in turn, has an impact on motivation. Both applications had high scores in perceived audiovisual adequacy, perceived feedback's adequacy, and perceived usability. These factors reflect either the applications' technical features or their comprehensiveness regarding the learning material. In this respect, these results are encouraging indicating well-designed applications.
The most commonly used factors in studies which examined the users’ learning experience when playing serious games were used for the development of a draft questionnaire. Though the sample was rather small and, consequently, it is not safe to draw de finite conclusions, nevertheless, the data analysis provided some useful insights regarding the factors that come into play. The next step is to record students' views on the seventeen open-ended questions in order to form the final questionnaire. A further statistical analysis will reveal which of the questions should remain in the questionnaire as the seventy-six questions are considered a fairly large number and, probably some questions are redundant.
I was deeply concentrated in the application If someone was talking to me, I couldn’t hear him I forgot about time passing while using the application I cannot tell that I was getting tired while using the application I feel detached from the outside world while using the application When I used the application, the virtual world was more real than the real world I think the application was fun I felt bored while using the application* I enjoy using the application I really enjoyed studying with this application It felt good to successfully complete the tasks in this application I felt frustrated* I feel that this application can ease the way I learn Applications are a much easier way to learn compared to the usual teaching Why use this application? There are easier ways to learn what I want to learn* The application can make learning more interesting I felt that the application increased my knowledge I felt that I caught the basic ideas of the knowledge taught I will definitely try to apply the knowledge I learned with this application There were explanations and examples of how to use the knowledge in this application The content and the presentation of the learning material in this application conveyed the impression that its content is worth knowing The visual display quality of the application distracted me from doing other things When interacting with the virtual objects, these interactions seemed like real There were times when the virtual objects seemed to be as real as the real ones The virtual objects seemed like the real objects to me I was captivated by the application's story from the beginning I enjoyed the fantasy or story provided by the application I was emotionally moved by the events in the application I could clearly understand the application's story I was very interested in seeing how the events in the application will progress I enjoyed the sound effects in the application I think the application’s audio fits the mood or style of the application I felt the application’s audio (e.g., sound effects, music) enhances my gaming experience I enjoyed the music in the application I enjoyed the application's graphics I think the application is visually appealing I think the graphics of the application fit the mood or style of the application Overall the application's goals were presented in the beginning of the application Overall the application's goals were presented clearly Intermediate goals were presented in the beginning of each scene I always knew how to achieve my goals/objectives in the application I received immediate feedback on my actions I was notified of new tasks immediately I received information on my success (or failure) of intermediate goals immediately I felt the application provides me the necessary information to accomplish a goal within the application I think it is easy to learn how to use the application I found the application unnecessarily complex* I would imagine that most people would learn to use this application very quickly I needed to learn a lot of things before I could get going with this application* I felt that I needed help from someone else to use the application because it was not easy for me to use it* It was easy for me to become skillful at using the application I found the controls of the application to be straightforward I found the application's interface to be easy to navigate I do not need to go through a lengthy tutorial or read a manual to use the application I found the application's menus to be user friendly I feel the application trained me well in all of its controls I think the information provided in the application (e.g., onscreen messages, help) is clear I felt skillful
I felt competent
I felt successful
I felt a sense of control over the application Perceived ade- The learning material was more difficult to understand than I would like for it to be* quacy of the In some cases, there was so much information that it was hard to remember the important points* learning material The exercises in this application were too difficult*
I could not really understand quite a bit of the material in this application*
The good organization of the content helped me to be confident that I would learn this material Motivation This application did not hold my attention*
The application was interesting and got my attention When using the application, I did not have the impulse to learn more about the learning subject* The application did not motivate me to learn* This application had things that stimulated my curiosity
The amount of repetition in this application caused me to get bored* Perceived rele- The content of this material was relevant to my interests vance to personal The application's content was not relevant to my needs because I already knew most of it* interests I could relate the content of this application to things I have seen, done, or thought about in my own life
It is clear to me how the content of the application is related to things I already know Note. * = Item for which its scoring was reversed