Untitled Bee Game: Be(e)ing mean to learn more about eco- sustainability Federico Bonetti1, Simone Bassanelli1,2, Antonio Bucchiarone1, Federica Gini1,2 and Annapaola Marconi1 1 Fondazione Bruno Kessler, Povo (TN), Italy 2 University of Trento, Trento, Italy Abstract The contemporary era witnesses a concerning rise in unsustainable practices, particularly evident in the staggering accumulation of plastic pollution and the detrimental impacts of non-eco-friendly elements on soil and aquatic ecosystems. These human-induced actions contribute to biodiversity decline, especially affecting crucial pollinators like wild bees, essential for sustaining the planet's life support systems. This paper explores the development and evaluation of a 3D serious game, named "Untitled Bee Game", to foster positive behavioral change toward eco-sustainability, by delivering knowledge in an engaging and enjoyable manner. The paper presents the design and development of the game, the results of a usability test, and an experiment conducted to evaluate it. While the initial usability test yielded suboptimal results, valuable feedback informed subsequent improvements, resulting in an overall enjoyable experience for users. The main experiment demonstrated the game's effectiveness in facilitating effective learning, with encouraging quiz results indicating increased awareness of eco-sustainability concepts among players. Participants also expressed a willingness to continue learning about eco-sustainability. Keywords Serious game, learning, gameful system, behavior change, eco-sustainability1 contributing to the decline of some species, especially 1. Introduction pollinators, such as wild bees [9]. Bees play a pivotal role in delivering a spectrum of ecosystem services that not In the contemporary era, the pivotal factor influencing only enhance human well-being but also sustain the the rise and persistence of environmental challenges is fundamental life support systems of our planet [10]. human behavior [1]. In fact, human action is making great These ecosystem services are inherently aligned with the changes within the ecosystem and the climate through pursuit of global sustainable development goals [11]. This numerous unsustainable behaviors [2]. underscores the necessity for a fundamental Every year, about 25% of the plastic we produce, transformation in people’s conduct [1]. It is imperative to which equates to a staggering 80.5 million tons, ends up replace practices such as consumerism, wasteful in our environment, making plastic a pervasive resource utilization, environmental degradation, environmental pollutant [3, 4]. Moreover, the use of non- contamination, selfish and unjust behaviors with eco-friendly elements, such as most detergents, causes sustainable habits, using the world’s resources in ways enormous problems, including soil structure that will allow human beings to continue to exist on Earth deterioration, with detrimental consequences for plant with an adequate quality of life [1]. With the continuous well-being [5], and eutrophication, which results from the upgrading of educational technology, gameful systems, discharge of detergents into water bodies, leading to known for being highly motivating, have often been depletion of dissolved oxygen levels in the water, which implemented to promote behavior change approaches or can be harmful to aquatic organisms such as fish, to support positive behaviors [12, 13] in different seaweed, corals and other aquatic life. domains, such as transportation and mobility [14, 15], Human action —e.g. urbanization, light pollution, health, well-being and physical exercise [16, 17], eco- chemical pollution, plastic pollution, deforestation, noise sustainability awareness and pro-environment behaviors pollution, and air pollution [6, 7, 8]— is therefore [18, 13], culture and tourism [19], and so forth. Overall, producing a decrease in terms of biodiversity, gameful systems present a reliable approach to 8th International GamiFIN Conference 2024 (GamiFIN 2024), April 2-5, 2024, Ruka, Finland. fbonetti@fbk.eu (F. Bonetti); sbassanelli@fbk.eu (S. Bassanelli); bucchiarone@fbk.eu (A. Bucchiarone); fgini@fbk.eu (F. Gini); marconi@fbk.eu (A. Marconi) 0000-0002-1884-6798 (F. Bonetti); 0000-0001-6061-8169 (S. Bassanelli); 0000-0003-1154-1382 (A. Bucchiarone); 0000-0003-3427- 3747 (F. Gini); 0000-0001-8699-7777 (A. Marconi) © 2024 Copyright for this paper by its authors. The use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings 159 enhancing and encouraging users to engage in the In some commercial video games, the ecological desired behavior, potentially fostering environmental component is pivotal and the educational purpose is part protection. In fact, some investigations concentrating on of the design itself. In this case, however, the goal of the connection between video games and ecological teaching something to players may also result in a less knowledge have demonstrated noteworthy quantitative playful system [23, 24] and a weaker flow experience outcomes. In particular, there is a notable increase in [25]. In recent years, numerous video games have been players’ motivation to grasp ecological concepts through developed with this purpose in mind —such as Eco8 [24], video games [20]. Plasticity9 [26, 27], and Bee Simulator10. These games, We therefore developed a serious game called which thus have some characteristics of serious games Untitled Bee Game2, in which players have the goal of while remaining commercial, present a more complicated discouraging non-player characters (NPCs) from engaging design procedure. Indeed, designing video games with a in specific non-eco-sustainable behaviors while learning well-defined purpose seems to be rather challenging. The notions related to pollution through dialogue with priority given in the development to the eco-sustainable flowers. The long-term goal of the game is to foster awareness component could lead these games to lack the positive behavioral change in terms of eco-sustainability, playful impact and the flow experience [25] that users by providing knowledge in an engaging and enjoyable often seek in commercial video games [23, 24]. manner. To assess the game’s usefulness and qualities, we formulated the following research questions: RQ1. To what extent was the game perceived by the 2.2. Pro-environment serious games players as playful and meaningful? RQ2. How would learning and perceived learning be The use of serious games has proven to be a useful affected by this game? element in promoting environmentally sustainable RQ3. What design directions should be taken in behaviors [28], and climate change awareness [29]. developing and improving such a game? In this paper, we Madani et al. [30] compiled a survey of serious games present the design, development, and evaluation related (both digital and physical) used in the context of tackling to the use of Untitled Bee Game. environmental challenges and awareness raising on eco- In Section 2 we present a brief overview of video sustainability. The authors report that the use of serious games and serious games related to raising awareness of games can increase motivation and engagement, with environmental issues; then, we present the design and beneficial effects on desired outcomes (i.e., development of Untitled Bee Game. In Section 3, we environmental awareness); however, the effectiveness of describe the design and development procedure to reach games is not universal for educational purposes, it the final version of the game. In Sections 4 and 5, we depends on the design and components of each game present the questionnaires and the methods for both and, for this reason, it is difficult to assess the accuracy of usability and experimental analyses, and then the results. the findings. An interesting example of a serious game is In Section 6, we discuss the findings after the analysis. Energy Chickens [31]. In the game, virtual chickens are Finally, in Section 7, we conclude the paper and present graphically manipulated to reflect the power the future work needed to improve the game, as well as consumption of the device running the game. The game some recommendations for future research. was found to contribute to being more energy conscious and the participants’ energy consumption decreased by 13%. Several serious games exist that are focused on city 2. Background and related work management [32], such as EnerCities [33], where players can manage different energy strategies for their city by Before starting the design phase, we analyzed choosing between fossil and renewable sources. In the commercial video games, and serious games used to raise field of VR games, [34] propose an immersive experience players’ awareness of eco-sustainability and biodiversity (Climate Connected: Outbreak) to engage players in the issues, or to teach about eco-sustainable behaviors in the topic of climate change, which was found to foster real world. learning and positive attitudes related to climate change. In the literature taken into consideration, to the best of our knowledge, there do not seem to be relevant 2.1. Pro-environment video games examples in terms of complex worlds and gameplay inspired by mainstream 3D games that also employ large In recent years, an increasing number of video games amounts of text. Some examples exist that try to achieve have been used to increase users’ environmental this result in a 3D environment with heavy use of text, in awareness. These include Terra Nil3, Donut County4 [21], the field of cyberbullying prevention [35], where Factorio5, Arma 36 [22], and Animal Crossing: New offensive text (to be erased or changed by players) is Horizons7 [20]. Despite being commercial video games, included in the form of messages from the characters or many of these games present elements of environmental graffiti. It was important that the teaching came from or biodiversity awareness simply because of the setting, virtual characters to merge it more naturally with the storytelling, mechanics, or dynamics of the game. game environment. We looked at existing commercial 2 The full game can be downloaded here: https://osf.io/ 6 https://arma3.com/ eznjw/?view_only=9cf4572c4b1e41e3a39be622a9ab433d 7 https://animalcrossing.nintendo.com/ 3 https://www.terranil.com/ 8 https://play.eco/ 4 http://donutcounty.com/ 9 https://plasticitygame.wixsite.com/about 5 https://www.factorio.com/ 10 https://beesimulator.com/ 160 games to find mechanics that could fit well with teaching the point of view from humans to animals and let players eco-sustainability facts to foster behavior change. In the correct or condemn human behavior as an external next sections, we discuss the inspiration for the game and observer. This choice was also dictated by the scarcity of the design choices that were made during the perspective change examples in the literature concerning development of Untitled Bee Game. serious games for sustainability, in the face of a relative abundance of such an approach in entertainment games. Moreover, the proposed perspective shift can be put in 2.3. Design frameworks for serious relation with recent literature that deals with the games abandonment of anthropocentrism, in the context of a posthuman turn that views ecology and climate change Serious games and gamification present a more from a non-human/other-than-human standpoint [55, complicated design procedure than video games, since 56, 57]. We also suspect that carrying out hostile in-game they are designed with a clear serious purpose. For this actions as a human instead of an animal could potentially reason, not all gameful systems and serious games are have negative educational consequences. The game effective [36, 37]. For a purpose-consistent design and might be thought to suggest that it is advisable to be effectiveness of these tools, numerous design hostile to peers who do not follow virtuous behavior. frameworks and procedures have been developed and The core mechanic of Untitled Goose Game consists adopted [38, 39, 40]. Design approaches can follow of playing tricks on, and stealing objects from, human different perspectives: user-centered, technology- non-playing characters (NPCs) to disturb their daily centered, game-centered, and context-centered [38, 40, activities. Albeit relying on such a simple mechanic, the 41]. Some of these methodologies present collections of game managed to sell one million copies in three months game elements that help the designer keep track of game [58]. In addition, its simple, toon-shaded low-poly 3D choices [42], while others are more complex graphics make for the ideal style for rapid prototyping methodologies that provide more of a guideline for the and development. design [29, 43], and also for the next steps [41]. Although there is a clear similarity with Bee Simulator Many frameworks are based on developments of pre- (see Section 2.1), the mechanics did not play a particular existing models: for instance, the works of [44] and [45] role in the design of UBG, although it may be used as a trace their roots to the MDA model [46]. This model model in the future to add new features. strives to amalgamate game design, development, critique, and technical game research by presenting a structural approach to understanding games. The 3.1. UBG game design pioneering 6D model by [47] has inspired numerous authors to create additional design frameworks (see [48, As a "game with a purpose", we carefully designed the 49, 50, 51]) or adapt it in different contexts (see [52, 53]). game from the contextual information and purpose. We However, although possible design options are relied on the GamiDoc design framework [41] to write a numerous, few design frameworks or procedures have game design document14 that would allow the team to been used or studied consistently. Specifically, few communicate and at the same time, keep track of the frameworks have been used to design tools promoting development in relation to the purpose. The game had no behavior change or climate change awareness. Among specific target and was inspired by a simple, cartoon the few frameworks for climate change awareness, in graphic style (used in Untitled Goose Game itself) that [34], the authors specifically provide guidelines for could be found pleasurable across different genders and serious games about climate change, including, among ages. The target of the experiments, however, was adults. others, adapting game design to the characteristics of the The idea behind UBG is to explore the potential of users, involving users actively, and presenting content in immersing players in a 3D virtual world inspired by recent new ways to aid learning. commercial games, with NPCs reacting to the avatar’s actions, to foster novel gameful interactions for learning and positive behavior change. 3. Design and development UBG lets players control a bee in a small town (see Figure 1). We selected a bee because of the importance Untitled Bee Game (henceforth UBG) borrows its title and of bees in the environment [10]. Indeed, facts about bees mechanics from Untitled Goose Game11, a 3D puzzle are part of the learning content of the game, which game for computers and consoles. Untitled Goose Game players are quizzed on. This choice also brought about is part of a rather recent video game genre that could be advantages under the technical aspect of the defined as “animal mayhem games” [54]. These games let implementation, such as having to deal with simpler players control animals that wreak havoc and punish animations. Imitating commercial games may imply human beings, generally relying on humor and disruptive adopting control schemes that are a bit difficult to game mechanics. Other such games are, for example, master. Therefore, at the beginning of the game, players Goat Simulator12 and DEEEER Simulator13. This strand of are asked to select the control style they prefer: overhead games seems appropriate to focus on the theme of camera or free camera. In both schemes, players control sustainability and environmental preservation: they shift the bee in the third person, but in the latter, they can also 11 https://goose.game 14 The game design document can be retrieved here along with the 12 https://www.goatsimulator3.com/it documents used for the textual content: 13 https://playism.com/en/game/deeeer-simulator/ https://osf.io/eznjw/?view_only=9cf4572c4b1e41e3a39be622a9ab433d 161 rotate the camera (and the bee) freely in all directions. the virtual environment better and more lively. The The overall objective is to dissuade the NPCs from mechanics of UBG are still rather simplistic, allowing as pursuing certain behaviors. If correct actions are taken by many participants as possible to complete the the player, the garden (a small area where flowers can experiment without too much effort. Apart from a couple appear) and the river are influenced: the garden of side mechanics, the task is rather straightforward. We produces flowers and more fish come to the river. discuss future implementations in Section 7. 3.2. Tools and resources UBG was developed using Unity3D15, an industry- standard environment in the field of video game development. Some of the 2D resources and images were found on Adobe Stock16, while others were created ad- hoc. The 3D models were either found in the Unity Asset Store17 and Quaternius.com18 or created using Blender 19, a popular 3D modeling tool. The music was either created Figure 1: The scenery of Untitled Bee Game. The ad hoc or found online for free20. screenshot shows the small town that is explorable in All the text presented in the game and the quiz the game. questions were created following an extensive literature search on the topic. Following an initial analysis, the final Two NPCs were implemented. The first one wastes data were selected on the basis of their association with plastic bottles, while the second one uses polluting player and NPC behaviors within the game. substances to wash her car. An icon above the head of the NPCs shows the player what behavior is being pursued. If it is a damaging behavior, players must sting 4. Methods the NPC or drop rubbish on their head. Stinging can be performed at any time, even if the behavior shown by the 4.1. Data collection method NPC is already correct. Therefore, we differentiate between two stinging actions the player can perform: Prior to deploying the game and administering it to correct and incorrect stinging. The game plays different the participants, we created a REST API based on a sounds accordingly. If correct stinging is performed, new Mongo21 database that communicated with the game flowers appear in the garden and new fish appear in the during the sessions. For this reason, players were asked river. Once a flower has appeared, players can interact to remain connected while playing. We collected the with it and read facts about the three topics (see Figure following information in JSON format: i) a game ID; ii) the 2), one paragraph at a time, shown in a classic video game number of times each NPC was stung; iii) the number of message box. Once a knowledge paragraph has been correct actions; iv) the number of incorrect actions; v) the unlocked, it becomes possible to read it again in the quiz scores; and vi) the paragraph reading times. pause menu. As a side mechanic, players may also pick up and recycle the rubbish they find. This grants them one 4.2. Participants and procedure additional fish. In addition, rubbish can be thrown at NPCs instead of stinging them to obtain the same effect. Players were mainly recruited in Italian university If players do not do anything for a certain amount of time, facilities (University of Trento, University of Verona, and fish start to disappear. University of Turin) and there were no exclusion criteria except being underage. Recruitment requests were circulated by two lecturers among bachelor students primarily. They could choose between a browser-based, a Mac, and a Windows version. They were both Italian and English speaking and it was possible for them to change the game language at the beginning of the game. The procedure consisted of playing for approximately 20 minutes and responding to an online questionnaire. The game assigned a random alphanumeric ID to each player to be entered in the final questionnaire, which granted complete anonymity. The procedure was employed in Figure 2: The messages displayed by pollinated flowers. two phases: the usability test and the main experiment. The main differences were (i) the version of the It is worth noting that there is no game over and the prototype, and (ii) the content of the final questionnaire, players’ actions are only driven by the possibility to make 15 http://www.unity.com/ 19 https://www.blender.org/ 16 http://stock.adobe.com/ 20 https://audionautix.com/ 17 https://assetstore.unity.com/ 21 https://www.mongodb.com/ 18 https://quaternius.com 162 administered after the post-session quiz, as described identified as male), among which N=18 completed the below. game, and N=24 completed the final questionnaire, of Game session. Players could play a desktop or a web which N=14 completed both the game and the final version of the game. After being shown their ID, they questionnaire. In this phase, players were administered a were taken to the initial quiz (pre-test) where they could refined version of the prototype, where many things express a choice among 4 to several questions related to were changed according to the previous round of eco-sustainability to establish a baseline of knowledge feedback. In particular, the number of paragraphs to read about sustainable behavior. This quiz established a and the number of quiz questions were both lowered to baseline for each participant. After the initial quiz, players 12 in total. After playing the game, participants had to fill had to complete a brief tutorial about the game out a questionnaire where they were asked to provide: (i) commands and mechanics. Then they could start demographic information and gaming habits; (ii) an exploring the environment and stinging the humans informal assessment of the degree of sustainability of showing undesirable behavior. In so doing they earned a their behavior; (iii) engagement and value provided by fish and a flower, and then had to pollinate a flower to the game by using the items from the GAMEFULQUEST, reveal a new fact. After revealing a certain number of the IMI, and the MEEGA360 described above, and two facts, players were taken to the second quiz (post-test) more items —Q5-"Thanks to the game, I am likely to with an additional 4 questions on the same topics as the inquire about how to adopt other sustainable behaviors initial one. in the future" and Q6-"Thanks to the things learned in the The content and order of the questions in the two game, I am likely to adopt more sustainable behaviors in quizzes were the same for all participants, both in the the future"— created ad-hoc to assess how much the usability test and the experimental phase. Accordingly, game may have influenced users to become informed the eco-sustainability facts were shown to all players in about environmental issues and adopt environmentally the same order. The game informed players about sustainable behaviors in the future. All the materials were whether their answer was correct or incorrect, and in the available both in English and Italian. latter case it also displayed the correct one. However, the pre-test and post-test quizzes presented radically different questions to prevent users from answering 5. Results correctly based on the previous answers. Usability test design. To collect usability data, we We report here both the results from the usability test used the System Usability Scale (SUS), which had to be (expressed mainly in terms of SUS scores and feedback filled out at the end of the game. The choice of using this received) and the main experiment. tool stems from its reliability and its widespread use for usability studies [59]. Moreover, we added the item "Do 5.1. Usability test you have any suggestions, feedback, comments that can help us in improving the game?" to collect more detailed The game’s SUS overall score (62.2) was below the set information about possible improvements in the game. threshold for a good level of usability (68), indicating We recruited N=9 participants (female = 4; male = 5) and marginal usability; hence, additional improvements were had them play the first version of the prototype, where needed [64]. For an exhaustive overview of SUS results, they had to unlock 16 flowers and answer 16 quiz see Table 1. questions in total (8 pre-session and 8 post-session). Each Questionnaire results. The results of the additional participant tried the game through the web or desktop items have identified that during this first application, not version and answered the online questionnaire on all users found the game completely entertaining (Q1 𝑀 Google Form22. Data in the literature [60] suggest that for = 3.33, 𝑆𝐷 = 1.11, 𝑀𝑑𝑛 = 4). However, they recognized its usability studies a sample size of 10 ± 2 is recommended educational power and its importance (Q2 𝑀 = 3.44, 𝑆𝐷 to detect at least 80% of usability problems. During the = 1.01, 𝑀𝑑𝑛 = 4; Q3 𝑀 = 3.66, 𝑆𝐷 = 1.11, 𝑀𝑑𝑛 = 4; Q4 𝑀 usability data collection, we still collected data on = 3.88, 𝑆𝐷 = 0.78, 𝑀d𝑛 = 4). playfulness, meaningfulness, and learning perception to Feedback comments. At least one participant (U1) check whether usability could in any way negatively found that the game showed too much text. Therefore, affect these aspects. Hence, we selected some items we lowered the number of flowers to 12. This meant less from reliable questionnaires: Q1-"This game gives me an text to read (12 facts instead of 16) and less playtime. overall playful experience" from the playfulness Three participants (U3, U4, U8) mentioned that the dimension of the GAMEFULQUEST [61], Q2-"I think this objective was not completely clear or that the icons were was an important activity" and Q3-"I believe this activity difficult to interpret. A participant wrote that they would has been beneficial to me" from the value/usefulness have liked to know how important it was to read the dimension of IMI [62], and Q4-"The game contributed to flower facts (U1). This would have allowed them to teach me some of the most relevant environmental answer more correctly to the final quiz. As a solution, we issues" from the perceived learning dimension of added a summary of the objectives in the pause menu. MEEGA360 [63]. We then interviewed the participants in Another participant (U9) mentioned the difficulty of a think-aloud session. understanding interactions with fish within the game. We Experimental design. We recruited N=28 participants therefore decided to specify this more clearly. The (age: 𝑀 = 26.62; 𝑆𝐷 = 8.80; 13 identified as female, 15 controls were a bit difficult for at least two participants 22 https://www.google.it/intl/it/forms/about/ 163 (U3, U7). This led us to offer a choice to the players at the from the questionnaire, the change in quiz performance beginning, asking which control scheme they preferred. in pre-test and post-test, and the feedback received at However, one participant (U1) liked how they worked. the end. Finally, one participant (U7) stated that the background music was a bit annoying. This stresses the importance of giving players an option menu to personalize the 5.2.1. Questionnaire results experience. All the analyses were conducted using RStudio23 (version 2023.03.0). Among the participants, N = 24 answered the SUS Item Mean SD Mdn questionnaire. The results (Figure 3) show an overall S1 2.00 0.86 2 positive evaluation of the tool. Answers were given on a S2 (R) 2.00 1.11 2 5-point Likert-type scale ranging from strongly disagree S3 3.22 1.20 3 to strongly agree (See Table 3). Most of the players found S4 (R) 1.20 0.66 1 the game playful (63%); however, a few subjects thought S5 3.22 0.97 3 the game was not playful, indicating the need to bring S6 (R) 1.88 1.36 1 changes to the game in the future (12%). Interestingly, S7 3.00 1.58 4 almost all the players found the game meaningful (96% S8 (R) 2.55 1.50 2 and 83% to Q2 and Q3 respectively), probably because of S9 3.33 1.41 4 the importance of the themes it dealt with, and they felt S10 (R) 2.22 1.39 2 they learned something from the experience (Q4 positive Overall SUS Score: 62.2/100 results = 88%). Lastly, 58% of the players reported the will Table 1: System Usability Scale item means, standard to inquire (Q5) and then adopt other sustainable deviations, and medians, as well as the final score. (R) behaviors in the future thanks to the game (Q6). We then refers to negative items. ran a linear regression to analyze whether the expressed playful level could be explained by users’ prior experience Think-aloud session. After the usability testing, we with video games. The results show no correlation interviewed the participants informally in a think-aloud between the two variables (𝐴𝑑𝑗𝑢𝑠𝑡𝑒𝑑𝑅² = −0.07, 𝐹1,12 = session to gather additional feedback. There were several 0.14, 𝑝 > 0.05), suggesting that the game is equally points that were raised in this session that had not perceived despite users’ experience with video games. To emerged as part of the written comments. We assess whether the meaningfulness and playfulness of summarize the main topics that emerged during the the game, and the perceived learning were conveyed by think-aloud session in Table 2 as coded comments. The the amount of information read, we ran some linear table includes both suggestions that were later regressions between the reading time and the implemented into the game and those that were not, playfulness value (Q1), the interaction of the items followed by an explanation. related to meaningfulness (Q2 and Q3), and the The suggestion “Add flowers to encourage perceived learning (Q4). The results show that no exploration” was not followed because the game only correlation occurred between the reading time and the allows to unlock flowers in a dedicated area to optimize expressed game meaningfulness (𝐴𝑑𝑗𝑢𝑠𝑡𝑒𝑑𝑅² = 0.58, the traveling time and the experiment duration, but will 𝐹1,12 = 2.54, 𝑝 > 0.05), and playfulness (𝐴𝑑𝑗𝑢𝑠𝑡𝑒𝑑𝑅² = be considered in the future; “Add a deeper narrative” −1.82, 𝐹1,12 = 0.301, 𝑝 > 0.05). The analysis showed a was not followed because, as mentioned, the game was significant relationship between reading time and optimized to be completed in a reasonable amount of perceived learning (𝐴𝑑𝑗𝑢𝑠𝑡𝑒𝑑𝑅² = 1, 𝐹1,12 = 5.79, 𝑝 < time, so we have not yet included a story behind the 0.001). characters or the avatar, but it is a direction for future Q6 8% 33% 58% work. Q5 17% 25% 58% Coded comments Added Add a progress bar for the quizzes Yes Q4 4% 8% 88% Make tutorial more visual Yes Q3 4% 12% 83% Include recycling mechanics Yes Make it easier to unlock more fish Yes Q2 4% 0% 96% Add flowers to encourage exploration No Q1 12% 25% 63% Add a deeper narrative No 100 50 0 50 100 Percentage Table 2: Suggestions from the think-aloud session, Response Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree with an indication as to whether they were followed. Figure 3: An overview of the users’ answers to the 5- point Likert-type items (Q1-Q6) in the questionnaire. 5.2. Experiment What follows is a summary of the results obtained with the main experiment, conducted after the usability test. We proceed to describe and analyze the data obtained 23 https://posit.co/ 164 Item Mean SD Mdn conducted a thematic analysis by coding all feedback Q1 3.66 1.05 4 manually under macro-themes. We summarize the Q2 4.37 0.71 4 comments here, grouped by theme. In Table 4, we Q3 4.16 0.82 4 provide a detailed breakdown of the main comments Q4 4.37 0.82 5 grouped by participants. Q5 3.58 1.10 4 Q6 3.75 0.94 4 Coded comments Participant IDs Table 3: The final questionnaire means, standard There should be additional areas E1, E2, E4 deviations and medians. Exploration is not encouraged E7 Text is too verbose E6, E11, E14, E16 Text is too formal E6, E10, E18 5.2.2. Quiz score improvement There should be a voiceover E3, E13, E16 The font is not very readable E18, E20, E24 Among the participants, N = 18 completed an entire game The game is fun/a nice experience E6, E11, E19 session, and did the post-test quizzes. To answer RQ2, we Playing as a bee is interesting E16 investigated the difference in scores between the pre- Stinging humans is fun/satisfying E9, E18 test and post-test quizzes. The post-test quiz obtained Input controls are easy E6 generally higher scores (𝑀 = 4.55; 𝑆𝐷 = 1.46; 𝑀𝑑𝑛 = 5) Input controls are difficult E10, E16, E17, than pre-test one (𝑀 = 3; 𝑆𝐷 = 1.23; 𝑀𝑑𝑛 = 2.5). At first E19 glance, it was possible to see that only 5 players did not Table 4: Suggestions from the experimental session, get an improvement, of which 4 maintained the same with an indication as to which participant provided them. score, and only one had a lower score. We then analyzed whether these differences in users’ answers before and T1. Repetitiveness: E1, E2, and E4 mentioned that after the use of the game were significant. A Shapiro-Wilk they would have liked to have different areas to explore test was conducted, indicating that the two distributions and to find flowers in. This mirrored what emerged of both variables differed from the normal distribution during the usability testing. E7 also mentioned that the (Pre: 𝑊 = 0.79, 𝑝 = 0.001; Post: 𝑊 = 0.83, 𝑝 = 0.005). positioning of the garden and the humans may be Considering this result, we opted for a non-parametric exploited to take shortcuts as the mechanics do not test. We then performed a one-tail Wilcoxon signed-rank encourage exploration. T2. Text style and length: many test [65] (𝑉 = 11, 𝑝 < 0.005) and found a statistically users expressed that they would have liked to read less meaningful disparity in the results at the in-game eco- or simplify the text. E6, E11, E14, and E16 found that the sustainability questionnaire before and after playing the text was too verbose. E6, E10, and E18 stated that they game. Effect size calculation (𝑟 = 0.49), which for the would have liked less formal language. T3. Voice over: Wilcoxon signed-rank test is based on the formula 𝑟 = this theme is largely connected to Theme 2. In particular, 𝑍⁄√𝑛 [66], suggested a moderate magnitude of the E3, E13, and E16 suggested employing text-to-speech effect [67]. The post-game results are not explained by techniques or voiceovers to decrease the amount of text the subjects’ prior experience with video games to be read, especially for people with dyslexia. T4. (𝐴𝑑𝑗𝑢𝑠𝑡𝑒𝑑𝑅² = 0.02, 𝐹1,12 = 1.32, 𝑝 > 0.05) or the flower Unclear goals: E8 stated that they found the stinging reading time (𝐴𝑑𝑗𝑢𝑠𝑡𝑒𝑑𝑅² = 0.76, 𝐹1,16 = 4.49, 𝑝 > 0.05). context button to be somewhat difficult to activate or interpret. E9, E14, and E19 asked for more succinct Quiz Pre Post indications as to what to do with the flower and the fish. 6 E19 suggested to include more visual cues. T5. Font readability: flowers taught players their facts with a pixelated font, which was chosen to be reminiscent of old-school video games. In this context, however, it 4 Results produced mixed results. E18, E20, and E24 had difficulties reading the font and this made them want to skip the messages (see Figure 2). T6. The fun in playing as a bee: 2 overall, the game was considered a nice experience by E6 and E19. E6 and E11 stated that the game was fun; E16 stated that they liked the concept of being a bee; E9 and Pre Post Quiz E18 explicitly stated that being mean to humans made Figure 4: A boxplot showing the pre-test and post-test them happy or satisfied. T7. Mastery of controls: scores on the eco-sustainability quizzes, taken at the although E6 explicitly stated that controls were easy to beginning and at the end of the game session. There master, E10, E16, E17, and E19 expressed their difficulty were 6 questions in each quiz. in mastering them. Finally, only 4 players did not report any comments. 5.2.3. Feedback comments 6. Discussion and limitations We were able to extract a substantial number of comments from the feedback collected with the final We assessed the enjoyability and usefulness of UBG in questionnaire, which helped us answer RQ3. We two phases: a usability testing phase and the main 165 experiment. All in all, the experience reported by users strictly towards reading and responding to quizzes, while was positive, although there were some problems in the it could be turned into a more relaxed activity in future usability phase. Comments were particularly useful in the versions. The second limitation is the limited number of usability testing phase since usability appeared to be participants involved in the study. A greater sample could suboptimal according to the SUS. Regarding the main be recruited or a follow-up knowledge retention test experiment, we observed that there was indeed an could be introduced to further support the learning trend increase in quiz performance, thanks to the knowledge observed. It could be argued that telling players about presented in the game, which is an encouraging finding the final quiz may have encouraged them to read more that contributes to answer RQ2. Interestingly, quiz carefully, thus influencing the final results. Future performance is not explained by self-reported experience directions may include a between-subjects experiment, with video games according to our analysis. This suggests where the treatment group does not know about the final that potential problems are not likely to have influenced quiz. learning. The level of reported playfulness does not seem to be influenced by experience with video games. In this respect, in addressing RQ1, we can say that in the end the 7. Conclusion and future work prototype was quite enjoyable for every participant and the difficulties with the controls did not invalidate the We developed a 3D serious game, mainly inspired by experience. Untitled Goose Game, about eco-sustainability and Several lessons can be learned by looking at the evaluated it with a usability test and an experiment. feedback collected to answer RQ3: Although the usability test turned out suboptimal, the 1. Consider implementing a voiceover: The proposal feedback collected allowed us to deliver an overall of removing some text and/or adding a voiceover was enjoyable experience in the following phase. Our quiz shared by 29% (7) of participants. In general, reading was results coming from the main experiment indicate that it fatiguing for players, and adding recorded speech or is indeed possible that players learn by means of such a using text-to-speech techniques seems like the sweet game. Moreover, most users felt encouraged to learn spot between teaching notions and not overwhelming more about eco-sustainability in the future. Some players, although we suspect that learning could be participants even stated that controlling a bee and influenced by the two different methods. Speaking of text stinging the humans was fun. However, the sample size more in general, our findings mirror those of [34], where involved in the study was limited. Future work should the authors suggest avoiding too much text and using investigate this tendency in larger samples and with a instead more visual and impactful feedback. more advanced prototype that includes, among others, a 2. Consider individual experiences: Four players narrative and more exploration. Future work may also thought that either the controls were difficult to master take advantage of more comprehensive questionnaires. or that they could be improved, while one player found Recommendations: The insights gained from UBG them easy to master. This underlines the importance of allow us to state some recommendations for future considering not only all the possible peripheral games on sustainability awareness. First, as noted in the configurations but also all the possible levels of literature [30], the design procedure matters: the experience with video games. Choosing the right game effectiveness of a game is not universal. Design genre is key: some games, like first-person shooters, for procedures should follow specific frameworks or example, require very specific skills such as aiming with procedures, and design choices should fit the game's the mouse. purpose and context (see [41] and [68]). Second, as 3. Too much freedom may lead to unclear goals: We suggested in the literature [69], fun can convey the obtained 24 complete questionnaires in the face of 18 effectiveness of serious games and gameful systems. completed game sessions. This fact is likely to be related Thus, it is necessary to analyze the context in which the in some way to T4 (Unclear goals), but it could also be game will be used and the target users to make the game influenced by the fact that the game, albeit giving fun. Third, interaction is essential in the learning process. instructions, leaves players rather free to explore. We As our data suggests, learning through interaction in play therefore recommend future endeavors in serious games is more fun than text-based learning, especially avoiding to consider the aspect of directing players to the goals in repetition of patterns. It is therefore necessary to the most straightforward and clear way possible. Finally, identify a method to make the way information is shown 5 players explicitly stated that either the game was a to players entertaining. Data in the literature suggest good experience or that controlling a bee and stinging the limiting repetition and giving space to novelty [42]. humans was fun. This contributes to addressing RQ1 and Serious games are developed with specific purposes in suggests that being mean to humans can be perceived as mind; hence, it is essential to include certain paths to satisfactory. follow, to prevent users from getting lost, clearly defining There are several limitations to the present study. the objectives of the game, and at the same time, leaving The serious game employed is still in an early version and some freedom for users. Lastly, as suggested by the data lacks a number of features that could make the in the literature [70] and player comments, consider experience more enjoyable (e.g., a more complex using narrative and storytelling to accompany the player narrative and high-level goals that go beyond the stinging along the way. task alone). Since one of the aims of the study was to assess the educational potential of the game, the instructions and the mechanics guided players rather 166 travel behavioural change, Cities 85 (2019) 140– Acknowledgements 149. [16] C. Y. Chow, R. R. Riantiningtyas, M. B. Kanstrup, M. We acknowledge the support of the PNRR project FAIR - Papavasileiou, G. D. Liem, A. Olsen, Can games Future AI Research (PE00000013), under the NRRP MUR change children’s eating behaviour? A review of program funded by the NextGenerationEU. gamification and serious games, Food Quality and Preference 80 (2020) 103823. References [17] D. Johnson, S. Deterding, K.-A. Kuhn, A. Staneva, S. Stoyanov, L. Hides, Gamification for health and [1] S. Oskamp, A sustainable future for humanity? How wellbeing: A systematic review of the literature, can psychology help?, American Psychologist 55 Internet interventions 6 (2016) 89–106. (2000) 496. [18] T. Wolf, Green gamification: How gamified [2] S. Celik, The effects of climate change on human information presentation affects pro- behaviors, Environment, climate, plant and environmental behavior., in: GamiFIN, 2020, pp. vegetation growth (2020) 577–589. 82–91. [3] O. S. Alimi, J. Farner Budarz, L. M. Hernandez, N. [19] F. Xu, F. Tian, D. Buhalis, J. Weber, H. Zhang, Tufenkji, Microplastics and nanoplastics in aquatic Tourists as mobile gamers: Gamification for tourism environments: aggregation, deposition, and marketing, Journal of Travel & Tourism Marketing enhanced contaminant transport, Environmental 33 (2016) 1124–1142. science & technology 52 (2018) 1704–1724. [20] S. Coroller, C. Flinois, Video games as a tool for [4] A. Unep, I. R. R. ASSESSMENT, The rise of ecological learning: the case of Animal Crossing, environmental crime, Nairobi: UNEP (2016). Ecosphere 14 (2023) e4463. [5] S. A. Mousavi, F. Khodadoost, Effects of detergents [21] B. Nicoll, Enjoyment in the Anthropocene: the on natural ecosystems and wastewater treatment extimacy of ecological catastrophe in Donut processes: a review, Environmental Science and County, Distinktion: Journal of Social Theory (2023) Pollution Research 26 (2019) 26439–26448. 1–19. [6] J. R. Barker, D. T. Tingey, Air pollution effects on [22] B. Abraham, Video game visions of climate futures: biodiversity, Springer Science & Business Media, Arma 3 and implications for games and persuasion, 2012. Games and Culture 13 (2018) 71–91. [7] F. Hölker, C. Wolter, E. K. Perkin, K. Tockner, Light [23] R. Buday, T. Baranowski, D. Thompson, Fun and pollution as a biodiversity threat, Trends in ecology games and boredom, GAMES FOR HEALTH: & evolution 25 (2010) 681–682. Research, Development, and Clinical Applications 1 [8] R. Sordello, O. Ratel, F. Flamerie De Lachapelle, C. (2012) 257–261. Leger, A. Dambry, S. Vanpeene, Evidence of the [24] K. S. Fjællingsdal, C. A. Klöckner, Gaming green: the impact of noise pollution on biodiversity: A educational potential of eco–a digital simulated systematic map, Environmental Evidence 9 (2020) ecosystem, Frontiers in psychology 10 (2019) 2846. 1–27. [25] M. Csikszentmihalyi, Flow, Munksgaard [9] S. G. Potts, V. Imperatriz-Fonseca, H. T. Ngo, M. A. København, 1991. Aizen, J. C. Biesmeijer, T. D. Breeze, L. V. Dicks, L. A. [26] M. E. Larreina-Morales, C. Gunella, Video games for Garibaldi, R. Hill, J. Settele, et al., Safeguarding environmental awareness: the accessibility gap, pollinators and their values to human well-being, Universal Access in the Information Society (2023) Nature 540 (2016) 220–229. 1–13. [10] V. Patel, N. Pauli, E. Biggs, L. Barbour, B. Boruff, [27] V.-L. Bekoum Essokolo, E. Robinot, «let’s go deep Why bees are critical for achieving sustainable into the game to save our planet!» how an development, Ambio 50 (2021) 49–59. immersive and educational video game reduces [11] T. J. Wood, I. Kaplan, Z. Szendrei, Wild bee pollen psychological distance and raises awareness, diets reveal patterns of seasonal foraging resources Sustainability 14 (2022) 5774. for honey bees, Frontiers in Ecology and Evolution [28] M. Stanitsas, K. Kirytopoulos, E. Vareilles, 6 (2018) 210. Facilitating sustainability transition through serious [12] S. Bassanelli, N. Vasta, A. Bucchiarone, A. Marconi, games: A systematic literature review, Journal of Gamification for behavior change: A scientometric cleaner production 208 (2019) 924–936. review, Acta Psychologica 228 (2022) 103657. [29] D. Fernández Galeote, J. Hamari, Game-based [13] A.-R. Seidler, C. Henkel, M. Fiedler, J. Kranz, A. climate change engagement: analyzing the Ixmeier, K. S. Strunk, Promoting eco-sustainable potential of entertainment and serious games, behavior with gamification: An experimental study Proceedings of the ACM on Human-Computer on the alignment of competing goals., in: ICIS, 2020. Interaction 5 (2021) 1–21. [14] M. Ferron, E. Loria, A. Marconi, P. Massa, Play&go, [30] K. Madani, T. W. Pierce, A. Mirchi, Serious games on an urban game promoting behaviour change for environmental management, Sustainable Cities sustainable mobility, Interaction Des. Archit. J 40 and Society 29 (2017) 1–11. doi:https: (2019) 24–25. //doi.org/10.1016/j.scs.2016.11.007. [15] B. T. Yen, C. Mulley, M. Burke, Gamification in [31] B. Orland, N. Ram, D. Lang, K. Houser, N. Kling, M. transport interventions: Another way to improve Coccia, Saving energy in an office environment: A serious game intervention, Energy and Buildings 74 167 (2014) 43–52. doi:https: [44] A. C. T. Klock, I. Gasparini, M. S. Pimenta, 5w2h //doi.org/10.1016/j.enbuild.2014.01.036. framework: a guide to design, develop and evaluate [32] A. Diniz dos Santos, F. Strada, A. Bottino, the user-centered gamification, in: Proceedings of Approaching sustainability learning via digital the 15th Brazilian Symposium on Human Factors in serious games, IEEE Transactions on Learning Computing Systems, 2016, pp. 1–10. Technologies 12 (2019) 303–320. doi:10.1109/ [45] A. Mora, P. Zaharias, C. González, J. Arnedo- TLT.2018.2858770. Moreno, Fraggle: a framework for agile [33] E. Knol, P. De Vries, Enercities: Educational game gamification of learning experiences, in: about energy, in: Central Europe towards International Conference on Games and Learning Sustainable Building ’10 (CESB10)’, 2010. Alliance, Springer, 2015, pp. 530–539. [34] D. Fernández Galeote, N.-Z. Legaki, J. Hamari, [46] R. Hunicke, M. LeBlanc, R. Zubek, Mda: A formal Climate Connected: An immersive VR and PC game approach to game design and game research, in: for climate change engagement, in: Companion Proceedings of the AAAI Workshop on Challenges Proceedings of the Annual Symposium on in Game AI, volume 4, San Jose, CA, 2004, p. 1722. Computer-Human Interaction in Play, ACM, [47] D. Hunter, K. Werbach, For the win, Wharton digital Stratford, ON, Canada, 2023, pp. 266–273. URL: press, 2012. https://dl.acm.org/doi/10.1145/3573382.3616053 [48] A. C. Tomé Klock, L. F. d. Cunha, M. F. d. Carvalho, . doi:10.1145/3573382.3616053. B. Eduardo Rosa, A. Jaqueline Anton, I. Gasparini, [35] F. Bonetti, S. Tonelli, An analysis of abusive Gamification in e-learning systems: A conceptual language data collected through a game with a model to engage students and its application in an purpose, in: Proceedings of the 9th Workshop on adaptive e-learning system, in: International Games and Natural Language Processing within the Conference on Learning and Collaboration 13th Language Resources and Evaluation Technologies, Springer, 2015, pp. 595–607. Conference, European Language Resources [49] K. Julius, J. Salo, Designing gamification, Marketing Association, Marseille, France, 2022, pp. 1–6. (2013). [36] J. Koivisto, J. Hamari, The rise of motivational [50] R. Schönen, Gamification in change management information systems: A review of gamification processes, Munich University, Munich. Bachelor research, International Journal of Information Thesis (2014). Management 45 (2019) 191–210. [51] J. Brito, V. Vieira, A. Duran, Towards a framework [37] P. Backlund, M. Hendrix, Educational games-are for gamification design on crowdsourcing systems: they worth the effort? A literature survey of the The game approach, in: 2015 12th International effectiveness of serious games, in: 2013 5th Conference on Information Technology-New international conference on games and virtual Generations, IEEE, 2015, pp. 445–450. worlds for serious applications (VS-GAMES), IEEE, [52] A. Yusof, N. A. Atan, J. Harun, S. Salleh, N. Khamis, 2013, pp. 1–8. Gamified hybrid service-learning for students [38] A. Mora, D. Riera, C. González, J. Arnedo-Moreno, engagement in computer network course Gamification: a systematic review of design (GAMYSEL), in: Proceedings of the International frameworks, Journal of Computing in Higher university carnival on e-learning (IUCEL) 2022, Education 29 (2017) 516–548. 2022, pp. 502–506. [39] D. Ávila-Pesántez, L. A. Rivera, M. S. Alban, [53] S. Gomez-Jaramillo, J. Moreno-Cadavid, C. M. Approaches for serious game design: A systematic Zapata-Jaramillo, Adaptation of the 6d gamification literature review., Computers in education journal model in a software development course, in: 2018 8 (2017). XIII Latin American Conference on Learning [40] D. Fernández Galeote, M. Rajanen, D. Rajanen, N.- Technologies (LACLO), IEEE, 2018, pp. 85–88. Z. Legaki, D. J. Langley, J. Hamari, Gamification for [54] Caracciolo, Marco, Animal mayhem games and climate change engagement: A user-centered nonhuman-oriented thinking, GAME STUDIES design agenda, in: Proceedings of the 26th (COPENHAGEN) 21 (2021) 16. International Academic Mindtrek Conference, [55] O. Cielemęcka, C. Daigle, Posthuman sustainability: 2023, pp. 45–56. an ethos for our anthropocenic future, Theory, [41] S. Bassanelli, A. Bucchiarone, Gamidoc: A tool for Culture & Society 36 (2019) 67–87. designing and evaluating gamified solutions, in: doi:10.1177/0263276419873710. Extended Abstracts of the 2022 Annual Symposium [56] C. Mengozzi, Outside the anthropological machine: on Computer-Human Interaction in Play, 2022, pp. crossing the human-animal divide and other exit 203–208. strategies, Routledge, 2020. [42] Y.-k. Chou, Actionable gamification: Beyond points, [57] L. Bernaerts, M. Caracciolo, L. Herman, B. Vervaeck, badges, and leaderboards, Packt Publishing Ltd, The storied lives of non-human narrators, Narrative 2019. 22 (2014) 68–93. doi:10.1353/nar.2014.0002. [43] B. Morschheuser, J. Hamari, K. Werder, J. Abe, How [58] Forbes, Indie darling ‘Untitled Goose Game’ clears to gamify? A method for designing gamification, in: one million copies sold in three months, 2019. URL: Proceedings of the 50th Hawaii International https://www.forbes.com/sites/lizlanier/2020/12/ Conference on System Sciences 2017, University of 31/indie-darling-untitled-goose-game-clears- one- Hawai’i at Manoa, 2017. million-copies-sold-in-three-months/?sh= 2f1695b84884. 168 [59] J. R. Lewis, The system usability scale: past, present, and future, International Journal of Human– Computer Interaction 34 (2018) 577–590. [60] W. Hwang, G. Salvendy, Number of people required for usability evaluation: the 10±2 rule, Communications of the ACM 53 (2010) 130–133. [61] J. Högberg, J. Hamari, E. Wästlund, Gameful experience questionnaire (GAMEFULQUEST): an instrument for measuring the perceived gamefulness of system use, User Modeling and User-Adapted Interaction 29 (2019) 619–660. [62] E. McAuley, T. Duncan, V. V. Tammen, Psychometric properties of the intrinsic motivation inventory in a competitive sport setting: A confirmatory factor analysis, Research quarterly for exercise and sport 60 (1989) 48–58. [63] F. Gini, S. Bassanelli, The relationship between fun and the overall score of the MEEGA360 scale, 2022. (in press). [64] A. Bangor, P. Kortum, J. Miller, Determining what individual sus scores mean: Adding an adjective rating scale, Journal of usability studies 4 (2009) 114–123. [65] R. F. Woolson, Wilcoxon signed-rank test, Wiley encyclopedia of clinical trials (2007) 1–3. [66] C. O. Fritz, P. E. Morris, J. J. Richler, Effect size estimates: current use, calculations, and interpretation., Journal of experimental psychology: General 141 (2012) 2. [67] M. Tomczak, E. Tomczak, The need to report effect size estimates revisited. an overview of some recommended measures of effect size, TRENDS in Sport Sciences (2014). [68] K. Kölln, Maybe we don’t need a new gamification framework after all, in: Extended Abstracts of the 2022 Annual Symposium on Computer-Human Interaction in Play, 2022, pp. 384–387. [69] G. Tisza, The role of fun in learning, in: Extended Abstracts of the 2021 Annual Symposium on Computer-Human Interaction in Play, 2021, pp. 391–393. [70] P. Palomino, A. Toda, W. Oliveira, L. Rodrigues, A. Cristea, S. Isotani, Exploring content game elements to support gamification design in educational systems: narrative and storytelling, in: Brazilian symposium on computers in education (Simpósio brasileiro de informática na educação- SBIE), volume 30, 2019, p. 773. 169