Untitled Bee Game: Be(e)ing mean to learn more about eco- sustainability

Abstract

The contemporary era witnesses a concerning rise in unsustainable practices, particularly evident in the staggering accumulation of plastic pollution and the detrimental impacts of non-eco-friendly elements on soil and aquatic ecosystems. These human-induced actions contribute to biodiversity decline, especially affecting crucial pollinators like wild bees, essential for sustaining the planet's life support systems. This paper explores the development and evaluation of a 3D serious game, named "Untitled Bee Game", to foster positive behavioral change toward eco-sustainability, by delivering knowledge in an engaging and enjoyable manner. The paper presents the design and development of the game, the results of a usability test, and an experiment conducted to evaluate it. While the initial usability test yielded suboptimal results, valuable feedback informed subsequent improvements, resulting in an overall enjoyable experience for users. The main experiment demonstrated the game's effectiveness in facilitating effective learning, with encouraging quiz results indicating increased awareness of eco-sustainability concepts among players. Participants also expressed a willingness to continue learning about eco-sustainability.

Keywords

• eol>Serious game
• learning
• gameful system
• behavior change
• eco-sustainability1

1. Introduction

In the contemporary era, the pivotal factor influencing the rise and persistence of environmental challenges is human behavior [ 1 ]. In fact, human action is making great changes within the ecosystem and the climate through numerous unsustainable behaviors [ 2 ].

Every year, about 25% of the plastic we produce, which equates to a staggering 80.5 million tons, ends up in our environment, making plastic a pervasive environmental pollutant [ 3, 4 ]. Moreover, the use of noneco-friendly elements, such as most detergents, causes enormous problems, including soil structure deterioration, with detrimental consequences for plant well-being [ 5 ], and eutrophication, which results from the discharge of detergents into water bodies, leading to depletion of dissolved oxygen levels in the water, which can be harmful to aquatic organisms such as fish, seaweed, corals and other aquatic life.

Human action —e.g. urbanization, light pollution, chemical pollution, plastic pollution, deforestation, noise pollution, and air pollution [ 6, 7, 8 ]— is therefore producing a decrease in terms of biodiversity,

0000-0002-1884-6798 (F. Bonetti); 0000-0001-6061-8169 (S. Bassanelli); 0000-0003-1154-1382 (A. Bucchiarone); 0000-0003-34273747 (F. Gini); 0000-0001-8699-7777 (A. Marconi) © 2024 Copyright for this paper by its authors. The use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). contributing to the decline of some species, especially pollinators, such as wild bees [ 9 ]. Bees play a pivotal role in delivering a spectrum of ecosystem services that not only enhance human well-being but also sustain the fundamental life support systems of our planet [ 10 ]. These ecosystem services are inherently aligned with the pursuit of global sustainable development goals [ 11 ]. This underscores the necessity for a fundamental transformation in people’s conduct [ 1 ]. It is imperative to replace practices such as consumerism, wasteful resource utilization, environmental degradation, contamination, selfish and unjust behaviors with sustainable habits, using the world’s resources in ways that will allow human beings to continue to exist on Earth with an adequate quality of life [ 1 ]. With the continuous upgrading of educational technology, gameful systems, known for being highly motivating, have often been implemented to promote behavior change approaches or to support positive behaviors [ 12, 13 ] in different domains, such as transportation and mobility [ 14, 15 ], health, well-being and physical exercise [ 16, 17 ], ecosustainability awareness and pro-environment behaviors [ 18, 13 ], culture and tourism [ 19 ], and so forth. Overall, gameful systems present a reliable approach to enhancing and encouraging users to engage in the desired behavior, potentially fostering environmental protection. In fact, some investigations concentrating on the connection between video games and ecological knowledge have demonstrated noteworthy quantitative outcomes. In particular, there is a notable increase in players’ motivation to grasp ecological concepts through video games [ 20 ].

We therefore developed a serious game called Untitled Bee Game2, in which players have the goal of discouraging non-player characters (NPCs) from engaging in specific non-eco-sustainable behaviors while learning notions related to pollution through dialogue with flowers. The long-term goal of the game is to foster positive behavioral change in terms of eco-sustainability, by providing knowledge in an engaging and enjoyable manner. To assess the game’s usefulness and qualities, we formulated the following research questions:

RQ1. To what extent was the game perceived by the players as playful and meaningful?

RQ2. How would learning and perceived learning be affected by this game?

RQ3. What design directions should be taken in developing and improving such a game? In this paper, we present the design, development, and evaluation related to the use of Untitled Bee Game.

In Section 2 we present a brief overview of video games and serious games related to raising awareness of environmental issues; then, we present the design and development of Untitled Bee Game. In Section 3, we describe the design and development procedure to reach the final version of the game. In Sections 4 and 5, we present the questionnaires and the methods for both usability and experimental analyses, and then the results. In Section 6, we discuss the findings after the analysis. Finally, in Section 7, we conclude the paper and present the future work needed to improve the game, as well as some recommendations for future research.

2. Background and related work

Before starting the design phase, we analyzed commercial video games, and serious games used to raise players’ awareness of eco-sustainability and biodiversity issues, or to teach about eco-sustainable behaviors in the real world.

2.1. Pro-environment video games

In recent years, an increasing number of video games have been used to increase users’ environmental awareness. These include Terra Nil3, Donut County4 [ 21 ], Factorio5, Arma 36 [ 22 ], and Animal Crossing: New Horizons7 [ 20 ]. Despite being commercial video games, many of these games present elements of environmental or biodiversity awareness simply because of the setting, storytelling, mechanics, or dynamics of the game. 2 The full game can be downloaded here: https://osf.io/ eznjw/?view_only=9cf4572c4b1e41e3a39be622a9ab433d 3 https://www.terranil.com/ 4 http://donutcounty.com/ 5 https://www.factorio.com/

In some commercial video games, the ecological component is pivotal and the educational purpose is part of the design itself. In this case, however, the goal of teaching something to players may also result in a less playful system [ 23, 24 ] and a weaker flow experience [ 25 ]. In recent years, numerous video games have been developed with this purpose in mind —such as Eco8 [ 24 ], Plasticity9 [ 26, 27 ], and Bee Simulator10. These games, which thus have some characteristics of serious games while remaining commercial, present a more complicated design procedure. Indeed, designing video games with a well-defined purpose seems to be rather challenging. The priority given in the development to the eco-sustainable awareness component could lead these games to lack the playful impact and the flow experience [ 25 ] that users often seek in commercial video games [ 23, 24 ].

2.2. Pro-environment serious games

The use of serious games has proven to be a useful element in promoting environmentally sustainable behaviors [ 28 ], and climate change awareness [ 29 ]. Madani et al. [ 30 ] compiled a survey of serious games (both digital and physical) used in the context of tackling environmental challenges and awareness raising on ecosustainability. The authors report that the use of serious games can increase motivation and engagement, with beneficial effects on desired outcomes (i.e., environmental awareness); however, the effectiveness of games is not universal for educational purposes, it depends on the design and components of each game and, for this reason, it is difficult to assess the accuracy of the findings. An interesting example of a serious game is Energy Chickens [ 31 ]. In the game, virtual chickens are graphically manipulated to reflect the power consumption of the device running the game. The game was found to contribute to being more energy conscious and the participants’ energy consumption decreased by 13%. Several serious games exist that are focused on city management [ 32 ], such as EnerCities [ 33 ], where players can manage different energy strategies for their city by choosing between fossil and renewable sources. In the field of VR games, [ 34 ] propose an immersive experience (Climate Connected: Outbreak) to engage players in the topic of climate change, which was found to foster learning and positive attitudes related to climate change.

In the literature taken into consideration, to the best of our knowledge, there do not seem to be relevant examples in terms of complex worlds and gameplay inspired by mainstream 3D games that also employ large amounts of text. Some examples exist that try to achieve this result in a 3D environment with heavy use of text, in the field of cyberbullying prevention [ 35 ], where offensive text (to be erased or changed by players) is included in the form of messages from the characters or graffiti. It was important that the teaching came from virtual characters to merge it more naturally with the game environment. We looked at existing commercial 6 https://arma3.com/ 7 https://animalcrossing.nintendo.com/ 8 https://play.eco/ 9 https://plasticitygame.wixsite.com/about 10 https://beesimulator.com/ games to find mechanics that could fit well with teaching eco-sustainability facts to foster behavior change. In the next sections, we discuss the inspiration for the game and the design choices that were made during the development of Untitled Bee Game.

2.3. Design frameworks for serious games

Serious games and gamification present a more complicated design procedure than video games, since they are designed with a clear serious purpose. For this reason, not all gameful systems and serious games are effective [ 36, 37 ]. For a purpose-consistent design and effectiveness of these tools, numerous design frameworks and procedures have been developed and adopted [ 38, 39, 40 ]. Design approaches can follow different perspectives: user-centered, technologycentered, game-centered, and context-centered [ 38, 40, 41 ]. Some of these methodologies present collections of game elements that help the designer keep track of game choices [ 42 ], while others are more complex methodologies that provide more of a guideline for the design [ 29, 43 ], and also for the next steps [ 41 ].

Many frameworks are based on developments of preexisting models: for instance, the works of [ 44 ] and [ 45 ] trace their roots to the MDA model [ 46 ]. This model strives to amalgamate game design, development, critique, and technical game research by presenting a structural approach to understanding games. The pioneering 6D model by [ 47 ] has inspired numerous authors to create additional design frameworks (see [ 48, 49, 50, 51 ]) or adapt it in different contexts (see [ 52, 53 ]). However, although possible design options are numerous, few design frameworks or procedures have been used or studied consistently. Specifically, few frameworks have been used to design tools promoting behavior change or climate change awareness. Among the few frameworks for climate change awareness, in [ 34 ], the authors specifically provide guidelines for serious games about climate change, including, among others, adapting game design to the characteristics of the users, involving users actively, and presenting content in new ways to aid learning.

3. Design and development

Untitled Bee Game (henceforth UBG) borrows its title and mechanics from Untitled Goose Game11, a 3D puzzle game for computers and consoles. Untitled Goose Game is part of a rather recent video game genre that could be defined as “animal mayhem games” [ 54 ]. These games let players control animals that wreak havoc and punish human beings, generally relying on humor and disruptive game mechanics. Other such games are, for example, Goat Simulator12 and DEEEER Simulator13. This strand of games seems appropriate to focus on the theme of sustainability and environmental preservation: they shift 11 https://goose.game 12 https://www.goatsimulator3.com/it 13 https://playism.com/en/game/deeeer-simulator/ the point of view from humans to animals and let players correct or condemn human behavior as an external observer. This choice was also dictated by the scarcity of perspective change examples in the literature concerning serious games for sustainability, in the face of a relative abundance of such an approach in entertainment games. Moreover, the proposed perspective shift can be put in relation with recent literature that deals with the abandonment of anthropocentrism, in the context of a posthuman turn that views ecology and climate change from a non-human/other-than-human standpoint [ 55, 56, 57 ]. We also suspect that carrying out hostile in-game actions as a human instead of an animal could potentially have negative educational consequences. The game might be thought to suggest that it is advisable to be hostile to peers who do not follow virtuous behavior.

The core mechanic of Untitled Goose Game consists of playing tricks on, and stealing objects from, human non-playing characters (NPCs) to disturb their daily activities. Albeit relying on such a simple mechanic, the game managed to sell one million copies in three months [ 58 ]. In addition, its simple, toon-shaded low-poly 3D graphics make for the ideal style for rapid prototyping and development.

Although there is a clear similarity with Bee Simulator (see Section 2.1), the mechanics did not play a particular role in the design of UBG, although it may be used as a model in the future to add new features.

3.1. UBG game design

As a "game with a purpose", we carefully designed the game from the contextual information and purpose. We relied on the GamiDoc design framework [ 41 ] to write a game design document14 that would allow the team to communicate and at the same time, keep track of the development in relation to the purpose. The game had no specific target and was inspired by a simple, cartoon graphic style (used in Untitled Goose Game itself) that could be found pleasurable across different genders and ages. The target of the experiments, however, was adults.

The idea behind UBG is to explore the potential of immersing players in a 3D virtual world inspired by recent commercial games, with NPCs reacting to the avatar’s actions, to foster novel gameful interactions for learning and positive behavior change.

UBG lets players control a bee in a small town (see Figure 1). We selected a bee because of the importance of bees in the environment [ 10 ]. Indeed, facts about bees are part of the learning content of the game, which players are quizzed on. This choice also brought about advantages under the technical aspect of the implementation, such as having to deal with simpler animations. Imitating commercial games may imply adopting control schemes that are a bit difficult to master. Therefore, at the beginning of the game, players are asked to select the control style they prefer: overhead camera or free camera. In both schemes, players control the bee in the third person, but in the latter, they can also 14 The game design document can be retrieved here along with the documents used for the textual content: https://osf.io/eznjw/?view_only=9cf4572c4b1e41e3a39be622a9ab433d rotate the camera (and the bee) freely in all directions. The overall objective is to dissuade the NPCs from pursuing certain behaviors. If correct actions are taken by the player, the garden (a small area where flowers can appear) and the river are influenced: the garden produces flowers and more fish come to the river.

Two NPCs were implemented. The first one wastes plastic bottles, while the second one uses polluting substances to wash her car. An icon above the head of the NPCs shows the player what behavior is being pursued. If it is a damaging behavior, players must sting the NPC or drop rubbish on their head. Stinging can be performed at any time, even if the behavior shown by the NPC is already correct. Therefore, we differentiate between two stinging actions the player can perform: correct and incorrect stinging. The game plays different sounds accordingly. If correct stinging is performed, new flowers appear in the garden and new fish appear in the river. Once a flower has appeared, players can interact with it and read facts about the three topics (see Figure 2), one paragraph at a time, shown in a classic video game message box. Once a knowledge paragraph has been unlocked, it becomes possible to read it again in the pause menu.

As a side mechanic, players may also pick up and recycle the rubbish they find. This grants them one additional fish. In addition, rubbish can be thrown at NPCs instead of stinging them to obtain the same effect. If players do not do anything for a certain amount of time, fish start to disappear. the virtual environment better and more lively. The mechanics of UBG are still rather simplistic, allowing as many participants as possible to complete the experiment without too much effort. Apart from a couple of side mechanics, the task is rather straightforward. We discuss future implementations in Section 7.

3.2. Tools and resources

UBG was developed using Unity3D15, an industrystandard environment in the field of video game development. Some of the 2D resources and images were found on Adobe Stock16, while others were created adhoc. The 3D models were either found in the Unity Asset Store17 and Quaternius.com18 or created using Blender19, a popular 3D modeling tool. The music was either created ad hoc or found online for free20.

All the text presented in the game and the quiz questions were created following an extensive literature search on the topic. Following an initial analysis, the final data were selected on the basis of their association with player and NPC behaviors within the game.

4. Methods

4.1. Data collection method

Prior to deploying the game and administering it to the participants, we created a REST API based on a Mongo21 database that communicated with the game during the sessions. For this reason, players were asked to remain connected while playing. We collected the following information in JSON format: i) a game ID; ii) the number of times each NPC was stung; iii) the number of correct actions; iv) the number of incorrect actions; v) the quiz scores; and vi) the paragraph reading times.

4.2. Participants and procedure

Players were mainly recruited in Italian university facilities (University of Trento, University of Verona, and University of Turin) and there were no exclusion criteria except being underage. Recruitment requests were circulated by two lecturers among bachelor students primarily. They could choose between a browser-based, a Mac, and a Windows version. They were both Italian and English speaking and it was possible for them to change the game language at the beginning of the game. The procedure consisted of playing for approximately 20 minutes and responding to an online questionnaire. The game assigned a random alphanumeric ID to each player to be entered in the final questionnaire, which granted complete anonymity. The procedure was employed in two phases: the usability test and the main experiment. The main differences were (i) the version of the prototype, and (ii) the content of the final questionnaire, 19 https://www.blender.org/ 20 https://audionautix.com/ 21 https://www.mongodb.com/ administered after the post-session quiz, as described below.

Game session. Players could play a desktop or a web version of the game. After being shown their ID, they were taken to the initial quiz (pre-test) where they could express a choice among 4 to several questions related to eco-sustainability to establish a baseline of knowledge about sustainable behavior. This quiz established a baseline for each participant. After the initial quiz, players had to complete a brief tutorial about the game commands and mechanics. Then they could start exploring the environment and stinging the humans showing undesirable behavior. In so doing they earned a fish and a flower, and then had to pollinate a flower to reveal a new fact. After revealing a certain number of facts, players were taken to the second quiz (post-test) with an additional 4 questions on the same topics as the initial one.

The content and order of the questions in the two quizzes were the same for all participants, both in the usability test and the experimental phase. Accordingly, the eco-sustainability facts were shown to all players in the same order. The game informed players about whether their answer was correct or incorrect, and in the latter case it also displayed the correct one. However, the pre-test and post-test quizzes presented radically different questions to prevent users from answering correctly based on the previous answers.

Usability test design. To collect usability data, we used the System Usability Scale (SUS), which had to be filled out at the end of the game. The choice of using this tool stems from its reliability and its widespread use for usability studies [ 59 ]. Moreover, we added the item "Do you have any suggestions, feedback, comments that can help us in improving the game?" to collect more detailed information about possible improvements in the game. We recruited N=9 participants (female = 4; male = 5) and had them play the first version of the prototype, where they had to unlock 16 flowers and answer 16 quiz questions in total (8 pre-session and 8 post-session). Each participant tried the game through the web or desktop version and answered the online questionnaire on Google Form22. Data in the literature [ 60 ] suggest that for usability studies a sample size of 10 ± 2 is recommended to detect at least 80% of usability problems. During the usability data collection, we still collected data on playfulness, meaningfulness, and learning perception to check whether usability could in any way negatively affect these aspects. Hence, we selected some items from reliable questionnaires: Q1-"This game gives me an overall playful experience" from the playfulness dimension of the GAMEFULQUEST [ 61 ], Q2-"I think this was an important activity" and Q3-"I believe this activity has been beneficial to me" from the value/usefulness dimension of IMI [ 62 ], and Q4-"The game contributed to teach me some of the most relevant environmental issues" from the perceived learning dimension of MEEGA360 [ 63 ]. We then interviewed the participants in a think-aloud session.

Experimental design. We recruited N=28 participants (age: = 26.62; = 8.80; 13 identified as female, 15 22 https://www.google.it/intl/it/forms/about/ identified as male), among which N=18 completed the game, and N=24 completed the final questionnaire, of which N=14 completed both the game and the final questionnaire. In this phase, players were administered a refined version of the prototype, where many things were changed according to the previous round of feedback. In particular, the number of paragraphs to read and the number of quiz questions were both lowered to 12 in total. After playing the game, participants had to fill out a questionnaire where they were asked to provide: (i) demographic information and gaming habits; (ii) an informal assessment of the degree of sustainability of their behavior; (iii) engagement and value provided by the game by using the items from the GAMEFULQUEST, the IMI, and the MEEGA360 described above, and two more items —Q5-"Thanks to the game, I am likely to inquire about how to adopt other sustainable behaviors in the future" and Q6-"Thanks to the things learned in the game, I am likely to adopt more sustainable behaviors in the future"— created ad-hoc to assess how much the game may have influenced users to become informed about environmental issues and adopt environmentally sustainable behaviors in the future. All the materials were available both in English and Italian.

5. Results

We report here both the results from the usability test (expressed mainly in terms of SUS scores and feedback received) and the main experiment.

5.1. Usability test

The game’s SUS overall score (62.2) was below the set threshold for a good level of usability (68), indicating marginal usability; hence, additional improvements were needed [ 64 ]. For an exhaustive overview of SUS results, see Table 1.

Questionnaire results. The results of the additional items have identified that during this first application, not all users found the game completely entertaining (Q1 = 3.33, = 1.11, = 4). However, they recognized its educational power and its importance (Q2 = 3.44, = 1.01, = 4; Q3 = 3.66, = 1.11, = 4; Q4 = 3.88, = 0.78, d = 4).

Feedback comments. At least one participant (U1) found that the game showed too much text. Therefore, we lowered the number of flowers to 12. This meant less text to read (12 facts instead of 16) and less playtime. Three participants (U3, U4, U8) mentioned that the objective was not completely clear or that the icons were difficult to interpret. A participant wrote that they would have liked to know how important it was to read the flower facts (U1). This would have allowed them to answer more correctly to the final quiz. As a solution, we added a summary of the objectives in the pause menu. Another participant (U9) mentioned the difficulty of understanding interactions with fish within the game. We therefore decided to specify this more clearly. The controls were a bit difficult for at least two participants (U3, U7). This led us to offer a choice to the players at the beginning, asking which control scheme they preferred. However, one participant (U1) liked how they worked. Finally, one participant (U7) stated that the background music was a bit annoying. This stresses the importance of giving players an option menu to personalize the experience.

Think-aloud session. After the usability testing, we interviewed the participants informally in a think-aloud session to gather additional feedback. There were several points that were raised in this session that had not emerged as part of the written comments. We summarize the main topics that emerged during the think-aloud session in Table 2 as coded comments. The table includes both suggestions that were later implemented into the game and those that were not, followed by an explanation.

The suggestion “Add flowers to encourage exploration” was not followed because the game only allows to unlock flowers in a dedicated area to optimize the traveling time and the experiment duration, but will be considered in the future; “Add a deeper narrative” was not followed because, as mentioned, the game was optimized to be completed in a reasonable amount of time, so we have not yet included a story behind the characters or the avatar, but it is a direction for future work.

Coded comments Added Add a progress bar for the quizzes Yes Make tutorial more visual Yes Include recycling mechanics Yes Make it easier to unlock more fish Yes Add flowers to encourage exploration No Add a deeper narrative No Table 2: Suggestions from the think-aloud session, with an indication as to whether they were followed.

5.2. Experiment

What follows is a summary of the results obtained with the main experiment, conducted after the usability test. We proceed to describe and analyze the data obtained 23 https://posit.co/ from the questionnaire, the change in quiz performance in pre-test and post-test, and the feedback received at the end.

5.2.1. Questionnaire results

All the analyses were conducted using RStudio23 (version 2023.03.0). Among the participants, N = 24 answered the questionnaire. The results (Figure 3) show an overall positive evaluation of the tool. Answers were given on a 5-point Likert-type scale ranging from strongly disagree to strongly agree (See Table 3). Most of the players found the game playful (63%); however, a few subjects thought the game was not playful, indicating the need to bring changes to the game in the future (12%). Interestingly, almost all the players found the game meaningful (96% and 83% to Q2 and Q3 respectively), probably because of the importance of the themes it dealt with, and they felt they learned something from the experience (Q4 positive results = 88%). Lastly, 58% of the players reported the will to inquire (Q5) and then adopt other sustainable behaviors in the future thanks to the game (Q6). We then ran a linear regression to analyze whether the expressed playful level could be explained by users’ prior experience with video games. The results show no correlation between the two variables ( ² = −0.07, 1,12 = 0.14, > 0.05), suggesting that the game is equally perceived despite users’ experience with video games. To assess whether the meaningfulness and playfulness of the game, and the perceived learning were conveyed by the amount of information read, we ran some linear regressions between the reading time and the playfulness value (Q1), the interaction of the items related to meaningfulness (Q2 and Q3), and the perceived learning (Q4). The results show that no correlation occurred between the reading time and the expressed game meaningfulness ( ² = 0.58, 1,12 = 2.54, > 0.05), and playfulness ( ² = −1.82, 1,12 = 0.301, > 0.05). The analysis showed a significant relationship between reading time and perceived learning ( ² = 1, 1,12 = 5.79, < 0.001).

Q6 8% Q5 17% Q4 4% Q3 4% Q2 4% Q1 12% 33% 25% 8% 12% 0% 25% 58% 58% 88% 83% 96% 63% 100 50 Perce0ntage 50 100 Response Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree Figure 3: An overview of the users’ answers to the 5point Likert-type items (Q1-Q6) in the questionnaire.

5.2.2. Quiz score improvement

Among the participants, N = 18 completed an entire game session, and did the post-test quizzes. To answer RQ2, we investigated the difference in scores between the pretest and post-test quizzes. The post-test quiz obtained generally higher scores ( = 4.55; = 1.46; = 5) than pre-test one ( = 3; = 1.23; = 2.5). At first glance, it was possible to see that only 5 players did not get an improvement, of which 4 maintained the same score, and only one had a lower score. We then analyzed whether these differences in users’ answers before and after the use of the game were significant. A Shapiro-Wilk test was conducted, indicating that the two distributions of both variables differed from the normal distribution (Pre: = 0.79, = 0.001; Post: = 0.83, = 0.005). Considering this result, we opted for a non-parametric test. We then performed a one-tail Wilcoxon signed-rank test [ 65 ] ( = 11, < 0.005) and found a statistically meaningful disparity in the results at the in-game ecosustainability questionnaire before and after playing the game. Effect size calculation ( = 0.49), which for the Wilcoxon signed-rank test is based on the formula = ⁄√ [ 66 ], suggested a moderate magnitude of the effect [ 67 ]. The post-game results are not explained by the subjects’ prior experience with video games ( ² = 0.02, 1,12 = 1.32, > 0.05) or the flower reading time ( ² = 0.76, 1,16 = 4.49, > 0.05).

Quiz

Pre

Post 6 lt4 s u s e R 2

Pre

Post

Quiz Figure 4: A boxplot showing the pre-test and post-test scores on the eco-sustainability quizzes, taken at the beginning and at the end of the game session. There were 6 questions in each quiz.

5.2.3. Feedback comments

We were able to extract a substantial number of comments from the feedback collected with the final questionnaire, which helped us answer RQ3. We conducted a thematic analysis by coding all feedback manually under macro-themes. We summarize the comments here, grouped by theme. In Table 4, we provide a detailed breakdown of the main comments grouped by participants.

Coded comments Participant IDs There should be additional areas E1, E2, E4 Exploration is not encouraged E7 Text is too verbose E6, E11, E14, E16 Text is too formal E6, E10, E18 There should be a voiceover E3, E13, E16 The font is not very readable E18, E20, E24 The game is fun/a nice experience E6, E11, E19 Playing as a bee is interesting E16 Stinging humans is fun/satisfying E9, E18 Input controls are easy E6 Input controls are difficult E10, E16, E17, E19

Table 4: Suggestions from the experimental session, with an indication as to which participant provided them.

T1. Repetitiveness: E1, E2, and E4 mentioned that they would have liked to have different areas to explore and to find flowers in. This mirrored what emerged during the usability testing. E7 also mentioned that the positioning of the garden and the humans may be exploited to take shortcuts as the mechanics do not encourage exploration. T2. Text style and length: many users expressed that they would have liked to read less or simplify the text. E6, E11, E14, and E16 found that the text was too verbose. E6, E10, and E18 stated that they would have liked less formal language. T3. Voice over: this theme is largely connected to Theme 2. In particular, E3, E13, and E16 suggested employing text-to-speech techniques or voiceovers to decrease the amount of text to be read, especially for people with dyslexia. T4. Unclear goals: E8 stated that they found the stinging context button to be somewhat difficult to activate or interpret. E9, E14, and E19 asked for more succinct indications as to what to do with the flower and the fish. E19 suggested to include more visual cues. T5. Font readability: flowers taught players their facts with a pixelated font, which was chosen to be reminiscent of old-school video games. In this context, however, it produced mixed results. E18, E20, and E24 had difficulties reading the font and this made them want to skip the messages (see Figure 2). T6. The fun in playing as a bee: overall, the game was considered a nice experience by E6 and E19. E6 and E11 stated that the game was fun; E16 stated that they liked the concept of being a bee; E9 and E18 explicitly stated that being mean to humans made them happy or satisfied. T7. Mastery of controls: although E6 explicitly stated that controls were easy to master, E10, E16, E17, and E19 expressed their difficulty in mastering them. Finally, only 4 players did not report any comments.

6. Discussion and limitations

We assessed the enjoyability and usefulness of UBG in two phases: a usability testing phase and the main experiment. All in all, the experience reported by users was positive, although there were some problems in the usability phase. Comments were particularly useful in the usability testing phase since usability appeared to be suboptimal according to the SUS. Regarding the main experiment, we observed that there was indeed an increase in quiz performance, thanks to the knowledge presented in the game, which is an encouraging finding that contributes to answer RQ2. Interestingly, quiz performance is not explained by self-reported experience with video games according to our analysis. This suggests that potential problems are not likely to have influenced learning. The level of reported playfulness does not seem to be influenced by experience with video games. In this respect, in addressing RQ1, we can say that in the end the prototype was quite enjoyable for every participant and the difficulties with the controls did not invalidate the experience.

Several lessons can be learned by looking at the feedback collected to answer RQ3:

1. Consider implementing a voiceover: The proposal of removing some text and/or adding a voiceover was shared by 29% (7) of participants. In general, reading was fatiguing for players, and adding recorded speech or using text-to-speech techniques seems like the sweet spot between teaching notions and not overwhelming players, although we suspect that learning could be influenced by the two different methods. Speaking of text more in general, our findings mirror those of [ 34 ], where the authors suggest avoiding too much text and using instead more visual and impactful feedback.

2. Consider individual experiences: Four players thought that either the controls were difficult to master or that they could be improved, while one player found them easy to master. This underlines the importance of considering not only all the possible peripheral configurations but also all the possible levels of experience with video games. Choosing the right game genre is key: some games, like first-person shooters, for example, require very specific skills such as aiming with the mouse.

3. Too much freedom may lead to unclear goals: We obtained 24 complete questionnaires in the face of 18 completed game sessions. This fact is likely to be related in some way to T4 (Unclear goals), but it could also be influenced by the fact that the game, albeit giving instructions, leaves players rather free to explore. We therefore recommend future endeavors in serious games to consider the aspect of directing players to the goals in the most straightforward and clear way possible. Finally, 5 players explicitly stated that either the game was a good experience or that controlling a bee and stinging the humans was fun. This contributes to addressing RQ1 and suggests that being mean to humans can be perceived as satisfactory.

There are several limitations to the present study. The serious game employed is still in an early version and lacks a number of features that could make the experience more enjoyable (e.g., a more complex narrative and high-level goals that go beyond the stinging task alone). Since one of the aims of the study was to assess the educational potential of the game, the instructions and the mechanics guided players rather strictly towards reading and responding to quizzes, while it could be turned into a more relaxed activity in future versions. The second limitation is the limited number of participants involved in the study. A greater sample could be recruited or a follow-up knowledge retention test could be introduced to further support the learning trend observed. It could be argued that telling players about the final quiz may have encouraged them to read more carefully, thus influencing the final results. Future directions may include a between-subjects experiment, where the treatment group does not know about the final quiz.

7. Conclusion and future work

We developed a 3D serious game, mainly inspired by Untitled Goose Game, about eco-sustainability and evaluated it with a usability test and an experiment. Although the usability test turned out suboptimal, the feedback collected allowed us to deliver an overall enjoyable experience in the following phase. Our quiz results coming from the main experiment indicate that it is indeed possible that players learn by means of such a game. Moreover, most users felt encouraged to learn more about eco-sustainability in the future. Some participants even stated that controlling a bee and stinging the humans was fun. However, the sample size involved in the study was limited. Future work should investigate this tendency in larger samples and with a more advanced prototype that includes, among others, a narrative and more exploration. Future work may also take advantage of more comprehensive questionnaires.

Recommendations: The insights gained from UBG allow us to state some recommendations for future games on sustainability awareness. First, as noted in the literature [ 30 ], the design procedure matters: the effectiveness of a game is not universal. Design procedures should follow specific frameworks or procedures, and design choices should fit the game's purpose and context (see [ 41 ] and [ 68 ]). Second, as suggested in the literature [ 69 ], fun can convey the effectiveness of serious games and gameful systems. Thus, it is necessary to analyze the context in which the game will be used and the target users to make the game fun. Third, interaction is essential in the learning process. As our data suggests, learning through interaction in play is more fun than text-based learning, especially avoiding repetition of patterns. It is therefore necessary to identify a method to make the way information is shown to players entertaining. Data in the literature suggest limiting repetition and giving space to novelty [ 42 ]. Serious games are developed with specific purposes in mind; hence, it is essential to include certain paths to follow, to prevent users from getting lost, clearly defining the objectives of the game, and at the same time, leaving some freedom for users. Lastly, as suggested by the data in the literature [ 70 ] and player comments, consider using narrative and storytelling to accompany the player along the way.

Acknowledgements

We acknowledge the support of the PNRR project FAIR Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU.

References

[1] S. Oskamp , A sustainable future for humanity? How can psychology help? , American Psychologist 55 ( 2000 ) 496 .

[2] S. Celik , The effects of climate change on human behaviors, Environment, climate, plant and vegetation growth ( 2020 ) 577 - 589 .

[3] O. S. Alimi , J. Farner Budarz , L. M. Hernandez , N. Tufenkji , Microplastics and nanoplastics in aquatic environments: aggregation, deposition, and enhanced contaminant transport , Environmental science & technology 52 ( 2018 ) 1704 - 1724 .

[4] A. Unep , I. R. R. ASSESSMENT , The rise of environmental crime, Nairobi: UNEP ( 2016 ).

[5] S. A. Mousavi , F. Khodadoost , Effects of detergents on natural ecosystems and wastewater treatment processes: a review , Environmental Science and Pollution Research 26 ( 2019 ) 26439 - 26448 .

[6] J. R. Barker , D. T. Tingey, Air pollution effects on biodiversity, Springer Science & Business Media , 2012 .

[7] F. Hölker , C. Wolter , E. K. Perkin , K. Tockner , Light pollution as a biodiversity threat , Trends in ecology & evolution 25 ( 2010 ) 681 - 682 .

[8] R. Sordello , O. Ratel , F. Flamerie De Lachapelle , C. Leger , A. Dambry , S. Vanpeene , Evidence of the impact of noise pollution on biodiversity: A systematic map , Environmental Evidence 9 ( 2020 ) 1 - 27 .

[9] S. G. Potts , V. Imperatriz-Fonseca , H. T. Ngo , M. A. Aizen , J. C. Biesmeijer , T. D. Breeze , L. V. Dicks , L. A. Garibaldi , R. Hill , J. Settele , et al., Safeguarding pollinators and their values to human well-being , Nature 540 ( 2016 ) 220 - 229 .

[10] V. Patel , N. Pauli , E. Biggs , L. Barbour , B. Boruff , Why bees are critical for achieving sustainable development , Ambio 50 ( 2021 ) 49 - 59 .

[11] T. J. Wood , I. Kaplan , Z. Szendrei , Wild bee pollen diets reveal patterns of seasonal foraging resources for honey bees , Frontiers in Ecology and Evolution 6 ( 2018 ) 210 .

[12] S. Bassanelli , N. Vasta , A. Bucchiarone , A. Marconi , Gamification for behavior change: A scientometric review , Acta Psychologica 228 ( 2022 ) 103657 .

[13] A.-R. Seidler , C. Henkel , M. Fiedler , J. Kranz , A. Ixmeier , K. S. Strunk , Promoting eco-sustainable behavior with gamification: An experimental study on the alignment of competing goals ., in: ICIS , 2020 .

[14] M. Ferron , E. Loria , A. Marconi , P. Massa , Play&go, an urban game promoting behaviour change for sustainable mobility , Interaction Des. Archit. J 40 ( 2019 ) 24 - 25 .

[15] B. T. Yen , C. Mulley , M. Burke , Gamification in transport interventions: Another way to improve travel behavioural change , Cities 85 ( 2019 ) 140 - 149 .

[16] C. Y. Chow , R. R. Riantiningtyas , M. B. Kanstrup , M. Papavasileiou , G. D. Liem , A. Olsen , Can games change children's eating behaviour? A review of gamification and serious games , Food Quality and Preference 80 ( 2020 ) 103823 .

[17] D. Johnson , S. Deterding, K.-A. Kuhn , A. Staneva , S. Stoyanov , L. Hides, Gamification for health and wellbeing: A systematic review of the literature , Internet interventions 6 ( 2016 ) 89 - 106 .

[18] T. Wolf , Green gamification: How gamified information presentation affects proenvironmental behavior ., in: GamiFIN, 2020 , pp. 82 - 91 .

[19] F. Xu , F. Tian , D. Buhalis , J. Weber , H. Zhang , Tourists as mobile gamers: Gamification for tourism marketing , Journal of Travel & Tourism Marketing 33 ( 2016 ) 1124 - 1142 .

[20] S. Coroller , C. Flinois , Video games as a tool for ecological learning: the case of Animal Crossing, Ecosphere 14 ( 2023 ) e4463 .

[21] B. Nicoll , Enjoyment in the Anthropocene: the extimacy of ecological catastrophe in Donut County , Distinktion: Journal of Social Theory ( 2023 ) 1 - 19 .

[22] B. Abraham , Video game visions of climate futures: Arma 3 and implications for games and persuasion , Games and Culture 13 ( 2018 ) 71 - 91 .

[23] R. Buday , T. Baranowski , D. Thompson , Fun and games and boredom , GAMES FOR HEALTH: Research, Development, and Clinical Applications 1 ( 2012 ) 257 - 261 .

[24] K. S. Fjaellingsdal , C. A. Klöckner , Gaming green: the educational potential of eco-a digital simulated ecosystem , Frontiers in psychology 10 ( 2019 ) 2846 .

[25] M. Csikszentmihalyi , Flow, Munksgaard København, 1991 .

[26] M. E. Larreina-Morales , C. Gunella , Video games for environmental awareness: the accessibility gap , Universal Access in the Information Society ( 2023 ) 1 - 13 .

[27] V.-L. Bekoum Essokolo , E. Robinot, « let's go deep into the game to save our planet!» how an immersive and educational video game reduces psychological distance and raises awareness , Sustainability 14 ( 2022 ) 5774 .

[28] M. Stanitsas , K. Kirytopoulos , E. Vareilles, Facilitating sustainability transition through serious games: A systematic literature review , Journal of cleaner production 208 ( 2019 ) 924 - 936 .

[29] D. Fernández Galeote , J. Hamari , Game-based climate change engagement: analyzing the potential of entertainment and serious games , Proceedings of the ACM on Human-Computer Interaction 5 ( 2021 ) 1 - 21 .

[30] K. Madani , T. W. Pierce , A. Mirchi , Serious games on environmental management , Sustainable Cities and Society 29 ( 2017 ) 1 - 11 . doi:https: //doi.org/10.1016/j.scs. 2016 . 11 .007.

[31] B. Orland , N. Ram , D. Lang , K. Houser , N. Kling , M. Coccia , Saving energy in an office environment: A serious game intervention , Energy and Buildings 74 ( 2014 ) 43 - 52 . doi:https: //doi.org/10.1016/j.enbuild. 2014 . 01 .036.

[32] A. Diniz dos Santos , F. Strada , A. Bottino , Approaching sustainability learning via digital serious games , IEEE Transactions on Learning Technologies 12 ( 2019 ) 303 - 320 . doi: 10 .1109/ TLT. 2018 . 2858770 .

[33] E. Knol , P. De Vries, Enercities: Educational game about energy , in: Central Europe towards Sustainable Building ' 10 ( CESB10 )', 2010 .

[34] D. Fernández Galeote , N.- Z. Legaki , J. Hamari , Climate Connected: An immersive VR and PC game for climate change engagement , in: Companion Proceedings of the Annual Symposium on Computer-Human Interaction in Play , ACM, Stratford, ON , Canada, 2023 , pp. 266 - 273 . URL: https://dl.acm.org/doi/10.1145/3573382.3616053 . doi: 10 .1145/3573382.3616053.

[35] F. Bonetti , S. Tonelli , An analysis of abusive language data collected through a game with a purpose , in: Proceedings of the 9th Workshop on Games and Natural Language Processing within the 13th Language Resources and Evaluation Conference , European Language Resources Association, Marseille, France, 2022 , pp. 1 - 6 .

[36] J. Koivisto , J. Hamari, The rise of motivational information systems: A review of gamification research , International Journal of Information Management 45 ( 2019 ) 191 - 210 .

[37] P. Backlund , M. Hendrix , Educational games-are they worth the effort? A literature survey of the effectiveness of serious games, in: 2013 5th international conference on games and virtual worlds for serious applications (VS-GAMES) , IEEE, 2013 , pp. 1 - 8 .

[38] A. Mora , D. Riera , C. González , J. Arnedo-Moreno , Gamification: a systematic review of design frameworks , Journal of Computing in Higher Education 29 ( 2017 ) 516 - 548 .

[39] D. Ávila-Pesántez , L. A. Rivera , M. S. Alban , Approaches for serious game design: A systematic literature review ., Computers in education journal 8 ( 2017 ).

[40] D. Fernández Galeote , M. Rajanen , D. Rajanen , N.- Z. Legaki , D. J. Langley , J. Hamari , Gamification for climate change engagement: A user-centered design agenda , in: Proceedings of the 26th International Academic Mindtrek Conference , 2023 , pp. 45 - 56 .

[41] S. Bassanelli , A. Bucchiarone , Gamidoc: A tool for designing and evaluating gamified solutions , in: Extended Abstracts of the 2022 Annual Symposium on Computer-Human Interaction in Play , 2022 , pp. 203 - 208 .

[42] Y. -k. Chou, Actionable gamification: Beyond points, badges, and leaderboards , Packt Publishing Ltd , 2019 .

[43] B. Morschheuser , J. Hamari , K. Werder , J. Abe , How to gamify? A method for designing gamification , in: Proceedings of the 50th Hawaii International Conference on System Sciences 2017 , University of Hawai'i at Manoa, 2017 .

[44] A. C. T. Klock , I. Gasparini, M. S. Pimenta , 5w2h framework: a guide to design, develop and evaluate the user-centered gamification , in: Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems , 2016 , pp. 1 - 10 .

[45] A. Mora , P. Zaharias , C. González , J. ArnedoMoreno , Fraggle: a framework for agile gamification of learning experiences , in: International Conference on Games and Learning Alliance , Springer, 2015 , pp. 530 - 539 .

[46] R. Hunicke , M. LeBlanc, R. Zubek, Mda: A formal approach to game design and game research , in: Proceedings of the AAAI Workshop on Challenges in Game AI , volume 4 , San Jose, CA, 2004 , p. 1722 .

[47] D. Hunter , K. Werbach, For the win , Wharton digital press, 2012 .

[48] A. C. Tomé Klock , L. F. d. Cunha , M. F. d. Carvalho , B. Eduardo Rosa , A. Jaqueline Anton , I. Gasparini , Gamification in e-learning systems: A conceptual model to engage students and its application in an adaptive e-learning system , in: International Conference on Learning and Collaboration Technologies , Springer, 2015 , pp. 595 - 607 .

[49] K. Julius , J. Salo , Designing gamification, Marketing ( 2013 ).

[50] R. Schönen , Gamification in change management processes , Munich University, Munich. Bachelor Thesis ( 2014 ).

[51] J. Brito , V. Vieira , A. Duran , Towards a framework for gamification design on crowdsourcing systems: The game approach , in: 2015 12th International Conference on Information Technology-New Generations, IEEE , 2015 , pp. 445 - 450 .

[52] A. Yusof , N. A. Atan , J. Harun , S. Salleh , N. Khamis , Gamified hybrid service-learning for students engagement in computer network course (GAMYSEL), in: Proceedings of the International university carnival on e-learning ( IUCEL) 2022 , 2022 , pp. 502 - 506 .

[53] S. Gomez-Jaramillo , J. Moreno-Cadavid , C. M. Zapata-Jaramillo , Adaptation of the 6d gamification model in a software development course , in: 2018 XIII Latin American Conference on Learning Technologies (LACLO) , IEEE, 2018 , pp. 85 - 88 .

[54] Caracciolo , Marco, Animal mayhem games and nonhuman-oriented thinking , GAME STUDIES (COPENHAGEN) 21 ( 2021 ) 16 .

[55] O. Cielemęcka , C. Daigle , Posthuman sustainability: an ethos for our anthropocenic future , Theory, Culture & Society 36 ( 2019 ) 67 - 87 . doi: 10 .1177/0263276419873710.

[56] C. Mengozzi , Outside the anthropological machine: crossing the human-animal divide and other exit strategies , Routledge , 2020 .

[57] L. Bernaerts , M. Caracciolo , L. Herman , B. Vervaeck , The storied lives of non-human narrators , Narrative 22 ( 2014 ) 68 - 93 . doi: 10 .1353/nar. 2014 . 0002 .

[58] Forbes , Indie darling 'Untitled Goose Game' clears one million copies sold in three months , 2019 . URL: https://www.forbes.com/sites/lizlanier/2020/12/ 31/indie-darling -untitled-goose-game-clears- onemillion-copies-sold-in-three-months/?sh= 2f1695b84884.

[59] J. R. Lewis , The system usability scale: past, present, and future , International Journal of HumanComputer Interaction 34 ( 2018 ) 577 - 590 .

[60] W. Hwang , G. Salvendy, Number of people required for usability evaluation: the 10±2 rule , Communications of the ACM 53 ( 2010 ) 130 - 133 .

[61] J. Högberg , J. Hamari , E. Wästlund, Gameful experience questionnaire (GAMEFULQUEST): an instrument for measuring the perceived gamefulness of system use, User Modeling and User-Adapted Interaction 29 ( 2019 ) 619 - 660 .

[62] E. McAuley , T. Duncan , V. V. Tammen , Psychometric properties of the intrinsic motivation inventory in a competitive sport setting: A confirmatory factor analysis, Research quarterly for exercise and sport 60 ( 1989 ) 48 - 58 .

[63] F. Gini , S. Bassanelli, The relationship between fun and the overall score of the MEEGA360 scale , 2022 . (in press).

[64] A. Bangor , P. Kortum , J. Miller , Determining what individual sus scores mean: Adding an adjective rating scale , Journal of usability studies 4 ( 2009 ) 114 - 123 .

[65] R. F. Woolson , Wilcoxon signed-rank test, Wiley encyclopedia of clinical trials ( 2007 ) 1 - 3 .

[66] C. O. Fritz , P. E. Morris , J. J. Richler , Effect size estimates: current use, calculations, and interpretation ., Journal of experimental psychology: General 141 ( 2012 ) 2 .

[67] M. Tomczak , E. Tomczak, The need to report effect size estimates revisited. an overview of some recommended measures of effect size, TRENDS in Sport Sciences ( 2014 ).

[68] K. Kölln , Maybe we don't need a new gamification framework after all , in: Extended Abstracts of the 2022 Annual Symposium on Computer-Human Interaction in Play , 2022 , pp. 384 - 387 .

[69] G. Tisza , The role of fun in learning , in: Extended Abstracts of the 2021 Annual Symposium on Computer-Human Interaction in Play , 2021 , pp. 391 - 393 .

[70] P. Palomino , A. Toda , W. Oliveira , L. Rodrigues , A. Cristea , S. Isotani , Exploring content game elements to support gamification design in educational systems: narrative and storytelling, in: Brazilian symposium on computers in education (Simpósio brasileiro de informática na educaçãoSBIE) , volume 30 , 2019 , p. 773 .