=Paper=
{{Paper
|id=Vol-3669/paper15
|storemode=property
|title=Untitled Bee Game: Be(e)ing mean to learn more about eco-sustainability
|pdfUrl=https://ceur-ws.org/Vol-3669/paper15.pdf
|volume=Vol-3669
|authors=Federico Bonetti,Simone Bassanelli,Antonio Bucchiarone,Federica Gini,Annapaola Marconi
|dblpUrl=https://dblp.org/rec/conf/gamifin/BonettiBBGM24
}}
==Untitled Bee Game: Be(e)ing mean to learn more about eco-sustainability==
https://ceur-ws.org/Vol-3669/paper15.pdf
Untitled Bee Game: Be(e)ing mean to learn more about eco-
sustainability
Federico Bonetti1, Simone Bassanelli1,2, Antonio Bucchiarone1, Federica Gini1,2 and
Annapaola Marconi1
1 Fondazione Bruno Kessler, Povo (TN), Italy
2 University of Trento, Trento, Italy
Abstract
The contemporary era witnesses a concerning rise in unsustainable practices, particularly evident in the
staggering accumulation of plastic pollution and the detrimental impacts of non-eco-friendly elements on
soil and aquatic ecosystems. These human-induced actions contribute to biodiversity decline, especially
affecting crucial pollinators like wild bees, essential for sustaining the planet's life support systems. This paper
explores the development and evaluation of a 3D serious game, named "Untitled Bee Game", to foster
positive behavioral change toward eco-sustainability, by delivering knowledge in an engaging and enjoyable
manner. The paper presents the design and development of the game, the results of a usability test, and an
experiment conducted to evaluate it. While the initial usability test yielded suboptimal results, valuable
feedback informed subsequent improvements, resulting in an overall enjoyable experience for users. The
main experiment demonstrated the game's effectiveness in facilitating effective learning, with encouraging
quiz results indicating increased awareness of eco-sustainability concepts among players. Participants also
expressed a willingness to continue learning about eco-sustainability.
Keywords
Serious game, learning, gameful system, behavior change, eco-sustainability1
contributing to the decline of some species, especially
1. Introduction pollinators, such as wild bees [9]. Bees play a pivotal role
in delivering a spectrum of ecosystem services that not
In the contemporary era, the pivotal factor influencing
only enhance human well-being but also sustain the
the rise and persistence of environmental challenges is
fundamental life support systems of our planet [10].
human behavior [1]. In fact, human action is making great
These ecosystem services are inherently aligned with the
changes within the ecosystem and the climate through
pursuit of global sustainable development goals [11]. This
numerous unsustainable behaviors [2].
underscores the necessity for a fundamental
Every year, about 25% of the plastic we produce,
transformation in people’s conduct [1]. It is imperative to
which equates to a staggering 80.5 million tons, ends up
replace practices such as consumerism, wasteful
in our environment, making plastic a pervasive
resource utilization, environmental degradation,
environmental pollutant [3, 4]. Moreover, the use of non-
contamination, selfish and unjust behaviors with
eco-friendly elements, such as most detergents, causes
sustainable habits, using the world’s resources in ways
enormous problems, including soil structure
that will allow human beings to continue to exist on Earth
deterioration, with detrimental consequences for plant
with an adequate quality of life [1]. With the continuous
well-being [5], and eutrophication, which results from the
upgrading of educational technology, gameful systems,
discharge of detergents into water bodies, leading to
known for being highly motivating, have often been
depletion of dissolved oxygen levels in the water, which
implemented to promote behavior change approaches or
can be harmful to aquatic organisms such as fish,
to support positive behaviors [12, 13] in different
seaweed, corals and other aquatic life.
domains, such as transportation and mobility [14, 15],
Human action —e.g. urbanization, light pollution,
health, well-being and physical exercise [16, 17], eco-
chemical pollution, plastic pollution, deforestation, noise
sustainability awareness and pro-environment behaviors
pollution, and air pollution [6, 7, 8]— is therefore
[18, 13], culture and tourism [19], and so forth. Overall,
producing a decrease in terms of biodiversity,
gameful systems present a reliable approach to
8th International GamiFIN Conference 2024 (GamiFIN 2024), April 2-5,
2024, Ruka, Finland.
fbonetti@fbk.eu (F. Bonetti); sbassanelli@fbk.eu (S. Bassanelli);
bucchiarone@fbk.eu (A. Bucchiarone); fgini@fbk.eu (F. Gini);
marconi@fbk.eu (A. Marconi)
0000-0002-1884-6798 (F. Bonetti); 0000-0001-6061-8169 (S.
Bassanelli); 0000-0003-1154-1382 (A. Bucchiarone); 0000-0003-3427-
3747 (F. Gini); 0000-0001-8699-7777 (A. Marconi)
© 2024 Copyright for this paper by its authors. The use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
159
enhancing and encouraging users to engage in the In some commercial video games, the ecological
desired behavior, potentially fostering environmental component is pivotal and the educational purpose is part
protection. In fact, some investigations concentrating on of the design itself. In this case, however, the goal of
the connection between video games and ecological teaching something to players may also result in a less
knowledge have demonstrated noteworthy quantitative playful system [23, 24] and a weaker flow experience
outcomes. In particular, there is a notable increase in [25]. In recent years, numerous video games have been
players’ motivation to grasp ecological concepts through developed with this purpose in mind —such as Eco8 [24],
video games [20]. Plasticity9 [26, 27], and Bee Simulator10. These games,
We therefore developed a serious game called which thus have some characteristics of serious games
Untitled Bee Game2, in which players have the goal of while remaining commercial, present a more complicated
discouraging non-player characters (NPCs) from engaging design procedure. Indeed, designing video games with a
in specific non-eco-sustainable behaviors while learning well-defined purpose seems to be rather challenging. The
notions related to pollution through dialogue with priority given in the development to the eco-sustainable
flowers. The long-term goal of the game is to foster awareness component could lead these games to lack the
positive behavioral change in terms of eco-sustainability, playful impact and the flow experience [25] that users
by providing knowledge in an engaging and enjoyable often seek in commercial video games [23, 24].
manner. To assess the game’s usefulness and qualities,
we formulated the following research questions:
RQ1. To what extent was the game perceived by the
2.2. Pro-environment serious games
players as playful and meaningful?
RQ2. How would learning and perceived learning be The use of serious games has proven to be a useful
affected by this game? element in promoting environmentally sustainable
RQ3. What design directions should be taken in behaviors [28], and climate change awareness [29].
developing and improving such a game? In this paper, we Madani et al. [30] compiled a survey of serious games
present the design, development, and evaluation related (both digital and physical) used in the context of tackling
to the use of Untitled Bee Game. environmental challenges and awareness raising on eco-
In Section 2 we present a brief overview of video sustainability. The authors report that the use of serious
games and serious games related to raising awareness of games can increase motivation and engagement, with
environmental issues; then, we present the design and beneficial effects on desired outcomes (i.e.,
development of Untitled Bee Game. In Section 3, we environmental awareness); however, the effectiveness of
describe the design and development procedure to reach games is not universal for educational purposes, it
the final version of the game. In Sections 4 and 5, we depends on the design and components of each game
present the questionnaires and the methods for both and, for this reason, it is difficult to assess the accuracy of
usability and experimental analyses, and then the results. the findings. An interesting example of a serious game is
In Section 6, we discuss the findings after the analysis. Energy Chickens [31]. In the game, virtual chickens are
Finally, in Section 7, we conclude the paper and present graphically manipulated to reflect the power
the future work needed to improve the game, as well as consumption of the device running the game. The game
some recommendations for future research. was found to contribute to being more energy conscious
and the participants’ energy consumption decreased by
13%. Several serious games exist that are focused on city
2. Background and related work management [32], such as EnerCities [33], where players
can manage different energy strategies for their city by
Before starting the design phase, we analyzed choosing between fossil and renewable sources. In the
commercial video games, and serious games used to raise field of VR games, [34] propose an immersive experience
players’ awareness of eco-sustainability and biodiversity (Climate Connected: Outbreak) to engage players in the
issues, or to teach about eco-sustainable behaviors in the topic of climate change, which was found to foster
real world. learning and positive attitudes related to climate change.
In the literature taken into consideration, to the best
of our knowledge, there do not seem to be relevant
2.1. Pro-environment video games examples in terms of complex worlds and gameplay
inspired by mainstream 3D games that also employ large
In recent years, an increasing number of video games amounts of text. Some examples exist that try to achieve
have been used to increase users’ environmental this result in a 3D environment with heavy use of text, in
awareness. These include Terra Nil3, Donut County4 [21], the field of cyberbullying prevention [35], where
Factorio5, Arma 36 [22], and Animal Crossing: New offensive text (to be erased or changed by players) is
Horizons7 [20]. Despite being commercial video games, included in the form of messages from the characters or
many of these games present elements of environmental graffiti. It was important that the teaching came from
or biodiversity awareness simply because of the setting, virtual characters to merge it more naturally with the
storytelling, mechanics, or dynamics of the game. game environment. We looked at existing commercial
2 The full game can be downloaded here: https://osf.io/ 6 https://arma3.com/
eznjw/?view_only=9cf4572c4b1e41e3a39be622a9ab433d 7 https://animalcrossing.nintendo.com/
3 https://www.terranil.com/ 8 https://play.eco/
4 http://donutcounty.com/ 9 https://plasticitygame.wixsite.com/about
5 https://www.factorio.com/ 10 https://beesimulator.com/
160
games to find mechanics that could fit well with teaching the point of view from humans to animals and let players
eco-sustainability facts to foster behavior change. In the correct or condemn human behavior as an external
next sections, we discuss the inspiration for the game and observer. This choice was also dictated by the scarcity of
the design choices that were made during the perspective change examples in the literature concerning
development of Untitled Bee Game. serious games for sustainability, in the face of a relative
abundance of such an approach in entertainment games.
Moreover, the proposed perspective shift can be put in
2.3. Design frameworks for serious relation with recent literature that deals with the
games abandonment of anthropocentrism, in the context of a
posthuman turn that views ecology and climate change
Serious games and gamification present a more from a non-human/other-than-human standpoint [55,
complicated design procedure than video games, since 56, 57]. We also suspect that carrying out hostile in-game
they are designed with a clear serious purpose. For this actions as a human instead of an animal could potentially
reason, not all gameful systems and serious games are have negative educational consequences. The game
effective [36, 37]. For a purpose-consistent design and might be thought to suggest that it is advisable to be
effectiveness of these tools, numerous design hostile to peers who do not follow virtuous behavior.
frameworks and procedures have been developed and The core mechanic of Untitled Goose Game consists
adopted [38, 39, 40]. Design approaches can follow of playing tricks on, and stealing objects from, human
different perspectives: user-centered, technology- non-playing characters (NPCs) to disturb their daily
centered, game-centered, and context-centered [38, 40, activities. Albeit relying on such a simple mechanic, the
41]. Some of these methodologies present collections of game managed to sell one million copies in three months
game elements that help the designer keep track of game [58]. In addition, its simple, toon-shaded low-poly 3D
choices [42], while others are more complex graphics make for the ideal style for rapid prototyping
methodologies that provide more of a guideline for the and development.
design [29, 43], and also for the next steps [41]. Although there is a clear similarity with Bee Simulator
Many frameworks are based on developments of pre- (see Section 2.1), the mechanics did not play a particular
existing models: for instance, the works of [44] and [45] role in the design of UBG, although it may be used as a
trace their roots to the MDA model [46]. This model model in the future to add new features.
strives to amalgamate game design, development,
critique, and technical game research by presenting a
structural approach to understanding games. The
3.1. UBG game design
pioneering 6D model by [47] has inspired numerous
authors to create additional design frameworks (see [48, As a "game with a purpose", we carefully designed the
49, 50, 51]) or adapt it in different contexts (see [52, 53]). game from the contextual information and purpose. We
However, although possible design options are relied on the GamiDoc design framework [41] to write a
numerous, few design frameworks or procedures have game design document14 that would allow the team to
been used or studied consistently. Specifically, few communicate and at the same time, keep track of the
frameworks have been used to design tools promoting development in relation to the purpose. The game had no
behavior change or climate change awareness. Among specific target and was inspired by a simple, cartoon
the few frameworks for climate change awareness, in graphic style (used in Untitled Goose Game itself) that
[34], the authors specifically provide guidelines for could be found pleasurable across different genders and
serious games about climate change, including, among ages. The target of the experiments, however, was adults.
others, adapting game design to the characteristics of the The idea behind UBG is to explore the potential of
users, involving users actively, and presenting content in immersing players in a 3D virtual world inspired by recent
new ways to aid learning. commercial games, with NPCs reacting to the avatar’s
actions, to foster novel gameful interactions for learning
and positive behavior change.
3. Design and development UBG lets players control a bee in a small town (see
Figure 1). We selected a bee because of the importance
Untitled Bee Game (henceforth UBG) borrows its title and of bees in the environment [10]. Indeed, facts about bees
mechanics from Untitled Goose Game11, a 3D puzzle are part of the learning content of the game, which
game for computers and consoles. Untitled Goose Game players are quizzed on. This choice also brought about
is part of a rather recent video game genre that could be advantages under the technical aspect of the
defined as “animal mayhem games” [54]. These games let implementation, such as having to deal with simpler
players control animals that wreak havoc and punish animations. Imitating commercial games may imply
human beings, generally relying on humor and disruptive adopting control schemes that are a bit difficult to
game mechanics. Other such games are, for example, master. Therefore, at the beginning of the game, players
Goat Simulator12 and DEEEER Simulator13. This strand of are asked to select the control style they prefer: overhead
games seems appropriate to focus on the theme of camera or free camera. In both schemes, players control
sustainability and environmental preservation: they shift the bee in the third person, but in the latter, they can also
11 https://goose.game 14 The game design document can be retrieved here along with the
12 https://www.goatsimulator3.com/it documents used for the textual content:
13 https://playism.com/en/game/deeeer-simulator/ https://osf.io/eznjw/?view_only=9cf4572c4b1e41e3a39be622a9ab433d
161
rotate the camera (and the bee) freely in all directions. the virtual environment better and more lively. The
The overall objective is to dissuade the NPCs from mechanics of UBG are still rather simplistic, allowing as
pursuing certain behaviors. If correct actions are taken by many participants as possible to complete the
the player, the garden (a small area where flowers can experiment without too much effort. Apart from a couple
appear) and the river are influenced: the garden of side mechanics, the task is rather straightforward. We
produces flowers and more fish come to the river. discuss future implementations in Section 7.
3.2. Tools and resources
UBG was developed using Unity3D15, an industry-
standard environment in the field of video game
development. Some of the 2D resources and images were
found on Adobe Stock16, while others were created ad-
hoc. The 3D models were either found in the Unity Asset
Store17 and Quaternius.com18 or created using Blender 19,
a popular 3D modeling tool. The music was either created
Figure 1: The scenery of Untitled Bee Game. The ad hoc or found online for free20.
screenshot shows the small town that is explorable in All the text presented in the game and the quiz
the game. questions were created following an extensive literature
search on the topic. Following an initial analysis, the final
Two NPCs were implemented. The first one wastes data were selected on the basis of their association with
plastic bottles, while the second one uses polluting player and NPC behaviors within the game.
substances to wash her car. An icon above the head of
the NPCs shows the player what behavior is being
pursued. If it is a damaging behavior, players must sting 4. Methods
the NPC or drop rubbish on their head. Stinging can be
performed at any time, even if the behavior shown by the 4.1. Data collection method
NPC is already correct. Therefore, we differentiate
between two stinging actions the player can perform: Prior to deploying the game and administering it to
correct and incorrect stinging. The game plays different the participants, we created a REST API based on a
sounds accordingly. If correct stinging is performed, new Mongo21 database that communicated with the game
flowers appear in the garden and new fish appear in the during the sessions. For this reason, players were asked
river. Once a flower has appeared, players can interact to remain connected while playing. We collected the
with it and read facts about the three topics (see Figure following information in JSON format: i) a game ID; ii) the
2), one paragraph at a time, shown in a classic video game number of times each NPC was stung; iii) the number of
message box. Once a knowledge paragraph has been correct actions; iv) the number of incorrect actions; v) the
unlocked, it becomes possible to read it again in the quiz scores; and vi) the paragraph reading times.
pause menu.
As a side mechanic, players may also pick up and
recycle the rubbish they find. This grants them one 4.2. Participants and procedure
additional fish. In addition, rubbish can be thrown at
NPCs instead of stinging them to obtain the same effect. Players were mainly recruited in Italian university
If players do not do anything for a certain amount of time, facilities (University of Trento, University of Verona, and
fish start to disappear. University of Turin) and there were no exclusion criteria
except being underage. Recruitment requests were
circulated by two lecturers among bachelor students
primarily. They could choose between a browser-based,
a Mac, and a Windows version. They were both Italian
and English speaking and it was possible for them to
change the game language at the beginning of the game.
The procedure consisted of playing for approximately 20
minutes and responding to an online questionnaire. The
game assigned a random alphanumeric ID to each player
to be entered in the final questionnaire, which granted
complete anonymity. The procedure was employed in
Figure 2: The messages displayed by pollinated flowers. two phases: the usability test and the main experiment.
The main differences were (i) the version of the
It is worth noting that there is no game over and the prototype, and (ii) the content of the final questionnaire,
players’ actions are only driven by the possibility to make
15 http://www.unity.com/ 19 https://www.blender.org/
16 http://stock.adobe.com/ 20 https://audionautix.com/
17 https://assetstore.unity.com/ 21 https://www.mongodb.com/
18 https://quaternius.com
162
administered after the post-session quiz, as described identified as male), among which N=18 completed the
below. game, and N=24 completed the final questionnaire, of
Game session. Players could play a desktop or a web which N=14 completed both the game and the final
version of the game. After being shown their ID, they questionnaire. In this phase, players were administered a
were taken to the initial quiz (pre-test) where they could refined version of the prototype, where many things
express a choice among 4 to several questions related to were changed according to the previous round of
eco-sustainability to establish a baseline of knowledge feedback. In particular, the number of paragraphs to read
about sustainable behavior. This quiz established a and the number of quiz questions were both lowered to
baseline for each participant. After the initial quiz, players 12 in total. After playing the game, participants had to fill
had to complete a brief tutorial about the game out a questionnaire where they were asked to provide: (i)
commands and mechanics. Then they could start demographic information and gaming habits; (ii) an
exploring the environment and stinging the humans informal assessment of the degree of sustainability of
showing undesirable behavior. In so doing they earned a their behavior; (iii) engagement and value provided by
fish and a flower, and then had to pollinate a flower to the game by using the items from the GAMEFULQUEST,
reveal a new fact. After revealing a certain number of the IMI, and the MEEGA360 described above, and two
facts, players were taken to the second quiz (post-test) more items —Q5-"Thanks to the game, I am likely to
with an additional 4 questions on the same topics as the inquire about how to adopt other sustainable behaviors
initial one. in the future" and Q6-"Thanks to the things learned in the
The content and order of the questions in the two game, I am likely to adopt more sustainable behaviors in
quizzes were the same for all participants, both in the the future"— created ad-hoc to assess how much the
usability test and the experimental phase. Accordingly, game may have influenced users to become informed
the eco-sustainability facts were shown to all players in about environmental issues and adopt environmentally
the same order. The game informed players about sustainable behaviors in the future. All the materials were
whether their answer was correct or incorrect, and in the available both in English and Italian.
latter case it also displayed the correct one. However, the
pre-test and post-test quizzes presented radically
different questions to prevent users from answering 5. Results
correctly based on the previous answers.
Usability test design. To collect usability data, we We report here both the results from the usability test
used the System Usability Scale (SUS), which had to be (expressed mainly in terms of SUS scores and feedback
filled out at the end of the game. The choice of using this received) and the main experiment.
tool stems from its reliability and its widespread use for
usability studies [59]. Moreover, we added the item "Do 5.1. Usability test
you have any suggestions, feedback, comments that can
help us in improving the game?" to collect more detailed
The game’s SUS overall score (62.2) was below the set
information about possible improvements in the game.
threshold for a good level of usability (68), indicating
We recruited N=9 participants (female = 4; male = 5) and
marginal usability; hence, additional improvements were
had them play the first version of the prototype, where
needed [64]. For an exhaustive overview of SUS results,
they had to unlock 16 flowers and answer 16 quiz
see Table 1.
questions in total (8 pre-session and 8 post-session). Each
Questionnaire results. The results of the additional
participant tried the game through the web or desktop
items have identified that during this first application, not
version and answered the online questionnaire on
all users found the game completely entertaining (Q1 𝑀
Google Form22. Data in the literature [60] suggest that for
= 3.33, 𝑆𝐷 = 1.11, 𝑀𝑑𝑛 = 4). However, they recognized its
usability studies a sample size of 10 ± 2 is recommended
educational power and its importance (Q2 𝑀 = 3.44, 𝑆𝐷
to detect at least 80% of usability problems. During the
= 1.01, 𝑀𝑑𝑛 = 4; Q3 𝑀 = 3.66, 𝑆𝐷 = 1.11, 𝑀𝑑𝑛 = 4; Q4 𝑀
usability data collection, we still collected data on
= 3.88, 𝑆𝐷 = 0.78, 𝑀d𝑛 = 4).
playfulness, meaningfulness, and learning perception to
Feedback comments. At least one participant (U1)
check whether usability could in any way negatively
found that the game showed too much text. Therefore,
affect these aspects. Hence, we selected some items
we lowered the number of flowers to 12. This meant less
from reliable questionnaires: Q1-"This game gives me an
text to read (12 facts instead of 16) and less playtime.
overall playful experience" from the playfulness
Three participants (U3, U4, U8) mentioned that the
dimension of the GAMEFULQUEST [61], Q2-"I think this
objective was not completely clear or that the icons were
was an important activity" and Q3-"I believe this activity
difficult to interpret. A participant wrote that they would
has been beneficial to me" from the value/usefulness
have liked to know how important it was to read the
dimension of IMI [62], and Q4-"The game contributed to
flower facts (U1). This would have allowed them to
teach me some of the most relevant environmental
answer more correctly to the final quiz. As a solution, we
issues" from the perceived learning dimension of
added a summary of the objectives in the pause menu.
MEEGA360 [63]. We then interviewed the participants in
Another participant (U9) mentioned the difficulty of
a think-aloud session.
understanding interactions with fish within the game. We
Experimental design. We recruited N=28 participants
therefore decided to specify this more clearly. The
(age: 𝑀 = 26.62; 𝑆𝐷 = 8.80; 13 identified as female, 15
controls were a bit difficult for at least two participants
22 https://www.google.it/intl/it/forms/about/
163
(U3, U7). This led us to offer a choice to the players at the from the questionnaire, the change in quiz performance
beginning, asking which control scheme they preferred. in pre-test and post-test, and the feedback received at
However, one participant (U1) liked how they worked. the end.
Finally, one participant (U7) stated that the background
music was a bit annoying. This stresses the importance of
giving players an option menu to personalize the
5.2.1. Questionnaire results
experience.
All the analyses were conducted using RStudio23 (version
2023.03.0). Among the participants, N = 24 answered the
SUS Item Mean SD Mdn questionnaire. The results (Figure 3) show an overall
S1 2.00 0.86 2 positive evaluation of the tool. Answers were given on a
S2 (R) 2.00 1.11 2 5-point Likert-type scale ranging from strongly disagree
S3 3.22 1.20 3 to strongly agree (See Table 3). Most of the players found
S4 (R) 1.20 0.66 1 the game playful (63%); however, a few subjects thought
S5 3.22 0.97 3 the game was not playful, indicating the need to bring
S6 (R) 1.88 1.36 1 changes to the game in the future (12%). Interestingly,
S7 3.00 1.58 4 almost all the players found the game meaningful (96%
S8 (R) 2.55 1.50 2 and 83% to Q2 and Q3 respectively), probably because of
S9 3.33 1.41 4 the importance of the themes it dealt with, and they felt
S10 (R) 2.22 1.39 2 they learned something from the experience (Q4 positive
Overall SUS Score: 62.2/100 results = 88%). Lastly, 58% of the players reported the will
Table 1: System Usability Scale item means, standard to inquire (Q5) and then adopt other sustainable
deviations, and medians, as well as the final score. (R) behaviors in the future thanks to the game (Q6). We then
refers to negative items. ran a linear regression to analyze whether the expressed
playful level could be explained by users’ prior experience
Think-aloud session. After the usability testing, we with video games. The results show no correlation
interviewed the participants informally in a think-aloud between the two variables (𝐴𝑑𝑗𝑢𝑠𝑡𝑒𝑑𝑅² = −0.07, 𝐹1,12 =
session to gather additional feedback. There were several 0.14, 𝑝 > 0.05), suggesting that the game is equally
points that were raised in this session that had not perceived despite users’ experience with video games. To
emerged as part of the written comments. We assess whether the meaningfulness and playfulness of
summarize the main topics that emerged during the the game, and the perceived learning were conveyed by
think-aloud session in Table 2 as coded comments. The the amount of information read, we ran some linear
table includes both suggestions that were later regressions between the reading time and the
implemented into the game and those that were not, playfulness value (Q1), the interaction of the items
followed by an explanation. related to meaningfulness (Q2 and Q3), and the
The suggestion “Add flowers to encourage perceived learning (Q4). The results show that no
exploration” was not followed because the game only correlation occurred between the reading time and the
allows to unlock flowers in a dedicated area to optimize expressed game meaningfulness (𝐴𝑑𝑗𝑢𝑠𝑡𝑒𝑑𝑅² = 0.58,
the traveling time and the experiment duration, but will 𝐹1,12 = 2.54, 𝑝 > 0.05), and playfulness (𝐴𝑑𝑗𝑢𝑠𝑡𝑒𝑑𝑅² =
be considered in the future; “Add a deeper narrative” −1.82, 𝐹1,12 = 0.301, 𝑝 > 0.05). The analysis showed a
was not followed because, as mentioned, the game was significant relationship between reading time and
optimized to be completed in a reasonable amount of perceived learning (𝐴𝑑𝑗𝑢𝑠𝑡𝑒𝑑𝑅² = 1, 𝐹1,12 = 5.79, 𝑝 <
time, so we have not yet included a story behind the 0.001).
characters or the avatar, but it is a direction for future
Q6 8% 33% 58%
work.
Q5 17% 25% 58%
Coded comments Added
Add a progress bar for the quizzes Yes Q4 4% 8% 88%
Make tutorial more visual Yes Q3 4% 12% 83%
Include recycling mechanics Yes
Make it easier to unlock more fish Yes Q2 4% 0% 96%
Add flowers to encourage exploration No Q1 12% 25% 63%
Add a deeper narrative No 100 50 0 50 100
Percentage
Table 2: Suggestions from the think-aloud session, Response Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree
with an indication as to whether they were followed. Figure 3: An overview of the users’ answers to the 5-
point Likert-type items (Q1-Q6) in the questionnaire.
5.2. Experiment
What follows is a summary of the results obtained with
the main experiment, conducted after the usability test.
We proceed to describe and analyze the data obtained
23 https://posit.co/
164
Item Mean SD Mdn conducted a thematic analysis by coding all feedback
Q1 3.66 1.05 4 manually under macro-themes. We summarize the
Q2 4.37 0.71 4 comments here, grouped by theme. In Table 4, we
Q3 4.16 0.82 4 provide a detailed breakdown of the main comments
Q4 4.37 0.82 5 grouped by participants.
Q5 3.58 1.10 4
Q6 3.75 0.94 4 Coded comments Participant IDs
Table 3: The final questionnaire means, standard There should be additional areas E1, E2, E4
deviations and medians. Exploration is not encouraged E7
Text is too verbose E6, E11, E14, E16
Text is too formal E6, E10, E18
5.2.2. Quiz score improvement There should be a voiceover E3, E13, E16
The font is not very readable E18, E20, E24
Among the participants, N = 18 completed an entire game The game is fun/a nice experience E6, E11, E19
session, and did the post-test quizzes. To answer RQ2, we Playing as a bee is interesting E16
investigated the difference in scores between the pre- Stinging humans is fun/satisfying E9, E18
test and post-test quizzes. The post-test quiz obtained Input controls are easy E6
generally higher scores (𝑀 = 4.55; 𝑆𝐷 = 1.46; 𝑀𝑑𝑛 = 5) Input controls are difficult E10, E16, E17,
than pre-test one (𝑀 = 3; 𝑆𝐷 = 1.23; 𝑀𝑑𝑛 = 2.5). At first E19
glance, it was possible to see that only 5 players did not Table 4: Suggestions from the experimental session,
get an improvement, of which 4 maintained the same with an indication as to which participant provided them.
score, and only one had a lower score. We then analyzed
whether these differences in users’ answers before and T1. Repetitiveness: E1, E2, and E4 mentioned that
after the use of the game were significant. A Shapiro-Wilk they would have liked to have different areas to explore
test was conducted, indicating that the two distributions and to find flowers in. This mirrored what emerged
of both variables differed from the normal distribution during the usability testing. E7 also mentioned that the
(Pre: 𝑊 = 0.79, 𝑝 = 0.001; Post: 𝑊 = 0.83, 𝑝 = 0.005). positioning of the garden and the humans may be
Considering this result, we opted for a non-parametric exploited to take shortcuts as the mechanics do not
test. We then performed a one-tail Wilcoxon signed-rank encourage exploration. T2. Text style and length: many
test [65] (𝑉 = 11, 𝑝 < 0.005) and found a statistically users expressed that they would have liked to read less
meaningful disparity in the results at the in-game eco- or simplify the text. E6, E11, E14, and E16 found that the
sustainability questionnaire before and after playing the text was too verbose. E6, E10, and E18 stated that they
game. Effect size calculation (𝑟 = 0.49), which for the would have liked less formal language. T3. Voice over:
Wilcoxon signed-rank test is based on the formula 𝑟 = this theme is largely connected to Theme 2. In particular,
𝑍⁄√𝑛 [66], suggested a moderate magnitude of the E3, E13, and E16 suggested employing text-to-speech
effect [67]. The post-game results are not explained by techniques or voiceovers to decrease the amount of text
the subjects’ prior experience with video games to be read, especially for people with dyslexia. T4.
(𝐴𝑑𝑗𝑢𝑠𝑡𝑒𝑑𝑅² = 0.02, 𝐹1,12 = 1.32, 𝑝 > 0.05) or the flower Unclear goals: E8 stated that they found the stinging
reading time (𝐴𝑑𝑗𝑢𝑠𝑡𝑒𝑑𝑅² = 0.76, 𝐹1,16 = 4.49, 𝑝 > 0.05). context button to be somewhat difficult to activate or
interpret. E9, E14, and E19 asked for more succinct
Quiz Pre Post
indications as to what to do with the flower and the fish.
6 E19 suggested to include more visual cues. T5. Font
readability: flowers taught players their facts with a
pixelated font, which was chosen to be reminiscent of
old-school video games. In this context, however, it
4
Results
produced mixed results. E18, E20, and E24 had difficulties
reading the font and this made them want to skip the
messages (see Figure 2). T6. The fun in playing as a bee:
2 overall, the game was considered a nice experience by E6
and E19. E6 and E11 stated that the game was fun; E16
stated that they liked the concept of being a bee; E9 and
Pre Post
Quiz E18 explicitly stated that being mean to humans made
Figure 4: A boxplot showing the pre-test and post-test them happy or satisfied. T7. Mastery of controls:
scores on the eco-sustainability quizzes, taken at the although E6 explicitly stated that controls were easy to
beginning and at the end of the game session. There master, E10, E16, E17, and E19 expressed their difficulty
were 6 questions in each quiz. in mastering them. Finally, only 4 players did not report
any comments.
5.2.3. Feedback comments
6. Discussion and limitations
We were able to extract a substantial number of
comments from the feedback collected with the final We assessed the enjoyability and usefulness of UBG in
questionnaire, which helped us answer RQ3. We two phases: a usability testing phase and the main
165
experiment. All in all, the experience reported by users strictly towards reading and responding to quizzes, while
was positive, although there were some problems in the it could be turned into a more relaxed activity in future
usability phase. Comments were particularly useful in the versions. The second limitation is the limited number of
usability testing phase since usability appeared to be participants involved in the study. A greater sample could
suboptimal according to the SUS. Regarding the main be recruited or a follow-up knowledge retention test
experiment, we observed that there was indeed an could be introduced to further support the learning trend
increase in quiz performance, thanks to the knowledge observed. It could be argued that telling players about
presented in the game, which is an encouraging finding the final quiz may have encouraged them to read more
that contributes to answer RQ2. Interestingly, quiz carefully, thus influencing the final results. Future
performance is not explained by self-reported experience directions may include a between-subjects experiment,
with video games according to our analysis. This suggests where the treatment group does not know about the final
that potential problems are not likely to have influenced quiz.
learning. The level of reported playfulness does not seem
to be influenced by experience with video games. In this
respect, in addressing RQ1, we can say that in the end the 7. Conclusion and future work
prototype was quite enjoyable for every participant and
the difficulties with the controls did not invalidate the We developed a 3D serious game, mainly inspired by
experience. Untitled Goose Game, about eco-sustainability and
Several lessons can be learned by looking at the evaluated it with a usability test and an experiment.
feedback collected to answer RQ3: Although the usability test turned out suboptimal, the
1. Consider implementing a voiceover: The proposal feedback collected allowed us to deliver an overall
of removing some text and/or adding a voiceover was enjoyable experience in the following phase. Our quiz
shared by 29% (7) of participants. In general, reading was results coming from the main experiment indicate that it
fatiguing for players, and adding recorded speech or is indeed possible that players learn by means of such a
using text-to-speech techniques seems like the sweet game. Moreover, most users felt encouraged to learn
spot between teaching notions and not overwhelming more about eco-sustainability in the future. Some
players, although we suspect that learning could be participants even stated that controlling a bee and
influenced by the two different methods. Speaking of text stinging the humans was fun. However, the sample size
more in general, our findings mirror those of [34], where involved in the study was limited. Future work should
the authors suggest avoiding too much text and using investigate this tendency in larger samples and with a
instead more visual and impactful feedback. more advanced prototype that includes, among others, a
2. Consider individual experiences: Four players narrative and more exploration. Future work may also
thought that either the controls were difficult to master take advantage of more comprehensive questionnaires.
or that they could be improved, while one player found Recommendations: The insights gained from UBG
them easy to master. This underlines the importance of allow us to state some recommendations for future
considering not only all the possible peripheral games on sustainability awareness. First, as noted in the
configurations but also all the possible levels of literature [30], the design procedure matters: the
experience with video games. Choosing the right game effectiveness of a game is not universal. Design
genre is key: some games, like first-person shooters, for procedures should follow specific frameworks or
example, require very specific skills such as aiming with procedures, and design choices should fit the game's
the mouse. purpose and context (see [41] and [68]). Second, as
3. Too much freedom may lead to unclear goals: We suggested in the literature [69], fun can convey the
obtained 24 complete questionnaires in the face of 18 effectiveness of serious games and gameful systems.
completed game sessions. This fact is likely to be related Thus, it is necessary to analyze the context in which the
in some way to T4 (Unclear goals), but it could also be game will be used and the target users to make the game
influenced by the fact that the game, albeit giving fun. Third, interaction is essential in the learning process.
instructions, leaves players rather free to explore. We As our data suggests, learning through interaction in play
therefore recommend future endeavors in serious games is more fun than text-based learning, especially avoiding
to consider the aspect of directing players to the goals in repetition of patterns. It is therefore necessary to
the most straightforward and clear way possible. Finally, identify a method to make the way information is shown
5 players explicitly stated that either the game was a to players entertaining. Data in the literature suggest
good experience or that controlling a bee and stinging the limiting repetition and giving space to novelty [42].
humans was fun. This contributes to addressing RQ1 and Serious games are developed with specific purposes in
suggests that being mean to humans can be perceived as mind; hence, it is essential to include certain paths to
satisfactory. follow, to prevent users from getting lost, clearly defining
There are several limitations to the present study. the objectives of the game, and at the same time, leaving
The serious game employed is still in an early version and some freedom for users. Lastly, as suggested by the data
lacks a number of features that could make the in the literature [70] and player comments, consider
experience more enjoyable (e.g., a more complex using narrative and storytelling to accompany the player
narrative and high-level goals that go beyond the stinging along the way.
task alone). Since one of the aims of the study was to
assess the educational potential of the game, the
instructions and the mechanics guided players rather
166
travel behavioural change, Cities 85 (2019) 140–
Acknowledgements 149.
[16] C. Y. Chow, R. R. Riantiningtyas, M. B. Kanstrup, M.
We acknowledge the support of the PNRR project FAIR -
Papavasileiou, G. D. Liem, A. Olsen, Can games
Future AI Research (PE00000013), under the NRRP MUR
change children’s eating behaviour? A review of
program funded by the NextGenerationEU.
gamification and serious games, Food Quality and
Preference 80 (2020) 103823.
References [17] D. Johnson, S. Deterding, K.-A. Kuhn, A. Staneva, S.
Stoyanov, L. Hides, Gamification for health and
[1] S. Oskamp, A sustainable future for humanity? How wellbeing: A systematic review of the literature,
can psychology help?, American Psychologist 55 Internet interventions 6 (2016) 89–106.
(2000) 496. [18] T. Wolf, Green gamification: How gamified
[2] S. Celik, The effects of climate change on human information presentation affects pro-
behaviors, Environment, climate, plant and environmental behavior., in: GamiFIN, 2020, pp.
vegetation growth (2020) 577–589. 82–91.
[3] O. S. Alimi, J. Farner Budarz, L. M. Hernandez, N. [19] F. Xu, F. Tian, D. Buhalis, J. Weber, H. Zhang,
Tufenkji, Microplastics and nanoplastics in aquatic Tourists as mobile gamers: Gamification for tourism
environments: aggregation, deposition, and marketing, Journal of Travel & Tourism Marketing
enhanced contaminant transport, Environmental 33 (2016) 1124–1142.
science & technology 52 (2018) 1704–1724. [20] S. Coroller, C. Flinois, Video games as a tool for
[4] A. Unep, I. R. R. ASSESSMENT, The rise of ecological learning: the case of Animal Crossing,
environmental crime, Nairobi: UNEP (2016). Ecosphere 14 (2023) e4463.
[5] S. A. Mousavi, F. Khodadoost, Effects of detergents [21] B. Nicoll, Enjoyment in the Anthropocene: the
on natural ecosystems and wastewater treatment extimacy of ecological catastrophe in Donut
processes: a review, Environmental Science and County, Distinktion: Journal of Social Theory (2023)
Pollution Research 26 (2019) 26439–26448. 1–19.
[6] J. R. Barker, D. T. Tingey, Air pollution effects on [22] B. Abraham, Video game visions of climate futures:
biodiversity, Springer Science & Business Media, Arma 3 and implications for games and persuasion,
2012. Games and Culture 13 (2018) 71–91.
[7] F. Hölker, C. Wolter, E. K. Perkin, K. Tockner, Light [23] R. Buday, T. Baranowski, D. Thompson, Fun and
pollution as a biodiversity threat, Trends in ecology games and boredom, GAMES FOR HEALTH:
& evolution 25 (2010) 681–682. Research, Development, and Clinical Applications 1
[8] R. Sordello, O. Ratel, F. Flamerie De Lachapelle, C. (2012) 257–261.
Leger, A. Dambry, S. Vanpeene, Evidence of the [24] K. S. Fjællingsdal, C. A. Klöckner, Gaming green: the
impact of noise pollution on biodiversity: A educational potential of eco–a digital simulated
systematic map, Environmental Evidence 9 (2020) ecosystem, Frontiers in psychology 10 (2019) 2846.
1–27. [25] M. Csikszentmihalyi, Flow, Munksgaard
[9] S. G. Potts, V. Imperatriz-Fonseca, H. T. Ngo, M. A. København, 1991.
Aizen, J. C. Biesmeijer, T. D. Breeze, L. V. Dicks, L. A. [26] M. E. Larreina-Morales, C. Gunella, Video games for
Garibaldi, R. Hill, J. Settele, et al., Safeguarding environmental awareness: the accessibility gap,
pollinators and their values to human well-being, Universal Access in the Information Society (2023)
Nature 540 (2016) 220–229. 1–13.
[10] V. Patel, N. Pauli, E. Biggs, L. Barbour, B. Boruff, [27] V.-L. Bekoum Essokolo, E. Robinot, «let’s go deep
Why bees are critical for achieving sustainable into the game to save our planet!» how an
development, Ambio 50 (2021) 49–59. immersive and educational video game reduces
[11] T. J. Wood, I. Kaplan, Z. Szendrei, Wild bee pollen psychological distance and raises awareness,
diets reveal patterns of seasonal foraging resources Sustainability 14 (2022) 5774.
for honey bees, Frontiers in Ecology and Evolution [28] M. Stanitsas, K. Kirytopoulos, E. Vareilles,
6 (2018) 210. Facilitating sustainability transition through serious
[12] S. Bassanelli, N. Vasta, A. Bucchiarone, A. Marconi, games: A systematic literature review, Journal of
Gamification for behavior change: A scientometric cleaner production 208 (2019) 924–936.
review, Acta Psychologica 228 (2022) 103657. [29] D. Fernández Galeote, J. Hamari, Game-based
[13] A.-R. Seidler, C. Henkel, M. Fiedler, J. Kranz, A. climate change engagement: analyzing the
Ixmeier, K. S. Strunk, Promoting eco-sustainable potential of entertainment and serious games,
behavior with gamification: An experimental study Proceedings of the ACM on Human-Computer
on the alignment of competing goals., in: ICIS, 2020. Interaction 5 (2021) 1–21.
[14] M. Ferron, E. Loria, A. Marconi, P. Massa, Play&go, [30] K. Madani, T. W. Pierce, A. Mirchi, Serious games on
an urban game promoting behaviour change for environmental management, Sustainable Cities
sustainable mobility, Interaction Des. Archit. J 40 and Society 29 (2017) 1–11. doi:https:
(2019) 24–25. //doi.org/10.1016/j.scs.2016.11.007.
[15] B. T. Yen, C. Mulley, M. Burke, Gamification in [31] B. Orland, N. Ram, D. Lang, K. Houser, N. Kling, M.
transport interventions: Another way to improve Coccia, Saving energy in an office environment: A
serious game intervention, Energy and Buildings 74
167
(2014) 43–52. doi:https: [44] A. C. T. Klock, I. Gasparini, M. S. Pimenta, 5w2h
//doi.org/10.1016/j.enbuild.2014.01.036. framework: a guide to design, develop and evaluate
[32] A. Diniz dos Santos, F. Strada, A. Bottino, the user-centered gamification, in: Proceedings of
Approaching sustainability learning via digital the 15th Brazilian Symposium on Human Factors in
serious games, IEEE Transactions on Learning Computing Systems, 2016, pp. 1–10.
Technologies 12 (2019) 303–320. doi:10.1109/ [45] A. Mora, P. Zaharias, C. González, J. Arnedo-
TLT.2018.2858770. Moreno, Fraggle: a framework for agile
[33] E. Knol, P. De Vries, Enercities: Educational game gamification of learning experiences, in:
about energy, in: Central Europe towards International Conference on Games and Learning
Sustainable Building ’10 (CESB10)’, 2010. Alliance, Springer, 2015, pp. 530–539.
[34] D. Fernández Galeote, N.-Z. Legaki, J. Hamari, [46] R. Hunicke, M. LeBlanc, R. Zubek, Mda: A formal
Climate Connected: An immersive VR and PC game approach to game design and game research, in:
for climate change engagement, in: Companion Proceedings of the AAAI Workshop on Challenges
Proceedings of the Annual Symposium on in Game AI, volume 4, San Jose, CA, 2004, p. 1722.
Computer-Human Interaction in Play, ACM, [47] D. Hunter, K. Werbach, For the win, Wharton digital
Stratford, ON, Canada, 2023, pp. 266–273. URL: press, 2012.
https://dl.acm.org/doi/10.1145/3573382.3616053 [48] A. C. Tomé Klock, L. F. d. Cunha, M. F. d. Carvalho,
. doi:10.1145/3573382.3616053. B. Eduardo Rosa, A. Jaqueline Anton, I. Gasparini,
[35] F. Bonetti, S. Tonelli, An analysis of abusive Gamification in e-learning systems: A conceptual
language data collected through a game with a model to engage students and its application in an
purpose, in: Proceedings of the 9th Workshop on adaptive e-learning system, in: International
Games and Natural Language Processing within the Conference on Learning and Collaboration
13th Language Resources and Evaluation Technologies, Springer, 2015, pp. 595–607.
Conference, European Language Resources [49] K. Julius, J. Salo, Designing gamification, Marketing
Association, Marseille, France, 2022, pp. 1–6. (2013).
[36] J. Koivisto, J. Hamari, The rise of motivational [50] R. Schönen, Gamification in change management
information systems: A review of gamification processes, Munich University, Munich. Bachelor
research, International Journal of Information Thesis (2014).
Management 45 (2019) 191–210. [51] J. Brito, V. Vieira, A. Duran, Towards a framework
[37] P. Backlund, M. Hendrix, Educational games-are for gamification design on crowdsourcing systems:
they worth the effort? A literature survey of the The game approach, in: 2015 12th International
effectiveness of serious games, in: 2013 5th Conference on Information Technology-New
international conference on games and virtual Generations, IEEE, 2015, pp. 445–450.
worlds for serious applications (VS-GAMES), IEEE, [52] A. Yusof, N. A. Atan, J. Harun, S. Salleh, N. Khamis,
2013, pp. 1–8. Gamified hybrid service-learning for students
[38] A. Mora, D. Riera, C. González, J. Arnedo-Moreno, engagement in computer network course
Gamification: a systematic review of design (GAMYSEL), in: Proceedings of the International
frameworks, Journal of Computing in Higher university carnival on e-learning (IUCEL) 2022,
Education 29 (2017) 516–548. 2022, pp. 502–506.
[39] D. Ávila-Pesántez, L. A. Rivera, M. S. Alban, [53] S. Gomez-Jaramillo, J. Moreno-Cadavid, C. M.
Approaches for serious game design: A systematic Zapata-Jaramillo, Adaptation of the 6d gamification
literature review., Computers in education journal model in a software development course, in: 2018
8 (2017). XIII Latin American Conference on Learning
[40] D. Fernández Galeote, M. Rajanen, D. Rajanen, N.- Technologies (LACLO), IEEE, 2018, pp. 85–88.
Z. Legaki, D. J. Langley, J. Hamari, Gamification for [54] Caracciolo, Marco, Animal mayhem games and
climate change engagement: A user-centered nonhuman-oriented thinking, GAME STUDIES
design agenda, in: Proceedings of the 26th (COPENHAGEN) 21 (2021) 16.
International Academic Mindtrek Conference, [55] O. Cielemęcka, C. Daigle, Posthuman sustainability:
2023, pp. 45–56. an ethos for our anthropocenic future, Theory,
[41] S. Bassanelli, A. Bucchiarone, Gamidoc: A tool for Culture & Society 36 (2019) 67–87.
designing and evaluating gamified solutions, in: doi:10.1177/0263276419873710.
Extended Abstracts of the 2022 Annual Symposium [56] C. Mengozzi, Outside the anthropological machine:
on Computer-Human Interaction in Play, 2022, pp. crossing the human-animal divide and other exit
203–208. strategies, Routledge, 2020.
[42] Y.-k. Chou, Actionable gamification: Beyond points, [57] L. Bernaerts, M. Caracciolo, L. Herman, B. Vervaeck,
badges, and leaderboards, Packt Publishing Ltd, The storied lives of non-human narrators, Narrative
2019. 22 (2014) 68–93. doi:10.1353/nar.2014.0002.
[43] B. Morschheuser, J. Hamari, K. Werder, J. Abe, How [58] Forbes, Indie darling ‘Untitled Goose Game’ clears
to gamify? A method for designing gamification, in: one million copies sold in three months, 2019. URL:
Proceedings of the 50th Hawaii International https://www.forbes.com/sites/lizlanier/2020/12/
Conference on System Sciences 2017, University of 31/indie-darling-untitled-goose-game-clears- one-
Hawai’i at Manoa, 2017. million-copies-sold-in-three-months/?sh=
2f1695b84884.
168
[59] J. R. Lewis, The system usability scale: past, present,
and future, International Journal of Human–
Computer Interaction 34 (2018) 577–590.
[60] W. Hwang, G. Salvendy, Number of people required
for usability evaluation: the 10±2 rule,
Communications of the ACM 53 (2010) 130–133.
[61] J. Högberg, J. Hamari, E. Wästlund, Gameful
experience questionnaire (GAMEFULQUEST): an
instrument for measuring the perceived
gamefulness of system use, User Modeling and
User-Adapted Interaction 29 (2019) 619–660.
[62] E. McAuley, T. Duncan, V. V. Tammen,
Psychometric properties of the intrinsic motivation
inventory in a competitive sport setting: A
confirmatory factor analysis, Research quarterly for
exercise and sport 60 (1989) 48–58.
[63] F. Gini, S. Bassanelli, The relationship between fun
and the overall score of the MEEGA360 scale, 2022.
(in press).
[64] A. Bangor, P. Kortum, J. Miller, Determining what
individual sus scores mean: Adding an adjective
rating scale, Journal of usability studies 4 (2009)
114–123.
[65] R. F. Woolson, Wilcoxon signed-rank test, Wiley
encyclopedia of clinical trials (2007) 1–3.
[66] C. O. Fritz, P. E. Morris, J. J. Richler, Effect size
estimates: current use, calculations, and
interpretation., Journal of experimental
psychology: General 141 (2012) 2.
[67] M. Tomczak, E. Tomczak, The need to report effect
size estimates revisited. an overview of some
recommended measures of effect size, TRENDS in
Sport Sciences (2014).
[68] K. Kölln, Maybe we don’t need a new gamification
framework after all, in: Extended Abstracts of the
2022 Annual Symposium on Computer-Human
Interaction in Play, 2022, pp. 384–387.
[69] G. Tisza, The role of fun in learning, in: Extended
Abstracts of the 2021 Annual Symposium on
Computer-Human Interaction in Play, 2021, pp.
391–393.
[70] P. Palomino, A. Toda, W. Oliveira, L. Rodrigues, A.
Cristea, S. Isotani, Exploring content game
elements to support gamification design in
educational systems: narrative and storytelling, in:
Brazilian symposium on computers in education
(Simpósio brasileiro de informática na educação-
SBIE), volume 30, 2019, p. 773.
169