<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta>
      <journal-title-group>
        <journal-title>April</journal-title>
      </journal-title-group>
    </journal-meta>
    <article-meta>
      <title-group>
        <article-title>gameful experience dimensions associated with intrinsic and extrinsic motivation? Results from an online vignette study</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Eetu Wallius</string-name>
          <email>eetu.wallius@tuni.fi</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Ana C.T Klock</string-name>
          <email>ana.tomeklock@tuni.fi</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Juho Hamari</string-name>
          <email>juho.hamari@tuni.fi</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Gamification Group, Tampere university</institution>
          ,
          <addr-line>Kalevantie 4, 33100, Tampere</addr-line>
          ,
          <country country="FI">Finland</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2023</year>
      </pub-date>
      <volume>1</volume>
      <fpage>8</fpage>
      <lpage>21</lpage>
      <abstract>
        <p>Gamification is one of the leading behavioral change strategies whose central premise is to afford experiences that resemble those present in games to provide intrinsic motivation. At the same time, gamification has an underlying utilitarian purpose and therefore its use is additionally driven by expected benefits. However, while gamification designs can afford a wide range of experiences, the corpus needs studies that examine which of these experiences induce intrinsic motivation, and which are associated with extrinsic motivation related to the utilitarian outcomes. Therefore, this study deploys a vignette-based online study (n=937) to examine the relationships between gameful experience dimensions (i.e., social playfulness), and intrinsic and extrinsic motivation in a case study related to social distancing during the COVID-19 pandemic. Based on the results, accomplishment, social experience, competition, and playfulness Gamification, gameful experience, intrinsic motivation, extrinsic motivation CEUR Workshop Proceedings (CEUR-WS.org)</p>
      </abstract>
      <kwd-group>
        <kwd>accomplishment</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>are
positively
associated
with intrinsic
motivation. Moreover, accomplishment, social experience, guided, and playfulness are
positively associated with extrinsic motivation.</p>
    </sec>
    <sec id="sec-2">
      <title>1. Introduction</title>
      <p>During the last decade, gamification has
become
technology,
prominent type
whose
purpose
of
is
motivational
to
induce
behavioral change that serves user goals or
benefits a larger collective of individuals [1]. To
achieve
change
in
individual
behavior,
gamification uses game design in contexts and
environments that are not normally considered to
be games and has received substantial attention
among practitioners and academia since its outset
[1]. Some of the domains where gamification has
been demonstrated to induce positive behavioral</p>
      <p>
        2020 Copyright for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
change include health, exercise, safety, and
security, among others [
        <xref ref-type="bibr" rid="ref18 ref27 ref45 ref54">1–4</xref>
        ].
      </p>
      <p>Gamification builds on the notion of gameplay
being an activity in which people are willing to
invest effort despite receiving no tangible benefits
or experiencing external pressure to do so [5].
Accordingly, central premise of gamification is to
promote
experiences
experienced
behavioral</p>
      <p>change
that
when
resemble
playing</p>
      <p>by
those
providing
that</p>
      <p>
        are
games, including
competition, a sense of achievement, or becoming
immersed in the game environment, among others
[
        <xref ref-type="bibr" rid="ref4">6,7</xref>
        ]. Ideally, by affording one or more gameful
experiences, gamification can induce intrinsic
motivation, transforming the activity that is being
gamified so that individuals derive pleasure from
engaging in it, while leading to sustained behavior
change [8]. Gamification, therefore, is a type of
hedonic-utilitarian system design, whose use is
driven by the pleasures and expected utilitarian
benefits it affords [9].
      </p>
      <p>While gamification draws inspiration from
game design, and the experiences games and
gamified systems provide resemble one another,
the underlying utilitarian purpose of gamification
clearly distinguishes it from games, which are
usually exclusively autotelic systems [1,6,10].
Therefore, the experiences gamification provides
do not solely serve towards enjoyment and
intrinsic motivation but may relate to the expected
benefits or other goals external to the gamified
system or activity - i.e., extrinsic motivation [6].
Prior research has also found that in some cases
gamification can work similarly to extrinsic
incentives [11,12]. In such cases, gamification
poses a risk of hampering intrinsic need
satisfaction and intrinsic motivation [13].
Although extrinsically motivating gamification
can indeed lead to behavioral change it might only
be effective in the short term, and deteriorate
performance quality [11,14].</p>
      <p>Despite the ability of gamification to foster
intrinsic motivation is one of the central reasons
for why it works, and studies have examined how
different gamification designs foster intrinsic
motivation [15], no existing studies have
comprehensively examined which of the various
experiences afforded by gamification can
transform behaviors to become intrinsically
motivated. On the other hand, we also lack an
understanding of the experiential qualities of
gamification that relate to extrinsic motivation. To
address these gaps, we examine the relationships
between the seven gameful experience
dimensions of accomplishment, challenge, social
experience, immersion, competition, guided and
playfulness as identified by [6], and intrinsic and
extrinsic motivation [16]. We use gamified social
distancing (i.e., maintaining a physical distance
from other individuals to keep themselves and
others safe) during the COVID-19 pandemic as a
case study, while deploying data from a
vignettebased online survey.</p>
    </sec>
    <sec id="sec-3">
      <title>2. Background</title>
    </sec>
    <sec id="sec-4">
      <title>2.1. Intrinsic motivation and extrinsic</title>
      <p>Self-Determination Theory (SDT) is one of the
prominent theories explaining the human
motivation to engage in various activities [17].
SDT is also the most widely used theory in the
corpus of gamification research, although
applying it to explain the motivation stemming
from games and gamification is not without
criticism [18,19].</p>
      <p>According to the SDT, activities can be
extrinsically or intrinsically motivated [17].
Whereas the former is predominantly performed
for consequences that are separable from the
activity, such as expected benefits, punishment
avoidance or social recognition, the latter is
pursued for the satisfaction inherent to the activity
itself. In other words, if a behavior is extrinsically
motivated, individuals do it for an outcome that is
external. However, if behaviors are intrinsically
motivated, they become something individuals
‘want to do’ for the sake of doing it [20].
Although both intrinsic and extrinsic motivation
drive human behavior, prior studies have shown
that intrinsic motivation is a more prominent
predictor of sustained behavioral change [21].</p>
      <p>SDT posits that the satisfaction of innate
psychological needs explains why some activities
are intrinsically motivated. These basic needs
include relatedness (i.e., feeling connected to
other people, and caring for them), competence
(i.e., feeling of mastery), and autonomy (i.e.,
feeling of volition and freedom) [17]. The
satisfaction of these three needs during an activity
is likely to make that given activity become
enjoyable, consequently supporting individual
well-being and sustained engagement [17,20]. For
example, gameplay is likely to offer the player
choices over actions and tasks (i.e., autonomy),
provide challenges and positive feedback (i.e.,
competence), and facilitate interactions with other
players in multiplayer environments (i.e.,
relatedness), therefore providing an intrinsically
motivating experience [5].
2.2.</p>
    </sec>
    <sec id="sec-5">
      <title>Gameful experience</title>
      <p>Researchers have increasingly pursued to
understand the unique characteristics of the
experiences that stem from playing games [22].
However, within gamification research, the
gameful experience has remained an
underdeveloped concept [6]. Moreover, despite
the gameful experiences being subjective, and
dependent on the characteristics of the individual
as well as the gameful system, only a few attempts
have been made to develop instruments for
measuring the gameful experience [6,23].</p>
      <p>Due to the diversity in different types of games
and gamified systems alike, it has been argued
that the gameful experience is not a single type of
experience but comprises multiple dimensions. In
[6], the authors built on prior research on game
studies and insights from an empirical inquiry into
the experiences of gamification users. The authors
identified seven dimensions comprising the
gameful experience and developed an instrument
for measuring each dimension. Although the
seven-dimensional model of the gameful
experience still lacks validation in different
contexts and across different types of
gamification designs, to date it remains the most
comprehensive inquiry that aims to holistically
capture these different dimensions, while also
providing an instrument for measuring each
dimension. Therefore, we decided to adopt the
model and instrument from [6] for the purposes of
this study.</p>
      <p>Social experience relates to creating and
maintaining social relations through gameplay
[6]. One of the characteristics of games that make
them appealing is that gameplay is, in many cases,
a predominantly social activity, whether related to
collaborating with other players to achieve a
common goal or maintaining existing and creating
new social relations through chatting and
networking [24]. Accordingly, social experiences
induced by gamification can take a variety of
different forms, including for example, teamwork,
socializing, and other ways to foster
connectedness with others [6]. Social experience
is, therefore, likely to tap into the need for
relatedness of the SDT and induce intrinsic
motivation, but the collaboration and teamwork
associated with social experience might also
increase social pressure to behave in a desired
manner, and therefore serve a mainly utilitarian
function [17].</p>
      <p>Competition can stem from competitiveness
against others or the gamified system [6]. In many
cases, competition emerges because of a social
situation where the user aims to overcome their
opponents. Competition can be induced using
gamification elements, such as leaderboards or
ranking that enable individuals to compare their
performance with other users and strive for
outperforming them. Competition can be seen as
a social activity, which can be expected to tap into
the relatedness need of the SDT. However,
existing studies in other domains have shown that
competition can also foster extrinsic motivation,
depending on the characteristics of the individual
[25]. Moreover, some studies have argued that the
requisite for motivational benefits is that
competition ought to be fair and that those
competing should be at a relatively even
performance level, and that its main motivational
mechanism might be social pressure [25,26].</p>
      <p>Immersion is an experience that is
characterized by being absorbed into the game
environment or alternate reality [6]. Immersive
experiences might cause a feeling of detachment
from the “real world”, losing the sense of self, and
the time passing quickly as it consumes the user's
attention [27]. Immersion can be facilitated by
creating a compelling storyline, game characters,
and a game world that enable the user to feel that
they are being involved in a self-directed activity,
while their actions have consequences – i.e.
“being part of the story” [5,28]. Prior studies have
shown that immersion-oriented gamification is
associated with autonomy need satisfaction [28].</p>
      <p>Playfulness refers to a non-serious, carefree
approach toward activities and not being confined
by goals or rules [29]. It relates to being involved
in voluntary and pleasurable settings while
leaving room for exploration, spontaneity, and
behaviors driven by imagination [6]. The
autonomous nature of playfulness resonates with
the autonomy need satisfaction of the SDT, while
its rule-free nature can be seen to contrast with
extrinsic regulation and motivations driven by
utility [17].</p>
      <p>
        Challenge, referring to the effort players need
to invest to achieve the gameplay goals, has been
argued to be one of the characteristics that define,
and make games enjoyable [
        <xref ref-type="bibr" rid="ref40">30</xref>
        ]. The experience
of the challenge stems from overcoming obstacles
and skill-building related to a progressing
difficulty level [6]. Challenge, therefore,
resonates with the competence need satisfaction
of intrinsic motivation [6]. Prior research has,
however, shown that for example gamification
that uses level-based progression to challenge the
user might lead to extrinsic motivation, while
challenges can become demotivating if they
require too much effort or are disproportionately
difficult to overcome [11,31].
      </p>
      <p>Accomplishment relates to goals and
completing tasks that are imposed by the gamified
system [6]. Accomplishment stems from
skillprogression, advancement and successfully
overcoming obstacles imposed by the gameful
system or those present in the real-world [6].
Similar to challenge, accomplishment resonates
with the need for competence, while tasks and
goals being induced by a gamified system can
serve as external motivators.</p>
      <p>Guided relates to the experience of being
helped by the gamification design in structuring
one’s tasks and sticking to plans [6]. Hence,
guided mainly relates to the user goals outside of
the gameful system and differs from the other
dimensions of the gameful experience as it does
not relate to hedonic aspects of using gamified
systems, but instead has a utilitarian focus,
allowing users to better achieve their goals that are
external to the activity [6]. However, prior studies
have shown that providing clear structure and
expectations regarding an activity supports
autonomy and, consequently, intrinsic motivation
[32]. Conversely, if the guidance stemming from
a gameful system is perceived as coercive, or if
the behaviors that are being promoted are not
aligned with individual goals, being guided poses
a risk of hampering autonomy and, consequently,
intrinsic motivation.</p>
      <p>While the literature supports the argument that
gamification designs can foster intrinsic and
extrinsic motivation [11–13,28], prior research
has not comprehensively investigated which
dimensions of the gameful experience lead to each
type of motivation. Different individuals may
experience gamification designs differently, and
therefore prior studies examining the relationships
between different types of gamification and
motivation do not provide direct evidence of the
relationships between different types of gameful
experiences and motivations per se. Therefore,
this study explores the associations between each
gameful experience dimension and both intrinsic
and extrinsic motivation.</p>
    </sec>
    <sec id="sec-6">
      <title>3. Materials and methods</title>
    </sec>
    <sec id="sec-7">
      <title>3.1. Procedure</title>
      <p>We conducted a vignette-based online
study to explore the relationships between the
different dimensions of the gameful experience,
and intrinsic and extrinsic motivation. In a
vignette study, the participants are presented with
a hypothetical situation to elicit their perceptions
and beliefs. This method allows effective
manipulation of the study stimulus, and data
collection from a large number of participants
[33]. We chose the vignette method as it allows
using distinct gamification designs to create
distinct gameful experiences in the chosen
context, which might not be easily achieved using
other methods such as surveys.</p>
      <p>We used social distancing as the context for
our study. During the COVID–19 pandemic,
many countries opted to limit and reshape
individual mobility patterns, most notably by
encouraging engagement in social distancing, i.e.,
keeping a distance from others in day-to-day life
to contain the spread of the virus and keep
individuals safe. We created five interactive user
interfaces to represent different ways to gamify
such social distancing behavior, each based on the
idea of tracking ‘contacts’ (i.e., situations where
the user comes within a 2-meter distance of
another user, thus causing a risk of infection) that
serve as input for gamification (see appendices
15).</p>
      <p>Gamification implementations are commonly
categorized into achievement and
progressionbased, social (including competitive) and
immersive designs, and the types of gamification
applied in this study were designed to correspond
to these categories to comprehensively create
distinct gameful experiences [1]. Version 1 uses
achievement and progression-based gamification
features, allowing users to gain badges and levels
for reaching social distancing goals they set
themselves while visualizing their social
distancing behaviors using a progress bar and
statistics in terms of e.g., contactless days, and the
average number of contacts. Version 2 allows the
user to compete with friends in having the least
contacts each day, week, and month, while using
a leaderboard. The version also displays a podium
of the top three social distancers for each period
with awards. Version 3 allows users to form a
virtual city with their friends and keep this city
safe by social distancing and sending others social
support in the form of emojis. Version 4 immerses
the user into an alternate storyline from Lord of
the Rings with the purpose of reframing social
distancing behavior as an adventure, allowing the
user to role-play as Frodo. The interface reveals a
new part of the storyline to the user at the end of
each day and whether Frodo stayed safe on his
journey to Mount Doom depended on whether the
user reached the social distancing goal they set
themselves. The gameful motivational
affordances implemented in each version are
depicted in Table 1. Version 5 is the non-gamified
version that displays the number of daily contacts
to the user without employing gamification.</p>
      <sec id="sec-7-1">
        <title>Version 1</title>
      </sec>
      <sec id="sec-7-2">
        <title>Version 2</title>
      </sec>
      <sec id="sec-7-3">
        <title>Version 3</title>
      </sec>
      <sec id="sec-7-4">
        <title>Version 4</title>
      </sec>
      <sec id="sec-7-5">
        <title>Non-gamified</title>
      </sec>
      <sec id="sec-7-6">
        <title>Levels, progress visualization, badges, performance statistics, goals</title>
      </sec>
      <sec id="sec-7-7">
        <title>Competition, leaderboard, ingame rewards</title>
      </sec>
      <sec id="sec-7-8">
        <title>Networking, common goals, cooperation</title>
      </sec>
      <sec id="sec-7-9">
        <title>Narrative, role-play, goals</title>
      </sec>
      <sec id="sec-7-10">
        <title>None</title>
        <p>Each participant was randomly assigned one
version of the interactive interface and a video that
described how the corresponding application
would work in everyday life (i.e., at work, in the
park, at the bus stop) during a pandemic. After
familiarizing themselves with the application
through the interface and the video, the
participants were asked to imagine themselves
using the application during a pandemic and
evaluate it in terms of gameful experience,
intrinsic motivation, and extrinsic motivation.
Thus, gameful experience and intrinsic and
extrinsic motivation were solely based on these
different designs presented to the participants.</p>
        <p>The data analysis was conducted using
SmartPLS 4, where a factor weighing scheme was
used to assess the reliability and validity
measures, and a path weighing scheme with 5000
subsamples bootstrapping for assessing the path
model. Gameful experiences are subjective, and
the premise is that different gameful and even
non-gamified designs can in some cases induce
overlapping experiences. Therefore, we analyzed
the responses based on all gamification versions
(including non-gamified) together.
3.2.</p>
      </sec>
    </sec>
    <sec id="sec-8">
      <title>Measurement</title>
      <p>Expected intrinsic motivation was measured
using a four-item scale adapted from [16]. We
used the perceived benefits as a proxy for extrinsic
motivation, which was measured using four items
adapted from [16]. To measure the expected
gameful experience, we used the instrument
adapted from [6]. The adaptations focused on
transforming the items to fit the social distancing
context. Participants evaluated all items on a
7point Likert scale (1 strongly disagree - 7 strongly
agree). The measurement tool is presented in the
appendix 6.
3.3.</p>
    </sec>
    <sec id="sec-9">
      <title>Participants</title>
      <p>The participants were recruited through the
Prolific crowdsourcing platform. Each participant
who completed the study received monetary
compensation (2.77 £). A total of 937 valid
responses were received, while incomplete,
nonengaged responses and those that failed to pass the
implemented attention checks were removed
(n=187 - 39 in Version 1, 39 in Version 2, 36 in
Version 3, 38 in Version 4 and 35 in non-gamified
version). Of the participants, 184 were assigned
the gamification Version 1, 176 the Version 2, 182
the Version 3, 170 the Version 4, and 225 were
assigned the non-gamified version. The
participants represented 59 different nationalities,
the most predominant being British (205
respondents, 21.9 %), Polish (144 respondents,
15.4 %), and South African (133, 14.2 %). The
participants were between 18 and 78 years of age,
30.0 on average (SD=10.5, Mdn=27). Of the
participants, 508 identified as men, 420 women, 6
non-binary, 1 queer, and 2 preferred not to
disclose. 391 participants were employed
fulltime, 276 students, 86 self-employed, 84
employed part-time, 67 unemployed, 33 others
(e.g., homemakers, retired, long-term sick leave,
disabled).
3.4.</p>
    </sec>
    <sec id="sec-10">
      <title>Reliability and validity</title>
      <p>Reliability and validity measures for the
reflective constructs are depicted in Table 3.
Cronbach’s alpha exceeded the threshold of 0.7
for all constructs, implying an acceptable level of
reliability [34]. AVE values exceeded 0.5 for all
constructs, while composite reliability values
were more than 0.7, indicating that convergent
validity was obtained [35].
Comp</p>
      <p>Ext</p>
      <p>G</p>
      <p>Imm</p>
      <p>Int</p>
      <p>P</p>
      <p>We assessed discriminant validity using the
Fornell-Larcker criterion (Table 4) and
heterotrait-monotrait ratio (HTMT) (Table 5).
Fornell-Larcker criterion, which states that the
square root of AVE of a construct (bolded) should
be higher than its correlation between other latent
constructs was satisfied for all cases [35]. The
HTMT criteria, which states that each value
should be less than 0.9 [36] was satisfied for all
cases, except challenge-accomplishment, for
which it was 0.934. However, the value of 1 did
not fall into the 10% confidence interval for the
HTMT criterion ([0.904, 0.961]), suggesting that
discriminant validity was established [36].</p>
    </sec>
    <sec id="sec-11">
      <title>4. Results</title>
      <p>The model accounted for 67.5% of the variance
for intrinsic motivation and 57.2% for extrinsic
motivation, therefore gameful experiences
explained a substantial amount of the variances in
motivations [37]. Social experience was
positively associated with intrinsic (β=0.204,
p&lt;.001) and extrinsic motivation (β=0.161,
p&lt;.001) (Table 6). Competition was associated
with intrinsic motivation (β=0.208, p&lt;.001), but
not with extrinsic at the p&lt;0.05 level, although
some evidence for this association was found.
Immersion was not associated with intrinsic or
extrinsic motivations, similar to challenge.
Accomplishment was positively associated with
intrinsic (β=0.200, p&lt;.001) and extrinsic
(β=0.324, p&lt;.001) motivation. Similarly,
playfulness was positively associated with
intrinsic (β=0.270, p&lt;.001) motivation and with
extrinsic (β=0.177, p&lt;.001) motivation. Guided
was associated with extrinsic (β=0.152, p&lt;.001)
motivation, but not intrinsic. The results show that
the strongest contributor to intrinsic motivation
was playfulness followed by competition, and the
strongest for extrinsic was accomplishment,
followed by playfulness. Overall, four dimensions
were associated with intrinsic motivation, and
four with extrinsic.</p>
      <p>We also examined how the associations differ
across gamification types (Table 7). Among those
assigned to Version 1 (achievement &amp;
progression-based gamification), competition was
associated with intrinsic motivation (β=0.184,
p=0.028), accomplishment with intrinsic
motivation (β=0.184, p=0.019), and playful with
both intrinsic (β=0.418, p&lt;.001) and extrinsic
(β=0.405, p=0.001) motivation. Among those
assigned to Version 2 (competitive gamification),
playful was associated with intrinsic (β=0.433,
p&lt;.001) and extrinsic (β=0.345, p&lt;.001)
motivation, accomplishment with extrinsic
motivation (β=0.201, p=0.028) and guided with
extrinsic motivation (β=0.262, p&lt;.001). Among
respondents assigned to Version 3 (social
gamification), social experience was associated
with intrinsic (β=0.198, p=0.016) and extrinsic
(β=0.237, p=0.017) motivation, competition with
intrinsic motivation (β=0.294, p=0.001),
accomplishment with extrinsic motivation
(β=0.397, p=0.003), and playful with intrinsic
motivation (β=0.319, p=0.001). Finally, among
those assigned to Version 4 (immersive
gamification), social experience was associated
with intrinsic (β=0.176, p=0.034) and extrinsic
(β=0.260, p=0.005) motivation, competition with
intrinsic motivation (β=0.239, p=0.011),
immersion with intrinsic motivation (β=0.225,
p=0.008), and accomplishment with extrinsic
motivation (β=0.296, p=0.011).</p>
    </sec>
    <sec id="sec-12">
      <title>5. Discussion</title>
      <p>This study used data from an online
vignettebased study to explore the relationships between
the different gameful experience dimensions and
intrinsic and extrinsic motivation, using social
distancing to keep individuals safe during
pandemics as the context. While prior studies
have examined how different gamification
designs provide intrinsic and extrinsic motivation
[11,28], the corpus lacked studies to examine the
specific subjective experiences stemming from
using gamification that transform activities to
become intrinsically or extrinsically motivated.
Therefore, this study provides a unique
contribution to the existing corpus of gamification
research by addressing this gap.</p>
      <p>Based on the results, the gameful experience
dimensions stemming from using gamified
systems are associated with both intrinsic and
extrinsic motivation. One of the central premises
of gamification is that it transforms activities to
provide similar positive experiences as games do,
generally making them intrinsically motivated
and therefore direct users towards desirable
behaviors [8,22]. Our results support this central
argument for why gamification can bring about a
sustained behavioral change as four of the seven
dimensions of the gameful experience were
associated with intrinsic motivation. Moreover,
while our results reveal that four gameful
experience dimensions are also associated with
extrinsic motivation, only one (guided) was solely
associated with extrinsic motivation, without
providing intrinsic motivation.</p>
      <p>The gameful experience dimensions
associated with intrinsic motivation were social
experience, competition, accomplishment, and
playfulness. These dimensions of the gameful
experience can be seen to reflect the three basic
needs that are associated with intrinsic motivation
according to the SDT [17]. Whereas social
experience is likely to tap into the need for
relatedness by fostering connectedness to other
gamification users, competition and
accomplishment can be expected to satisfy the
need for competence as they provide feedback on
the user's behavior, either based on individual
goals imposed by the gamified system or in
relation to other users [6,17]. Playfulness, on the
other hand, is likely to satisfy the need for
autonomy as it is associated with exploration and
spontaneity [6]. Moreover, these results resonate
with prior research which has demonstrated that
social and achievement-based gamification are
associated with autonomy, competence and
relatedness need satisfaction, contributing to
intrinsic motivation [28].</p>
      <p>
        Somewhat unexpectedly, immersion,
challenge and guided were not associated with
intrinsic motivation. The non-significant
associations between immersion and intrinsic
motivation and challenge and intrinsic motivation
were especially surprising as overcoming
challenges and being immersed are some of the
defining characteristics of games and qualities
that make games motivating and enjoyable
[
        <xref ref-type="bibr" rid="ref40">24,30</xref>
        ]. A possible explanation is that in the social
distancing setting that entails factors that are
uncontrollable by the individual, yet affect social
distancing behavior, challenge and immersion fail
to support the basic needs of autonomy and
competence, and therefore provide intrinsic
motivation. For example, even if the individual is
challenged to social distance, they might not be
able to meet the demands due to the circumstances
in their daily lives (e.g., working and family
arrangements), which hinders the sense of
autonomy and competence. Additionally, the
utilitarian focus of guided that is targeted toward
fulfilling goals that are external to the gamified
system is likely to explain why guided was not
associated with intrinsic motivation but only the
perceived benefits [6].
      </p>
      <p>Besides guided, social experience,
accomplishment and playfulness were positively
associated with extrinsic motivation, implying
that these dimensions are perceived to serve a
utilitarian goal in addition to providing an
intrinsically motivating experience. The
association between playfulness and extrinsic
motivation was surprising, as playfulness is
characterized by being carefree, while engaging in
free exploration that is not confined by rules. One
possible explanation relates to the context of this
study, as social distancing can be mentally
straining, and injecting such activity with a sense
of carefree playfulness might be perceived as
providing extrinsic benefits to individual
wellbeing.</p>
      <p>However, the associations between different
gameful experience dimensions and motivations
were not completely consistent across
gamification types. For example, the sense of
accomplishment that stems from using
achievement &amp; progression -based gamification
features (Version 1) induced intrinsic motivation,
whereas the sense of achievement stemming from
using competitive (Version 2), social (Version 3)
or immersive (Version 4) gamification were
associated with extrinsic motivation. This implies
that the gameful experiences cannot be thought of
as separate from the gamification designs that
induce them. Instead, how the gameful
experiences are induced affect their overall
motivational qualities.</p>
    </sec>
    <sec id="sec-13">
      <title>6. Limitations</title>
      <p>This study has four main limitations. First,
while the vignette methodology allows creating
situations that resemble real-life and easily
manipulate the stimulus that is presented to each
participant [33], studies conducted in real-world
contexts might yield different results regarding
the associations between gameful experience
dimensions and types of motivation. Second, the
study was conducted in the social distancing
context, which might hinder the generalizability
of the results. During COVID-19, the pressure to
social distance imposed by social settings was
prominent, and therefore the relationships
between gameful experience dimensions and
motivations might differ when targeting different
safety or other prosocial behaviors that are
perceived as ‘voluntary’ or charitable. Third,
while the discriminant validity measures satisfied
the laxer criteria, some gameful experience
dimensions were highly related [36]. Fourth, we
combined different commonly used gamification
elements in the versions that were presented to the
participants, potentially hampering understanding
of the granular effects of considering individual
affordances in isolation that might have yielded
different results in terms of the relationships
between different gameful experiences and
motivations.</p>
    </sec>
    <sec id="sec-14">
      <title>7. Acknowledgements</title>
      <p>
        This work was supported by Academy of Finland
Flagship Programme (Forest-Human-Machine
Interplay (UNITE)) [grant No 337653]; the
European Union Regional Development Fund,
SataDiLogis project [grant No A74723]; and the
European Union’s Horizon 2020 research and
innovation programme under the Marie
Sklodowska-Curie [grant No 101029543,
GamInclusive]. Icons and pictures used in the
Social Distancer versions were created by
FlatIcon and Lord-Pillock (DeviantArt).
8. References
[1] Koivisto J, Hamari J. The rise of
motivational information systems: A review
of gamification research. Int J Inf Manage
2019;45:191–210.
[
        <xref ref-type="bibr" rid="ref18">2</xref>
        ] Hamari J, Koivisto J. “Working out for
likes”: An empirical study on social
influence in exercise gamification. Comput
Human Behav 2015;50:333–47.
[3] Wallius E, Klock ACT, Hamari J. Playing it
safe: A literature review and research agenda
on motivational technologies in
transportation safety. Reliab Eng Syst Saf
2022;223.
[
        <xref ref-type="bibr" rid="ref54">4</xref>
        ] Bayuk J, Altobello SA. Can gamification
improve financial behavior? The moderating
[35] Fornell C, Larcker DF. Evaluating Structural
Equation Models with Unobservable
Variables and Measurement Error. vol. 18.
1981.
[36] Henseler J, Ringle CM, Sarstedt M. A new
criterion for assessing discriminant validity
in variance-based structural equation
modeling. J Acad Mark Sci 2015;43:115–35.
https://doi.org/10.1007/s11747-014-0403-8.
[37] Jacob C. Statistical Power Analysis for the
Behavioral Sciences. New Jersey: Hillsdale;
1988
      </p>
      <sec id="sec-14-1">
        <title>Appendix 1. Version 1</title>
      </sec>
      <sec id="sec-14-2">
        <title>Appendix 4. Version 4</title>
      </sec>
      <sec id="sec-14-3">
        <title>Appendix 2. Version 2</title>
      </sec>
      <sec id="sec-14-4">
        <title>Appendix 5. Non-gamified version</title>
      </sec>
      <sec id="sec-14-5">
        <title>Appendix 3. Version 3</title>
        <p>Appendix 6. The measurement instrument</p>
        <sec id="sec-14-5-1">
          <title>Construct</title>
        </sec>
        <sec id="sec-14-5-2">
          <title>Item</title>
          <p>Accomplishment
...using this application would
be advantageous to me.
…using this application would
provide gains to me.
…using this application would
result in benefits to me.
...using this application would
be favorable to me.
…give me a sense of being
directed in my social
distancing endeavors
…give me a sense of knowing
what I need to do to …do
better in social distancing
…give me feedback for
adapting my social distancing
behavior
…make me feel absorbed in
social distancing
…make me feel deeply
involved in social distancing
…cause me to feel deeply
engaged in social distancing
…make social distancing feel
like an adventure
…give me a sense of
discovery through social
distancing
…make social distancing
appeal to my curiosity
…give me the feeling that I’m
not on my own while social
distancing
…give me a sense of social
support while social
distancing
…give me a feeling of being
connected to others while
social distancing
…make me feel like I am
socially involved with others
while social distancing
…give me a sense of having
someone to share my social
distancing endeavors with
…make social distancing feel
like it has a social aspect</p>
          <p>I think that during the
COVID-19 pandemic...
0.858
0.876</p>
        </sec>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          <source>Bank Marketing</source>
          <year>2019</year>
          ;
          <volume>37</volume>
          :
          <fpage>951</fpage>
          -
          <lpage>75</lpage>
          . [5]
          <string-name>
            <surname>Ryan</surname>
            <given-names>RM</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rigby</surname>
            <given-names>CS</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Przybylski</surname>
            <given-names>A</given-names>
          </string-name>
          . The
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          https://doi.org/10.1007/s11031-006-9051-8. [6]
          <string-name>
            <surname>Högberg</surname>
            <given-names>J</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hamari</surname>
            <given-names>J</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wästlund</surname>
            <given-names>E</given-names>
          </string-name>
          . Gameful
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>https://doi.org/10.1007/s11257-019-09223-</mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          w. [7]
          <string-name>
            <surname>Huotari</surname>
            <given-names>K</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hamari</surname>
            <given-names>J</given-names>
          </string-name>
          . Defining Gamification
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          <string-name>
            <surname>- A Service Marketing Perspective</surname>
          </string-name>
          .
          <year>2012</year>
          . [8]
          <string-name>
            <surname>Hamari</surname>
            <given-names>J.</given-names>
          </string-name>
          <string-name>
            <surname>Gamification</surname>
          </string-name>
          . The Blackwell
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          John Wiley &amp; Sons, Ltd;
          <year>2019</year>
          , p.
          <fpage>1</fpage>
          -
          <lpage>3</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>https://doi.org/10.1002/9781405165518.wbe</mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          os1321. [9]
          <string-name>
            <surname>Hamari</surname>
            <given-names>J</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Koivisto</surname>
            <given-names>J</given-names>
          </string-name>
          . Why do people use
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          https://doi.org/10.1016/j.ijinfomgt.
          <year>2015</year>
          .
          <volume>04</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          006. [10]
          <string-name>
            <surname>Seaborn</surname>
            <given-names>K</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fels</surname>
            <given-names>DI</given-names>
          </string-name>
          .
          <article-title>Gamification in theory</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          <source>Human Computer Studies</source>
          <year>2015</year>
          ;
          <volume>74</volume>
          :
          <fpage>14</fpage>
          -
          <lpage>31</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          https://doi.org/10.1016/j.ijhcs.
          <year>2014</year>
          .
          <volume>09</volume>
          .006. [11]
          <string-name>
            <surname>Mekler</surname>
            <given-names>ED</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Brühlmann</surname>
            <given-names>F</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tuch</surname>
            <given-names>AN</given-names>
          </string-name>
          , Opwis
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          <source>Human Behav</source>
          <year>2017</year>
          ;
          <volume>71</volume>
          :
          <fpage>525</fpage>
          -
          <lpage>34</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          https://doi.org/10.1016/j.chb.
          <year>2015</year>
          .
          <volume>08</volume>
          .048. [12]
          <string-name>
            <surname>Landers</surname>
            <given-names>RN</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Collmus</surname>
            <given-names>AB</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Williams</surname>
            <given-names>H</given-names>
          </string-name>
          . The
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          https://doi.org/10.1016/j.ijhcs.
          <year>2018</year>
          .
          <volume>09</volume>
          .011. [13]
          <string-name>
            <surname>Mitchell</surname>
            <given-names>R</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Schuster</surname>
            <given-names>L</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Jin</surname>
            <given-names>HS</given-names>
          </string-name>
          . Gamification
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          <source>Res</source>
          <year>2020</year>
          ;
          <volume>106</volume>
          :
          <fpage>323</fpage>
          -
          <lpage>30</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          https://doi.org/10.1016/j.jbusres.
          <year>2018</year>
          .
          <volume>11</volume>
          .02
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          2. [14]
          <string-name>
            <surname>Buckley</surname>
            <given-names>P</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Doyle</surname>
            <given-names>E</given-names>
          </string-name>
          . Gamification and
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          <source>Environments</source>
          <year>2016</year>
          ;
          <volume>24</volume>
          :
          <fpage>1162</fpage>
          -
          <lpage>75</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          https://doi.org/10.1080/10494820.
          <year>2014</year>
          .964
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          263. [15]
          <string-name>
            <surname>Seaborn</surname>
            <given-names>K</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fels</surname>
            <given-names>DI</given-names>
          </string-name>
          .
          <article-title>Gamification in theory</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          <source>Human Computer Studies</source>
          <year>2015</year>
          ;
          <volume>74</volume>
          :
          <fpage>14</fpage>
          -
          <lpage>31</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          https://doi.org/10.1016/j.ijhcs.
          <year>2014</year>
          .
          <volume>09</volume>
          .006. [16]
          <string-name>
            <surname>Bulgurcu</surname>
          </string-name>
          , Cavusoglu, Benbasat. Information
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          <source>Quarterly</source>
          <year>2010</year>
          ;
          <volume>34</volume>
          :
          <fpage>523</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          https://doi.org/10.2307/25750690. [17]
          <string-name>
            <surname>Deci</surname>
            <given-names>EL</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ryan</surname>
            <given-names>RM</given-names>
          </string-name>
          .
          <article-title>The “what” and “why”</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>https://doi.org/10.1207/S15327965PLI1104</mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          _
          <fpage>01</fpage>
          . [18]
          <string-name>
            <surname>Krath</surname>
            <given-names>J</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Schürmann</surname>
            <given-names>L</given-names>
          </string-name>
          , von Korflesch HFO.
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          learning.
          <source>Comput Human Behav</source>
          <year>2021</year>
          ;
          <fpage>125</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          https://doi.org/10.1016/j.chb.
          <year>2021</year>
          .
          <volume>106963</volume>
          . [19]
          <string-name>
            <surname>Tyack</surname>
            <given-names>A</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Mekler</surname>
            <given-names>ED</given-names>
          </string-name>
          . Self-Determination
        </mixed-citation>
      </ref>
      <ref id="ref30">
        <mixed-citation>
          <string-name>
            <surname>Proceedings</surname>
          </string-name>
          , vol. 2020- January, Association
        </mixed-citation>
      </ref>
      <ref id="ref31">
        <mixed-citation>
          <string-name>
            <surname>for Computing</surname>
            <given-names>Machinery</given-names>
          </string-name>
          ;
          <year>2020</year>
          . [20]
          <string-name>
            <surname>Martela</surname>
            <given-names>F</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hankonen</surname>
            <given-names>N</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ryan</surname>
            <given-names>RM</given-names>
          </string-name>
          ,
        </mixed-citation>
      </ref>
      <ref id="ref32">
        <mixed-citation>
          <source>principles for COVID-19 and other</source>
        </mixed-citation>
      </ref>
      <ref id="ref33">
        <mixed-citation>
          <source>Psychol</source>
          <year>2021</year>
          ;
          <volume>32</volume>
          :
          <fpage>305</fpage>
          -
          <lpage>47</lpage>
          . [21]
          <string-name>
            <surname>Deci</surname>
          </string-name>
          et al.
          <article-title>Effects of externally mediated</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref34">
        <mixed-citation>
          <source>rewards on intrinsic motivation 1</source>
          . vol.
          <volume>18</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref35">
        <mixed-citation>
          <year>1971</year>
          . [22]
          <string-name>
            <surname>Landers</surname>
            <given-names>RN</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tondello</surname>
            <given-names>GF</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kappen</surname>
            <given-names>DL</given-names>
          </string-name>
          ,
        </mixed-citation>
      </ref>
      <ref id="ref36">
        <mixed-citation>
          <source>Human Computer Studies</source>
          <year>2019</year>
          ;
          <volume>127</volume>
          :
          <fpage>81</fpage>
          -
          <lpage>94</lpage>
          . [23]
          <string-name>
            <surname>Eppmann</surname>
            <given-names>R</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bekk</surname>
            <given-names>M</given-names>
          </string-name>
          ,
          <article-title>Klein</article-title>
          K. Gameful
        </mixed-citation>
      </ref>
      <ref id="ref37">
        <mixed-citation>
          <source>Marketing</source>
          <year>2018</year>
          ;
          <volume>43</volume>
          :
          <fpage>98</fpage>
          -
          <lpage>115</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref38">
        <mixed-citation>
          https://doi.org/10.1016/j.intmar.
          <year>2018</year>
          .
          <volume>03</volume>
          .
          <volume>002</volume>
          [24]
          <string-name>
            <surname>Yee</surname>
            <given-names>N.</given-names>
          </string-name>
          <article-title>Motivations for Play in Online</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref39">
        <mixed-citation>
          Games. vol.
          <volume>9</volume>
          .
          <year>2006</year>
          . [25]
          <string-name>
            <surname>Conti</surname>
            <given-names>R</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Collins</surname>
            <given-names>MA</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Picariello</surname>
            <given-names>ML</given-names>
          </string-name>
          . The
        </mixed-citation>
      </ref>
      <ref id="ref40">
        <mixed-citation>
          30.
          <year>2001</year>
          . [26]
          <string-name>
            <surname>Sailer</surname>
            <given-names>M</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hense</surname>
            <given-names>JU</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Mayr</surname>
            <given-names>SK</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Mandl</surname>
            <given-names>H.</given-names>
          </string-name>
        </mixed-citation>
      </ref>
      <ref id="ref41">
        <mixed-citation>
          https://doi.org/10.1016/j.chb.
          <year>2016</year>
          .
          <volume>12</volume>
          .033. [27]
          <string-name>
            <surname>Brown</surname>
            <given-names>E</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cairns</surname>
            <given-names>P</given-names>
          </string-name>
          . A Grounded
        </mixed-citation>
      </ref>
      <ref id="ref42">
        <mixed-citation>
          <source>Investigation of Game Immersion</source>
          .
          <year>2004</year>
          . [28]
          <string-name>
            <surname>Xi</surname>
            <given-names>N</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hamari</surname>
            <given-names>J</given-names>
          </string-name>
          . Does gamification satisfy
        </mixed-citation>
      </ref>
      <ref id="ref43">
        <mixed-citation>
          satisfaction.
          <source>Int J Inf Manage</source>
          <year>2019</year>
          ;
          <volume>46</volume>
          :
          <fpage>210</fpage>
          -
        </mixed-citation>
      </ref>
      <ref id="ref44">
        <mixed-citation>
          https://doi.org/10.1016/j.ijinfomgt.
          <year>2018</year>
          .
          <volume>12</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref45">
        <mixed-citation>
          002. [29]
          <string-name>
            <surname>Lucero</surname>
            <given-names>A</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Karapanos</surname>
            <given-names>E</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Arrasvuori</surname>
            <given-names>J</given-names>
          </string-name>
          ,
        </mixed-citation>
      </ref>
      <ref id="ref46">
        <mixed-citation>
          <source>Interactions</source>
          <year>2014</year>
          ;
          <volume>21</volume>
          :
          <fpage>34</fpage>
          -
          <lpage>9</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref47">
        <mixed-citation>
          https://doi.org/10.1145/2590973. [30]
          <string-name>
            <surname>Denisova</surname>
            <given-names>A</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cairns</surname>
            <given-names>P</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Guckelsberger</surname>
            <given-names>C</given-names>
          </string-name>
          ,
        </mixed-citation>
      </ref>
      <ref id="ref48">
        <mixed-citation>
          <source>Studies</source>
          <year>2020</year>
          ;
          <volume>137</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref49">
        <mixed-citation>
          https://doi.org/10.1016/j.ijhcs.
          <year>2019</year>
          .
          <volume>102383</volume>
          . [31]
          <string-name>
            <surname>Cao</surname>
            <given-names>Y</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gong</surname>
            <given-names>SY</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wang</surname>
            <given-names>Z</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cheng</surname>
            <given-names>Y</given-names>
          </string-name>
          , Wang
        </mixed-citation>
      </ref>
      <ref id="ref50">
        <mixed-citation>
          <source>Learn</source>
          <year>2022</year>
          ;
          <volume>38</volume>
          :
          <fpage>845</fpage>
          -
          <lpage>60</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref51">
        <mixed-citation>
          https://doi.org/10.1111/jcal.12652. [32]
          <string-name>
            <surname>Vansteenkiste</surname>
            <given-names>M</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sierens</surname>
            <given-names>E</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Goossens</surname>
            <given-names>L</given-names>
          </string-name>
          ,
        </mixed-citation>
      </ref>
      <ref id="ref52">
        <mixed-citation>
          <source>Instr</source>
          <year>2012</year>
          ;
          <volume>22</volume>
          :
          <fpage>431</fpage>
          -
          <lpage>9</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref53">
        <mixed-citation>
          https://doi.org/10.1016/j.learninstruc.
          <year>2012</year>
          .0
        </mixed-citation>
      </ref>
      <ref id="ref54">
        <mixed-citation>
          4.
          <fpage>002</fpage>
          . [33]
          <string-name>
            <surname>Alexander</surname>
            <given-names>CS</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Becker</surname>
            <given-names>HJ</given-names>
          </string-name>
          . The Use of
        </mixed-citation>
      </ref>
      <ref id="ref55">
        <mixed-citation>
          <year>1978</year>
          ;
          <volume>42</volume>
          :
          <fpage>93</fpage>
          . https://doi.org/10.1086/268432. [34]
          <string-name>
            <surname>Nunnally</surname>
            <given-names>J. Psychometric</given-names>
          </string-name>
          <string-name>
            <surname>Theory</surname>
          </string-name>
          . New
        </mixed-citation>
      </ref>
      <ref id="ref56">
        <mixed-citation>
          <string-name>
            <surname>York: McGraw-Hill</surname>
          </string-name>
          ;
          <year>1978</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>