=Paper=
{{Paper
|id=Vol-2730/paper1
|storemode=property
|title=Investigating motivation and time perception in traditional and technology-enhanced methods for psychological assessment
|pdfUrl=https://ceur-ws.org/Vol-2730/paper1.pdf
|volume=Vol-2730
|authors=Daniela Pacella,Davide Marocco
|dblpUrl=https://dblp.org/rec/conf/psychobit/PacellaM20
}}
==Investigating motivation and time perception in traditional and technology-enhanced methods for psychological assessment==
https://ceur-ws.org/Vol-2730/paper1.pdf
Investigating motivation and time perception in
traditional and technology-enhanced methods
for psychological assessment
Daniela Pacella1 and Davide Marocco2
1
University of Naples Federico II
Department of Public Health, Naples, Italy
2
University of Naples Federico II
Department of Humanistic Studies, Naples, Italy
{daniela.pacella,davide.marocco}@unina.it
Abstract. The recent rise in the adoption of technology-enhanced me-
thodologies for the assessment of psychological abilities (such as cogni-
tive skills and soft skills) has shown the several advantages that these
novel technologies have in comparison with traditional methods, includ-
ing their interactive nature, immediate feedback and the ability to simu-
late real-life situations. However, the differences between these methods
are often difficult to quantify and conceptualize. The aim of the present
study is to take a step forward in investigating these differences by eval-
uating the implicit and explicit motivation in the completion of two tests
that assess the ability to manage interpersonal conflict. A sample of 100
participants was asked to complete two psychological tests for conflict
management: the ROCI-II, a traditional test and Enact, a serious game,
followed by a self-report questionnaire. The motivation, pleasantness and
perceived time spent on the two tests were compared. Results show that
the technology-enhanced platform was evaluated as significantly more
pleasant and interesting. Additionally, graduates/postgraduates in other
disciplines than psychology significantly underestimated the time spent
interacting with the technological platform in comparison with the stan-
dard paper and pencil test.
·
· Motivation · Time perception.
Keywords: Psychological assessment Technology-enhanced assessment
1 Introduction
Serious games and technology-enhanced tests, in particular role-play and simu-
lation-based, are considered powerful tools thanks to their interactive [15] and
reflective [8] elements, as well as their immersive ability. These platforms can be
more engaging and entertaining than traditional paper-and-pencil tools, in par-
ticular thanks to their potential to foster user motivation. Motivation, in fact, is
Copyright © 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
2 D. Pacella and D. Marocco
a key factor to ensure high quality training and performance [5]. Although previ-
ous studies emphasized these aspects as being an important advantage pertaining
technology-enhanced tools, only few studies actually investigated the differences
between traditional and game-based tests [7], while most recent studies focused
instead on the difference between paper-and-pencil tests and their computerized
counterparts with the same structure (e.g. [13]). With the present study we wish
to make a step towards a more comprehensive investigation of the differences
in pleasantness and motivation between paper-and-pencil and interactive tests.
In particular, we aim at testing the hypothesis that an interactive methodology
has game-like sources of internal motivation, and is thus perceived as more in-
volving and motivating in comparison with traditional methodologies. To pursue
this aim, we adopted two different tests to assess conflict management ability
based on the same psychological framework: the ROCI-II, a traditional paper-
and-pencil test, and Enact, a simulation-based serious game, and we evaluated
the generated implicit and explicit motivation using a self-report questionnaire.
The results aim at testing if the adoption of game-based platforms can improve
engagement and personal motivation to complete a given task, also providing
new insights for the future development of simulation-based platforms for psy-
chological assessment.
To investigate explicit motivation, as described in the next section, an ad
hoc questionnaire was developed, while implicit motivation and pleasantness
was measured with the construct of time perception.
Concerning time perception, several studies analysed changes in the indi-
vidual perception of time, trying to evaluate the relationships between objec-
tive time and subjective time and how these are related to several internal
and external conditions as well as subjective distortions [6]. Time perception
has already been used as a measurement of implicit pleasantness or motivation.
For example, Thayer [12] investigated how pleasantness impacted time percep-
tion by asking participants to estimate how long they spent in an eye-contact
social task and manipulated the facial expression of the individual they were
looking at, a critical variable that can affect the perceived valence of the so-
cial interaction. The eye-contact was estimated as longer when the stimulus
(facial expression) was negative-unpleasant (scowling-angry) than when it was
positive-pleasant (smiling-friendly). Concerning time perception and motivation,
Watt [14] investigated the effect of boredom proneness on perception of time.
In the experiment, undergraduate students were divided into two groups, highly
boredom-prone and low boredom-prone and were asked to complete a repetitive
number circling-task. Highly boredom-prone individuals reported subjective time
as passing more slowly during the task.
2 Materials and Methods
2.1 The psychological tests
The Rahim Organizational Conflict Inventory II, or ROCI-II [1] is a standard-
ised psychological test that provides a score for each of the five possible styles
Investigating motivation and time perception in psychological assessment 3
of conflict management defined in Rahim and Bonoma’s theory [11], namely In-
tegrating, Obliging, Dominating, Avoiding and Compromising. The test is fully
paper-and-pencil and consists of 28 items on a 5-Point Likert scale.
The Enact game platform is a simulation-based serious game in which users
are asked to negotiate with one virtual agent in a real-life conflict situation
selecting one among five different sentences, each reflecting one of the styles
defined in Rahim and Bonoma’s theory. The game is scenario-based and in each
scenario the virtual agent displays a different conflict management style, for a
total of 5 [9, 10].
2.2 The questionnaire
For the purpose of the research, an ad hoc self-report questionnaire was devel-
oped, as a methodology for assessing the explicit and implicit levels of motivation
and pleasantness of the participants after the administration of the two tests.
The questionnaire was composed as follows:
– information sheet with informed consent
– demographic questionnaire
– pleasantness and time perception self-report
In the self-report questionnaire, divided into three sections, participants were
asked to respond to items using a three or five-point Likert scale. The question-
naire evaluated:
1. the personal perception of time, interest in performing the two tests, pleas-
antness and attention;
2. the perception of having acquired greater awareness of their style of manag-
ing conflict after each test, and the consequent motivation and will to deepen
or not the subject;
3. estimation of the time spent in the completion of the two assessment tools.
The research was conducted at the Natural and Artificial Cognition (NAC)
Lab at the University of Naples Federico II. The sample was selected randomly
and consists of 100 participants equally distributed between males and females
(N = 100 of which M = 50 and F = 50). The age of the sample is between 18 and
35 (mean = 23.98, st.d. = 2.749, female mean = 24, male mean = 23.96). 49%
of the sample consists of BSc or MSc psychology students. The remaining 51%
is made of other BSc or MSc students in 9% language, 7% law, 7% philosophy,
5% literature, 2% history, 2% physical education, 2% biology, and 1% from the
following courses: history of art, phonology, physics, geology, speech therapy,
engineering, digital cultures, political science, music, economics. Finally, 2% are
interns and PhD students, and 5% did not specify. Each participant was uniquely
identified with a code to ensure anonymity.
4 D. Pacella and D. Marocco
3 Results
For convenience, the sample was divided into two groups: a group formed by
graduates or postgraduates in psychology and one formed by graduates or post-
graduates in other disciplines. Regardless of their gender, participants on average
spent about 6 minutes to complete the ROCI-II and 12 to complete Enact.
3.1 Results on the explicit pleasantness
Regarding the explicit perception of pleasantness, data from the answers given
to the self-report was analyzed. The answer frequencies for each section of the
self-report questionnaire are shown respectively in Table 1, Table 2 and Table 3
Item Enact ROCI-II I don’t know
You took more time to finish 87% 9% 4%
You found more interesting 91% 5% 4%
You were more keen on completing 84% 10% 6%
You paid more attention completing 77% 20% 3%
Table 1. Frequencies of the answers in Section 1 of the self-report questionnaire
Strongly Neither agree Strongly
Item Disagree Agree
Disagree nor disagree Agree
I think it took me too
much time to finish
6% 37% 28% 26% 3%
the Enact game,
compared to the ROCI-II
I felt bored completing
the ROCI-II test
4% 27% 29% 34% 6%
compared to the
Enact game
I felt more pleased
completing ROCI-II
19% 53% 18% 10% 0%
questions than those
of the Enact game
I got distracted several
times while completing
14% 38% 24% 19% 5%
the ROCI-II compared
to the Enact game.
Table 2. Frequencies of the answers in Section 2 of the self-report questionnaire
From the analysis of the data of the first section, the following results emerged:
Investigating motivation and time perception in psychological assessment 5
Strongly Neither agree Strongly
Item Disagree Agree
Disagree nor disagree Agree
While completing the
Enact game, I gained
1% 23% 26% 48% 2%
knowledge concerning
my ability to handle conflicts
I am motivated to further deepen
the knowledge gained in Enact
0% 11% 30% 48% 11%
(concerning my ability to
manage conflicts)
While completing the ROCI-II
test, I gained
8% 45% 29% 17% 1%
knowledge concerning my
ability to handle conflicts
I am motivated to further deepen
the knowledge gained in ROCI-II
1% 20% 31% 40% 8%
(concerning my ability to
manage conflicts)
Table 3. Frequencies of the answers in Section 3 of the self-report questionnaire
1. Most of the sample (87%) believed that it took longer to complete Enact
than ROCI-II.
2. Although students perceived that more time was spent on Enact, most of
them found Enact more interesting (91%) and pleasant (84%) than the
ROCI-II. Also the level of attention was considered by most students (77%)
higher while completing the serious game compared to the traditional test.
From the analysis of the data of the second section of the questionnaire, it is
possible to evidence that:
1. 43% of the sample, thus the relative majority, did not believe that it took
too long to finish Enact compared to ROCI II, while only 29% believed the
opposite is true.
2. 40% of the sample declared that completing the ROCI-II was a more boring
task compared to Enact.
3. The majority of the sample (72%) preferred to complete Enact rather than
the ROCI-II.
4. More than half (52%) of the sample declared that they were not distracted
during the execution of ROCI-II compared to Enact, and only 24% declared
the opposite.
From the analysis of the data of the third section, the following results emerged:
1. The majority (51%) of the sample declared that, during the interaction with
Enact, new knowledge regarding the ability to manage conflict was acquired
and even more (59%) declared that they were motivated to deepen this
knowledge.
6 D. Pacella and D. Marocco
2. On the other hand, 53% of the sample stated that they have not learned
anything new during the administration of the ROCI-II.
Then, we analysed gender differences in the distribution of the answers to the
questionnaire. The following statistically significant differences were found:
– the serious game Enact was evaluated differently in terms of the perception
of learning new knowledge concerning conflict management skills between
males and females (p = 0.001). Specifically, most female participants (66%)
evaluated Enact positively, (22% do not take a position on this and 12%
evaluate it negatively); while male students are almost equally divided be-
tween those who positively rate it (34%), those who do not take a position
(30%) and those who evaluate it negatively (36%);
– the ROCI-II test is evaluated differently in terms of perception of new ac-
quired knowledge regarding their conflict management skills by participants
based on their gender (p = 0.038). Specifically, almost half of the female
participants (46%) assessed ROCI II negatively, while the other half either
takes no position (30%) or evaluates it positively (24%); the percentage of
male students that rate it negatively instead is much higher (60%), while
28% do not take a position on it and only 12% rate it positively.
Summarizing the results, most participants correctly estimated that it took
longer to complete Enact rather than the ROCI-II. However, the serious game
was perceived as more interesting and enjoyable to play, while the traditional
test was less appreciated. More than half of the sample also declared to have
learned new knowledge about their way of managing conflict with others thanks
to the use of the serious game Enact (while the opposite was true about the
traditional test) and was motivated to deepen this knowledge.
3.2 Results on the implicit pleasantness
Regarding the implicit perception of pleasantness, the first analysis performed
aimed at comparing the real time spent to complete the two assessment tools
and the subjective time spent as perceived by the participants. Table 4 shows
the statistical averages and the standard deviations referring to the real time
(expressed in minutes) used to perform the ROCI-II test and the Enact game.
There are no significant differences between males and females in the real
time spent to complete Enact (p = 0.697) nor the real time spent to complete
the ROCI-II (p = 0.931).
To express their perception of time, participants could choose one option
among: less than 10 minutes, between 10 and 15 minutes and more than 15
minutes. The results are summarized in Table 5.
In contrast to the lack on any difference on real time spent, the gender
variable determines effects on the perceived duration of the two tests: the per-
ceived time spent to complete Enact, in fact, is estimated differently according
to the gender (p = 0.034). In particular female students tend to more accu-
rately estimate the administration time of Enact, while male students tend to
Investigating motivation and time perception in psychological assessment 7
Gender Stat Real Time Enact Real Time ROCI-II
F Average 12.38 5.50
St. Dev. 3.557 1.741
M Average 12.00 6.42
St. Dev. 3.338 1.727
TOT Average 12.19 5.96
St. Dev. 3.437 1.786
Table 4. Average time (minutes) taken by the participants to complete the two as-
sessment tools divided by gender.
Less than Between 10 More than
Item
10 minutes and 15 minutes 15 minutes
Enact 18% 76% 6%
ROCI-II 88% 12% 0%
Table 5. Time estimated spent on the two assessment tools by the participants
Fig. 1. Gender differences in the perception of time spent completing Enact. Frequen-
cies are displayed.
8 D. Pacella and D. Marocco
underestimate it, as shown in Figure 1. Time perception also appears to be
influenced by the type of degree (graduate/postgraduate in psychology versus
graduate/postgraduate in other subjects). The two groups significantly differ in
terms of perceived time spent on Enact (p = 0.015, measured with the Mann-
Whitney U test). Figure 2 shows the distribution of the answers given to the
self-report by the two groups, concerning the time perceived in executing Enact.
Fig. 2. Degree-based differences in the perception of time spent completing Enact.
Frequencies are displayed.
It emerged that participants in the group ”Psychology graduates/postgradu-
ates” were able to correctly estimate the time spent completing Enact, while
those enrolled or graduated in other disciplines did present a bias towards the
underestimation of this time.
4 Discussion
From the results, it emerged that the majority of the sample correctly estimated
the time taken to perform the two evaluation tools, recognizing that the serious
game requires more time. However, the serious game was perceived as more
interesting and enjoyable to play, while the traditional test was less liked. This
data confirms the hypothesis that the perception of explicit pleasantness of the
participants, in reference to the two evaluation tools, is higher towards the serious
game Enact.
Investigating motivation and time perception in psychological assessment 9
The gender variable does not determine effects on the actual duration of
Enact and ROCI-II, but it significantly determines the subjective perception of
time: female subjects tended to estimate more correctly the execution time of
Enact, while males tended to slightly underestimate it. Females are also more
likely to positively evaluate Enact (in terms of perception of acquiring new knowl-
edge regarding their ability to manage conflict), while males evaluate ROCI-II,
more negatively than females. Data related to gender differences sees the female
sample having a significantly higher score in the items that express a greater ap-
preciation explicit towards Enact (unlike males who maintain on average more
neutral responses. This is in accordance with a previous study about Enact pleas-
antness [3] where it was found that female participants, compared to males, gave
significantly higher scores in items that expressed appreciation towards Enact.
The difference emerged between the sample group that was graduate/postgrad-
uate in psychology and the group that was graduate/postgraduate in other dis-
ciplines concerning the perception of time in reference to Enact sparks particular
interest: participants that were graduated/postgraduate in disciplines other than
psychology tended to underestimate the time spent on Enact, and this can be in-
terpreted as an evaluation bias introduced by their interest and engagement with
the game. However, graduates/postgraduates in psychology were not affected by
this bias. Indeed, previous literature showed that graduates/postgraduates in
psychology are more aware of the subjective aspects that influence our percep-
tion of time, as they, during their academic journey, come into contact with
psychological constructs such as human metacognition and the biases associated
with it. Theories on metacognition argue that there is a causal link between
metacognitive reflections, control processes and cognitive performance [4], also
concerning time perception [2]. Future studies will aim at further investigate and
control for this aspect, which could be extremely important for the development
of more engaging technology-enhanced tests.
5 Conclusion
The present research aimed to explore the differences in pleasantness and motiva-
tion during the administration of standard paper and pencil tests and technology-
enhanced tests for psychological assessment. For this purpose, participants were
asked to complete two assessment tools that test the ability to manage inter-
personal conflict, in particular the ROCI-II test (a paper-and-pencil test) and
the Enact game platform (a simulation-based serious game). Then, participants
were asked to fill a self-report questionnaire. Results show that, even if the se-
rious game Enact took much longer to complete, it was generally assessed as
significantly more pleasant and interesting, regardless of the gender and degree
of the participant. Additionally, perceived time spent on the Enact platform
was significantly underestimated, but only by participants who were not gradu-
ates/postgraduates in psychology. Also, the majority of the sample reported that
new knowledge regarding negotiation and conflict management was acquired dur-
ing the Enact gameplay, while this was not reported for the ROCI-II.
10 D. Pacella and D. Marocco
References
1. Afzalur Rahim, M., Antonioni, D., Psenicka, C.: A structural equations model
of leader power, subordinates’ styles of handling conflict, and job perfor-
mance. International Journal of Conflict Management 12(3), 191–211 (mar 2001).
https://doi.org/10.1108/eb022855
2. Chakroun, N., Izaute, M., Lamotte, M., Droit-Volet, S.: Metacognitive Question-
naire on Time: Feeling of the Passage of Time. Timing & Time Perception 2(3),
339–359 (nov 2014). https://doi.org/10.1163/22134468-00002031
3. Dell’Aquila, E., Marocco, D., Ponticorvo, M., Di Ferdinando, A., Schembri, M.,
Miglino, O.: Educational Games for Soft-Skills Training in Digital Environments.
Springer International Publishing, Cham (2017). https://doi.org/10.1007/978-3-
319-06311-9
4. Deroy, O., Spence, C., Noppeney, U.: Metacognition in Multisensory
Perception. Trends in Cognitive Sciences 20(10), 736–747 (oct 2016).
https://doi.org/10.1016/j.tics.2016.08.006
5. Filsecker, M., Hickey, D.T.: A multilevel analysis of the effects of external rewards
on elementary students’ motivation, engagement and learning in an educational
game. Computers & Education 75, 136–148 (2014)
6. Grondin, S.: Timing and time perception: A review of recent behavioral and
neuroscience findings and theoretical directions. Attention, Perception, and Psy-
chophysics 72(3), 561–582 (apr 2010). https://doi.org/10.3758/APP.72.3.561
7. Jones, M.B., Dunlap, W.P., Bilodeau, I.M.: Comparison of video game and con-
ventional test performance. Simulation & Games 17(4), 435–446 (1986)
8. Laurillard, D.: Rethinking university teaching: a framework for the effective use of
learning technologies. TechTrends 69 (2010)
9. Marocco, D., Pacella, D., Dell’Aquila, E., Di Ferdinando, A.: Grounding serious
game design on scientific findings: The case of ENACT on soft skills training and
assessment. In: Lecture Notes in Computer Science (including subseries Lecture
Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9307, pp.
441–446. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24258-337
10. Pacella, D., Dell’Aquila, E., Marocco, D., Furnell, S.: Toward an automatic classi-
fication of negotiation styles using natural language processing. vol. 10498 LNAI,
pp. 339–342. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67401-843
11. Rahim, A., Bonoma, T.V.: Managing Organizational Conflict: a Model for
Diagnosis and Intervention. Psychological Reports 44(3c), 1323–1344 (1979).
https://doi.org/10.2466/pr0.1979.44.3c.1323
12. Thayer, S., Schiff, W.: Eye-contact, facial expression, and the experi-
ence of time. Journal of Social Psychology 95(1), 117–124 (feb 1975).
https://doi.org/10.1080/00224545.1975.9923242
13. Vereecken, C.A., Maes, L.: Comparison of a computer-administered and paper-
and-pencil-administered questionnaire on health and lifestyle behaviors. Journal
of Adolescent Health 38(4), 426–432 (2006)
14. Watt, J.D.: Effect of boredom proneness on time perception. Psychological reports
69(1), 323–327 (aug 1991). https://doi.org/10.2466/pr0.1991.69.1.323
15. Wills, S., Leigh, E., Ip, A.: The power of role-based e-learning: Designing and
moderating online role play. Routledge (2011)