Investigating motivation and time perception in traditional and technology-enhanced methods for psychological assessment Daniela Pacella1 and Davide Marocco2 1 University of Naples Federico II Department of Public Health, Naples, Italy 2 University of Naples Federico II Department of Humanistic Studies, Naples, Italy {daniela.pacella,davide.marocco}@unina.it Abstract. The recent rise in the adoption of technology-enhanced me- thodologies for the assessment of psychological abilities (such as cogni- tive skills and soft skills) has shown the several advantages that these novel technologies have in comparison with traditional methods, includ- ing their interactive nature, immediate feedback and the ability to simu- late real-life situations. However, the differences between these methods are often difficult to quantify and conceptualize. The aim of the present study is to take a step forward in investigating these differences by eval- uating the implicit and explicit motivation in the completion of two tests that assess the ability to manage interpersonal conflict. A sample of 100 participants was asked to complete two psychological tests for conflict management: the ROCI-II, a traditional test and Enact, a serious game, followed by a self-report questionnaire. The motivation, pleasantness and perceived time spent on the two tests were compared. Results show that the technology-enhanced platform was evaluated as significantly more pleasant and interesting. Additionally, graduates/postgraduates in other disciplines than psychology significantly underestimated the time spent interacting with the technological platform in comparison with the stan- dard paper and pencil test. · · Motivation · Time perception. Keywords: Psychological assessment Technology-enhanced assessment 1 Introduction Serious games and technology-enhanced tests, in particular role-play and simu- lation-based, are considered powerful tools thanks to their interactive [15] and reflective [8] elements, as well as their immersive ability. These platforms can be more engaging and entertaining than traditional paper-and-pencil tools, in par- ticular thanks to their potential to foster user motivation. Motivation, in fact, is Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 2 D. Pacella and D. Marocco a key factor to ensure high quality training and performance [5]. Although previ- ous studies emphasized these aspects as being an important advantage pertaining technology-enhanced tools, only few studies actually investigated the differences between traditional and game-based tests [7], while most recent studies focused instead on the difference between paper-and-pencil tests and their computerized counterparts with the same structure (e.g. [13]). With the present study we wish to make a step towards a more comprehensive investigation of the differences in pleasantness and motivation between paper-and-pencil and interactive tests. In particular, we aim at testing the hypothesis that an interactive methodology has game-like sources of internal motivation, and is thus perceived as more in- volving and motivating in comparison with traditional methodologies. To pursue this aim, we adopted two different tests to assess conflict management ability based on the same psychological framework: the ROCI-II, a traditional paper- and-pencil test, and Enact, a simulation-based serious game, and we evaluated the generated implicit and explicit motivation using a self-report questionnaire. The results aim at testing if the adoption of game-based platforms can improve engagement and personal motivation to complete a given task, also providing new insights for the future development of simulation-based platforms for psy- chological assessment. To investigate explicit motivation, as described in the next section, an ad hoc questionnaire was developed, while implicit motivation and pleasantness was measured with the construct of time perception. Concerning time perception, several studies analysed changes in the indi- vidual perception of time, trying to evaluate the relationships between objec- tive time and subjective time and how these are related to several internal and external conditions as well as subjective distortions [6]. Time perception has already been used as a measurement of implicit pleasantness or motivation. For example, Thayer [12] investigated how pleasantness impacted time percep- tion by asking participants to estimate how long they spent in an eye-contact social task and manipulated the facial expression of the individual they were looking at, a critical variable that can affect the perceived valence of the so- cial interaction. The eye-contact was estimated as longer when the stimulus (facial expression) was negative-unpleasant (scowling-angry) than when it was positive-pleasant (smiling-friendly). Concerning time perception and motivation, Watt [14] investigated the effect of boredom proneness on perception of time. In the experiment, undergraduate students were divided into two groups, highly boredom-prone and low boredom-prone and were asked to complete a repetitive number circling-task. Highly boredom-prone individuals reported subjective time as passing more slowly during the task. 2 Materials and Methods 2.1 The psychological tests The Rahim Organizational Conflict Inventory II, or ROCI-II [1] is a standard- ised psychological test that provides a score for each of the five possible styles Investigating motivation and time perception in psychological assessment 3 of conflict management defined in Rahim and Bonoma’s theory [11], namely In- tegrating, Obliging, Dominating, Avoiding and Compromising. The test is fully paper-and-pencil and consists of 28 items on a 5-Point Likert scale. The Enact game platform is a simulation-based serious game in which users are asked to negotiate with one virtual agent in a real-life conflict situation selecting one among five different sentences, each reflecting one of the styles defined in Rahim and Bonoma’s theory. The game is scenario-based and in each scenario the virtual agent displays a different conflict management style, for a total of 5 [9, 10]. 2.2 The questionnaire For the purpose of the research, an ad hoc self-report questionnaire was devel- oped, as a methodology for assessing the explicit and implicit levels of motivation and pleasantness of the participants after the administration of the two tests. The questionnaire was composed as follows: – information sheet with informed consent – demographic questionnaire – pleasantness and time perception self-report In the self-report questionnaire, divided into three sections, participants were asked to respond to items using a three or five-point Likert scale. The question- naire evaluated: 1. the personal perception of time, interest in performing the two tests, pleas- antness and attention; 2. the perception of having acquired greater awareness of their style of manag- ing conflict after each test, and the consequent motivation and will to deepen or not the subject; 3. estimation of the time spent in the completion of the two assessment tools. The research was conducted at the Natural and Artificial Cognition (NAC) Lab at the University of Naples Federico II. The sample was selected randomly and consists of 100 participants equally distributed between males and females (N = 100 of which M = 50 and F = 50). The age of the sample is between 18 and 35 (mean = 23.98, st.d. = 2.749, female mean = 24, male mean = 23.96). 49% of the sample consists of BSc or MSc psychology students. The remaining 51% is made of other BSc or MSc students in 9% language, 7% law, 7% philosophy, 5% literature, 2% history, 2% physical education, 2% biology, and 1% from the following courses: history of art, phonology, physics, geology, speech therapy, engineering, digital cultures, political science, music, economics. Finally, 2% are interns and PhD students, and 5% did not specify. Each participant was uniquely identified with a code to ensure anonymity. 4 D. Pacella and D. Marocco 3 Results For convenience, the sample was divided into two groups: a group formed by graduates or postgraduates in psychology and one formed by graduates or post- graduates in other disciplines. Regardless of their gender, participants on average spent about 6 minutes to complete the ROCI-II and 12 to complete Enact. 3.1 Results on the explicit pleasantness Regarding the explicit perception of pleasantness, data from the answers given to the self-report was analyzed. The answer frequencies for each section of the self-report questionnaire are shown respectively in Table 1, Table 2 and Table 3 Item Enact ROCI-II I don’t know You took more time to finish 87% 9% 4% You found more interesting 91% 5% 4% You were more keen on completing 84% 10% 6% You paid more attention completing 77% 20% 3% Table 1. Frequencies of the answers in Section 1 of the self-report questionnaire Strongly Neither agree Strongly Item Disagree Agree Disagree nor disagree Agree I think it took me too much time to finish 6% 37% 28% 26% 3% the Enact game, compared to the ROCI-II I felt bored completing the ROCI-II test 4% 27% 29% 34% 6% compared to the Enact game I felt more pleased completing ROCI-II 19% 53% 18% 10% 0% questions than those of the Enact game I got distracted several times while completing 14% 38% 24% 19% 5% the ROCI-II compared to the Enact game. Table 2. Frequencies of the answers in Section 2 of the self-report questionnaire From the analysis of the data of the first section, the following results emerged: Investigating motivation and time perception in psychological assessment 5 Strongly Neither agree Strongly Item Disagree Agree Disagree nor disagree Agree While completing the Enact game, I gained 1% 23% 26% 48% 2% knowledge concerning my ability to handle conflicts I am motivated to further deepen the knowledge gained in Enact 0% 11% 30% 48% 11% (concerning my ability to manage conflicts) While completing the ROCI-II test, I gained 8% 45% 29% 17% 1% knowledge concerning my ability to handle conflicts I am motivated to further deepen the knowledge gained in ROCI-II 1% 20% 31% 40% 8% (concerning my ability to manage conflicts) Table 3. Frequencies of the answers in Section 3 of the self-report questionnaire 1. Most of the sample (87%) believed that it took longer to complete Enact than ROCI-II. 2. Although students perceived that more time was spent on Enact, most of them found Enact more interesting (91%) and pleasant (84%) than the ROCI-II. Also the level of attention was considered by most students (77%) higher while completing the serious game compared to the traditional test. From the analysis of the data of the second section of the questionnaire, it is possible to evidence that: 1. 43% of the sample, thus the relative majority, did not believe that it took too long to finish Enact compared to ROCI II, while only 29% believed the opposite is true. 2. 40% of the sample declared that completing the ROCI-II was a more boring task compared to Enact. 3. The majority of the sample (72%) preferred to complete Enact rather than the ROCI-II. 4. More than half (52%) of the sample declared that they were not distracted during the execution of ROCI-II compared to Enact, and only 24% declared the opposite. From the analysis of the data of the third section, the following results emerged: 1. The majority (51%) of the sample declared that, during the interaction with Enact, new knowledge regarding the ability to manage conflict was acquired and even more (59%) declared that they were motivated to deepen this knowledge. 6 D. Pacella and D. Marocco 2. On the other hand, 53% of the sample stated that they have not learned anything new during the administration of the ROCI-II. Then, we analysed gender differences in the distribution of the answers to the questionnaire. The following statistically significant differences were found: – the serious game Enact was evaluated differently in terms of the perception of learning new knowledge concerning conflict management skills between males and females (p = 0.001). Specifically, most female participants (66%) evaluated Enact positively, (22% do not take a position on this and 12% evaluate it negatively); while male students are almost equally divided be- tween those who positively rate it (34%), those who do not take a position (30%) and those who evaluate it negatively (36%); – the ROCI-II test is evaluated differently in terms of perception of new ac- quired knowledge regarding their conflict management skills by participants based on their gender (p = 0.038). Specifically, almost half of the female participants (46%) assessed ROCI II negatively, while the other half either takes no position (30%) or evaluates it positively (24%); the percentage of male students that rate it negatively instead is much higher (60%), while 28% do not take a position on it and only 12% rate it positively. Summarizing the results, most participants correctly estimated that it took longer to complete Enact rather than the ROCI-II. However, the serious game was perceived as more interesting and enjoyable to play, while the traditional test was less appreciated. More than half of the sample also declared to have learned new knowledge about their way of managing conflict with others thanks to the use of the serious game Enact (while the opposite was true about the traditional test) and was motivated to deepen this knowledge. 3.2 Results on the implicit pleasantness Regarding the implicit perception of pleasantness, the first analysis performed aimed at comparing the real time spent to complete the two assessment tools and the subjective time spent as perceived by the participants. Table 4 shows the statistical averages and the standard deviations referring to the real time (expressed in minutes) used to perform the ROCI-II test and the Enact game. There are no significant differences between males and females in the real time spent to complete Enact (p = 0.697) nor the real time spent to complete the ROCI-II (p = 0.931). To express their perception of time, participants could choose one option among: less than 10 minutes, between 10 and 15 minutes and more than 15 minutes. The results are summarized in Table 5. In contrast to the lack on any difference on real time spent, the gender variable determines effects on the perceived duration of the two tests: the per- ceived time spent to complete Enact, in fact, is estimated differently according to the gender (p = 0.034). In particular female students tend to more accu- rately estimate the administration time of Enact, while male students tend to Investigating motivation and time perception in psychological assessment 7 Gender Stat Real Time Enact Real Time ROCI-II F Average 12.38 5.50 St. Dev. 3.557 1.741 M Average 12.00 6.42 St. Dev. 3.338 1.727 TOT Average 12.19 5.96 St. Dev. 3.437 1.786 Table 4. Average time (minutes) taken by the participants to complete the two as- sessment tools divided by gender. Less than Between 10 More than Item 10 minutes and 15 minutes 15 minutes Enact 18% 76% 6% ROCI-II 88% 12% 0% Table 5. Time estimated spent on the two assessment tools by the participants Fig. 1. Gender differences in the perception of time spent completing Enact. Frequen- cies are displayed. 8 D. Pacella and D. Marocco underestimate it, as shown in Figure 1. Time perception also appears to be influenced by the type of degree (graduate/postgraduate in psychology versus graduate/postgraduate in other subjects). The two groups significantly differ in terms of perceived time spent on Enact (p = 0.015, measured with the Mann- Whitney U test). Figure 2 shows the distribution of the answers given to the self-report by the two groups, concerning the time perceived in executing Enact. Fig. 2. Degree-based differences in the perception of time spent completing Enact. Frequencies are displayed. It emerged that participants in the group ”Psychology graduates/postgradu- ates” were able to correctly estimate the time spent completing Enact, while those enrolled or graduated in other disciplines did present a bias towards the underestimation of this time. 4 Discussion From the results, it emerged that the majority of the sample correctly estimated the time taken to perform the two evaluation tools, recognizing that the serious game requires more time. However, the serious game was perceived as more interesting and enjoyable to play, while the traditional test was less liked. This data confirms the hypothesis that the perception of explicit pleasantness of the participants, in reference to the two evaluation tools, is higher towards the serious game Enact. Investigating motivation and time perception in psychological assessment 9 The gender variable does not determine effects on the actual duration of Enact and ROCI-II, but it significantly determines the subjective perception of time: female subjects tended to estimate more correctly the execution time of Enact, while males tended to slightly underestimate it. Females are also more likely to positively evaluate Enact (in terms of perception of acquiring new knowl- edge regarding their ability to manage conflict), while males evaluate ROCI-II, more negatively than females. Data related to gender differences sees the female sample having a significantly higher score in the items that express a greater ap- preciation explicit towards Enact (unlike males who maintain on average more neutral responses. This is in accordance with a previous study about Enact pleas- antness [3] where it was found that female participants, compared to males, gave significantly higher scores in items that expressed appreciation towards Enact. The difference emerged between the sample group that was graduate/postgrad- uate in psychology and the group that was graduate/postgraduate in other dis- ciplines concerning the perception of time in reference to Enact sparks particular interest: participants that were graduated/postgraduate in disciplines other than psychology tended to underestimate the time spent on Enact, and this can be in- terpreted as an evaluation bias introduced by their interest and engagement with the game. However, graduates/postgraduates in psychology were not affected by this bias. Indeed, previous literature showed that graduates/postgraduates in psychology are more aware of the subjective aspects that influence our percep- tion of time, as they, during their academic journey, come into contact with psychological constructs such as human metacognition and the biases associated with it. Theories on metacognition argue that there is a causal link between metacognitive reflections, control processes and cognitive performance [4], also concerning time perception [2]. Future studies will aim at further investigate and control for this aspect, which could be extremely important for the development of more engaging technology-enhanced tests. 5 Conclusion The present research aimed to explore the differences in pleasantness and motiva- tion during the administration of standard paper and pencil tests and technology- enhanced tests for psychological assessment. For this purpose, participants were asked to complete two assessment tools that test the ability to manage inter- personal conflict, in particular the ROCI-II test (a paper-and-pencil test) and the Enact game platform (a simulation-based serious game). Then, participants were asked to fill a self-report questionnaire. Results show that, even if the se- rious game Enact took much longer to complete, it was generally assessed as significantly more pleasant and interesting, regardless of the gender and degree of the participant. Additionally, perceived time spent on the Enact platform was significantly underestimated, but only by participants who were not gradu- ates/postgraduates in psychology. Also, the majority of the sample reported that new knowledge regarding negotiation and conflict management was acquired dur- ing the Enact gameplay, while this was not reported for the ROCI-II. 10 D. Pacella and D. Marocco References 1. Afzalur Rahim, M., Antonioni, D., Psenicka, C.: A structural equations model of leader power, subordinates’ styles of handling conflict, and job perfor- mance. International Journal of Conflict Management 12(3), 191–211 (mar 2001). https://doi.org/10.1108/eb022855 2. Chakroun, N., Izaute, M., Lamotte, M., Droit-Volet, S.: Metacognitive Question- naire on Time: Feeling of the Passage of Time. Timing & Time Perception 2(3), 339–359 (nov 2014). https://doi.org/10.1163/22134468-00002031 3. Dell’Aquila, E., Marocco, D., Ponticorvo, M., Di Ferdinando, A., Schembri, M., Miglino, O.: Educational Games for Soft-Skills Training in Digital Environments. Springer International Publishing, Cham (2017). https://doi.org/10.1007/978-3- 319-06311-9 4. Deroy, O., Spence, C., Noppeney, U.: Metacognition in Multisensory Perception. Trends in Cognitive Sciences 20(10), 736–747 (oct 2016). https://doi.org/10.1016/j.tics.2016.08.006 5. Filsecker, M., Hickey, D.T.: A multilevel analysis of the effects of external rewards on elementary students’ motivation, engagement and learning in an educational game. Computers & Education 75, 136–148 (2014) 6. Grondin, S.: Timing and time perception: A review of recent behavioral and neuroscience findings and theoretical directions. Attention, Perception, and Psy- chophysics 72(3), 561–582 (apr 2010). https://doi.org/10.3758/APP.72.3.561 7. Jones, M.B., Dunlap, W.P., Bilodeau, I.M.: Comparison of video game and con- ventional test performance. Simulation & Games 17(4), 435–446 (1986) 8. Laurillard, D.: Rethinking university teaching: a framework for the effective use of learning technologies. TechTrends 69 (2010) 9. Marocco, D., Pacella, D., Dell’Aquila, E., Di Ferdinando, A.: Grounding serious game design on scientific findings: The case of ENACT on soft skills training and assessment. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9307, pp. 441–446. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24258-337 10. Pacella, D., Dell’Aquila, E., Marocco, D., Furnell, S.: Toward an automatic classi- fication of negotiation styles using natural language processing. vol. 10498 LNAI, pp. 339–342. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67401-843 11. Rahim, A., Bonoma, T.V.: Managing Organizational Conflict: a Model for Diagnosis and Intervention. Psychological Reports 44(3c), 1323–1344 (1979). https://doi.org/10.2466/pr0.1979.44.3c.1323 12. Thayer, S., Schiff, W.: Eye-contact, facial expression, and the experi- ence of time. Journal of Social Psychology 95(1), 117–124 (feb 1975). https://doi.org/10.1080/00224545.1975.9923242 13. Vereecken, C.A., Maes, L.: Comparison of a computer-administered and paper- and-pencil-administered questionnaire on health and lifestyle behaviors. Journal of Adolescent Health 38(4), 426–432 (2006) 14. Watt, J.D.: Effect of boredom proneness on time perception. Psychological reports 69(1), 323–327 (aug 1991). https://doi.org/10.2466/pr0.1991.69.1.323 15. Wills, S., Leigh, E., Ip, A.: The power of role-based e-learning: Designing and moderating online role play. Routledge (2011)