Computer simulations for teaching an AI algorithm: the case of psychology students struggling with Support Vector Machines

Abstract

Computer simulations can be a great support in education, especially in learning contexts deemed too complex such as AI. In this work we examined how a web-based simulation implementing Support Vector Machines (an AI classification algorithm) can affect perceived learning and cognitive load in a group of psychology students. Results of this preliminary work suggest that the specific implemented simulation did not affect perceived learning but may have differently affected cognitive load in students with different educational backgrounds (scientific vs humanistic). Differences related to the high-school educational background are discussed and future perspectives are drawn.

Keywords

• Simulations
• AI
• Support Vector Machines
• education
• learning

-

Computer simulations under different forms have been used not only for basic research [ 1 ] [ 2 ] [ 3 ] [ 4 ] [ 5 ] but also for educational or edutainment (education plus entertainment) purposes (see for example [ 6 ] [ 7 ]). Simulation in education can be considered a methodological support as the individual can directly manipulate and observe the phenomena under study, thus fostering inquiry-based learning [ 8 ]. Inquiry learning is defined as a process of exploration and manipulation which leads to asking questions, discovering solutions and testing the findings in search of new understanding of experience. The use of simulation for teaching reduces the gap between theory and practice, between concept and application. Students can become familiar with complex tools or skills, directly observe the operation, hypothesize and design experiments.

Serious games for example [ 9 ][ 10 ] and simulated worlds are meant to ease the daunting endeavor of learning topics requiring a dynamical perspective; it is the case of social psychology theories [ 10 ] in which agent-based modeling is crucial in simulating social phenomena. Moreover, simulations are very useful especially when acquiring a certain knowledge can be potentially risky and costly, for example in the case of flight-learning.

Many of the above cited simulations make use of some form of Artificial Intelligence (AI) under the hood. However, in this historical moment of expansion of AI algorithms everywhere (see for example the many App boasting AI in their name across various App stores), we assume it can be useful to disseminate to the general public, or at least to professionals who may get some advantage for their own work, AI principles.

In fact, for professionals not inclined to hard disciplines like mathematics, computer science and the like, AI may appear as a magic or, at the most, a very complicated topic; however, they could benefit from learning some AI principles [ 11 ].

Accordingly, the aim of the present pilot study was to investigate perceived learning and perceived cognitive load with respect to simulation-enhanced learning activity. Perceived learning is indicative, across its three domains (cognitive, affective and psychomotor) of how a learner has acquired a specific knowledge without the need to resort upon traditional form of grading requiring specific questions for each topic [ 12 ]. In particular, the affective component highlights the personal attitude towards a specific topic. This latter aspect to us is critical because it may mark a future interest in using that technology for professional use.

Cognitive load or effort is defined as the load that performing a specific task imposes on the individual's cognitive system [ 13 ]. Particularly, our initial hypothesis was that performing a simulation while reading a related topic content, may have resulted in a higher perceived learning and in a lower cognitive load. 2 2.1

Materials and Methods

Participants 65 Psychology students [Table 1] of the University of Naples “Federico” II were involved in the experimental session and randomly allocated either to the group without simulation (Text condition) or the group with simulation (Software condition). Selection criteria for participants recruitment included normal or corrected-to-normal vision. Both groups completed an initial questionnaire on general information: gender, age, qualification and high school field of study (humanistic or scientific). Informed consent was obtained from all participants.

Gender Educational background

Age

Males

Females Humanistic Scientific

Group Text (No simulation)

Group Software

(With simulation) N = 36

N = 29 16 20 21.6 (1.4) 19 Users facing difficulty in understanding multimedia education systems, can overload their mental resources. In order to increase student success in learning, overload should be avoided. To this end, it is useful to measure the cognitive load with respect to the use of simulations in order to determine if these multimedia environments are effective and helpful for learning.

To obtain a measure of perceived cognitive effort the Cognitive Load Scale (CLS) [ 13 ] [ 14 ] was administered to participants at the end of the session: CLS is a 9-point self-rating scale to assess cognitive load in multimedia learning environments, based on the cognitive load theory paradigm. CLS measures how much mental effort the individual invested in studying or in solving a task in a complex learning environment, from very, very low mental effort to very, very high mental effort.

Scientific evidence suggests that self-reports of perceived learning may be a valid measure of learning. In order to measure perceived learning, we administered the CAP Perceived Learning Scale [ 12 ], that addresses three overlapping domains: cognitive, affective, and psychomotor learning. CAP is a 9-item self-rating scale used to evaluate the effectiveness of learning and educational environments. According to the theoretical background of the CAP scale, Cognitive learning is the recognition of knowledge and development of intellectual skills, Affective learning is defined as an increase of positive attitudes toward the content or subject matter, and Psychomotor learning is associated with skills relating to manual tasks and physical movement that refers to operations with the physical system, as the computer. Experimental sessions, due to COVID-19 restrictions, took place via a web-app.

First, participants completed the initial questionnaire on general information. Then, we presented to the two groups of students an only text learning condition (Text Group) and a text plus simulation condition (Software Group) and asked both to spend at least 30 minutes studying the provided materials. The text to learn was a descriptive text on Support Vector Machines (SVM), an AI classification algorithm for breast cancer diagnostics; the simulation software, developed by [ 15 ], allowed students to directly experience the SVM simulator during the text reading.

After each session (only text or text plus simulation), students were administered with Perceived Learning Scale and Cognitive Load Scale. Again, we hypothesized a role of simulation in lowering cognitive load and leveraging perceived learning. 3

Results

Statistical analyses were conducted using JASP software (Version 0.12.2) [ 16 ]. All 65 participants were included in the analysis.

Regarding Cognitive Load, two-way ANOVA (with Group and Gender set as independent variables) demonstrated that there was no variability between Text and Software groups, F (1.04) = 0.422, p = 0.518, η2 = 0.006, thus both groups perceived almost the same cognitive effort during the learning task (“Text Group” M = 5.722; “Software Group” M = 6.069) between “neither low nor high mental effort” or “rather high mental effort”. Similarly, the main effect of the gender variable was not significant, F (1.27) = 2.422, p = 0.125, η2 = 0.035.

A second two-way ANOVA was then conducted, always setting the Cognitive Load as dependent variable and the Group (Text or Software) and the type of high school formation (Scientific or Humanistic) as independent variables. The main effect of the group variable was not significant, F (1.10) = 0.895, p = 0.348, η2 = 0.013, as well as the main effect of the study field F (1.07) = 0.675, p = 0.414 , η2 = 0.010. However, a significant interaction Group * Field of study resulted (F (1.61) = 5,384, p 0.024, η2 = 0.079).

Descriptive plots (Figure 3) revealed that, while the scientific studies group perceived almost the same level of cognitive effort in both the software condition (M = 5.80) and text-only condition (M = 6.18), with a slight increase in the latter condition, the group of students with a humanistic high school educational background perceived more cognitive effort in the software condition (M = 6.21) compared to the text-only condition (M = 5.32).

We then analyzed any group differences in Perceived Learning Scale general index (CAP): two-way ANOVA results showed significant difference regarding Gender, F (1.41) = 9.590 , p = 0.003, η2 = 0.127), but not regarding Group, F (1,11) = 2.717, p = 0.104, η2 = 0.036) or regarding interaction effect Gender * Group, F (1,10) = 2.424, p = 0.125, η2 = 0.032. Particularly, females in both Text (M = 35.050) and Software (M = 34.895) Group expressed higher perceived learning than males in Text (M = 32.438) and especially in the Software condition (M = 27.000) (Figure 2). To better explore the data with respect to the single subscales of the Perceived Learning Scale, two-way ANOVA was conducted for Cognitive, Affective and Psychomotor Subscales. Results showed that the perception of affective learning did differ across Gender, F (1.11) = 13.724, p = <.001, η2 = 0.172, as the gender main effect was statistically significant: particularly females demonstrated higher perception of affective learning both in Text and Software group. 4

Discussion

Analysis results do not reveal significant differences between the two groups, the one who reads the text together with a Support Vector Machine simulator [ 15 ] and the one who reads the text without the simulator, regarding perceived learning and cognitive load. In particular, the perceived cognitive load [ 13 ] was rather high in both conditions.

The results, however, highlighted a difference with respect to the educational background, as the students with scientific higher-education perceived less cognitive effort in the simulator condition than in the text-only condition, while the students with humanistic background perceived more cognitive effort in the Software condition rather than in the Text condition. A more balanced sample with respect to the school background is necessary in order to draw conclusions on these results, however it could be argued that a more developed inquiry learning style [ 8 ], such as that promoted by high schools oriented to scientific studies, may help students to be more inclined to use a simulator for learning rather than just reading the text. This hypothesis should be explored in future works.

An interesting aspect, however, emerged with respect to gender: results highlighted that females demonstrated higher perception of affective learning in both groups, Text and Software. In addition, affective learning perception was slightly higher in the group with simulation than in the group without simulation. In contrast, males perceived more affective learning in the Text group than in the Software group and both measures were lower than the female’s ones.

Considering the limits of the present study given by the low sample size, which is even not gender-balanced, it would be interesting to explore in more details the gender perception difference to understand whether this difference could be interpreted in the light of the target topic. Breast cancer topic is particularly pregnant for females, who may have felt emotionally closer to the subject treated and may have perceived more attitude to the topic. Therefore, the topic itself may have been the factor that influenced the perception of greater emotional learning. In addition, the females who were in the group with the simulator perceived even more affective learning: we cannot draw conclusions about it, however, it would be interesting to understand if the addition of a simulation software together with meaningful topics for female participants could involve them more in affective learning. However, due to the disproportion between male and female participants, these are only hypotheses and there is a need for further studies with a larger and gender-balanced sample.

It is also possible to reflect on the fact that higher affective involvement in the learning process could have had an impact on the perceived cognitive load of females in the Software group: this hypothesis would be in line with literature data that underline how emotion may relate to cognitive load during learning [ 17 ]. Learners’ emotional states during learning could affect learning outcomes [ 18 ]. The processing of emotion and cognition involves highly connected cortical networks; thus, information encoding could be affected by emotion and emotion may directly affect cognitive resources and mental effort investment [ 19 ] [ 20 ]. It would be interesting for future studies to evaluate both cognitive load and perceived learning comparing non neutral and neutral topics of the text and of simulation, to better understand the role that a specific theme has on general and affective perceived learning, both for males and females.

It is necessary to take in account some limits of the present study: first of all, the Support Vector Machine simulator was used in its original form, according to [ 15 ]'s study, and not adapted to the sample, so in the future it would be useful to simplify and tailor the instrument to the specific target. As already mentioned, a larger and more gender-balanced sample, as well as a more educational background-balanced one, would have made it possible to obtain more reliable and significant results, especially regarding the affective scale’s scores. Another limitation of the study is that the CAP Perceived Learning Scale [ 11 ] has been translated specifically for the present study as no validation with an Italian sample is provided. In the future it would be necessary to validate the Italian version in order to repeat the study with a validated scale. Finally, given the restrictions due to the Covid-19 pandemic, it was not possible to carry out the experimental sessions live, and the remote procedure did not provide for adequate control of the setting.

Conclusion

Studying the effects of simulations on perceived learning and cognitive load could be useful in helping to tune specific software (as that used in for the present paper) to target a specific population. Probably, the simulator used in the present study is still too complex and could have increased cognitive effort instead of decreasing it; therefore it would be useful for the future to simplify the usability of the software, perhaps adding a virtual tutor able to guide the inquiry-based learning; without an efficient guide, in fact, the risk to hinder rather foster learning is real [ 8 ]. Moreover, the usability and engagement of technological tools is a key criterion that motivate the learning process [ 21 ] [ 22 ].

In any case, the present study has highlighted a difference, albeit subtle, between genders with respect to perceived affective learning, and new research hypotheses have been put forward to understand whether this difference could be related to the specific dataset (breast cancer data) used to train support vector machines and to explain how the AI algorithm works. A difference with respect to scientific and humanistic educational backgrounds also emerged, that must be explored in more depth. Finally, future works should focus on using measures of perceived learning and cognitive load to tune simulation-based learning tools in order to maximize their educational impact on specific target audiences.

References

1. Miglino , O. , Pagliarini , L. , Cardaci , M. , Gigliotta , O. : TeamSim: An Educational Microworld for the Teaching of Team Dynamics . In: Tsihrintzis, G.A. , Virvou , M. , Howlett , R.J. , and Jain , L.C. (eds.) New Directions in Intelligent Interactive Multimedia, pp. 417 - 425 . Springer, Berlin, Heidelberg ( 2008 ).

2. Ponticorvo , M. , Gigliotta , O. , Miglino , O. : Simulative Models to Understand Numerical Cognition . In: Ferrández Vicente, J.M. , Álvarez-Sánchez , J.R. , de la Paz López, F. , Toledo Moreo , J. , and Adeli , H. (eds.) Natural and Artificial Computation for Biomedicine and Neuroscience , pp. 75 - 84 . Springer International Publishing, Cham ( 2017 ).

3. Pacella , D. , Ponticorvo , M. , Gigliotta , O. Miglino , O. : Basic emotions and adaptation. A computational and evolutionary model . PLOS ONE 12 , 1 - 20 ( 2017 ).

4. Gigliotta , O. , Bartolomeo , P. , Miglino , O. : Approaching neuropsychological tasks through adaptive neurorobots . Connection Science 27 , 153 - 163 ( 2015 ).

5. Rotondaro , F. , Ponticorvo , M. , Gigliotta , O. , Pinto , M. , Pellegrino , M. , Gazzellini , S. , Dolce , P. , Miglino , O. , Doricchi , F. : The Number Interval Position Effect (NIPE) in the mental bisection of numerical intervals might reflect the influence of the decimal-number system on the Gaussian representations of numerosities: A combined developmental and computational-modeling study . Cortex 114 , 164 - 175 ( 2019 ).

6. Miglino , O. , Gigliotta , O. , Ponticorvo , M. , Nolfi , S. : Breedbot: An edutainment robotics system to link digital and real world . Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 4693 LNAI , 74 - 81 ( 2007 ).

7. Council , N.R. : Learning Science Through Computer Games and Simulations . Washington, DC: The National Academies Press ( 2011 ).

8. Jong, T. de: Technological Advances in Inquiry Learning. Science . 312 , 532 - 533 ( 2006 ).

9. Rubinacci , F. , Ponticorvo , M. , Gigliotta , O. Miglino , O. Breeding robots to learn how to rule complex systems . In: Merdan M. , Lepuschitz W. , Koppensteiner G. , Balogh R . (eds.) Robotics in Education. Advances in Intelligent Systems and Computing , pp. 137 - 142 . Springer International Publishing, Cham ( 2017 ).

10. Dell'Aquila , E. , Marocco , Davide, Ponticorvo, M. , Di Ferdinando , A. , Schembri , M. , Miglino , O. : Educational Games for Soft-Skills Training in Digital Environments . Springer International Publishing, Cham ( 2017 ).

11. Gigliotta , O. , Miglino , O. , Schembri , M. , Di Ferdinando , A. : Building up serious games with an artificial life approach: Two case studies . In: Evolution, Complexity and Artificial Life , pp. 149 - 158 ( 2014 ).

12. Rovai , A.P. , Wighting , M.J. , Baker , J.D. , Grooms , L.D. : Development of an instrument to measure perceived cognitive, affective, and psychomotor learning in traditional and virtual classroom higher education settings . The Internet and Higher Education 12 , 7 - 13 ( 2009 ).

13. Paas , F. , Ayres , P. , Pachman , M. : Assessment of cognitive load in multimedia learning: theory, methods and applications. In Recent innovations in educational technology that facilitate student learning , pp. 11 - 35 . Information Age Publishing Inc., Charlotte, NC ( 2008 ).

14. Schmeck , A. , Opfermann , M. , Van Gog , T. , Paas , F. , Leutner , D. : Measuring cognitive load with subjective rating scales during problem solving: differences between immediate and delayed ratings . Instructional Science 43 ( 1 ), 93 - 114 ( 2015 ).

15. Walsh , P. , Lynch , J. , Kelly , B. , Palu , C. , Gigliotta , O. , Fuccio , D.: Demonstrating Machine Learning for Cancer Diagnostics . In: Collaborative European Research Conference , pp. 185 - 194 , ( 2019 ).

16. JASP Team ( 2020 ). JASP (Version 0 .12.2) [Computer software].

17. Plass , J. L. , Kalyuga , S. : Four ways of considering emotion in cognitive load theory . Educational Psychology Review 31 ( 2 ), 1 - 21 , ( 2019 ).

18. Heidig , S. , Müller , J. , Reichelt , M. : Emotional design in multimedia learning: Differentiation on relevant design features and their effects on emotions and learning . Computers in Human Behavior 44 , 81 - 95 ( 2015 ).

19. Lazarus , R. S. Thoughts on the relations between emotion and cognition . American psychologist , 37 ( 9 ), 1019 ( 1982 ).

20. Pessoa , L. : On the relationship between emotion and cognition . Nature Reviews Neuroscience 9 ( 2 ), 148 - 158 ( 2008 ).

21. Ibili , E. , Billinghurst , M. : Assessing the Relationship between Cognitive Load and the Usability of a Mobile Augmented Reality Tutorial System: A Study of Gender Effects . International Journal of Assessment Tools in Education 6 ( 3 ), 378 - 395 ( 2019 ).

22. Soffer , T. , Yaron , E.: Perceived Learning and Students' Perceptions toward Using Tablets for Learning: The Mediating Role of Perceived Engagement among High School Students . Journal of Educational Computing Research 55 , 951 - 973 ( 2017 ).