Emotions Evaluation on Virtual Learning Environments: Students perceptions

Abstract

When evaluating a web platform, it is important to analyze how the user feels during its use. Therefore, it is important to know the perceptions of the interaction that users/students of a Virtual Learning Environment have with these platforms. This article leave a list of emotions to evaluate the functionalities of educational platforms, validated by students and experts. This list of emotions in Spanish will be very useful for the field of Affectivity and Educational Technology, since it has the support of recognized professionals as well as students of higher education, of different ages and countries of origin, which enriches the validity and heterogeneity of the study.

Keywords

• Human computer interaction
• Affective Computing
• User Experience
• Emotional Evaluation
• Virtual Learning Environments

-

The boom in the use of virtual platforms nowadays is enhanced by the COVID-19 pandemic [ 1 ][ 2 ]. In the Virtual Learning Environment (VLE, for its acronym in English), the student interacts with the platform to follow the course lessons [ 3 ]. The need to design and provide educational platforms and resources that are not only accessible but also attractive to students has been emphasized [ 4 ][ 5 ]. Therefore, it is important to know the perceptions of the interaction that users/students of a VLE have with these platforms [ 6 ].

The User Experience (UX) constitutes a holistic evaluation of the interaction with a product [ 7 ]. This is an evaluation that includes various facets, including the emotional, which takes into account satisfaction attributes from the affective point of view of the user [ 8 ], [ 9 ].

The functionalities of these VLEs may vary, but in general, they all have basic functions that are common to most of them. The work cited in [ 10 ], was taken as a basis, which investigated the most important characteristics to take into account in the evaluation of a VLE. Those authors determine the following characteristics to take into account when evaluating a VLE:  Pedagogical facility: this category refers to the functionalities that the environment provides to teachers and students for the development of the teaching and learning process.  Support, help and documentation: information of this type should be easy to find, focused on the user's task, with the list of concrete steps that will be carried out, and not be too long.  Content: the content or courseware is the learning material that is made available to the student.  User interface: consists of those aspects of the system which the user comes into contact, physically, perceptually or conceptually.  Error handling: cancellation, revocation of actions, error prevention, error detection, error recognition, visible and clear error messages, recovery mechanisms and error solution.  Tools: used for the management of academic content, improving user intercommunication.  Flexibility: refers to the multiplicity of ways in which the user and the system exchange information. It involves providing user control, substitution capacity and adaptability.  Standards: refers to how the web application conforms to norms, standards, conventions, or design guides in the web domain.

After identifying the characteristics of VLE, it is necessary to know the existing methods to evaluate these characteristics from the affective aspect, considering that the UX emphasizes emotional aspects. Table 1 show some tools for affective evaluations.

User emotion assessment tool that uses

graphic scales for dimensions: Valence “pleasant” to “unpleasant”; Arousal “excited” to “calm”; Dominance: “out of control” to “in control”

Cartoons express eight discrete emotions of users: Joy, Desire, Fascination, Satisfaction, Sadness, Disgust, Boredom, and Dissatisfaction.

PrEmo (Product Emotion) [15]– [18]

Self-report instrument that non-verbally

measures, various user emotions, both satisfactory and unpleasant. "Geneva Emotions Wheel +" Emotional state of the user during the (GEW +) [ 19 ] based on [ 20 ] evaluation

These questionnaires use emotion scales that may be inaccurate and/or confusing for the people who participate in the assessments, in addition to the fact that not all of them can be adapted to assess the different functionalities of a VLE.

In order to offer an alternative to existing instruments, this work aims to obtain a scale of emotions and consider it when evaluating a VLE from an affective point of view.

The emotions scale used in PrEmo [ 21 ] was chosen as a base, due to the ease with which the emotions of the participants about the characteristics of a product can be consulted, in addition to being one of the best known and utilized instruments worldwide, therefore, one of the most tested methods and available in different languages, including Spanish.

In previous publication [ 22 ], we interviewed a group of experts in the field of Affectivity and Educational Technology, from several Spanish-speaking countries, those most significant emotions that a higher education student may express during the use of each of the VLE functionalities. The aim of the present work is to validate with students that list of emotions [ 22 ].

The article is organized as follows: section two presents related works, on the one hand related to UX studies, and on the other hand a summary of works that analyzed emotions and affectivity for VLE. Then, section three details the study conducted with high-level students, compiling the emotions that they can perceive when evaluating VLE. Subsequently, section four analyzes the results of the study to validate the list of emotions obtained previously from experts, leaving as a result the final list of appropriate emotions to evaluate each VLE functionality. Finally, in section five the conclusions and future work of the research are presented. 2

Related Works

The works related to the subject of this article are presented. On the one hand, a review of UX evaluation studies related to Affectivity, and on the other hand, previous works that analyze emotions and affectivity for VLE. 2.1

The related research regarding UX evaluation and affectivity

In [ 23 ] their authors researched emotion and how it influences motivation and moods, defining models for emotional design and the dimensions of emotion.

The authors of [ 24 ] establish the conceptualization of interaction design, interaction paradigms, how affective aspects influence, expressive interfaces and the four basic activities of interaction design in conjunction with Human-Computer Interaction (HCI) lifecycle models.

In addition, aspects of the affective interface in the game (challenge, aesthetics and visualization, narrative and characters, usability and interface and flow) are determined in [ 25 ].

In the work cited in [ 26 ], the authors characterized emotions, proposed a framework for affective design, and established methods for performing emotion assessment.

Within this framework, the work cited in [ 27 ] contributes to the study of emotional design by defining user experience design, providing guidelines for e-learning applications and providing content strategies.

Regarding the evaluation of VLEs, in [ 28 ] the authors propose some ideas to achieve an evaluation model that incorporates usability criteria. 2.2

Background on Affective Evaluations for VLEs

However, it is noteworthy that up to date few studies have been found [ 29-36 ], that clearly defines a list of relevant emotions to evaluate VLE:

With respect to attitude, enjoyment, experience, self-efficacy, and speed, the work cited in [ 29 ] provides an updated framework of the human factors that positively influence the success and effectiveness of VLE, which have the greatest influence on user satisfaction.

Two categories of emotions were used in [ 30 ], explicit: collected directly from the learner through self-reported surveys; and implicit emotions: those inferred discretely during the learning process. The authors indicate that VLEs pose several challenges in understanding the different types of affect experienced by a learner. Uses sentiment analysis with a bag-of-words model and does not consider other implicit measures of emotions.

Learnability and system effectiveness had a significant influence on user satisfaction. The study published in [ 31 ] examined the relationship of each usability attribute with the user's emotional responses. Should improve the user interface in terms of usability and user's emotional responses while using the system, to improve user satisfaction.

To evaluate the quality of an application at low cost using two scale-like methods: AttrakDiff and Hedonic Utility Scale. In [ 32 ] a UX evaluation on the Edmodo educational mobile application. It is necessary to adapt these methods to provide a more complete UX report, allowing users to subjectively report their experiences and, consequently, identify the problems that affected the UX.

Strengthening students' self-esteem and emotions have a decisive influence on whether students continue or abandon this educational process. In study [ 33 ], the most common web accessibility errors presented by the VLE and how these errors can induce different emotions in students were identified. The emotions were obtained from surveys answered by students.

In [ 34 ] developed UX evaluation metrics for VLEs. For the hedonic quality dimension, stimulation and novelty. While For the pragmatic dimension of quality with transparency, efficiency and reliability. The same lead author in [ 35 ] conducts a literature review and concludes that no UX evaluation method for VLE assesses user feelings.

To culminate this bibliographical review, the authors of the paper cited in [ 36 ], leave recommendations:  Have students detail their experiences of how they felt about using a product and the difficulties they had  Be specific using semantic differential scales  Specify which aspects are being evaluated by adjectives or let students explain their evaluation  As the VLE has many different characteristics and an adjective may fit one but not another  Evaluation should be quick and simple  Use quick and simple evaluation methods, such as semantic differential scales.

These contributions allow us to affirm the importance of performing an evaluation of the UX of VLEs from an affective point of view, this type of evaluations will contribute to improve UX and user satisfaction. However, so far, there are no existing or no known yet publications that provide a valid list to evaluate the functionalities of a VLE. 3

Students Test

When evaluating an educational platform, it is important to analyze how the user feels during its use. Therefore, it is important to know the perceptions of the interaction that users/students of a Virtual Learning Environment have with these platforms.

In a previous experience, experts in the field of Affectivity and Educational Technology were consulted on possible student emotions when use a VLE, resulting in an initial list of emotions [ 22 ]. As a complement to that experience, in this opportunity a test with graduate and postgraduate students was designed, with the purpose of improving the list of emotions proposed by experts in order to provide reliability and validity to it. 3.1

Questionnaire

A questionnaire was designed consisting of eight "multiple choice" questions with emotions, one for each VLE characteristic mentioned above, noting that they could choose more than one emotion or none at all. A copy of this questionnaire can be accessed from the following link https://drive.google.com/file/d/1pzKk-pS0FZ5G07V4EciqH4qryMtctVH/view?usp=sharing.

The questions were along the lines of "For Pedagogical Facility: When using the functionalities provided by the educational platform for the teaching/learning process such as: doing group work, tracking schedules and class attendance, evaluations...it is possible that it makes me feel the following emotions....” and so on for all characteristics.

This questionnaire was online, in Google Forms, resulting in a simple, economical and direct way of obtaining the opinion of the respondents. Students from Argentina, Colombia, Spain, and Brazil participated without any difficulty, and from any device, even from their own cell phones, thus avoiding the inconveniences of distance, time and synchronization. 3.2

Selection of Emotions

Participants were asked to choose from a set of emotions for each VLE characteristic, based on the initial list generated with expert support [ 22 ]. The student could choose more than one emotion from the suggested list (multiple choice) and even add emotions that were not in the proposed options and/or leave comments or recommendations through open-ended responses.

The scale of emotions emerged as result of the previous experience [ 22 ], in which the following negative emotions were defined with the support of experts: ─ Indignation: rage, anger due to unmanageable behavior. ─ Contempt: contempt, lack of affection. ─ Dissatisfaction: Unsatisfactory experience. ─ Disappointment: does not meet expectations ─ Unpleasant Surprise: Unexpected negative characteristic. ─ Boredom: Uninteresting characteristics. ─ Tension: Nervousness, stress, lack of tranquility. ─ Frustration: failure to achieve a goal, impossibility of satisfying a need ─ Confusion: lack of understanding.

In addition, the following positive emotions were defined: ─ Desire: drive to use the tool. ─ Inspiration: motivation to do new things ─ Admiration: Appreciation of the tool. ─ Satisfaction: Satisfied with the available features ─ Fun: Fun experience. ─ Commitment: obligation to comply 3.3

Participants

In order to validate the list of emotions for functionalities/characteristics of a VLE defined by experts [ 22 ], teachers from Argentina, Colombia, Spain and Brazil from different educational levels were asked to distribute the survey to their students. As a result, a sample of 36 students was obtained. The subjects were from different countries, of different age ranges, who participated voluntarily. In addition, each participant answered his or her own questionnaire without knowing the answers of his or her peers, thus avoiding response bias. Regarding age, 17 belonged to the 18-25 age group, 15 were over 30 years old and 4 were 26-30 years old. In terms of nationality, 19 students were from Argentina, 10 from Colombia, 5 from Spain and 2 from Brazil. This is an important aspect to take into account to indicate the representativeness and heterogeneity of the participants in terms of culture, language and preferences, enriching the validity of the study. 3.4

Test Results

For each functionalities of a VLE, were analyzed the students' responses. The results are shown in graphical form, for better reading and visualization. For this study, was performed a frequency analysis. As can be seen, in Table 2, the votes of the 36 students for each emotion in the top row with respect to the “Pedagogical Facility” functionality were marked in green. t n e m e g a g n P E P P P P P P P P P P P P P P P P P P P P

Then, in the "Total" row, was obtained the frequency of votes for each emotion for this functionality. In the next row, the percentage of the total that it represents. The most voted emotion for this functionality was "engagement" (compromiso in Spanish), with 21 votes, corresponding to 58% of the total votes (green box), more than half of the total votes. The least chosen emotion was "contempt" (desprecio in Spanish), with 3 votes, 8% of the total (orange box). The emotion "disgust" was not chosen for this functionality (0 zero votes).

This same analysis was repeated for all emotions and for each VLE functionality. Table 3 shows in summary the number of emotions associated with each characteristic VLE according to the survey applied. As can be seen in the Table 3, where the first column on the left indicates the set of emotions analyzed. The light blue line separates negative and positive emotions. The first row presents each characteristic of the VLE. In the right part of the table the averages, minimums and maximums of each emotion were calculated, as well as the averages of the set of negative emotions and average of the positive ones.

Disgust 0 0 1 1 4 0 Indignation 4 5 3 2 7 1

Contempt 3 4 3 2 5 1 Dissatisfaction 16 7 9 10 10 7 Disappointment 11 7 9 5 8 5

Surprise 5 3 4 4 7 1 Unpleasantness

Boredom 11 10 11 10 5 8

Tension 18 0 0 0 0 0 Frustration 13 7 6 5 7 7 Confusion 15 15 7 9 11 6

Desire 13 5 8 8 0 10 8 5 57 7,13 Inspiration 7 5 10 3 2 4 7 4 42 5,25 SAadtimsfiarcattiioonn 142 143 198 198 131 1107 190 188 15167 174,,0603 54,33

Fun 5 3 3 5 0 4 5 3 28 3,50

Engagement 21 0 0 0 0 0 5 0 26 3,25

Considering all features (table 3), the most chosen emotion was "satisfaction" (satisfacción in Spanish - green shading), while the least chosen emotion was "contempt" (desprecio in Spanish - orange shading). Then, doing an analysis respect to the emotions, it is observed that positive emotions were more chosen to evaluate all the functionalities of a VLE. However, as can be seen in Fig. 1, the choice of emotions behaves in a variable way for all functionalities, with a peak in the emotion "satisfaction" and average values in the rest. Being "disgust" the least chosen of all.

This list of emotions in Spanish will be very useful for evaluate affectively VLE functionalities, since it was constructed with the help of higher education students of different genders, ages and countries of origin, which enriches the validity and heterogeneity of the study. 4

Student Validation of Expert-defined Emotions

In previous publications [ 22 ], we interviewed a group of experts in the field of Affectivity and Educational Technology, from several Spanish-speaking countries, those most significant emotions that a higher education student may express during the use of each of the VLE functionalities [38].

The aim of the present work is to validate with students those list of emotions [ 22 ]. In order to compare the values of both experiences, were calculated the percentages of proportions of votes for each emotion in each VLE characteristic. That is, for example, if experts chose the emotion 5 times, the proportion results from dividing 5 by 12 (total number of experts who answered) and in percentage format would be 41%. In the case of students, it was divided into 36 (total number of students who responded).

We can appreciate the following similarities and differences: analyzing the votes for all VLE functionalities with respect to disgust (See table 4); we can notice that both, experts and students, did not choose it for “Pedagogical Facility” and “Support”, 0% in this cells.

Final validated Emotions for VLEs

As a result of both experiences, was obtained a validated list of emotions for each VLE characteristic. To determine the final list of emotions for functionalities of a VLE validated by experts and students, it was determined as an inclusion rule that emotions that were chosen at least once, by an expert or by a student, would be part of this final list.

Table 6 shows a summary of the negative emotions selected by characteristic. The emotions that were selected are marked in green. In addition, the table 7 shows the positive emotions selected. t p m e t n o P C P P P P P P P

Flexibility

Standards

References

1. H. Fardoun , C. González , C. A. Collazos , and M. Yousef , “ Exploratory Study in Iberoamerica on the Teaching-Learning Process and Assessment Proposal in the Pandemic Times = Estudio exploratorio en iberoamérica sobre procesos de enseñanza-aprendizaje y propuesta de evaluación en tiempos de pandemia,” Educ. Knowl. Soc. , vol. 21 , 2020 .

2. F. J. García-Peñalvo , A. Corell , V. Abella-García , and M. Grande , “ La evaluación online en la educación superior en tiempos de la COVID- 19 ,” Educ. Knowl. Soc. , vol. 21 , no. 0 , p. 26 , 2020 .

3. I. T. Plata and D. B. Alado , Evaluating the Perceived Usability of Virtual Learning Environment in Teaching ICT Courses , vol. 1 . 2015 .

4. E. Crisol-Moya , L. Herrera-Nieves , and R. Montes-Soldado , “ Educación virtual para todos: una revisión sistemática , ” Educ. Knowl. Soc. , vol. 21 , no. 0 , p. 13 , Jun . 2020 .

5. F. J. García-Peñalvo , A. Miguel , and S. Pardo , “ Una revisión actualizada del concepto de eLearning . Décimo Aniversario,” vol. 16 , no. 1 , 2015 .

6. E. Martínez Caro, “ E-Learning: Un Análisis Desde El Punto De Vista Del Alumno , ” RIED. Rev. Iberoam. Educ. a Distancia , vol. 11 , no. 2 , pp. 151 - 168 , 2012 .

7. M. Hassenzahl , “ The Thing and I: Understanding the Relationship Between User and Product ,” Springer, Cham, 2018 , pp. 301 - 313 .

8. K. Capota , M. van Hout , and T. van der Geest, “ Measuring the emotional impact of websites,” in Proceedings of the 2007 conference on Designing pleasurable products and interfaces - DPPI '07 , 2007 , p. 135 .

9. L. Masip Ardévol, “ User experience methodology for the design and evaluation of interactive systems ,” University of Lleida, 2013 .

10. J. I. Cocunubo-Suárez , J. A. Parra-Valencia , and J. E. Otálora-Luna , “ Propuesta para la evaluación de Entornos Virtuales de Enseñanza Aprendizaje con base en estándares de Usabilidad Evaluation of Virtual Teaching- Learning Environments based on usability standards Cómo citar / How to cite Propuesta para la evaluación de En,” vol. 21 , no. 41 , pp. 123 - 7799 , 2018 .

11. M. Hassenzahl , M. Burmester , and K. Franz , “ AttrakDiff: A questionnaire to measure perceived hedonic and pragmatic quality,” Mensch Comput ., vol. 57 , 2003 .

12. B. Laugwitz , T. Held , and M. Schrepp , “ Construction and Evaluation of a User Experience Questionnaire ,” Springer, Berlin, Heidelberg, 2008 , pp. 63 - 76 .

13. Bradley , M. M. and Lang , P. J., “ Measuring emotion: the self-assessment manikin and the semantic differential,” J. Behav . Ther. Exp. Psychiatry , vol. 25 , no. 1 , pp. 49 - 59 , 1994 .

14. G. Huisman, M. van Hout , E. van Dijk, T. van der Geest, and D. Heylen , “LEMtool,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI '13 , 2013 , p. 351 .

15. P. Desmet, “ Pieter Desmet Chapter 9 ,” Funology From usability to enjoyment, pp. 111 - 123 , 2005 .

16. P. M. A. Desmet , M. A. Blythe , A. F. Monk , K. Overbeeke , and P. C. Wright , “ Measuring Emotions Development and application of an instrument to measure emotional responses to products .”

17. P. M. Desmet , “ Measuring emotions: Development of an instrument to measure emotional responses to products,” Funology from usability to enjoyment . Kluwer Acad. Publ. Dordrecht, Boston, Londres, no. Blythe, M. A. ; Overbeeke , K. ; Monk , A. F. ; Wright , P. C. (ed.), 2003 .

18. G. Laurans and P. Desmet , “ Introducing Premo2 New Directions for the Non - Verbal Measurement of Emotion in Design,” Proc. 8th Int. Des. Emot. Conf. , 2012 .

19. Y. A. Méndez-Alegría , C. A. Collazos , T. Granollers , and R. Gil , “Rueda de emociones de Ginebra+ : instrumento para la valoración emocional de los usuarios mientras participan en una evaluación de sistemas interactivos , ” Dyna , vol. 91 , no. 2 , pp. 151 - 155 , 2016 .

20. M. Fritz, “ Reinventing the Wheel: Emotional Awareness Enhancement in Computer-Mediated Collaboration with the Dynamic Emotion Wheel .” 2015 .

21. P. Desmet, “Measuring Emotion: Development and Application of an Instrument to Measure Emotional Responses to Products ,” 2003 .

22. L. N. Aballay , S. V. Aciar , and C. A. Collazos , “ Emotions for Virtual Learning Environments,” IEEE-RITA , vol. 9 , no. 3 , 2021 . (Not yet published)

23. 23. V. Sacharin , K. Schlegel , and K. R. Scherer , “Geneva Emotion Wheel Rating Study,” 2012 .

24. M. Blythe and M. Hassenzahl , “The Semantics of Fun: Differentiating Enjoyable Experiences,” Springer, Cham, 2018 , pp. 375 - 387 .

25. W. Aarron, Designing for emotion (pp. 978 - 1 ). New York: A book apart. 2012 .

26. Y. -H. Shih and M. Liu, “ The Importance of Emotional Usability,” J. Educ . Technol. Syst., vol. 36 , no. 2 , pp. 203 - 218 , Dec. 2007 .

27. M. L. Wong , C. W. Khong , and H. Thwaites , “ Applied UX and UCD Design Process in Interface Design,” Procedia - Soc. Behav. Sci. , vol. 51 , pp. 703 - 708 , Jan. 2012 .

28. L. A. Hasan and K. T. Al-Sarayreh, “An Integrated Measurement Model for Evaluating Usability Attributes , ” Proc. Int. Conf. Intell. Inf. Process. Secur. Adv. Commun. , p. 94 , 2015 .

29. M. M. Alomari , H. El-Kanj , N. I. Alshdaifat , and A. Topal , “ A framework for the impact of human factors on the effectiveness of learning management systems , ” IEEE Access , vol. 8 , pp. 23542 - 23558 , 2020 .

30. M. Syed , M. Chetlur , S. Afzal , G. A. Ambrose , and N. V. Chawla , “ Implicit and explicit emotions in MOOCs,” EDM 2019 - Proc. 12th Int. Conf. Educ. Data Min ., no. Edm , pp. 432 - 437 , 2019 .

31. N. Phongphaew and A. Jiamsanguanwong , “ The Usability Evaluation Concerning Emotional Responses of Users on Learning Management System , ” Proc. 2016 6th Int. Work. Comput. Sci. Eng . (WCSE 2016 ), pp. 43 - 48 , 2016 .

32. L. Marques , W. Nakamura , N. Valentim , L. Rivero , and T. Conte, “ Do Scale Type Techniques Identify Problems that Affect User eXperience? User Experience Evaluation of a Mobile Application ,” 2018 .

33. T. Acosta and S. Luján-Mora , “Analysis of Emotion in the Use of Accessible Learning Management Systems by Students,” in 9th Annual International Conference of Education, Research and Innovation (ICERI 2016 ), 2016 .

34. W. T. Nakamura, L. C. Marques , L. Rivero , and E. H. T. De Oliveira, “ Are Generic UX Evaluation Techniques Enough? A study on the UX Evaluation of the Edmodo Learning Management System,” Brazilian Symp . Comput. Educ. (Simpósio Bras. Informática na Educ . - SBIE) , vol. 28 , no. 1 , p. 1007 , Oct . 2017 .

35. W. T. Nakamura , E. H. T. De Oliveira , and T. Conte, “ Usability and user experience evaluation of learning management systems a systematic mapping study ,” in ICEIS 2017 - Proceedings of the 19th International Conference on Enterprise Information Systems , 2017 , vol. 3 , pp. 97 - 108 .

36. W. T. Nakamura, L. C. Marques , L. Rivero , E. H. T. De Oliveira , and T. Conte, “ Are scalebased techniques enough for learners to convey their UX when using a Learning Management System?” Rev. Bras. Informática na Educ ., vol. 27 , no. 01 , p. 104 , Jan . 2019 .