Emotions Evaluation on Virtual Learning Environments: Students perceptions Laura N. Aballay1, César A. Collazos2 and Silvana V. Aciar3 1 Universidad Nacional de San Juan, San Juan, Argentina lnaballay@gmail.com 2 Universidad del Cauca, Popayán, Colombia ccollazo@unicauca.edu.co 3 Universidad Nacional de San Juan, San Juan, Argentina silvanav.aciar@gmail.com Abstract. When evaluating a web platform, it is important to analyze how the user feels during its use. Therefore, it is important to know the perceptions of the interaction that users/students of a Virtual Learning Environment have with these platforms. This article leave a list of emotions to evaluate the functionalities of educational platforms, validated by students and experts. This list of emotions in Spanish will be very useful for the field of Affectivity and Educational Technol- ogy, since it has the support of recognized professionals as well as students of higher education, of different ages and countries of origin, which enriches the validity and heterogeneity of the study. Keywords: Human computer interaction, Affective Computing, User Experi- ence, Emotional Evaluation, Virtual Learning Environments. 1 Introduction The boom in the use of virtual platforms nowadays is enhanced by the COVID-19 pan- demic [1][2]. In the Virtual Learning Environment (VLE, for its acronym in English), the student interacts with the platform to follow the course lessons [3]. The need to design and provide educational platforms and resources that are not only accessible but also attractive to students has been emphasized [4][5]. Therefore, it is important to know the perceptions of the interaction that users/students of a VLE have with these platforms [6]. The User Experience (UX) constitutes a holistic evaluation of the interaction with a product [7]. This is an evaluation that includes various facets, including the emotional, which takes into account satisfaction attributes from the affective point of view of the user [8], [9]. The functionalities of these VLEs may vary, but in general, they all have basic func- tions that are common to most of them. The work cited in [10], was taken as a basis, Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 2 which investigated the most important characteristics to take into account in the evalu- ation of a VLE. Those authors determine the following characteristics to take into ac- count when evaluating a VLE:  Pedagogical facility: this category refers to the functionalities that the environment provides to teachers and students for the development of the teaching and learning process.  Support, help and documentation: information of this type should be easy to find, focused on the user's task, with the list of concrete steps that will be carried out, and not be too long.  Content: the content or courseware is the learning material that is made available to the student.  User interface: consists of those aspects of the system which the user comes into contact, physically, perceptually or conceptually.  Error handling: cancellation, revocation of actions, error prevention, error detection, error recognition, visible and clear error messages, recovery mechanisms and error solution.  Tools: used for the management of academic content, improving user intercommu- nication.  Flexibility: refers to the multiplicity of ways in which the user and the system ex- change information. It involves providing user control, substitution capacity and adaptability.  Standards: refers to how the web application conforms to norms, standards, conven- tions, or design guides in the web domain. After identifying the characteristics of VLE, it is necessary to know the existing methods to evaluate these characteristics from the affective aspect, considering that the UX emphasizes emotional aspects. Table 1 show some tools for affective evaluations. Table 1. Affective questionnaires. Tool Emotions measured AttrakDiff [11] Both measure product acceptance. User Experience Questionnaire (UEQ) [12] SAM (Self-Assessment Mani- User emotion assessment tool that uses kin)[13] graphic scales for dimensions: Valence “pleasant” to “unpleasant”; Arousal “ex- cited” to “calm”; Dominance: “out of con- trol” to “in control” LEM-Tool [14] Cartoons express eight discrete emotions of users: Joy, Desire, Fascination, Satisfac- tion, Sadness, Disgust, Boredom, and Dis- satisfaction. 3 PrEmo (Product Emotion) [15]– Self-report instrument that non-verbally [18] measures, various user emotions, both satis- factory and unpleasant. "Geneva Emotions Wheel +" Emotional state of the user during the (GEW +) [19] based on [20] evaluation These questionnaires use emotion scales that may be inaccurate and/or confusing for the people who participate in the assessments, in addition to the fact that not all of them can be adapted to assess the different functionalities of a VLE. In order to offer an alternative to existing instruments, this work aims to obtain a scale of emotions and consider it when evaluating a VLE from an affective point of view. The emotions scale used in PrEmo [21] was chosen as a base, due to the ease with which the emotions of the participants about the characteristics of a product can be consulted, in addition to being one of the best known and utilized instruments world- wide, therefore, one of the most tested methods and available in different languages, including Spanish. In previous publication [22], we interviewed a group of experts in the field of Affec- tivity and Educational Technology, from several Spanish-speaking countries, those most significant emotions that a higher education student may express during the use of each of the VLE functionalities. The aim of the present work is to validate with students that list of emotions [22]. The article is organized as follows: section two presents related works, on the one hand related to UX studies, and on the other hand a summary of works that analyzed emotions and affectivity for VLE. Then, section three details the study conducted with high-level students, compiling the emotions that they can perceive when evaluating VLE. Subsequently, section four analyzes the results of the study to validate the list of emotions obtained previously from experts, leaving as a result the final list of appropri- ate emotions to evaluate each VLE functionality. Finally, in section five the conclusions and future work of the research are presented. 2 Related Works The works related to the subject of this article are presented. On the one hand, a review of UX evaluation studies related to Affectivity, and on the other hand, previous works that analyze emotions and affectivity for VLE. 2.1 The related research regarding UX evaluation and affectivity In [23] their authors researched emotion and how it influences motivation and moods, defining models for emotional design and the dimensions of emotion. The authors of [24] establish the conceptualization of interaction design, interaction paradigms, how affective aspects influence, expressive interfaces and the four basic 4 activities of interaction design in conjunction with Human-Computer Interaction (HCI) lifecycle models. In addition, aspects of the affective interface in the game (challenge, aesthetics and visualization, narrative and characters, usability and interface and flow) are determined in [25]. In the work cited in [26], the authors characterized emotions, proposed a framework for affective design, and established methods for performing emotion assessment. Within this framework, the work cited in [27] contributes to the study of emotional design by defining user experience design, providing guidelines for e-learning applica- tions and providing content strategies. Regarding the evaluation of VLEs, in [28] the authors propose some ideas to achieve an evaluation model that incorporates usability criteria. 2.2 Background on Affective Evaluations for VLEs However, it is noteworthy that up to date few studies have been found [29-36], that clearly defines a list of relevant emotions to evaluate VLE: With respect to attitude, enjoyment, experience, self-efficacy, and speed, the work cited in [29] provides an updated framework of the human factors that positively influ- ence the success and effectiveness of VLE, which have the greatest influence on user satisfaction. Two categories of emotions were used in [30], explicit: collected directly from the learner through self-reported surveys; and implicit emotions: those inferred discretely during the learning process. The authors indicate that VLEs pose several challenges in understanding the different types of affect experienced by a learner. Uses sentiment analysis with a bag-of-words model and does not consider other implicit measures of emotions. Learnability and system effectiveness had a significant influence on user satisfac- tion. The study published in [31] examined the relationship of each usability attribute with the user's emotional responses. Should improve the user interface in terms of usa- bility and user's emotional responses while using the system, to improve user satisfac- tion. To evaluate the quality of an application at low cost using two scale-like methods: AttrakDiff and Hedonic Utility Scale. In [32] a UX evaluation on the Edmodo educa- tional mobile application. It is necessary to adapt these methods to provide a more com- plete UX report, allowing users to subjectively report their experiences and, conse- quently, identify the problems that affected the UX. Strengthening students' self-esteem and emotions have a decisive influence on whether students continue or abandon this educational process. In study [33], the most common web accessibility errors presented by the VLE and how these errors can induce different emotions in students were identified. The emotions were obtained from sur- veys answered by students. 5 In [34] developed UX evaluation metrics for VLEs. For the hedonic quality dimen- sion, stimulation and novelty. While For the pragmatic dimension of quality with trans- parency, efficiency and reliability. The same lead author in [35] conducts a literature review and concludes that no UX evaluation method for VLE assesses user feelings. To culminate this bibliographical review, the authors of the paper cited in [36], leave recommendations:  Have students detail their experiences of how they felt about using a product and the difficulties they had  Be specific using semantic differential scales  Specify which aspects are being evaluated by adjectives or let students explain their evaluation  As the VLE has many different characteristics and an adjective may fit one but not another  Evaluation should be quick and simple  Use quick and simple evaluation methods, such as semantic differential scales. These contributions allow us to affirm the importance of performing an evaluation of the UX of VLEs from an affective point of view, this type of evaluations will con- tribute to improve UX and user satisfaction. However, so far, there are no existing or no known yet publications that provide a valid list to evaluate the functionalities of a VLE. 3 Students Test When evaluating an educational platform, it is important to analyze how the user feels during its use. Therefore, it is important to know the perceptions of the interaction that users/students of a Virtual Learning Environment have with these platforms. In a previous experience, experts in the field of Affectivity and Educational Tech- nology were consulted on possible student emotions when use a VLE, resulting in an initial list of emotions [22]. As a complement to that experience, in this opportunity a test with graduate and postgraduate students was designed, with the purpose of improv- ing the list of emotions proposed by experts in order to provide reliability and validity to it. 3.1 Questionnaire A questionnaire was designed consisting of eight "multiple choice" questions with emotions, one for each VLE characteristic mentioned above, noting that they could choose more than one emotion or none at all. A copy of this questionnaire can be ac- cessed from the following link https://drive.google.com/file/d/1pzKk-pS0F- Z5G07V4EciqH4qryMtctVH/view?usp=sharing. The questions were along the lines of "For Pedagogical Facility: When using the functionalities provided by the educational platform for the teaching/learning process such as: doing group work, tracking schedules and class attendance, evaluations...it is 6 possible that it makes me feel the following emotions....” and so on for all characteris- tics. This questionnaire was online, in Google Forms, resulting in a simple, economical and direct way of obtaining the opinion of the respondents. Students from Argentina, Colombia, Spain, and Brazil participated without any difficulty, and from any device, even from their own cell phones, thus avoiding the inconveniences of distance, time and synchronization. 3.2 Selection of Emotions Participants were asked to choose from a set of emotions for each VLE characteristic, based on the initial list generated with expert support [22]. The student could choose more than one emotion from the suggested list (multiple choice) and even add emotions that were not in the proposed options and/or leave comments or recommendations through open-ended responses. The scale of emotions emerged as result of the previous experience [22], in which the following negative emotions were defined with the support of experts: ─ Indignation: rage, anger due to unmanageable behavior. ─ Contempt: contempt, lack of affection. ─ Dissatisfaction: Unsatisfactory experience. ─ Disappointment: does not meet expectations ─ Unpleasant Surprise: Unexpected negative characteristic. ─ Boredom: Uninteresting characteristics. ─ Tension: Nervousness, stress, lack of tranquility. ─ Frustration: failure to achieve a goal, impossibility of satisfying a need ─ Confusion: lack of understanding. In addition, the following positive emotions were defined: ─ Desire: drive to use the tool. ─ Inspiration: motivation to do new things ─ Admiration: Appreciation of the tool. ─ Satisfaction: Satisfied with the available features ─ Fun: Fun experience. ─ Commitment: obligation to comply 3.3 Participants In order to validate the list of emotions for functionalities/characteristics of a VLE de- fined by experts [22], teachers from Argentina, Colombia, Spain and Brazil from dif- ferent educational levels were asked to distribute the survey to their students. As a re- sult, a sample of 36 students was obtained. The subjects were from different countries, of different age ranges, who participated voluntarily. In addition, each participant an- swered his or her own questionnaire without knowing the answers of his or her peers, thus avoiding response bias. Regarding age, 17 belonged to the 18-25 age group, 15 7 were over 30 years old and 4 were 26-30 years old. In terms of nationality, 19 students were from Argentina, 10 from Colombia, 5 from Spain and 2 from Brazil. This is an important aspect to take into account to indicate the representativeness and heterogene- ity of the participants in terms of culture, language and preferences, enriching the va- lidity of the study. 3.4 Test Results For each functionalities of a VLE, were analyzed the students' responses. The results are shown in graphical form, for better reading and visualization. For this study, was performed a frequency analysis. As can be seen, in Table 2, the votes of the 36 students for each emotion in the top row with respect to the “Pedagogical Facility” functionality were marked in green. Table 2. Votes for each Emotion for Pedagogical Facility Surprise Unpleasantness Disappointment Desire: Impulse Dissatisfaction Engagement Satisfaction Indignation Admiration Frustration Inspiration Confusion Contempt Boredom Tension Disgust Fun P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P Total 0 4 3 16 11 5 11 18 13 15 13 7 4 12 5 21 % 0% 11% 8% 44% 31% 14% 31% 50% 36% 42% 36% 19% 11% 33% 14% 58% 8 Then, in the "Total" row, was obtained the frequency of votes for each emotion for this functionality. In the next row, the percentage of the total that it represents. The most voted emotion for this functionality was "engagement" (compromiso in Spanish), with 21 votes, corresponding to 58% of the total votes (green box), more than half of the total votes. The least chosen emotion was "contempt" (desprecio in Spanish), with 3 votes, 8% of the total (orange box). The emotion "disgust" was not chosen for this functionality (0 zero votes). This same analysis was repeated for all emotions and for each VLE functionality. Table 3 shows in summary the number of emotions associated with each characteristic VLE according to the survey applied. As can be seen in the Table 3, where the first column on the left indicates the set of emotions analyzed. The light blue line separates negative and positive emotions. The first row presents each characteristic of the VLE. In the right part of the table the averages, minimums and maximums of each emotion were calculated, as well as the averages of the set of negative emotions and average of the positive ones. Table 3. Summary of e motions for each characteristic VLE average PositEmotions average NegEmotions Pedagogical facility Handling Errors User Interface Flexibility Standards TOTAL Support Content average Tools max min Disgust 0 0 1 1 4 0 0 2 8 1,00 Indignation 4 5 3 2 7 1 0 5 27 3,38 Contempt 3 4 3 2 5 1 1 4 23 2,88 Dissatisfaction 16 7 9 10 10 7 8 7 74 9,25 Disappointment 11 7 9 5 8 5 6 9 60 7,50 Surprise 42,50 5 3 4 4 7 1 3 3 30 3,75 Unpleasantness Boredom 11 10 11 10 5 8 5 7 67 8,38 Tension 18 0 0 0 0 0 0 0 18 2,25 8 117 Frustration 13 7 6 5 7 7 4 0 49 6,13 Confusion 15 15 7 9 11 6 6 0 69 8,63 Desire 13 5 8 8 0 10 8 5 57 7,13 Inspiration 7 5 10 3 2 4 7 4 42 5,25 Admiration 4 4 9 9 3 10 9 8 56 7,00 54,33 Satisfaction 12 13 18 18 11 17 10 18 117 14,63 Fun 5 3 3 5 0 4 5 3 28 3,50 Engagement 21 0 0 0 0 0 5 0 26 3,25 Considering all features (table 3), the most chosen emotion was "satisfaction" (satis- facción in Spanish - green shading), while the least chosen emotion was "contempt" (desprecio in Spanish - orange shading). Then, doing an analysis respect to the emo- 9 tions, it is observed that positive emotions were more chosen to evaluate all the func- tionalities of a VLE. However, as can be seen in Fig. 1, the choice of emotions behaves in a variable way for all functionalities, with a peak in the emotion "satisfaction" and average values in the rest. Being "disgust" the least chosen of all. This list of emotions in Spanish will be very useful for evaluate affectively VLE functionalities, since it was constructed with the help of higher education students of different genders, ages and countries of origin, which enriches the validity and hetero- geneity of the study. 4 Student Validation of Expert-defined Emotions In previous publications [22], we interviewed a group of experts in the field of Affec- tivity and Educational Technology, from several Spanish-speaking countries, those most significant emotions that a higher education student may express during the use of each of the VLE functionalities [38]. Fig. 1. General Average of Emotions The aim of the present work is to validate with students those list of emotions [22]. In order to compare the values of both experiences, were calculated the percentages of proportions of votes for each emotion in each VLE characteristic. That is, for example, if experts chose the emotion 5 times, the proportion results from dividing 5 by 12 (total number of experts who answered) and in percentage format would be 41%. In the case of students, it was divided into 36 (total number of students who responded). We can appreciate the following similarities and differences: analyzing the votes for all VLE functionalities with respect to disgust (See table 4); we can notice that both, experts and students, did not choose it for “Pedagogical Facility” and “Support”, 0% in this cells. 10 Table 4. Summary of negative emotions per functionalities-VLE/Student/Expert Pedagogical Facility Handling Errors User Interface Flexibility Standards Support Content Tools Student 0% 0% 3% 3% 11% 0% 0% 6% Disgust Expert 0% 0% 33% 25% 17% 8% 17% 17% Student 11% 14% 8% 6% 19% 3% 0% 14% Indignation Expert 58% 25% 25% 42% 83% 50% 42% 17% Student 8% 11% 8% 6% 14% 3% 3% 11% Contempt Expert 25% 33% 42% 33% 17% 42% 17% 33% Student 44% 19% 25% 28% 28% 19% 22% 19% Dissatisfaction Expert 100% 67% 75% 92% 100% 92% 58% 50% Student 31% 19% 25% 14% 22% 14% 17% 25% Disappointment Expert 75% 75% 83% 83% 58% 67% 67% 33% Surprise Student 14% 8% 11% 11% 19% 3% 8% 8% Unpleasantness Expert 25% 8% 50% 42% 58% 17% 42% 8% Student 31% 28% 31% 28% 14% 22% 14% 19% Boredom Expert 92% 58% 75% 75% 25% 67% 50% 33% Tension Student 50% 0% 0% 0% 0% 0% 0% 0% Frustration Student 36% 19% 17% 14% 19% 19% 11% 0% Confusion Student 42% 42% 19% 25% 31% 17% 17% 0% It should be noted that experts added the emotions tension, frustration, confusion and commitment in the previous instance, for this reason they were not chosen by the other experts since they were not part of the initial list at the time, which is why there are only student votes in these columns. Analyzing these values (see table 4), it can be noticed for example that tension was only chosen for the functionality "Pedagogical Facility", zeros (0) in that column except in the 1st row. Similarly, engagement (in table 5) only appears for "Pedagogical Facil- ity" and “Flexibility”. The rest of the emotions were chosen for almost all functionali- ties. The same means for negative emotions in Table 5. 11 Table 5. Summary of positive emotions per functionalities-VLE/Student/Expert Pedagogical Facility Handling Errors User Interface Flexibility Standards Support Content Tools Student 36% 14% 22% 22% 0% 28% 22% 14% Desire Expert 50% 33% 25% 42% 0% 33% 58% 25% Student 19% 14% 28% 8% 6% 11% 19% 11% Inspiration Expert 50% 25% 42% 33% 8% 42% 50% 17% Student 11% 11% 25% 25% 8% 28% 25% 22% Admiration Expert 17% 8% 17% 33% 17% 58% 50% 33% Student 33% 36% 50% 50% 31% 47% 28% 50% Satisfaction Expert 92% 58% 75% 75% 58% 100% 75% 75% Student 14% 8% 8% 14% 0% 11% 14% 8% Fun Expert 50% 8% 50% 58% 0% 42% 25% 8% Engagement Student 58% 0% 0% 0% 0% 0% 14% 0% 4.1 Final validated Emotions for VLEs As a result of both experiences, was obtained a validated list of emotions for each VLE characteristic. To determine the final list of emotions for functionalities of a VLE vali- dated by experts and students, it was determined as an inclusion rule that emotions that were chosen at least once, by an expert or by a student, would be part of this final list. Table 6 shows a summary of the negative emotions selected by characteristic. The emotions that were selected are marked in green. In addition, the table 7 shows the positive emotions selected. Table 6. Negative Emotions selected for characteristic VLE Surprise Unpleasantness Disappointment Dissatisfaction Negative Emotions Indignation Frustration Confusion Contempt Boredom Tension Disgust Pedagogical facility P P P P P P P P P Support P P P P P P P P Content P P P P P P P P P User Interface P P P P P P P P P Handling Errors P P P P P P P P P Tools P P P P P P P P Flexibility P P P P P P P Standards P P P P P P P 12 Table 7. Positive Emotions selected for characteristic VLE Desire: Impulse Engagement Satisfaction Admiration Inspiration Positive Emotions Fun Pedagogical facility P P P P P P Support P P P P P Content P P P P P User Interface P P P P P Handling Errors P P P Tools P P P P P Flexibility P P P P P P Standards P P P P P Finally, the emotions to evaluate during the use of each VLE functionality according to the opinion of experts and student users are listed in table 8. Table 8. Final list of emotions per functionality of an VLE according to experts and students Positive Negative Indignation, Contempt, Dissatisfac- Desire, Inspiration, Admira- Pedagogical tion, Disappointment, Unpleasant Sur- tion, Satisfaction, Fun, Engage- facility prise, Boredom, Tension-Nervousness, ment Frustration, Confusion Indignation, Contempt, Dissatisfac- Desire, Inspiration, Admira- tion, Disappointment, Unpleasant Sur- Support tion, Satisfaction, Fun prise, Boredom, Frustration, Confu- sion Disgust , Indignation, Contempt, Desire, Inspiration, Admira- Dissatisfaction, Disappointment, Un- Content tion, Satisfaction, Fun pleasant Surprise, Boredom, Frustra- tion, Confusion Disgust , Indignation, Contempt, Desire, Inspiration, Admira- User Inter- Dissatisfaction, Disappointment, Un- tion, Satisfaction, Fun face pleasant Surprise, Boredom, Frustra- tion, Confusion Disgust , Indignation, Contempt, Inspiration, Admiration, Sat- Handling Dissatisfaction, Disappointment, Un- isfaction Errors pleasant Surprise, Boredom, Frustra- tion, Confusion Indignation, Contempt, Dissatisfac- Desire, Inspiration, Admira- tion, Disappointment, Unpleasant Sur- Tools tion, Satisfaction, Fun prise, Boredom, Frustration, Confu- sion 13 Desire, Inspiration, Admira- Contempt, Dissatisfaction, Disap- Flexibility tion, Satisfaction, Fun, Engage- pointment, Unpleasant Surprise, Bore- ment dom, Frustration, Confusion Disgust , Indignation, Contempt, Desire, Inspiration, Admira- Standards Dissatisfaction, Disappointment, Un- tion, Satisfaction, Fun pleasant Surprise, Boredom 5 Conclusions and Further Works It is important that users/students of a VLE feel satisfied when interacting with these computer systems. Identifying positive and negative perceptions through such emo- tions. The main problem is that there is no validated list of emotions that students may experience when interacting with each of the functionalities of an VLE. As a contribution to evaluate affectively each of the functionalities of a VLE, a list of emotions, validated by experts and students, is left. In this sense, this work managed to define the emotions in Spanish according to each functionality of the VLE, contributing to the area of HCI in the context of education. This list of emotions in Spanish will be very useful for the field of Affectivity and Educational Technology, since it has the support of recognized professionals as well as students of higher education, of different ages and countries of origin, which enriches the validity and heterogeneity of the study. As future work, it is planned to determine the scale to be used to perform an affective evaluation of VLEs. Subsequently, it is planned to implement affective evaluations of the educational platforms with the proposal list of emotions, for validation and feed- back. Another useful aspect to be analyzed in the future is the possibility of determining which emotions are most related to each characteristic, in order to consider them in both the design and the evaluation of the platform. The HCI community should consider these issues to identify aspects that influence users' perceptions of their UX experience in order to develop better evaluation ap- proaches. 6 References 1. H. Fardoun, C. González, C. A. Collazos, and M. Yousef, “Exploratory Study in Iberoam- erica on the Teaching-Learning Process and Assessment Proposal in the Pandemic Times = Estudio exploratorio en iberoamérica sobre procesos de enseñanza-aprendizaje y propuesta de evaluación en tiempos de pandemia,” Educ. Knowl. Soc., vol. 21, 2020. 2. F. J. García-Peñalvo, A. Corell, V. Abella-García, and M. Grande, “La evaluación online en la educación superior en tiempos de la COVID-19,” Educ. Knowl. Soc., vol. 21, no. 0, p. 26, 2020. 3. I. T. Plata and D. B. Alado, Evaluating the Perceived Usability of Virtual Learning Environ- ment in Teaching ICT Courses, vol. 1. 2015. 14 4. E. Crisol-Moya, L. Herrera-Nieves, and R. Montes-Soldado, “Educación virtual para todos: una revisión sistemática,” Educ. Knowl. Soc., vol. 21, no. 0, p. 13, Jun. 2020. 5. F. J. García-Peñalvo, A. Miguel, and S. Pardo, “Una revisión actualizada del concepto de eLearning. Décimo Aniversario,” vol. 16, no. 1, 2015. 6. E. Martínez Caro, “E-Learning: Un Análisis Desde El Punto De Vista Del Alumno,” RIED. Rev. Iberoam. Educ. a Distancia, vol. 11, no. 2, pp. 151–168, 2012. 7. M. Hassenzahl, “The Thing and I: Understanding the Relationship Between User and Prod- uct,” Springer, Cham, 2018, pp. 301–313. 8. K. Capota, M. van Hout, and T. van der Geest, “Measuring the emotional impact of web- sites,” in Proceedings of the 2007 conference on Designing pleasurable products and inter- faces - DPPI ’07, 2007, p. 135. 9. L. Masip Ardévol, “User experience methodology for the design and evaluation of interac- tive systems,” University of Lleida, 2013. 10. J. I. Cocunubo-Suárez, J. A. Parra-Valencia, and J. E. Otálora-Luna, “Propuesta para la eva- luación de Entornos Virtuales de Enseñanza Aprendizaje con base en estándares de Usabi- lidad Evaluation of Virtual Teaching- Learning Environments based on usability standards Cómo citar / How to cite Propuesta para la evaluación de En,” vol. 21, no. 41, pp. 123–7799, 2018. 11. M. Hassenzahl, M. Burmester, and K. Franz, “AttrakDiff: A questionnaire to measure per- ceived hedonic and pragmatic quality,” Mensch Comput., vol. 57, 2003. 12. B. Laugwitz, T. Held, and M. Schrepp, “Construction and Evaluation of a User Experience Questionnaire,” Springer, Berlin, Heidelberg, 2008, pp. 63–76. 13. Bradley, M. M. and Lang, P. J., “Measuring emotion: the self-assessment manikin and the semantic differential,” J. Behav. Ther. Exp. Psychiatry, vol. 25, no. 1, pp. 49–59, 1994. 14. G. Huisman, M. van Hout, E. van Dijk, T. van der Geest, and D. Heylen, “LEMtool,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’13, 2013, p. 351. 15. P. Desmet, “Pieter Desmet Chapter 9,” Funology From usability to enjoyment, pp. 111–123, 2005. 16. P. M. A. Desmet, M. A. Blythe, A. F. Monk, K. Overbeeke, and P. C. Wright, “Measuring Emotions Development and application of an instrument to measure emotional responses to products.” 17. P. M. Desmet, “Measuring emotions: Development of an instrument to measure emotional responses to products,” Funology from usability to enjoyment. Kluwer Acad. Publ. Dor- drecht, Boston, Londres, no. Blythe, M. A.; Overbeeke, K.; Monk, A. F.; Wright, P. C. (ed.), 2003. 18. G. Laurans and P. Desmet, “Introducing Premo2 New Directions for the Non - Verbal Meas- urement of Emotion in Design,” Proc. 8th Int. Des. Emot. Conf., 2012. 19. Y. A. Méndez-Alegría, C. A. Collazos, T. Granollers, and R. Gil, “Rueda de emociones de Ginebra+: instrumento para la valoración emocional de los usuarios mientras participan en una evaluación de sistemas interactivos,” Dyna, vol. 91, no. 2, pp. 151–155, 2016. 20. M. Fritz, “Reinventing the Wheel: Emotional Awareness Enhancement in Computer-Medi- ated Collaboration with the Dynamic Emotion Wheel.” 2015. 21. P. Desmet, “Measuring Emotion: Development and Application of an Instrument to Measure Emotional Responses to Products,” 2003. 22. L. N. Aballay, S. V. Aciar, and C. A. Collazos, “Emotions for Virtual Learning Environ- ments,” IEEE-RITA, vol. 9, no. 3, 2021. (Not yet published) 23. 23. V. Sacharin, K. Schlegel, and K. R. Scherer, “Geneva Emotion Wheel Rating Study,” 2012. 15 24. M. Blythe and M. Hassenzahl, “The Semantics of Fun: Differentiating Enjoyable Experi- ences,” Springer, Cham, 2018, pp. 375–387. 25. W. Aarron, Designing for emotion (pp. 978-1). New York: A book apart. 2012. 26. Y.-H. Shih and M. Liu, “The Importance of Emotional Usability,” J. Educ. Technol. Syst., vol. 36, no. 2, pp. 203–218, Dec. 2007. 27. M. L. Wong, C. W. Khong, and H. Thwaites, “Applied UX and UCD Design Process in Interface Design,” Procedia - Soc. Behav. Sci., vol. 51, pp. 703–708, Jan. 2012. 28. L. A. Hasan and K. T. Al-Sarayreh, “An Integrated Measurement Model for Evaluating Us- ability Attributes,” Proc. Int. Conf. Intell. Inf. Process. Secur. Adv. Commun., p. 94, 2015. 29. M. M. Alomari, H. El-Kanj, N. I. Alshdaifat, and A. Topal, “A framework for the impact of human factors on the effectiveness of learning management systems,” IEEE Access, vol. 8, pp. 23542–23558, 2020. 30. M. Syed, M. Chetlur, S. Afzal, G. A. Ambrose, and N. V. Chawla, “Implicit and explicit emotions in MOOCs,” EDM 2019 - Proc. 12th Int. Conf. Educ. Data Min., no. Edm, pp. 432–437, 2019. 31. N. Phongphaew and A. Jiamsanguanwong, “The Usability Evaluation Concerning Emo- tional Responses of Users on Learning Management System,” Proc. 2016 6th Int. Work. Comput. Sci. Eng. (WCSE 2016), pp. 43–48, 2016. 32. L. Marques, W. Nakamura, N. Valentim, L. Rivero, and T. Conte, “Do Scale Type Tech- niques Identify Problems that Affect User eXperience? User Experience Evaluation of a Mobile Application,” 2018. 33. T. Acosta and S. Luján-Mora, “Analysis of Emotion in the Use of Accessible Learning Man- agement Systems by Students,” in 9th Annual International Conference of Education, Re- search and Innovation (ICERI 2016), 2016. 34. W. T. Nakamura, L. C. Marques, L. Rivero, and E. H. T. De Oliveira, “Are Generic UX Evaluation Techniques Enough? A study on the UX Evaluation of the Edmodo Learning Management System,” Brazilian Symp. Comput. Educ. (Simpósio Bras. Informática na Educ. - SBIE), vol. 28, no. 1, p. 1007, Oct. 2017. 35. W. T. Nakamura, E. H. T. De Oliveira, and T. Conte, “Usability and user experience evalu- ation of learning management systems a systematic mapping study,” in ICEIS 2017 - Pro- ceedings of the 19th International Conference on Enterprise Information Systems, 2017, vol. 3, pp. 97–108. 36. W. T. Nakamura, L. C. Marques, L. Rivero, E. H. T. De Oliveira, and T. Conte, “Are scale- based techniques enough for learners to convey their UX when using a Learning Manage- ment System?” Rev. Bras. Informática na Educ., vol. 27, no. 01, p. 104, Jan. 2019.