=Paper=
{{Paper
|id=Vol-3070/paper03
|storemode=property
|title=Emotions Evaluation on Virtual Learning Environments: Students Perceptions
|pdfUrl=https://ceur-ws.org/Vol-3070/paper03.pdf
|volume=Vol-3070
|authors=Laura N. Aballay,César A. Collazos,Silvana V. Aciar
}}
==Emotions Evaluation on Virtual Learning Environments: Students Perceptions==
https://ceur-ws.org/Vol-3070/paper03.pdf
Emotions Evaluation on Virtual Learning Environments:
Students perceptions
Laura N. Aballay1, César A. Collazos2 and Silvana V. Aciar3
1
Universidad Nacional de San Juan, San Juan, Argentina
lnaballay@gmail.com
2
Universidad del Cauca, Popayán, Colombia
ccollazo@unicauca.edu.co
3
Universidad Nacional de San Juan, San Juan, Argentina
silvanav.aciar@gmail.com
Abstract. When evaluating a web platform, it is important to analyze how the
user feels during its use. Therefore, it is important to know the perceptions of the
interaction that users/students of a Virtual Learning Environment have with these
platforms. This article leave a list of emotions to evaluate the functionalities of
educational platforms, validated by students and experts. This list of emotions in
Spanish will be very useful for the field of Affectivity and Educational Technol-
ogy, since it has the support of recognized professionals as well as students of
higher education, of different ages and countries of origin, which enriches the
validity and heterogeneity of the study.
Keywords: Human computer interaction, Affective Computing, User Experi-
ence, Emotional Evaluation, Virtual Learning Environments.
1 Introduction
The boom in the use of virtual platforms nowadays is enhanced by the COVID-19 pan-
demic [1][2]. In the Virtual Learning Environment (VLE, for its acronym in English),
the student interacts with the platform to follow the course lessons [3]. The need to
design and provide educational platforms and resources that are not only accessible but
also attractive to students has been emphasized [4][5]. Therefore, it is important to
know the perceptions of the interaction that users/students of a VLE have with these
platforms [6].
The User Experience (UX) constitutes a holistic evaluation of the interaction with a
product [7]. This is an evaluation that includes various facets, including the emotional,
which takes into account satisfaction attributes from the affective point of view of the
user [8], [9].
The functionalities of these VLEs may vary, but in general, they all have basic func-
tions that are common to most of them. The work cited in [10], was taken as a basis,
Copyright © 2021 for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
2
which investigated the most important characteristics to take into account in the evalu-
ation of a VLE. Those authors determine the following characteristics to take into ac-
count when evaluating a VLE:
Pedagogical facility: this category refers to the functionalities that the environment
provides to teachers and students for the development of the teaching and learning
process.
Support, help and documentation: information of this type should be easy to find,
focused on the user's task, with the list of concrete steps that will be carried out, and
not be too long.
Content: the content or courseware is the learning material that is made available to
the student.
User interface: consists of those aspects of the system which the user comes into
contact, physically, perceptually or conceptually.
Error handling: cancellation, revocation of actions, error prevention, error detection,
error recognition, visible and clear error messages, recovery mechanisms and error
solution.
Tools: used for the management of academic content, improving user intercommu-
nication.
Flexibility: refers to the multiplicity of ways in which the user and the system ex-
change information. It involves providing user control, substitution capacity and
adaptability.
Standards: refers to how the web application conforms to norms, standards, conven-
tions, or design guides in the web domain.
After identifying the characteristics of VLE, it is necessary to know the existing
methods to evaluate these characteristics from the affective aspect, considering that the
UX emphasizes emotional aspects. Table 1 show some tools for affective evaluations.
Table 1. Affective questionnaires.
Tool Emotions measured
AttrakDiff [11] Both measure product acceptance.
User Experience Questionnaire
(UEQ) [12]
SAM (Self-Assessment Mani- User emotion assessment tool that uses
kin)[13] graphic scales for dimensions: Valence
“pleasant” to “unpleasant”; Arousal “ex-
cited” to “calm”; Dominance: “out of con-
trol” to “in control”
LEM-Tool [14] Cartoons express eight discrete emotions
of users: Joy, Desire, Fascination, Satisfac-
tion, Sadness, Disgust, Boredom, and Dis-
satisfaction.
3
PrEmo (Product Emotion) [15]– Self-report instrument that non-verbally
[18] measures, various user emotions, both satis-
factory and unpleasant.
"Geneva Emotions Wheel +" Emotional state of the user during the
(GEW +) [19] based on [20] evaluation
These questionnaires use emotion scales that may be inaccurate and/or confusing for
the people who participate in the assessments, in addition to the fact that not all of them
can be adapted to assess the different functionalities of a VLE.
In order to offer an alternative to existing instruments, this work aims to obtain a
scale of emotions and consider it when evaluating a VLE from an affective point of
view.
The emotions scale used in PrEmo [21] was chosen as a base, due to the ease with
which the emotions of the participants about the characteristics of a product can be
consulted, in addition to being one of the best known and utilized instruments world-
wide, therefore, one of the most tested methods and available in different languages,
including Spanish.
In previous publication [22], we interviewed a group of experts in the field of Affec-
tivity and Educational Technology, from several Spanish-speaking countries, those
most significant emotions that a higher education student may express during the use
of each of the VLE functionalities. The aim of the present work is to validate with
students that list of emotions [22].
The article is organized as follows: section two presents related works, on the one
hand related to UX studies, and on the other hand a summary of works that analyzed
emotions and affectivity for VLE. Then, section three details the study conducted with
high-level students, compiling the emotions that they can perceive when evaluating
VLE. Subsequently, section four analyzes the results of the study to validate the list of
emotions obtained previously from experts, leaving as a result the final list of appropri-
ate emotions to evaluate each VLE functionality. Finally, in section five the conclusions
and future work of the research are presented.
2 Related Works
The works related to the subject of this article are presented. On the one hand, a review
of UX evaluation studies related to Affectivity, and on the other hand, previous works
that analyze emotions and affectivity for VLE.
2.1 The related research regarding UX evaluation and affectivity
In [23] their authors researched emotion and how it influences motivation and moods,
defining models for emotional design and the dimensions of emotion.
The authors of [24] establish the conceptualization of interaction design, interaction
paradigms, how affective aspects influence, expressive interfaces and the four basic
4
activities of interaction design in conjunction with Human-Computer Interaction (HCI)
lifecycle models.
In addition, aspects of the affective interface in the game (challenge, aesthetics and
visualization, narrative and characters, usability and interface and flow) are determined
in [25].
In the work cited in [26], the authors characterized emotions, proposed a framework
for affective design, and established methods for performing emotion assessment.
Within this framework, the work cited in [27] contributes to the study of emotional
design by defining user experience design, providing guidelines for e-learning applica-
tions and providing content strategies.
Regarding the evaluation of VLEs, in [28] the authors propose some ideas to achieve
an evaluation model that incorporates usability criteria.
2.2 Background on Affective Evaluations for VLEs
However, it is noteworthy that up to date few studies have been found [29-36], that
clearly defines a list of relevant emotions to evaluate VLE:
With respect to attitude, enjoyment, experience, self-efficacy, and speed, the work
cited in [29] provides an updated framework of the human factors that positively influ-
ence the success and effectiveness of VLE, which have the greatest influence on user
satisfaction.
Two categories of emotions were used in [30], explicit: collected directly from the
learner through self-reported surveys; and implicit emotions: those inferred discretely
during the learning process. The authors indicate that VLEs pose several challenges in
understanding the different types of affect experienced by a learner. Uses sentiment
analysis with a bag-of-words model and does not consider other implicit measures of
emotions.
Learnability and system effectiveness had a significant influence on user satisfac-
tion. The study published in [31] examined the relationship of each usability attribute
with the user's emotional responses. Should improve the user interface in terms of usa-
bility and user's emotional responses while using the system, to improve user satisfac-
tion.
To evaluate the quality of an application at low cost using two scale-like methods:
AttrakDiff and Hedonic Utility Scale. In [32] a UX evaluation on the Edmodo educa-
tional mobile application. It is necessary to adapt these methods to provide a more com-
plete UX report, allowing users to subjectively report their experiences and, conse-
quently, identify the problems that affected the UX.
Strengthening students' self-esteem and emotions have a decisive influence on
whether students continue or abandon this educational process. In study [33], the most
common web accessibility errors presented by the VLE and how these errors can induce
different emotions in students were identified. The emotions were obtained from sur-
veys answered by students.
5
In [34] developed UX evaluation metrics for VLEs. For the hedonic quality dimen-
sion, stimulation and novelty. While For the pragmatic dimension of quality with trans-
parency, efficiency and reliability. The same lead author in [35] conducts a literature
review and concludes that no UX evaluation method for VLE assesses user feelings.
To culminate this bibliographical review, the authors of the paper cited in [36], leave
recommendations:
Have students detail their experiences of how they felt about using a product and the
difficulties they had
Be specific using semantic differential scales
Specify which aspects are being evaluated by adjectives or let students explain their
evaluation
As the VLE has many different characteristics and an adjective may fit one but not
another
Evaluation should be quick and simple
Use quick and simple evaluation methods, such as semantic differential scales.
These contributions allow us to affirm the importance of performing an evaluation
of the UX of VLEs from an affective point of view, this type of evaluations will con-
tribute to improve UX and user satisfaction. However, so far, there are no existing or
no known yet publications that provide a valid list to evaluate the functionalities of a
VLE.
3 Students Test
When evaluating an educational platform, it is important to analyze how the user feels
during its use. Therefore, it is important to know the perceptions of the interaction that
users/students of a Virtual Learning Environment have with these platforms.
In a previous experience, experts in the field of Affectivity and Educational Tech-
nology were consulted on possible student emotions when use a VLE, resulting in an
initial list of emotions [22]. As a complement to that experience, in this opportunity a
test with graduate and postgraduate students was designed, with the purpose of improv-
ing the list of emotions proposed by experts in order to provide reliability and validity
to it.
3.1 Questionnaire
A questionnaire was designed consisting of eight "multiple choice" questions with
emotions, one for each VLE characteristic mentioned above, noting that they could
choose more than one emotion or none at all. A copy of this questionnaire can be ac-
cessed from the following link https://drive.google.com/file/d/1pzKk-pS0F-
Z5G07V4EciqH4qryMtctVH/view?usp=sharing.
The questions were along the lines of "For Pedagogical Facility: When using the
functionalities provided by the educational platform for the teaching/learning process
such as: doing group work, tracking schedules and class attendance, evaluations...it is
6
possible that it makes me feel the following emotions....” and so on for all characteris-
tics.
This questionnaire was online, in Google Forms, resulting in a simple, economical
and direct way of obtaining the opinion of the respondents. Students from Argentina,
Colombia, Spain, and Brazil participated without any difficulty, and from any device,
even from their own cell phones, thus avoiding the inconveniences of distance, time
and synchronization.
3.2 Selection of Emotions
Participants were asked to choose from a set of emotions for each VLE characteristic,
based on the initial list generated with expert support [22]. The student could choose
more than one emotion from the suggested list (multiple choice) and even add emotions
that were not in the proposed options and/or leave comments or recommendations
through open-ended responses.
The scale of emotions emerged as result of the previous experience [22], in which
the following negative emotions were defined with the support of experts:
─ Indignation: rage, anger due to unmanageable behavior.
─ Contempt: contempt, lack of affection.
─ Dissatisfaction: Unsatisfactory experience.
─ Disappointment: does not meet expectations
─ Unpleasant Surprise: Unexpected negative characteristic.
─ Boredom: Uninteresting characteristics.
─ Tension: Nervousness, stress, lack of tranquility.
─ Frustration: failure to achieve a goal, impossibility of satisfying a need
─ Confusion: lack of understanding.
In addition, the following positive emotions were defined:
─ Desire: drive to use the tool.
─ Inspiration: motivation to do new things
─ Admiration: Appreciation of the tool.
─ Satisfaction: Satisfied with the available features
─ Fun: Fun experience.
─ Commitment: obligation to comply
3.3 Participants
In order to validate the list of emotions for functionalities/characteristics of a VLE de-
fined by experts [22], teachers from Argentina, Colombia, Spain and Brazil from dif-
ferent educational levels were asked to distribute the survey to their students. As a re-
sult, a sample of 36 students was obtained. The subjects were from different countries,
of different age ranges, who participated voluntarily. In addition, each participant an-
swered his or her own questionnaire without knowing the answers of his or her peers,
thus avoiding response bias. Regarding age, 17 belonged to the 18-25 age group, 15
7
were over 30 years old and 4 were 26-30 years old. In terms of nationality, 19 students
were from Argentina, 10 from Colombia, 5 from Spain and 2 from Brazil. This is an
important aspect to take into account to indicate the representativeness and heterogene-
ity of the participants in terms of culture, language and preferences, enriching the va-
lidity of the study.
3.4 Test Results
For each functionalities of a VLE, were analyzed the students' responses. The results
are shown in graphical form, for better reading and visualization. For this study, was
performed a frequency analysis. As can be seen, in Table 2, the votes of the 36 students
for each emotion in the top row with respect to the “Pedagogical Facility” functionality
were marked in green.
Table 2. Votes for each Emotion for Pedagogical Facility
Surprise Unpleasantness
Disappointment
Desire: Impulse
Dissatisfaction
Engagement
Satisfaction
Indignation
Admiration
Frustration
Inspiration
Confusion
Contempt
Boredom
Tension
Disgust
Fun
P P P P P P P P P P P P
P P P P P
P P P P
P P P P P P P
P P P P
P P P
P P P
P P
P P P
P
P P P P
P P P P P
P P P P
P P P P
P P P P P P P P
P P
P P P P P P P P P
P P P
P P P P P P
P
P P P P P P
P P P P P P P P P P
P P P
P P P P P
P P P P P
P P P P P P P
P
P P P P
P P
P P P P P P P
P P
P P P P P
P P P P P
P P P
P P
P
Total 0 4 3 16 11 5 11 18 13 15 13 7 4 12 5 21
% 0% 11% 8% 44% 31% 14% 31% 50% 36% 42% 36% 19% 11% 33% 14% 58%
8
Then, in the "Total" row, was obtained the frequency of votes for each emotion for
this functionality. In the next row, the percentage of the total that it represents. The
most voted emotion for this functionality was "engagement" (compromiso in Spanish),
with 21 votes, corresponding to 58% of the total votes (green box), more than half of
the total votes. The least chosen emotion was "contempt" (desprecio in Spanish), with
3 votes, 8% of the total (orange box). The emotion "disgust" was not chosen for this
functionality (0 zero votes).
This same analysis was repeated for all emotions and for each VLE functionality.
Table 3 shows in summary the number of emotions associated with each characteristic
VLE according to the survey applied. As can be seen in the Table 3, where the first
column on the left indicates the set of emotions analyzed. The light blue line separates
negative and positive emotions. The first row presents each characteristic of the VLE.
In the right part of the table the averages, minimums and maximums of each emotion
were calculated, as well as the averages of the set of negative emotions and average of
the positive ones.
Table 3. Summary of e motions for each characteristic VLE
average PositEmotions
average NegEmotions
Pedagogical facility
Handling Errors
User Interface
Flexibility
Standards
TOTAL
Support
Content
average
Tools
max
min
Disgust 0 0 1 1 4 0 0 2 8 1,00
Indignation 4 5 3 2 7 1 0 5 27 3,38
Contempt 3 4 3 2 5 1 1 4 23 2,88
Dissatisfaction 16 7 9 10 10 7 8 7 74 9,25
Disappointment 11 7 9 5 8 5 6 9 60 7,50
Surprise 42,50
5 3 4 4 7 1 3 3 30 3,75
Unpleasantness
Boredom 11 10 11 10 5 8 5 7 67 8,38
Tension 18 0 0 0 0 0 0 0 18 2,25 8 117
Frustration 13 7 6 5 7 7 4 0 49 6,13
Confusion 15 15 7 9 11 6 6 0 69 8,63
Desire 13 5 8 8 0 10 8 5 57 7,13
Inspiration 7 5 10 3 2 4 7 4 42 5,25
Admiration 4 4 9 9 3 10 9 8 56 7,00
54,33
Satisfaction 12 13 18 18 11 17 10 18 117 14,63
Fun 5 3 3 5 0 4 5 3 28 3,50
Engagement 21 0 0 0 0 0 5 0 26 3,25
Considering all features (table 3), the most chosen emotion was "satisfaction" (satis-
facción in Spanish - green shading), while the least chosen emotion was "contempt"
(desprecio in Spanish - orange shading). Then, doing an analysis respect to the emo-
9
tions, it is observed that positive emotions were more chosen to evaluate all the func-
tionalities of a VLE. However, as can be seen in Fig. 1, the choice of emotions behaves
in a variable way for all functionalities, with a peak in the emotion "satisfaction" and
average values in the rest. Being "disgust" the least chosen of all.
This list of emotions in Spanish will be very useful for evaluate affectively VLE
functionalities, since it was constructed with the help of higher education students of
different genders, ages and countries of origin, which enriches the validity and hetero-
geneity of the study.
4 Student Validation of Expert-defined Emotions
In previous publications [22], we interviewed a group of experts in the field of Affec-
tivity and Educational Technology, from several Spanish-speaking countries, those
most significant emotions that a higher education student may express during the use
of each of the VLE functionalities [38].
Fig. 1. General Average of Emotions
The aim of the present work is to validate with students those list of emotions [22].
In order to compare the values of both experiences, were calculated the percentages of
proportions of votes for each emotion in each VLE characteristic. That is, for example,
if experts chose the emotion 5 times, the proportion results from dividing 5 by 12 (total
number of experts who answered) and in percentage format would be 41%. In the case
of students, it was divided into 36 (total number of students who responded).
We can appreciate the following similarities and differences: analyzing the votes for
all VLE functionalities with respect to disgust (See table 4); we can notice that both,
experts and students, did not choose it for “Pedagogical Facility” and “Support”, 0% in
this cells.
10
Table 4. Summary of negative emotions per functionalities-VLE/Student/Expert
Pedagogical Facility
Handling Errors
User Interface
Flexibility
Standards
Support
Content
Tools
Student 0% 0% 3% 3% 11% 0% 0% 6%
Disgust
Expert 0% 0% 33% 25% 17% 8% 17% 17%
Student 11% 14% 8% 6% 19% 3% 0% 14%
Indignation
Expert 58% 25% 25% 42% 83% 50% 42% 17%
Student 8% 11% 8% 6% 14% 3% 3% 11%
Contempt
Expert 25% 33% 42% 33% 17% 42% 17% 33%
Student 44% 19% 25% 28% 28% 19% 22% 19%
Dissatisfaction
Expert 100% 67% 75% 92% 100% 92% 58% 50%
Student 31% 19% 25% 14% 22% 14% 17% 25%
Disappointment
Expert 75% 75% 83% 83% 58% 67% 67% 33%
Surprise Student 14% 8% 11% 11% 19% 3% 8% 8%
Unpleasantness Expert
25% 8% 50% 42% 58% 17% 42% 8%
Student 31% 28% 31% 28% 14% 22% 14% 19%
Boredom
Expert 92% 58% 75% 75% 25% 67% 50% 33%
Tension Student 50% 0% 0% 0% 0% 0% 0% 0%
Frustration Student 36% 19% 17% 14% 19% 19% 11% 0%
Confusion Student 42% 42% 19% 25% 31% 17% 17% 0%
It should be noted that experts added the emotions tension, frustration, confusion and
commitment in the previous instance, for this reason they were not chosen by the other
experts since they were not part of the initial list at the time, which is why there are
only student votes in these columns.
Analyzing these values (see table 4), it can be noticed for example that tension was
only chosen for the functionality "Pedagogical Facility", zeros (0) in that column except
in the 1st row. Similarly, engagement (in table 5) only appears for "Pedagogical Facil-
ity" and “Flexibility”. The rest of the emotions were chosen for almost all functionali-
ties. The same means for negative emotions in Table 5.
11
Table 5. Summary of positive emotions per functionalities-VLE/Student/Expert
Pedagogical Facility
Handling Errors
User Interface
Flexibility
Standards
Support
Content
Tools
Student 36% 14% 22% 22% 0% 28% 22% 14%
Desire
Expert 50% 33% 25% 42% 0% 33% 58% 25%
Student 19% 14% 28% 8% 6% 11% 19% 11%
Inspiration
Expert 50% 25% 42% 33% 8% 42% 50% 17%
Student 11% 11% 25% 25% 8% 28% 25% 22%
Admiration
Expert 17% 8% 17% 33% 17% 58% 50% 33%
Student 33% 36% 50% 50% 31% 47% 28% 50%
Satisfaction
Expert 92% 58% 75% 75% 58% 100% 75% 75%
Student 14% 8% 8% 14% 0% 11% 14% 8%
Fun
Expert 50% 8% 50% 58% 0% 42% 25% 8%
Engagement Student 58% 0% 0% 0% 0% 0% 14% 0%
4.1 Final validated Emotions for VLEs
As a result of both experiences, was obtained a validated list of emotions for each VLE
characteristic. To determine the final list of emotions for functionalities of a VLE vali-
dated by experts and students, it was determined as an inclusion rule that emotions that
were chosen at least once, by an expert or by a student, would be part of this final list.
Table 6 shows a summary of the negative emotions selected by characteristic. The
emotions that were selected are marked in green. In addition, the table 7 shows the
positive emotions selected.
Table 6. Negative Emotions selected for characteristic VLE
Surprise Unpleasantness
Disappointment
Dissatisfaction
Negative Emotions
Indignation
Frustration
Confusion
Contempt
Boredom
Tension
Disgust
Pedagogical facility P P P P P P P P P
Support P P P P P P P P
Content P P P P P P P P P
User Interface P P P P P P P P P
Handling Errors P P P P P P P P P
Tools P P P P P P P P
Flexibility P P P P P P P
Standards P P P P P P P
12
Table 7. Positive Emotions selected for characteristic VLE
Desire: Impulse
Engagement
Satisfaction
Admiration
Inspiration
Positive Emotions
Fun
Pedagogical facility P P P P P P
Support P P P P P
Content P P P P P
User Interface P P P P P
Handling Errors P P P
Tools P P P P P
Flexibility P P P P P P
Standards P P P P P
Finally, the emotions to evaluate during the use of each VLE functionality according
to the opinion of experts and student users are listed in table 8.
Table 8. Final list of emotions per functionality of an VLE according to experts and students
Positive Negative
Indignation, Contempt, Dissatisfac-
Desire, Inspiration, Admira-
Pedagogical tion, Disappointment, Unpleasant Sur-
tion, Satisfaction, Fun, Engage-
facility prise, Boredom, Tension-Nervousness,
ment
Frustration, Confusion
Indignation, Contempt, Dissatisfac-
Desire, Inspiration, Admira-
tion, Disappointment, Unpleasant Sur-
Support tion, Satisfaction, Fun
prise, Boredom, Frustration, Confu-
sion
Disgust , Indignation, Contempt,
Desire, Inspiration, Admira-
Dissatisfaction, Disappointment, Un-
Content tion, Satisfaction, Fun
pleasant Surprise, Boredom, Frustra-
tion, Confusion
Disgust , Indignation, Contempt,
Desire, Inspiration, Admira-
User Inter- Dissatisfaction, Disappointment, Un-
tion, Satisfaction, Fun
face pleasant Surprise, Boredom, Frustra-
tion, Confusion
Disgust , Indignation, Contempt,
Inspiration, Admiration, Sat-
Handling Dissatisfaction, Disappointment, Un-
isfaction
Errors pleasant Surprise, Boredom, Frustra-
tion, Confusion
Indignation, Contempt, Dissatisfac-
Desire, Inspiration, Admira-
tion, Disappointment, Unpleasant Sur-
Tools tion, Satisfaction, Fun
prise, Boredom, Frustration, Confu-
sion
13
Desire, Inspiration, Admira- Contempt, Dissatisfaction, Disap-
Flexibility tion, Satisfaction, Fun, Engage- pointment, Unpleasant Surprise, Bore-
ment dom, Frustration, Confusion
Disgust , Indignation, Contempt,
Desire, Inspiration, Admira-
Standards Dissatisfaction, Disappointment, Un-
tion, Satisfaction, Fun
pleasant Surprise, Boredom
5 Conclusions and Further Works
It is important that users/students of a VLE feel satisfied when interacting with these
computer systems. Identifying positive and negative perceptions through such emo-
tions. The main problem is that there is no validated list of emotions that students may
experience when interacting with each of the functionalities of an VLE.
As a contribution to evaluate affectively each of the functionalities of a VLE, a list
of emotions, validated by experts and students, is left.
In this sense, this work managed to define the emotions in Spanish according to each
functionality of the VLE, contributing to the area of HCI in the context of education.
This list of emotions in Spanish will be very useful for the field of Affectivity and
Educational Technology, since it has the support of recognized professionals as well as
students of higher education, of different ages and countries of origin, which enriches
the validity and heterogeneity of the study.
As future work, it is planned to determine the scale to be used to perform an affective
evaluation of VLEs. Subsequently, it is planned to implement affective evaluations of
the educational platforms with the proposal list of emotions, for validation and feed-
back.
Another useful aspect to be analyzed in the future is the possibility of determining
which emotions are most related to each characteristic, in order to consider them in both
the design and the evaluation of the platform.
The HCI community should consider these issues to identify aspects that influence
users' perceptions of their UX experience in order to develop better evaluation ap-
proaches.
6 References
1. H. Fardoun, C. González, C. A. Collazos, and M. Yousef, “Exploratory Study in Iberoam-
erica on the Teaching-Learning Process and Assessment Proposal in the Pandemic Times =
Estudio exploratorio en iberoamérica sobre procesos de enseñanza-aprendizaje y propuesta
de evaluación en tiempos de pandemia,” Educ. Knowl. Soc., vol. 21, 2020.
2. F. J. García-Peñalvo, A. Corell, V. Abella-García, and M. Grande, “La evaluación online en
la educación superior en tiempos de la COVID-19,” Educ. Knowl. Soc., vol. 21, no. 0, p.
26, 2020.
3. I. T. Plata and D. B. Alado, Evaluating the Perceived Usability of Virtual Learning Environ-
ment in Teaching ICT Courses, vol. 1. 2015.
14
4. E. Crisol-Moya, L. Herrera-Nieves, and R. Montes-Soldado, “Educación virtual para todos:
una revisión sistemática,” Educ. Knowl. Soc., vol. 21, no. 0, p. 13, Jun. 2020.
5. F. J. García-Peñalvo, A. Miguel, and S. Pardo, “Una revisión actualizada del concepto de
eLearning. Décimo Aniversario,” vol. 16, no. 1, 2015.
6. E. Martínez Caro, “E-Learning: Un Análisis Desde El Punto De Vista Del Alumno,” RIED.
Rev. Iberoam. Educ. a Distancia, vol. 11, no. 2, pp. 151–168, 2012.
7. M. Hassenzahl, “The Thing and I: Understanding the Relationship Between User and Prod-
uct,” Springer, Cham, 2018, pp. 301–313.
8. K. Capota, M. van Hout, and T. van der Geest, “Measuring the emotional impact of web-
sites,” in Proceedings of the 2007 conference on Designing pleasurable products and inter-
faces - DPPI ’07, 2007, p. 135.
9. L. Masip Ardévol, “User experience methodology for the design and evaluation of interac-
tive systems,” University of Lleida, 2013.
10. J. I. Cocunubo-Suárez, J. A. Parra-Valencia, and J. E. Otálora-Luna, “Propuesta para la eva-
luación de Entornos Virtuales de Enseñanza Aprendizaje con base en estándares de Usabi-
lidad Evaluation of Virtual Teaching- Learning Environments based on usability standards
Cómo citar / How to cite Propuesta para la evaluación de En,” vol. 21, no. 41, pp. 123–7799,
2018.
11. M. Hassenzahl, M. Burmester, and K. Franz, “AttrakDiff: A questionnaire to measure per-
ceived hedonic and pragmatic quality,” Mensch Comput., vol. 57, 2003.
12. B. Laugwitz, T. Held, and M. Schrepp, “Construction and Evaluation of a User Experience
Questionnaire,” Springer, Berlin, Heidelberg, 2008, pp. 63–76.
13. Bradley, M. M. and Lang, P. J., “Measuring emotion: the self-assessment manikin and the
semantic differential,” J. Behav. Ther. Exp. Psychiatry, vol. 25, no. 1, pp. 49–59, 1994.
14. G. Huisman, M. van Hout, E. van Dijk, T. van der Geest, and D. Heylen, “LEMtool,” in
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI
’13, 2013, p. 351.
15. P. Desmet, “Pieter Desmet Chapter 9,” Funology From usability to enjoyment, pp. 111–123,
2005.
16. P. M. A. Desmet, M. A. Blythe, A. F. Monk, K. Overbeeke, and P. C. Wright, “Measuring
Emotions Development and application of an instrument to measure emotional responses to
products.”
17. P. M. Desmet, “Measuring emotions: Development of an instrument to measure emotional
responses to products,” Funology from usability to enjoyment. Kluwer Acad. Publ. Dor-
drecht, Boston, Londres, no. Blythe, M. A.; Overbeeke, K.; Monk, A. F.; Wright, P. C. (ed.),
2003.
18. G. Laurans and P. Desmet, “Introducing Premo2 New Directions for the Non - Verbal Meas-
urement of Emotion in Design,” Proc. 8th Int. Des. Emot. Conf., 2012.
19. Y. A. Méndez-Alegría, C. A. Collazos, T. Granollers, and R. Gil, “Rueda de emociones de
Ginebra+: instrumento para la valoración emocional de los usuarios mientras participan en
una evaluación de sistemas interactivos,” Dyna, vol. 91, no. 2, pp. 151–155, 2016.
20. M. Fritz, “Reinventing the Wheel: Emotional Awareness Enhancement in Computer-Medi-
ated Collaboration with the Dynamic Emotion Wheel.” 2015.
21. P. Desmet, “Measuring Emotion: Development and Application of an Instrument to Measure
Emotional Responses to Products,” 2003.
22. L. N. Aballay, S. V. Aciar, and C. A. Collazos, “Emotions for Virtual Learning Environ-
ments,” IEEE-RITA, vol. 9, no. 3, 2021. (Not yet published)
23. 23. V. Sacharin, K. Schlegel, and K. R. Scherer, “Geneva Emotion Wheel Rating Study,”
2012.
15
24. M. Blythe and M. Hassenzahl, “The Semantics of Fun: Differentiating Enjoyable Experi-
ences,” Springer, Cham, 2018, pp. 375–387.
25. W. Aarron, Designing for emotion (pp. 978-1). New York: A book apart. 2012.
26. Y.-H. Shih and M. Liu, “The Importance of Emotional Usability,” J. Educ. Technol. Syst.,
vol. 36, no. 2, pp. 203–218, Dec. 2007.
27. M. L. Wong, C. W. Khong, and H. Thwaites, “Applied UX and UCD Design Process in
Interface Design,” Procedia - Soc. Behav. Sci., vol. 51, pp. 703–708, Jan. 2012.
28. L. A. Hasan and K. T. Al-Sarayreh, “An Integrated Measurement Model for Evaluating Us-
ability Attributes,” Proc. Int. Conf. Intell. Inf. Process. Secur. Adv. Commun., p. 94, 2015.
29. M. M. Alomari, H. El-Kanj, N. I. Alshdaifat, and A. Topal, “A framework for the impact of
human factors on the effectiveness of learning management systems,” IEEE Access, vol. 8,
pp. 23542–23558, 2020.
30. M. Syed, M. Chetlur, S. Afzal, G. A. Ambrose, and N. V. Chawla, “Implicit and explicit
emotions in MOOCs,” EDM 2019 - Proc. 12th Int. Conf. Educ. Data Min., no. Edm, pp.
432–437, 2019.
31. N. Phongphaew and A. Jiamsanguanwong, “The Usability Evaluation Concerning Emo-
tional Responses of Users on Learning Management System,” Proc. 2016 6th Int. Work.
Comput. Sci. Eng. (WCSE 2016), pp. 43–48, 2016.
32. L. Marques, W. Nakamura, N. Valentim, L. Rivero, and T. Conte, “Do Scale Type Tech-
niques Identify Problems that Affect User eXperience? User Experience Evaluation of a
Mobile Application,” 2018.
33. T. Acosta and S. Luján-Mora, “Analysis of Emotion in the Use of Accessible Learning Man-
agement Systems by Students,” in 9th Annual International Conference of Education, Re-
search and Innovation (ICERI 2016), 2016.
34. W. T. Nakamura, L. C. Marques, L. Rivero, and E. H. T. De Oliveira, “Are Generic UX
Evaluation Techniques Enough? A study on the UX Evaluation of the Edmodo Learning
Management System,” Brazilian Symp. Comput. Educ. (Simpósio Bras. Informática na
Educ. - SBIE), vol. 28, no. 1, p. 1007, Oct. 2017.
35. W. T. Nakamura, E. H. T. De Oliveira, and T. Conte, “Usability and user experience evalu-
ation of learning management systems a systematic mapping study,” in ICEIS 2017 - Pro-
ceedings of the 19th International Conference on Enterprise Information Systems, 2017, vol.
3, pp. 97–108.
36. W. T. Nakamura, L. C. Marques, L. Rivero, E. H. T. De Oliveira, and T. Conte, “Are scale-
based techniques enough for learners to convey their UX when using a Learning Manage-
ment System?” Rev. Bras. Informática na Educ., vol. 27, no. 01, p. 104, Jan. 2019.