There is no doubt that e-learning, which is defined as "the learning supported by digital electronic tools and media" [1], especially after the Covid pandemic, has become a necessity in the educational sector globally, either it concerns school education, higher education or vocational education and training. Thus, it is reasonable that there is an increasing interest in optimizing e-learning procedures and tools in order to offer e-learning services of high quality. This interest derives not only from the educational organizations that wish to pioneer and to be competitive, but also from researchers and academics that constantly look for new ways to evaluate and improve e-learning environments. Therefore, it comes that the concept of quality is crucial when it relates to education and especially elearning, mainly because the latter is delivered via digital technologies, which in fact mediate the educational process and transform the connections among the participants (teachers and students).
In general, it is difficult to define quality in a unique way, because it is a subjective, relational and multidimensional concept. Actually, there is a long debate about its definition. Ehlers [2] distinguishes three components of this debate on quality, as shown in fig. 1. According to him, it is not easy to define quality because of a) different interpretations of quality, b) different players with different perspectives of quality, c) different forms of quality.
However, generally speaking and according to Juran, quality can be defined as "the degree to which a product / service meets or exceeds expectations / requirements of customers" and "fitness for use". According to Crosby, quality means "conformance to requirements". According to ISO 8402 [3], quality is "the characteristics of a product or service that bear on its ability to satisfy stated or implied needs". Finally, another definition of quality is given in the Qualinet White Paper [4]: "Quality is the outcome of an individual"s comparison and judgment process. It includes perception, reflection about the perception, and the description of the outcome. In contrast to definitions which see quality as a "set of inherent characteristics", quality is considered in terms of the evaluated excellence or goodness, of the degree of need fulfillment, and in terms of a "quality event"". As can be seen from the above, quality may refer either to services or to products and according to this basic distinction it could be categorized into further types, e.g. "quality of conventional services" and "quality of e-services". Following a conceptual path from quality to the central subject of our study, namely the quality of experience in e-learning, we present in what follows the state of the art knowledge regarding the related concepts.
Specifically about services, Ehlers [2] mentions Donabedian"s statement of the relational nature of quality: "quality is the degree of conformance between a performed service and the goals set for this service". Similarly, according to Parasuraman et al. [5], service quality can be defined as "perceptions (that) result from a comparison of consumer expectations with actual service performance". From this point of view, service quality is measured by calculating the difference between expectations and perceptions.
With the advent of the internet to most aspects of human activity in modern societies, more and more conventional services are extended to electronic, namely e-services. The latter are web-based services, which are delivered through electronic media (e.g. computers, smartphones, tablets). Thus, the characteristics of services and e-services vary to a great extent and, therefore, the quality dimensions of services and e-services vary too [6]. One well-known definition of the quality of eservices is the following: "E-service quality is defined as the extent to which a website facilitates efficient and effective shopping, purchasing, and delivery" [7]. Of course, this definition does not actually reflect the wide range of e-services that exist nowadays (e.g. e-learning, e-banking, egovernment, etc.), but we evaluate positively the fact that it focuses on two important elements, efficiency and effectiveness. According to the above definition, e-service quality is the degree of efficiency and effectiveness of services delivered to users on the internet.
Taking one step further, e-learning -as an educational service offered through digital resources on the internet -is undeniably a form of e-service. Quality of e-learning is defined as "the philosophy of achieving excellence in all aspects of education through e-learning technologies" [8].
Exploiting the previously described definition of quality, we consider the quality of e-learning as the degree to which an e-learning course meets or exceeds the expectations / requirements of learners, providing them with educational experiences. Quality of e-learning may refer to the educational technology, the educational content, the teaching-learning process, the student assessment, the administrative support etc.
Through e-learning environments, the participants acquire various educational experiences. An experience is defined as "an individual"s stream of perception and interpretation of one or multiple events" [4]. Experiences in e-learning may refer to teachers, to students or both. Moreover, experiences are connected to the quality that characterizes every e-learning environment (and every educational environment generally), meaning that there may be good or bad experiences. Ιn other words, there may be high or low quality of experience. The latter is defined as follows: "Quality of Experience (QoE) is the degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his or her expectations with respect to the utility and/or enjoyment of the application or service in the light of the user"s personality and current state" [4].
Discussing quality of experience in the context of e-learning, quality is an important factor for the future of e-learning [2]. On the other hand, experience of the learners is a crucial factor for the overall success of an e-learning course, because it directly affects their learning outcome(s). Through their experience, the learners fulfill their expectations and gain satisfaction and knowledge. So, from the learners" perspective, the concept of quality is perceived as their experience, based on their expectations and satisfaction.
The purpose of this paper is to explore the literature of the last decade, in order to determine and clarify the concept of QoE in e-learning. We also aim to gain deeper understanding of the concept, by searching for answers to research questions such as: Research Question 1: What is QoE in e-learning? and Research Question 2: Is QoE different from user experience?
To serve our purpose we conduct a scoping literature review, as this type of review is the most appropriate to gather knowledge, clarify concepts and identify gaps on a particular topic. The findings of this review hopefully provide a clear understanding of QoE to researchers and other stakeholders in the field of e-learning (e.g. academics, practitioners, organizations). They also indicate the recent trends of research on QoE in e-learning.
The paper is organized as follows. In Section 2 we present the methodology of the scoping literature review. In Section 3 we go on with the results of our review and in Section 4 we discuss the findings. In Section 5 we provide our conclusions.
A literature review is a form of research for knowledge synthesis. There are many literature review types, such as narrative review, systematic review, scoping review, mapping review etc. [9,10]. For the purpose of this paper, we consider the scoping review as the most appropriate. Scoping reviews belong to the category of rapid reviews [9]. A scoping review is suitable for exploring, identifying, mapping, reporting or discussing characteristics or concepts across a breadth of evidence sources [11]. It is used for clarifying concepts, identifying knowledge gaps and providing an overview of the existing knowledge. The present study is guided by the PRISMA 2020 statement [12] and the PRISMA extension for Scoping Reviews Checklist [10,13], and was conducted in 5 stages [9]: The purpose and the research questions of our research (Stage 1) are mentioned at the end of Section 1. Next, (as per Stage 2) we selected the database of Scopus and determined the search criteria to trace the relevant literature: boolean search of the terms "quality of experience" AND "elearning" in title-abstract-keywords, filtered with the year range "2014-2023". We continued with the inclusion or exclusion of the publications that had arisen (Stage 3). Publications that either were irrelevant to our research or did not provide any definition of QoE were excluded. We then studied the selected publications and extracted the useful information (Stage 4). Finally we summarized the findings (Stage 5).
Our search in Scopus initially brought 122 records as a result. After the removal of 4 records due to duplication, retraction and other reasons, 118 records remained. At the first screening 67 records were excluded as irrelevant. We proceeded to the second screening for retrieval of the reports related to our purpose, where 45 reports were retrieved. We assessed thoroughly all 45 retrieved reports for eligibility. 29 of them were excluded with reasons (no definitions detected). We finally included 16 studies in the review.
In Table 1 we provide the results of our review in chronological order. For each reviewed study we present the definitions/meanings of QoE mentioned in it. In addition, we present the research topics in the field of e-learning that the study relates to. The related research topics are of great interest because they offer more detail on the meaning of QoE. Moreover, we note the type of publication. QoE is one of the six measures for experience, along with hedonic user experience, pragmatic user experience, VRinduced symptoms, sense of presence, overall satisfaction.
Ali, Simba 2021 [19] conference paper video streaming QoE is the overall acceptability of an application or service, as perceived subjectively by the end-user.
Gao 2021 [20] journal article internet-based resources, English learning QoE is the user's point of view of the whole performance of a system. QoE is the degree to which users of an application or service are happy or annoyed. It is based on the user's personality, current state, and expectations for the utility and/or enjoyment of the application service. QoE ultimately leads to the acceptability of the system, application, or service. QoE is the construct concerning how well a system provides functions for a user to complete particular tasks. QoE is related to the evaluation of each participant towards the application.
Pal, Vanijja, Patra 2020 [23] conference paper online learning, videoconferencing applications, multimedia quality QoE is defined as "the overall acceptability of an application or service, as perceived subjectively by the end user". From the definition itself it is clear that QoE is a superset of QoS that includes all possible complete end-to-end system effects, along with the user expectations. Idrizi, Filiposka, Trajkovik 2018 [24] conference paper online education, learning styles
Qualities of Experience are the measurements of how students felt and were satisfied during the online classes.
Kong, Liu, An 2018 [25] conference paper
QoE is "the overall acceptability of an application or service, as perceived subjectively by an end-user". "QoE is the degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his or her expectations with respect to the utility and/or enjoyment of the application or service in the light of the user's personality and current state". QoE is a multidimensional quality that can be decomposed in a set of perceptual attributes called features. Davcev, Jakimovski, Scepanovic 2018 [26] conference paper mobile learning, multimedia content, mobile cloud computing QoE is used as an overall acceptability of an application or service, as perceived subjectively by the end-user and represents a multidimensional subjective concept that is not easy to evaluate.
2017 [27] conference paper mobile learning, multimedia QoE is a main factor contributing to user's engagement with multimedia services.
Fernandes, Cardoso, Marcelino 2015 [28] conference paper MOOC QoE is used to describe how students evaluate a service. QoE "is how a user perceives the usability of a service when in use". QoE is "the degree to which a system meets the tacit and explicit expectations of the user for the experience".
2015 [29] conference paper mobile learning, multimedia content, collaborative cloud service QoE is a metric that quantifies the multifaceted, multidimensional factors that influence the perceived quality. QoE is defined as the degree of delight or annoyance of a person whose experience involves an application, service, or system. QoE is no longer an expression for user satisfaction of using a service, but it is a degree of delight or annoyance, which is a more dynamic measure of personal experience.
10 articles out of 16 (62.5%) are published between 2020 and 2023 and the rest (37.5%) from 2015 to 2019. The majority of them (75%) are conference papers, which implies that the particular research area is still unexplored, thus very promising. The research topics of selected studies vary depending on the e-learning area, the applied technologies and the type of educational content. Topics that refer to the field or a sub-field of elearning are the following: video-based online learning [ [19,20,23,25,26] the delight or annoyance from an application or service [20,25,29] a subjective perception by the user [14,15,16,20,22,26,28] a multidimensional concept [25,26,29] a factor of user's engagement [27] From the above conceptual variety, we confirm that QoE in e-learning is multidimensional and subjective, thus it is defined depending on the dimensions of the research interest. However, very often the definitions overlap and have many elements in common. We infer that experience and satisfaction are related concepts, both user-centric and user-perceived, and their measurement determines QoE. We also infer that most of the studies adopt mainly two definitions of QoE, either directly or with some paraphrase, which are a) the definition by the ITU (i.e., "the overall acceptability of an application or service, as perceived subjectively by an end-user") and b) the definition by the Qualinet White Paper (i.e., "the degree of delight or annoyance of the user of an application or service"). At this point it is noteworthy to mention the statement of Karadimce and Davcev [29] that "QoE is no longer an expression for user satisfaction of using a service, but it is a degree of delight or annoyance, which is a more dynamic measure of personal experience". This indicates the subjective character of QoE.
According to the above definitions, QoE is affected by factors such as: the type of device and the quality of the Internet connectivity [14], presence, flow and symptoms in VR [16, 21] quality variation, stalling, and initial delay in a streaming session [17] user expectations and context [19] user"s personality, current state, and expectations for the utility and/or enjoyment of the application service [20] Additionally, as there is no single definition of QoE, it is difficult to evaluate the particular concept [26] or, at least, there is no single measure of it. The measures of QoE vary as dependable on the various factors that are taken into account in every research.
The answer to this question is that QoE and user experience (UX) are very similar concepts, since both of them refer to users" experience with digital technologies. However, they are not exactly the same. We infer that UX is the experience perceived by the user, while QoE is a measure for UX [16,18,21,29]. More specifically, Bieg et al. [18] mention an interesting description of QoE, "as one of the six measures for experience, along with hedonic user experience, pragmatic user experience, VRinduced symptoms, sense of presence, overall satisfaction". From this statement we conclude that QoE and UX are not the same.
Concluding our review, we find out that there is not a specific, widely accepted definition of QoE in e-learning. The adopted definitions in the field of e-learning are also used in other fields, as QoE is defined for all e-services. Moreover, most studies in this review approach e-learning from a technical view, mainly through computer science, thus QoE is mainly related with technical aspects of the elearning systems. QoE is a concept often used in a technical context [30]. We agree with Kist and Brodie [30], that the learning experience differs significantly from the general consumer experience. We also agree with Ehlers and Hilera [31], according to whom "quality in the field of e-learning is an especially diverse field, because it brings together the field of education, technology, and economy in order to contribute a) to societal development, b) to innovate formal, non formal, and informal learning opportunities, and c) empower learners as citizens to take part in our emerging learning and information societies". We believe that QoE in e-learning is a concept that refers to learning processes, through learning environments, where learning objectives exist and learning outcomes happen. Learners are more than consumers of a service, they acquire knowledge and they develop skills and attitudes in a long-term horizon, apart from their short-term experiences. So, QoE in elearning needs to be defined more specifically and this is certainly a research gap which should be studied in the near future.
This review also reveals that the research focus concerns only the learners" perspective. This was expected, as learners are the main end-users in e-learning and actually the user-perceived experience refers to the learners" experience. However, this fact can be seen as a research gap, because teachers are highly involved in e-learning too (except self-learning cases), and their experience may have a significance for the overall QoE, e.g. in higher education. This review also shows that QoE is not easy to evaluate and this can be considered as another research gap too.
This review has some limitations regarding the generalization of the findings. Due to the fact that it was conducted in only one database, Scopus, more research is needed, expanded in more databases in order to come to general results. The time span is also a limitation of this review.