Defining Quality of Experience in e-Learning: a Literature Review

Abstract

Our research aims to investigate the “quality of experience” in education and particularly in the context of e-learning. Firstly, we study separately the concepts of “quality” and “experience”, as well as the underlying meaning of these terms. Secondly, through a scoping review among papers of the last decade, we study how “quality of experience”, as a whole, is defined in e-learning. Our purpose is to set a solid basis for the clarification of “quality of experience” in e-learning, considering that it is important to have a clear view of concepts before proceeding with a research related to them. As a result of our review, we reach useful answers to questions such as “What is quality of experience in e-learning?”, “Is quality of experience different from user experience?” gaining deeper understanding of the examined concepts.

Keywords

• 1 quality
• quality of experience
• e-learning
• literature review
• scoping review

1. Introduction

There is no doubt that e-learning, which is defined as “the learning supported by digital electronic tools and media” [1], especially after the Covid pandemic, has become a necessity in the educational sector globally, either it concerns school education, higher education or vocational education and training. Thus, it is reasonable that there is an increasing interest in optimizing e-learning procedures and tools in order to offer e-learning services of high quality. This interest derives not only from the educational organizations that wish to pioneer and to be competitive, but also from researchers and academics that constantly look for new ways to evaluate and improve e-learning environments. Therefore, it comes that the concept of quality is crucial when it relates to education and especially elearning, mainly because the latter is delivered via digital technologies, which in fact mediate the educational process and transform the connections among the participants (teachers and students).

In general, it is difficult to define quality in a unique way, because it is a subjective, relational and multidimensional concept. Actually, there is a long debate about its definition. Ehlers [ 2 ] distinguishes three components of this debate on quality, as shown in fig. 1. According to him, it is not easy to define quality because of a) different interpretations of quality, b) different players with different perspectives of quality, c) different forms of quality.

However, generally speaking and according to Juran, quality can be defined as “the degree to which a product / service meets or exceeds expectations / requirements of customers” and “fitness for use”. According to Crosby, quality means “conformance to requirements”. According to ISO 8402 [ 3 ], quality is “the characteristics of a product or service that bear on its ability to satisfy stated or implied needs”. Finally, another definition of quality is given in the Qualinet White Paper [ 4 ]: “Quality is the outcome of an individual‟s comparison and judgment process. It includes perception, reflection about the perception, and the description of the outcome. In contrast to definitions which see quality as a „set of inherent characteristics‟, quality is considered in terms of the evaluated excellence or goodness, of the degree of need fulfillment, and in terms of a „quality event‟”.

As can be seen from the above, quality may refer either to services or to products and according to this basic distinction it could be categorized into further types, e.g. “quality of conventional services” and “quality of e-services”. Following a conceptual path from quality to the central subject of our study, namely the quality of experience in e-learning, we present in what follows the state of the art knowledge regarding the related concepts.

Specifically about services, Ehlers [ 2 ] mentions Donabedian‟s statement of the relational nature of quality: “quality is the degree of conformance between a performed service and the goals set for this service”. Similarly, according to Parasuraman et al. [ 5 ], service quality can be defined as “perceptions (that) result from a comparison of consumer expectations with actual service performance”. From this point of view, service quality is measured by calculating the difference between expectations and perceptions.

With the advent of the internet to most aspects of human activity in modern societies, more and more conventional services are extended to electronic, namely e-services. The latter are web-based services, which are delivered through electronic media (e.g. computers, smartphones, tablets). Thus, the characteristics of services and e-services vary to a great extent and, therefore, the quality dimensions of services and e-services vary too [ 6 ]. One well-known definition of the quality of eservices is the following: “E-service quality is defined as the extent to which a website facilitates efficient and effective shopping, purchasing, and delivery” [ 7 ]. Of course, this definition does not actually reflect the wide range of e-services that exist nowadays (e.g. e-learning, e-banking, egovernment, etc.), but we evaluate positively the fact that it focuses on two important elements, efficiency and effectiveness. According to the above definition, e-service quality is the degree of efficiency and effectiveness of services delivered to users on the internet.

Taking one step further, e-learning - as an educational service offered through digital resources on the internet - is undeniably a form of e-service. Quality of e-learning is defined as “the philosophy of achieving excellence in all aspects of education through e-learning technologies” [ 8 ].

Exploiting the previously described definition of quality, we consider the quality of e-learning as the degree to which an e-learning course meets or exceeds the expectations / requirements of learners, providing them with educational experiences. Quality of e-learning may refer to the educational technology, the educational content, the teaching-learning process, the student assessment, the administrative support etc.

Through e-learning environments, the participants acquire various educational experiences. An experience is defined as “an individual‟s stream of perception and interpretation of one or multiple events” [ 4 ]. Experiences in e-learning may refer to teachers, to students or both. Moreover, experiences are connected to the quality that characterizes every e-learning environment (and every educational environment generally), meaning that there may be good or bad experiences. Ιn other words, there may be high or low quality of experience. The latter is defined as follows: “Quality of Experience (QoE) is the degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his or her expectations with respect to the utility and/or enjoyment of the application or service in the light of the user‟s personality and current state” [ 4 ].

Discussing quality of experience in the context of e-learning, quality is an important factor for the future of e-learning [ 2 ]. On the other hand, experience of the learners is a crucial factor for the overall success of an e-learning course, because it directly affects their learning outcome(s). Through their experience, the learners fulfill their expectations and gain satisfaction and knowledge. So, from the learners‟ perspective, the concept of quality is perceived as their experience, based on their expectations and satisfaction.

The purpose of this paper is to explore the literature of the last decade, in order to determine and clarify the concept of QoE in e-learning. We also aim to gain deeper understanding of the concept, by searching for answers to research questions such as: Research Question 1: What is QoE in e-learning? and Research Question 2: Is QoE different from user experience?

To serve our purpose we conduct a scoping literature review, as this type of review is the most appropriate to gather knowledge, clarify concepts and identify gaps on a particular topic. The findings of this review hopefully provide a clear understanding of QoE to researchers and other stakeholders in the field of e-learning (e.g. academics, practitioners, organizations). They also indicate the recent trends of research on QoE in e-learning.

The paper is organized as follows. In Section 2 we present the methodology of the scoping literature review. In Section 3 we go on with the results of our review and in Section 4 we discuss the findings. In Section 5 we provide our conclusions.

2. Literature review methodology

A literature review is a form of research for knowledge synthesis. There are many literature review types, such as narrative review, systematic review, scoping review, mapping review etc. [ 9, 10 ]. For the purpose of this paper, we consider the scoping review as the most appropriate. Scoping reviews belong to the category of rapid reviews [ 9 ]. A scoping review is suitable for exploring, identifying, mapping, reporting or discussing characteristics or concepts across a breadth of evidence sources [ 11 ]. It is used for clarifying concepts, identifying knowledge gaps and providing an overview of the existing knowledge. The present study is guided by the PRISMA 2020 statement [ 12 ] and the PRISMA extension for Scoping Reviews Checklist [ 10, 13 ], and was conducted in 5 stages [ 9 ]: Stage 1: identifying the research question(s) Stage 2: identifying relevant publications / studies Stage 3: selection of publications / studies Stage 4: charting the data Stage 5: collating, summarizing and reporting the results

The purpose and the research questions of our research (Stage 1) are mentioned at the end of Section 1. Next, (as per Stage 2) we selected the database of Scopus and determined the search criteria to trace the relevant literature: boolean search of the terms “quality of experience” AND “elearning” in title-abstract-keywords, filtered with the year range “2014-2023”. We continued with the inclusion or exclusion of the publications that had arisen (Stage 3). Publications that either were irrelevant to our research or did not provide any definition of QoE were excluded. We then studied the selected publications and extracted the useful information (Stage 4). Finally we summarized the findings (Stage 5).

Our search in Scopus initially brought 122 records as a result. After the removal of 4 records due to duplication, retraction and other reasons, 118 records remained. At the first screening 67 records were excluded as irrelevant. We proceeded to the second screening for retrieval of the reports related to our purpose, where 45 reports were retrieved. We assessed thoroughly all 45 retrieved reports for eligibility. 29 of them were excluded with reasons (no definitions detected). We finally included 16 studies in the review.

In Table 1 we provide the results of our review in chronological order. For each reviewed study we present the definitions/meanings of QoE mentioned in it. In addition, we present the research topics in the field of e-learning that the study relates to. The related research topics are of great interest because they offer more detail on the meaning of QoE. Moreover, we note the type of publication.

3. Results

Idrizi, Filiposka, Trajkovik

4. Discussion of the findings

4.1. Basic characteristics of the studies

10 articles out of 16 (62.5%) are published between 2020 and 2023 and the rest (37.5%) from 2015 to 2019. The majority of them (75%) are conference papers, which implies that the particular research area is still unexplored, thus very promising.

The research topics of selected studies vary depending on the e-learning area, the applied technologies and the type of educational content. Topics that refer to the field or a sub-field of elearning are the following:  video-based online learning [ 14 ]  online learning [ 17, 23, 24 ]  digital training [ 18 ]  language learning [20, 22]  virtual training [25]  mobile learning [26, 27, 29]  mooc [28] Topics that refer to the technology or the content of e-learning are:  video streaming [ 14, 17, 19, 21 ]  machine learning [ 14 ]  web service for course selection [ 15 ]  virtual reality [ 16, 18, 21, 22, 25 ]  gamification [ 16, 22 ]  internet-based resources [20]  videoconferencing [23]  multimedia [23, 26, 27, 29]  learning styles [24]  cloud computing [26, 29]

Finally, another characteristic is that all 16 studies approach e-learning from the learners‟ perspective.

4.2. Ηow is QoE defined in e-learning?

4.2.1. What is QoE in e-learning?

According to the definitions presented in Table 1, QoE is considered as:  a measure of satisfaction [ 15, 17, 24 ]  a measure of experience [ 16, 18, 21, 29 ]  the acceptability of an application or service [19, 20, 23, 25, 26]  the delight or annoyance from an application or service [20, 25, 29]  a subjective perception by the user [ 14, 15, 16, 20, 22, 26, 28 ]  a multidimensional concept [25, 26, 29]  a factor of user's engagement [27]

From the above conceptual variety, we confirm that QoE in e-learning is multidimensional and subjective, thus it is defined depending on the dimensions of the research interest. However, very often the definitions overlap and have many elements in common. We infer that experience and satisfaction are related concepts, both user-centric and user-perceived, and their measurement determines QoE. We also infer that most of the studies adopt mainly two definitions of QoE, either directly or with some paraphrase, which are a) the definition by the ITU (i.e., “the overall acceptability of an application or service, as perceived subjectively by an end-user”) and b) the definition by the Qualinet White Paper (i.e., “the degree of delight or annoyance of the user of an application or service”). At this point it is noteworthy to mention the statement of Karadimce and Davcev [29] that “QoE is no longer an expression for user satisfaction of using a service, but it is a degree of delight or annoyance, which is a more dynamic measure of personal experience”. This indicates the subjective character of QoE.

According to the above definitions, QoE is affected by factors such as:  the type of device and the quality of the Internet connectivity [ 14 ],  presence, flow and symptoms in VR [ 16, 21 ]  quality variation, stalling, and initial delay in a streaming session [ 17 ]  user expectations and context [19]  user‟s personality, current state, and expectations for the utility and/or enjoyment of the application service [20]

Additionally, as there is no single definition of QoE, it is difficult to evaluate the particular concept [26] or, at least, there is no single measure of it. The measures of QoE vary as dependable on the various factors that are taken into account in every research.

4.2.2. Is QoE different from user experience?

The answer to this question is that QoE and user experience (UX) are very similar concepts, since both of them refer to users‟ experience with digital technologies. However, they are not exactly the same. We infer that UX is the experience perceived by the user, while QoE is a measure for UX [ 16, 18, 21, 29 ]. More specifically, Bieg et al. [ 18 ] mention an interesting description of QoE, “as one of the six measures for experience, along with hedonic user experience, pragmatic user experience, VRinduced symptoms, sense of presence, overall satisfaction”. From this statement we conclude that QoE and UX are not the same.

5. Conclusions

Concluding our review, we find out that there is not a specific, widely accepted definition of QoE in e-learning. The adopted definitions in the field of e-learning are also used in other fields, as QoE is defined for all e-services. Moreover, most studies in this review approach e-learning from a technical view, mainly through computer science, thus QoE is mainly related with technical aspects of the elearning systems. QoE is a concept often used in a technical context [30]. We agree with Kist and Brodie [30], that the learning experience differs significantly from the general consumer experience. We also agree with Ehlers and Hilera [31], according to whom “quality in the field of e-learning is an especially diverse field, because it brings together the field of education, technology, and economy in order to contribute a) to societal development, b) to innovate formal, non formal, and informal learning opportunities, and c) empower learners as citizens to take part in our emerging learning and information societies”. We believe that QoE in e-learning is a concept that refers to learning processes, through learning environments, where learning objectives exist and learning outcomes happen. Learners are more than consumers of a service, they acquire knowledge and they develop skills and attitudes in a long-term horizon, apart from their short-term experiences. So, QoE in elearning needs to be defined more specifically and this is certainly a research gap which should be studied in the near future.

This review also reveals that the research focus concerns only the learners‟ perspective. This was expected, as learners are the main end-users in e-learning and actually the user-perceived experience refers to the learners‟ experience. However, this fact can be seen as a research gap, because teachers are highly involved in e-learning too (except self-learning cases), and their experience may have a significance for the overall QoE, e.g. in higher education. This review also shows that QoE is not easy to evaluate and this can be considered as another research gap too.

This review has some limitations regarding the generalization of the findings. Due to the fact that it was conducted in only one database, Scopus, more research is needed, expanded in more databases in order to come to general results. The time span is also a limitation of this review.

6. References

[1] K. S. Basak, M. Wotto, P. Bélanger, E-learning, M-learning and D-learning: Conceptual definition and comparative analysis, E-Learning and Digital Media 15 (4) (2018) p. 191-216. https://doi.org/10.1177/2042753018785180 International Conference on Quality of Multimedia Experience (QoMEX) (Lippstadt, Germany), IEEE, 2022. https://doi.org/10.1109/QoMEX55416.2022.9900889 [19] M. K. Ali, F. Simba, Performance evaluation of 3G (UMTS Network) for E-Learning Video Streaming, IEEE AFRICON (Arusha, Tanzania), IEEE, 2021. https://doi.org/10.1109/AFRICON51333.2021.9570866 [20] H. L. Gao, The Impact of Quality of Experience of Chinese College Students on Internet-Based Resources English Learning, Future Internet 13 (7) no. 162 (2021). https://doi.org/10.3390/fi13070162 [21] I. Doumanis, D. Economou, L. Argyriou, Measuring and Comparing QoE of Hybrid VR Applications under Increased Network Load, 7th International Conference of the Immersive Learning Research Network (iLRN) (Eureka, CA, USA), IEEE, 2021. https://doi.org/10.23919/iLRN52045.2021.9459316 [22] R. D. Pinto, P. Monteiro, M. Melo, L. Cabral, M. Bessa, Does gamification in virtual reality improve second language learning?, International Conference on Graphics and Interaction (ICGI) (Porto, Portugal), IEEE, 2021. https://doi.org/10.1109/ICGI54032.2021.9655286 [23] D. Pal, V. Vanijja, S. Patra, Online Learning During COVID-19: Students‟ Perception of Multimedia Quality, 11th International Conference on Advances in Information Technology (IAIT) (Bangkok, Thailand), ACM, 2020. https://doi.org/10.1145/3406601.3406632 [24] E. Idrizi, S. Filiposka, V. Trajkovik, Character Traits in Online Education: Case Study, in: S.

Kalajdziski, N. Ackovska (Eds.) ICT Innovations 2018, Engineering and Life Sciences, ICT 2018, Communications in Computer and Information Science, 940, Springer, Cham, 2018, p. 247-258. https://doi.org/10.1007/978-3-030-00825-3_21 [25] X. Kong, Y. Liu, M. An, Study on the Quality of Experience Evaluation Metrics for Astronaut Virtual Training System, in: J. Chen, G. Fragomeni (Eds.) Virtual, Augmented and Mixed Reality: Interaction, Navigation, Visualization, Embodiment, and Simulation, VAMR 2018, Lecture Notes in Computer Science, 10909, Springer, Cham, 2018, p. 416-426. https://doi.org/10.1007/978-3-319-91581-4_31 [26] D. Davcev, G. Jakimovski, S. Scepanovic, Model of M-Learning by Multimedia Content Delivery from mCloud to Mobile Devices, in: M. Auer, D. Guralnick, I. Simonics (Eds.) Teaching and Learning in a Digital World, ICL 2017, Advances in Intelligent Systems and Computing, 715, Springer, Cham, 2018, p. 795-802. https://doi.org/10.1007/978-3-319-732107_92 [27] A.-N. Moldovan, I. Ghergulescu, C. H. Muntean, Analysis of Learner Interest, QoE and EEGBased Affective States in Multimedia Mobile Learning, 17th International Conference on Advanced Learning Technologies (ICALT) (Timisoara, Romania), IEEE, 2017, p. 398-402. https://doi.org/10.1109/ICALT.2017.93 [28] A. F. Fernandes, J. Cardoso, M. J. Marcelino, A Systematic Mapping Applied to MOOC‟s Study, in: Proceedings of the 7th International Conference on Computer Supported Education (CSEDU) (Lisbon, Portugal), Vol. 1, SciTePress, 2015, p. 444-449. https://doi.org/10.5220/0005483904440449 [29] A. Karadimce, D. Davcev, Collaborative cloud service model for delivering multimedia content in mCloud, 10th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom 2014) (Miami, USA), IEEE, 2015, p. 469-474. https://doi.org/10.4108/icst.collaboratecom.2014.257484 [30] A. A. Kist, L. Brodie, Quality of Service, Quality of Experience and Online Learning, 2012 Frontiers in Education Conference Proceedings (Seattle, WA, USA), IEEE, 2012. https://doi.org/10.1109/FIE.2012.6462223 [31] U.-D., Ehlers, J. R. Hilera, Special Issue on quality in e-learning, Journal of Computer Assisted Learning, 28 (1) (2012) p. 1-3. https://doi.org/10.1111/j.1365-2729.2011.00448.x

References

[2] U .-D. Ehlers , Quality in e-learning: the learner as a key quality assurance category , Vocational Training European Journal no. 29 ( 2003 ) p. 3 - 15 . https://www.cedefop.europa.eu/files/etv/Upload/Information_resources/Bookshop/349/29_en_ Eh lers .pdf

[3] ISO 8402 , Quality - Vocabulary, International Organisation for Standardisation, Geneva, 1986 .

[4] K. Brunnström , S. A. Beker , K. de Moor , A. Dooms , S. Egger et al., Qualinet White Paper on Definitions of Quality of Experience , Output from the 5th Qualinet meeting (Novi Sad) ( March 2013 ). https://hal.science/hal-00977812

[5] A. Parasuraman , V. A. Zeithaml , L. L. Berry , A Conceptual Model of Service Quality and Its Implications for Future Research , Journal of Marketing 49 ( 4 ) ( 1985 ) p. 41 - 50 . https://doi.org/10.1177/002224298504900403

[6] S. Firdous , R. Farooqi , Service Quality To E-Service Quality : A Paradigm Shift , in: Proceedings of the 9th International Conference on Industrial Engineering and Operations Management (Bangkok, Thailand), IEOM Society International , 2019 , p. 1656 - 1666 . http://www.ieomsociety.org/ieom2019/papers/404.pdf

[7] V. A. Zeithaml , A. Parasuraman , A. Malhotra , A conceptual framework for understanding eservice quality: implications for future research and managerial practice , Working Paper, Report No. 00-115 , MSI, Cambridge, MA, USA ( 2000 ). https://thearf-org-unifiedadmin. s3 .amazonaws.com/MSI/ 2020 /06/MSI_WP_ 00 - 115 .pdf

[8] M. R. M. Veeramanickam , P. Ramesh, Analysis on quality of learning in e-Learning platforms , Advances in Engineering Software 172 no. 103168 ( 2022 ), ISSN 0965- 9978 . https://doi.org/10.1016/j.advengsoft. 2022 .103168

[9] H. Arksey , L. O'Malley, Scoping studies: towards a methodological framework , International Journal of Social Research Methodology 8 ( 1 ) ( 2005 ) p. 19 - 32 . https://doi.org/10.1080/1364557032000119616

[10] F. Campbell , A. C. Tricco , Z. Munn et al., Mapping reviews, scoping reviews, and evidence and gap maps (EGMs): the same but different - the “Big Picture” review family , Systematic Reviews 12 no. 45 ( 2023 ). https://doi.org/10.1186/s13643-023-02178-5

[11] M. D. J. Peters , C. Marnie , H. Colquhoun et al., Scoping reviews: reinforcing and advancing the methodology and application , Systematic Reviews 10 no. 263 ( 2021 ). https://doi.org/10.1186/s13643-021-01821-3

[12] M. J. Page , J. E. McKenzie , P. M. Bossuyt et al., The PRISMA 2020 statement : an updated guideline for reporting systematic reviews , Systematic Reviews 10 no. 89 ( 2021 ). https://doi.org/10.1186/s13643-021-01626-4

[13] A. C. Tricco , E. Lillie , W. Zarin , K. K. O'Brien , H. Colquhoun , D. Levac , D. Moher , M. D. J. Peters , T. Horsley , L. Weeks , S. Hempel et al., PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation , Annals of Internal Medicine 169 ( 7 ) ( 2018 ) p. 467 - 473 . https://doi.org/10.7326/M18-0850

[14] I.-S. Comsa , A. Molnar , I. Tal, C. Imhof , P. Bergamin , G. -M. Muntean , C. H. Muntean , R. Trestian , Improved Quality of Online Education Using Prioritized Multi-Agent Reinforcement Learning for Video Traffic Scheduling , IEEE Transactions on Broadcasting 69 ( 2 ) ( 2023 ) p. 436 - 454 . https://doi.org/10.1109/TBC. 2023 .3246815

[15] M. Chakkaravarthy , G. S. Kumar , A Quality of Experience-based Recommender System for Elearning Resources , International Journal on Recent and Innovation Trends in Computing and Communication 11 ( 5s ) ( 2023 ). https://doi.org/10.17762/ijritcc.v11i5s. 6590

[16] A. D. Souchet , S. Philippe , A. Lévêque , F. Ober , L. Leroy, Short- and long-term learning of job interview with a serious game in virtual reality: influence of eyestrain, stereoscopy, and apparatus , Virtual Reality 26 ( 2 ) ( 2022 ) p. 583 - 600 . https://doi.org/10.1007/s10055-021-00548-9

[17] T. C. Thang , Y. Watanobe , R. U. Kiran and I. Paik , Towards QoE Management for PostPandemic Online Learning: Invited Paper, 14th International Conference on Knowledge and Systems Engineering (KSE) (Nha Trang, Vietnam) , IEEE, 2022 . https://doi.org/10.1109/KSE56063. 2022 .9953769

[18] T. Bieg , R. Schatz , S. Egger-Lampl , B. Roszipal , K. Kinzer , Better Experience, Better Performance? Results of a Study on VR Training Effectiveness in Healthcare, 14th