Defining Quality of Experience in e-Learning: a Literature Review Athanasia Kylafi1, Nikolaos Misirlis2 and Yiannis Nikolaidis1 1 University of Macedonia, 156 Egnatia Street, 546 36, Thessaloniki, Greece 2 HAN University of Applied Sciences, R31, 6824DG, Arnhem, The Netherlands Abstract Our research aims to investigate the “quality of experience” in education and particularly in the context of e-learning. Firstly, we study separately the concepts of “quality” and “experience”, as well as the underlying meaning of these terms. Secondly, through a scoping review among papers of the last decade, we study how “quality of experience”, as a whole, is defined in e-learning. Our purpose is to set a solid basis for the clarification of “quality of experience” in e-learning, considering that it is important to have a clear view of concepts before proceeding with a research related to them. As a result of our review, we reach useful answers to questions such as “What is quality of experience in e-learning?”, “Is quality of experience different from user experience?” gaining deeper understanding of the examined concepts. Keywords 1 quality, quality of experience, e-learning, literature review, scoping review 1. Introduction There is no doubt that e-learning, which is defined as “the learning supported by digital electronic tools and media” [1], especially after the Covid pandemic, has become a necessity in the educational sector globally, either it concerns school education, higher education or vocational education and training. Thus, it is reasonable that there is an increasing interest in optimizing e-learning procedures and tools in order to offer e-learning services of high quality. This interest derives not only from the educational organizations that wish to pioneer and to be competitive, but also from researchers and academics that constantly look for new ways to evaluate and improve e-learning environments. Therefore, it comes that the concept of quality is crucial when it relates to education and especially e- learning, mainly because the latter is delivered via digital technologies, which in fact mediate the educational process and transform the connections among the participants (teachers and students). In general, it is difficult to define quality in a unique way, because it is a subjective, relational and multidimensional concept. Actually, there is a long debate about its definition. Ehlers [2] distinguishes three components of this debate on quality, as shown in fig. 1. According to him, it is not easy to define quality because of a) different interpretations of quality, b) different players with different perspectives of quality, c) different forms of quality. However, generally speaking and according to Juran, quality can be defined as “the degree to which a product / service meets or exceeds expectations / requirements of customers” and “fitness for use”. According to Crosby, quality means “conformance to requirements”. According to ISO 8402 [3], quality is “the characteristics of a product or service that bear on its ability to satisfy stated or implied needs”. Finally, another definition of quality is given in the Qualinet White Paper [4]: “Quality is the outcome of an individual‟s comparison and judgment process. It includes perception, reflection about the perception, and the description of the outcome. In contrast to definitions which Proceedings for the 14th International Conference on e-Learning 2023, September 28-29, 2023, Belgrade, Serbia EMAIL: nancykylafi@uom.edu.gr (A. 1); nikolaos.misirlis@han.nl (A. 2); nikolai@uom.edu.gr (A. 3) ORCID: 0009-0004-2249-0618 (A. 1); 0000-0001-8815-2081 (A. 2); 0000-0003-4201-7786 (A. 3) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings 126 see quality as a „set of inherent characteristics‟, quality is considered in terms of the evaluated excellence or goodness, of the degree of need fulfillment, and in terms of a „quality event‟”. Figure 1: Dimensions of Quality [2] As can be seen from the above, quality may refer either to services or to products and according to this basic distinction it could be categorized into further types, e.g. “quality of conventional services” and “quality of e-services”. Following a conceptual path from quality to the central subject of our study, namely the quality of experience in e-learning, we present in what follows the state of the art knowledge regarding the related concepts. Specifically about services, Ehlers [2] mentions Donabedian‟s statement of the relational nature of quality: “quality is the degree of conformance between a performed service and the goals set for this service”. Similarly, according to Parasuraman et al. [5], service quality can be defined as “perceptions (that) result from a comparison of consumer expectations with actual service performance”. From this point of view, service quality is measured by calculating the difference between expectations and perceptions. With the advent of the internet to most aspects of human activity in modern societies, more and more conventional services are extended to electronic, namely e-services. The latter are web-based services, which are delivered through electronic media (e.g. computers, smartphones, tablets). Thus, the characteristics of services and e-services vary to a great extent and, therefore, the quality dimensions of services and e-services vary too [6]. One well-known definition of the quality of e- services is the following: “E-service quality is defined as the extent to which a website facilitates efficient and effective shopping, purchasing, and delivery” [7]. Of course, this definition does not actually reflect the wide range of e-services that exist nowadays (e.g. e-learning, e-banking, e- government, etc.), but we evaluate positively the fact that it focuses on two important elements, efficiency and effectiveness. According to the above definition, e-service quality is the degree of efficiency and effectiveness of services delivered to users on the internet. Taking one step further, e-learning - as an educational service offered through digital resources on the internet - is undeniably a form of e-service. Quality of e-learning is defined as “the philosophy of achieving excellence in all aspects of education through e-learning technologies” [8]. Exploiting the previously described definition of quality, we consider the quality of e-learning as the degree to which an e-learning course meets or exceeds the expectations / requirements of learners, providing them with educational experiences. Quality of e-learning may refer to the educational technology, the educational content, the teaching-learning process, the student assessment, the administrative support etc. Through e-learning environments, the participants acquire various educational experiences. An experience is defined as “an individual‟s stream of perception and interpretation of one or multiple events” [4]. Experiences in e-learning may refer to teachers, to students or both. Moreover, 127 experiences are connected to the quality that characterizes every e-learning environment (and every educational environment generally), meaning that there may be good or bad experiences. Ιn other words, there may be high or low quality of experience. The latter is defined as follows: “Quality of Experience (QoE) is the degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his or her expectations with respect to the utility and/or enjoyment of the application or service in the light of the user‟s personality and current state” [4]. Discussing quality of experience in the context of e-learning, quality is an important factor for the future of e-learning [2]. On the other hand, experience of the learners is a crucial factor for the overall success of an e-learning course, because it directly affects their learning outcome(s). Through their experience, the learners fulfill their expectations and gain satisfaction and knowledge. So, from the learners‟ perspective, the concept of quality is perceived as their experience, based on their expectations and satisfaction. The purpose of this paper is to explore the literature of the last decade, in order to determine and clarify the concept of QoE in e-learning. We also aim to gain deeper understanding of the concept, by searching for answers to research questions such as: Research Question 1: What is QoE in e-learning? and Research Question 2: Is QoE different from user experience? To serve our purpose we conduct a scoping literature review, as this type of review is the most appropriate to gather knowledge, clarify concepts and identify gaps on a particular topic. The findings of this review hopefully provide a clear understanding of QoE to researchers and other stakeholders in the field of e-learning (e.g. academics, practitioners, organizations). They also indicate the recent trends of research on QoE in e-learning. The paper is organized as follows. In Section 2 we present the methodology of the scoping literature review. In Section 3 we go on with the results of our review and in Section 4 we discuss the findings. In Section 5 we provide our conclusions. 2. Literature review methodology A literature review is a form of research for knowledge synthesis. There are many literature review types, such as narrative review, systematic review, scoping review, mapping review etc. [9, 10]. For the purpose of this paper, we consider the scoping review as the most appropriate. Scoping reviews belong to the category of rapid reviews [9]. A scoping review is suitable for exploring, identifying, mapping, reporting or discussing characteristics or concepts across a breadth of evidence sources [11]. It is used for clarifying concepts, identifying knowledge gaps and providing an overview of the existing knowledge. The present study is guided by the PRISMA 2020 statement [12] and the PRISMA extension for Scoping Reviews Checklist [10, 13], and was conducted in 5 stages [9]: Stage 1: identifying the research question(s) Stage 2: identifying relevant publications / studies Stage 3: selection of publications / studies Stage 4: charting the data Stage 5: collating, summarizing and reporting the results The purpose and the research questions of our research (Stage 1) are mentioned at the end of Section 1. Next, (as per Stage 2) we selected the database of Scopus and determined the search criteria to trace the relevant literature: boolean search of the terms “quality of experience” AND “e- learning” in title-abstract-keywords, filtered with the year range “2014-2023”. We continued with the inclusion or exclusion of the publications that had arisen (Stage 3). Publications that either were irrelevant to our research or did not provide any definition of QoE were excluded. We then studied the selected publications and extracted the useful information (Stage 4). Finally we summarized the findings (Stage 5). Our search in Scopus initially brought 122 records as a result. After the removal of 4 records due to duplication, retraction and other reasons, 118 records remained. At the first screening 67 records were excluded as irrelevant. We proceeded to the second screening for retrieval of the reports related 128 to our purpose, where 45 reports were retrieved. We assessed thoroughly all 45 retrieved reports for eligibility. 29 of them were excluded with reasons (no definitions detected). We finally included 16 studies in the review. 3. Results In Table 1 we provide the results of our review in chronological order. For each reviewed study we present the definitions/meanings of QoE mentioned in it. In addition, we present the research topics in the field of e-learning that the study relates to. The related research topics are of great interest because they offer more detail on the meaning of QoE. Moreover, we note the type of publication. Table 1 Definitions of QoE in e-learning and related research topics Author(s) & Type of Related research Definition/meaning of "quality of experience" Year publication topics in "e-learning" Comsa, Molnar, journal video-based online QoE is perceived by learners experiencing heterogeneous Tal, Imhof, article learning, video traffic (which means with different traffic loads) video services in Bergamin, prioritization with e-Learning. Muntean, Machine Learning Muntean, Trestian 2023 [14] Chakkaravarthy, journal web service QoE is a user-perceived metric representing service Kumar article recommendation system satisfaction. for course selection 2023 [15] Souchet, Philippe, journal virtual reality, serious QoE is the subjective experience by users, based on usual Lévêque, Ober, article games for learning variables considered in VR such as Presence and Flow. Leroy 2022 [16] Thang, conference online learning, video QoE is the extent users are satisfied with contents provided Watanobe, Kiran, paper streaming by applications or services. Paik 2022 [17] Bieg, Schatz, conference virtual reality, digital QoE is one of the six measures for experience, along with Egger-Lampl, paper training hedonic user experience, pragmatic user experience, VR- Roszipal, Kinzer induced symptoms, sense of presence, overall satisfaction. 2022 [18] Ali, Simba conference video streaming QoE is the overall acceptability of an application or service, paper as perceived subjectively by the end-user. 2021 [19] Gao journal internet-based resources, QoE is the user’s point of view of the whole performance of article English learning a system. QoE is the degree to which users of an application 2021 [20] or service are happy or annoyed. It is based on the user’s personality, current state, and expectations for the utility and/or enjoyment of the application service. QoE ultimately leads to the acceptability of the system, application, or service. QoE is the construct concerning how well a system provides functions for a user to complete particular tasks. 129 Doumanis, conference hybrid Virtual Reality, QoE is a measure of how users experience a hybrid VR Economou, paper 360-degree video application. Argyriou streaming 2021 [21] Pinto, Monteiro, conference Virtual Reality, QoE is related to the evaluation of each participant towards Melo, Cabral, paper gamification, second the application. Bessa language learning 2021 [22] Pal, Vanijja, Patra conference online learning, QoE is defined as “the overall acceptability of an application paper videoconferencing or service, as perceived subjectively by the end user”. From 2020 [23] applications, multimedia the definition itself it is clear that QoE is a superset of QoS quality that includes all possible complete end-to-end system effects, along with the user expectations. Idrizi, Filiposka, conference online education, Qualities of Experience are the measurements of how Trajkovik paper learning styles students felt and were satisfied during the online classes. 2018 [24] Kong, Liu, An conference Virtual Reality, virtual QoE is “the overall acceptability of an application or service, paper training system as perceived subjectively by an end-user”. “QoE is the 2018 [25] degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his or her expectations with respect to the utility and/or enjoyment of the application or service in the light of the user’s personality and current state”. QoE is a multidimensional quality that can be decomposed in a set of perceptual attributes called features. Davcev, conference mobile learning, QoE is used as an overall acceptability of an application or Jakimovski, paper multimedia content, service, as perceived subjectively by the end-user and Scepanovic mobile cloud computing represents a multidimensional subjective concept that is not easy to evaluate. 2018 [26] Moldovan, conference mobile learning, QoE is a main factor contributing to user’s engagement with Ghergulescu, paper multimedia multimedia services. Muntean 2017 [27] Fernandes, conference MOOC QoE is used to describe how students evaluate a service. Cardoso, paper QoE "is how a user perceives the usability of a service when Marcelino in use". QoE is "the degree to which a system meets the tacit and explicit expectations of the user for the 2015 [28] experience". Karadimce, conference mobile learning, QoE is a metric that quantifies the multifaceted, Davcev paper multimedia content, multidimensional factors that influence the perceived collaborative cloud quality. QoE is defined as the degree of delight or 2015 [29] service annoyance of a person whose experience involves an application, service, or system. QoE is no longer an expression for user satisfaction of using a service, but it is a degree of delight or annoyance, which is a more dynamic measure of personal experience. 4. Discussion of the findings 4.1. Basic characteristics of the studies 10 articles out of 16 (62.5%) are published between 2020 and 2023 and the rest (37.5%) from 2015 to 2019. The majority of them (75%) are conference papers, which implies that the particular research area is still unexplored, thus very promising. 130 The research topics of selected studies vary depending on the e-learning area, the applied technologies and the type of educational content. Topics that refer to the field or a sub-field of e- learning are the following:  video-based online learning [14]  online learning [17, 23, 24]  digital training [18]  language learning [20, 22]  virtual training [25]  mobile learning [26, 27, 29]  mooc [28] Topics that refer to the technology or the content of e-learning are:  video streaming [14, 17, 19, 21]  machine learning [14]  web service for course selection [15]  virtual reality [16, 18, 21, 22, 25]  gamification [16, 22]  internet-based resources [20]  videoconferencing [23]  multimedia [23, 26, 27, 29]  learning styles [24]  cloud computing [26, 29] Finally, another characteristic is that all 16 studies approach e-learning from the learners‟ perspective. 4.2. Ηow is QoE defined in e-learning? 4.2.1. What is QoE in e-learning? According to the definitions presented in Table 1, QoE is considered as:  a measure of satisfaction [15, 17, 24]  a measure of experience [16, 18, 21, 29]  the acceptability of an application or service [19, 20, 23, 25, 26]  the delight or annoyance from an application or service [20, 25, 29]  a subjective perception by the user [14, 15, 16, 20, 22, 26, 28]  a multidimensional concept [25, 26, 29]  a factor of user's engagement [27] From the above conceptual variety, we confirm that QoE in e-learning is multidimensional and subjective, thus it is defined depending on the dimensions of the research interest. However, very often the definitions overlap and have many elements in common. We infer that experience and satisfaction are related concepts, both user-centric and user-perceived, and their measurement determines QoE. We also infer that most of the studies adopt mainly two definitions of QoE, either directly or with some paraphrase, which are a) the definition by the ITU (i.e., “the overall acceptability of an application or service, as perceived subjectively by an end-user”) and b) the definition by the Qualinet White Paper (i.e., “the degree of delight or annoyance of the user of an application or service”). At this point it is noteworthy to mention the statement of Karadimce and Davcev [29] that “QoE is no longer an expression for user satisfaction of using a service, but it is a degree of delight or annoyance, which is a more dynamic measure of personal experience”. This indicates the subjective character of QoE. According to the above definitions, QoE is affected by factors such as:  the type of device and the quality of the Internet connectivity [14],  presence, flow and symptoms in VR [16, 21]  quality variation, stalling, and initial delay in a streaming session [17] 131  user expectations and context [19]  user‟s personality, current state, and expectations for the utility and/or enjoyment of the application service [20] Additionally, as there is no single definition of QoE, it is difficult to evaluate the particular concept [26] or, at least, there is no single measure of it. The measures of QoE vary as dependable on the various factors that are taken into account in every research. 4.2.2. Is QoE different from user experience? The answer to this question is that QoE and user experience (UX) are very similar concepts, since both of them refer to users‟ experience with digital technologies. However, they are not exactly the same. We infer that UX is the experience perceived by the user, while QoE is a measure for UX [16, 18, 21, 29]. More specifically, Bieg et al. [18] mention an interesting description of QoE, “as one of the six measures for experience, along with hedonic user experience, pragmatic user experience, VR- induced symptoms, sense of presence, overall satisfaction”. From this statement we conclude that QoE and UX are not the same. 5. Conclusions Concluding our review, we find out that there is not a specific, widely accepted definition of QoE in e-learning. The adopted definitions in the field of e-learning are also used in other fields, as QoE is defined for all e-services. Moreover, most studies in this review approach e-learning from a technical view, mainly through computer science, thus QoE is mainly related with technical aspects of the e- learning systems. QoE is a concept often used in a technical context [30]. We agree with Kist and Brodie [30], that the learning experience differs significantly from the general consumer experience. We also agree with Ehlers and Hilera [31], according to whom “quality in the field of e-learning is an especially diverse field, because it brings together the field of education, technology, and economy in order to contribute a) to societal development, b) to innovate formal, non formal, and informal learning opportunities, and c) empower learners as citizens to take part in our emerging learning and information societies”. We believe that QoE in e-learning is a concept that refers to learning processes, through learning environments, where learning objectives exist and learning outcomes happen. Learners are more than consumers of a service, they acquire knowledge and they develop skills and attitudes in a long-term horizon, apart from their short-term experiences. So, QoE in e- learning needs to be defined more specifically and this is certainly a research gap which should be studied in the near future. This review also reveals that the research focus concerns only the learners‟ perspective. This was expected, as learners are the main end-users in e-learning and actually the user-perceived experience refers to the learners‟ experience. However, this fact can be seen as a research gap, because teachers are highly involved in e-learning too (except self-learning cases), and their experience may have a significance for the overall QoE, e.g. in higher education. This review also shows that QoE is not easy to evaluate and this can be considered as another research gap too. This review has some limitations regarding the generalization of the findings. Due to the fact that it was conducted in only one database, Scopus, more research is needed, expanded in more databases in order to come to general results. The time span is also a limitation of this review. 6. References [1] K. S. Basak, M. Wotto, P. Bélanger, E-learning, M-learning and D-learning: Conceptual definition and comparative analysis, E-Learning and Digital Media 15 (4) (2018) p. 191-216. https://doi.org/10.1177/2042753018785180 132 [2] U.-D. Ehlers, Quality in e-learning: the learner as a key quality assurance category, Vocational Training European Journal no. 29 (2003) p. 3-15. https://www.cedefop.europa.eu/files/etv/Upload/Information_resources/Bookshop/349/29_en_Eh lers.pdf [3] ISO 8402, Quality - Vocabulary, International Organisation for Standardisation, Geneva, 1986. [4] K. Brunnström, S. A. Beker, K. de Moor, A. Dooms, S. Egger et al., Qualinet White Paper on Definitions of Quality of Experience, Output from the 5th Qualinet meeting (Novi Sad) (March 2013). https://hal.science/hal-00977812 [5] A. Parasuraman, V. A. Zeithaml, L. L. Berry, A Conceptual Model of Service Quality and Its Implications for Future Research, Journal of Marketing 49 (4) (1985) p. 41-50. https://doi.org/10.1177/002224298504900403 [6] S. Firdous, R. Farooqi, Service Quality To E-Service Quality: A Paradigm Shift, in: Proceedings of the 9th International Conference on Industrial Engineering and Operations Management (Bangkok, Thailand), IEOM Society International, 2019, p. 1656-1666. http://www.ieomsociety.org/ieom2019/papers/404.pdf [7] V. A. Zeithaml, A. Parasuraman, A. Malhotra, A conceptual framework for understanding e- service quality: implications for future research and managerial practice, Working Paper, Report No. 00-115, MSI, Cambridge, MA, USA (2000). https://thearf-org-unified- admin.s3.amazonaws.com/MSI/2020/06/MSI_WP_00-115.pdf [8] M. R. M. Veeramanickam, P. Ramesh, Analysis on quality of learning in e-Learning platforms, Advances in Engineering Software 172 no. 103168 (2022), ISSN 0965-9978. https://doi.org/10.1016/j.advengsoft.2022.103168 [9] H. Arksey, L. O'Malley, Scoping studies: towards a methodological framework, International Journal of Social Research Methodology 8 (1) (2005) p. 19-32. https://doi.org/10.1080/1364557032000119616 [10] F. Campbell, A. C. Tricco, Z. Munn et al., Mapping reviews, scoping reviews, and evidence and gap maps (EGMs): the same but different - the “Big Picture” review family, Systematic Reviews 12 no. 45 (2023). https://doi.org/10.1186/s13643-023-02178-5 [11] M. D. J. Peters, C. Marnie, H. Colquhoun et al., Scoping reviews: reinforcing and advancing the methodology and application, Systematic Reviews 10 no. 263 (2021). https://doi.org/10.1186/s13643-021-01821-3 [12] M. J. Page, J. E. McKenzie, P. M. Bossuyt et al., The PRISMA 2020 statement: an updated guideline for reporting systematic reviews, Systematic Reviews 10 no. 89 (2021). https://doi.org/10.1186/s13643-021-01626-4 [13] A. C. Tricco, E. Lillie, W. Zarin, K. K. O'Brien, H. Colquhoun, D. Levac, D. Moher, M. D. J. Peters, T. Horsley, L. Weeks, S. Hempel et al., PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation, Annals of Internal Medicine 169 (7) (2018) p. 467- 473. https://doi.org/10.7326/M18-0850 [14] I.-S. Comsa, A. Molnar, I. Tal, C. Imhof, P. Bergamin, G.-M. Muntean, C. H. Muntean, R. Trestian, Improved Quality of Online Education Using Prioritized Multi-Agent Reinforcement Learning for Video Traffic Scheduling, IEEE Transactions on Broadcasting 69 (2) (2023) p. 436- 454. https://doi.org/10.1109/TBC.2023.3246815 [15] M. Chakkaravarthy, G. S. Kumar, A Quality of Experience-based Recommender System for E- learning Resources, International Journal on Recent and Innovation Trends in Computing and Communication 11 (5s) (2023). https://doi.org/10.17762/ijritcc.v11i5s.6590 [16] A. D. Souchet, S. Philippe, A. Lévêque, F. Ober, L. Leroy, Short- and long-term learning of job interview with a serious game in virtual reality: influence of eyestrain, stereoscopy, and apparatus, Virtual Reality 26 (2) (2022) p. 583-600. https://doi.org/10.1007/s10055-021-00548-9 [17] T. C. Thang, Y. Watanobe, R. U. Kiran and I. Paik, Towards QoE Management for Post- Pandemic Online Learning: Invited Paper, 14th International Conference on Knowledge and Systems Engineering (KSE) (Nha Trang, Vietnam), IEEE, 2022. https://doi.org/10.1109/KSE56063.2022.9953769 [18] T. Bieg, R. Schatz, S. Egger-Lampl, B. Roszipal, K. Kinzer, Better Experience, Better Performance? Results of a Study on VR Training Effectiveness in Healthcare, 14th 133 International Conference on Quality of Multimedia Experience (QoMEX) (Lippstadt, Germany), IEEE, 2022. https://doi.org/10.1109/QoMEX55416.2022.9900889 [19] M. K. Ali, F. Simba, Performance evaluation of 3G (UMTS Network) for E-Learning Video Streaming, IEEE AFRICON (Arusha, Tanzania), IEEE, 2021. https://doi.org/10.1109/AFRICON51333.2021.9570866 [20] H. L. Gao, The Impact of Quality of Experience of Chinese College Students on Internet-Based Resources English Learning, Future Internet 13 (7) no. 162 (2021). https://doi.org/10.3390/fi13070162 [21] I. Doumanis, D. Economou, L. Argyriou, Measuring and Comparing QoE of Hybrid VR Applications under Increased Network Load, 7th International Conference of the Immersive Learning Research Network (iLRN) (Eureka, CA, USA), IEEE, 2021. https://doi.org/10.23919/iLRN52045.2021.9459316 [22] R. D. Pinto, P. Monteiro, M. Melo, L. Cabral, M. Bessa, Does gamification in virtual reality improve second language learning?, International Conference on Graphics and Interaction (ICGI) (Porto, Portugal), IEEE, 2021. https://doi.org/10.1109/ICGI54032.2021.9655286 [23] D. Pal, V. Vanijja, S. Patra, Online Learning During COVID-19: Students‟ Perception of Multimedia Quality, 11th International Conference on Advances in Information Technology (IAIT) (Bangkok, Thailand), ACM, 2020. https://doi.org/10.1145/3406601.3406632 [24] E. Idrizi, S. Filiposka, V. Trajkovik, Character Traits in Online Education: Case Study, in: S. Kalajdziski, N. Ackovska (Eds.) ICT Innovations 2018, Engineering and Life Sciences, ICT 2018, Communications in Computer and Information Science, 940, Springer, Cham, 2018, p. 247-258. https://doi.org/10.1007/978-3-030-00825-3_21 [25] X. Kong, Y. Liu, M. An, Study on the Quality of Experience Evaluation Metrics for Astronaut Virtual Training System, in: J. Chen, G. Fragomeni (Eds.) Virtual, Augmented and Mixed Reality: Interaction, Navigation, Visualization, Embodiment, and Simulation, VAMR 2018, Lecture Notes in Computer Science, 10909, Springer, Cham, 2018, p. 416-426. https://doi.org/10.1007/978-3-319-91581-4_31 [26] D. Davcev, G. Jakimovski, S. Scepanovic, Model of M-Learning by Multimedia Content Delivery from mCloud to Mobile Devices, in: M. Auer, D. Guralnick, I. Simonics (Eds.) Teaching and Learning in a Digital World, ICL 2017, Advances in Intelligent Systems and Computing, 715, Springer, Cham, 2018, p. 795-802. https://doi.org/10.1007/978-3-319-73210- 7_92 [27] A.-N. Moldovan, I. Ghergulescu, C. H. Muntean, Analysis of Learner Interest, QoE and EEG- Based Affective States in Multimedia Mobile Learning, 17th International Conference on Advanced Learning Technologies (ICALT) (Timisoara, Romania), IEEE, 2017, p. 398-402. https://doi.org/10.1109/ICALT.2017.93 [28] A. F. Fernandes, J. Cardoso, M. J. Marcelino, A Systematic Mapping Applied to MOOC‟s Study, in: Proceedings of the 7th International Conference on Computer Supported Education (CSEDU) (Lisbon, Portugal), Vol. 1, SciTePress, 2015, p. 444-449. https://doi.org/10.5220/0005483904440449 [29] A. Karadimce, D. Davcev, Collaborative cloud service model for delivering multimedia content in mCloud, 10th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom 2014) (Miami, USA), IEEE, 2015, p. 469-474. https://doi.org/10.4108/icst.collaboratecom.2014.257484 [30] A. A. Kist, L. Brodie, Quality of Service, Quality of Experience and Online Learning, 2012 Frontiers in Education Conference Proceedings (Seattle, WA, USA), IEEE, 2012. https://doi.org/10.1109/FIE.2012.6462223 [31] U.-D., Ehlers, J. R. Hilera, Special Issue on quality in e-learning, Journal of Computer Assisted Learning, 28 (1) (2012) p. 1-3. https://doi.org/10.1111/j.1365-2729.2011.00448.x 134