=Paper=
{{Paper
|id=Vol-2217/paper-iva
|storemode=property
|title=Usability and Quality Parameters for E-Learning Environments and Systems
|pdfUrl=https://ceur-ws.org/Vol-2217/paper-iva.pdf
|volume=Vol-2217
|authors=Mirjana Ivanović,Aleksandra Klašnja Milićević,Maria Ganzha,Amelia Bădică,Marcin Paprzycki,Costin Bădică
|dblpUrl=https://dblp.org/rec/conf/sqamia/IvanovicKGBPB18
}}
==Usability and Quality Parameters for E-Learning Environments and Systems==
https://ceur-ws.org/Vol-2217/paper-iva.pdf
6
Usability and Quality Parameters for E-Learning
Environments and Systems
MIRJANA IVANOVIĆ, University of Novi Sad
ALEKSANDRA KLAŠNJA MILICEVIĆ, University of Novi Sad
MARIA GANZHA, Polish Academy of Sciences and Warsaw University of Technology
AMELIA BĂDICĂ, University of Craiova
MARCIN PAPRZYCKI, Polish Academy of Sciences and Warsaw Management Academy
COSTIN BĂDICĂ, University of Craiova
An important issue of wide-range of e-learning environments and systems, for different educational purposes and domains, is
their usability and quality. In this paper we present several aspects of measuring and comparing usability and quality of wide
range of e-learning systems. Here, we focus mainly on communication interfaces of such systems. Our approach, and
particular case study on usability and quality of wide range of e-learning systems, is presented.
1. INTRODUCTION
Use of technology in educational environments that started several decades ago, is an inevitable
force of changing traditional teaching methodologies for every society. As time goes by, wide range of
studies, activities, forums and various discussions on this topic have been published. Essential parts
of these comprehensive efforts are devoted to problems like: usability and designs, cultural issues
and learning cultures, influences and effects of technology enhanced learning on key learning
activities: e-teaching, e-learning and e-assessment [Ogunbase 2016]. An important issue, connected
to development and use of wide-range of e-learning environments and platforms (like: Learning
Management Systems (LMS), educational tools and software for different educational purposes and
domains), is usability of such systems. Further we can say that usability is not a quality existing in
an absolute sense, but it can be seen as a general quality of the “appropriateness to a purpose” of
some particular artefact. From the point of view of relatively high-level assessment of usefulness of
e-learning environments and platforms, usability testing is an extremely important instrument.
Usability testing represents a general technique to evaluate the effectiveness, user-friendliness,
comfort and easiness of a user-centered design of different computer systems including e-learning
systems. Since usefulness and usability of the system highly depends on success of user in
completing appropriate tasks, usability testing plays crucial role. Users of an interactive system
(including e-learning one) will select/decide to use user friendly versions so the failure or success of a
system will affect its productivity. With rapid development of technologies and modern interactive
systems and their influences on humans’ everyday activities, concept of usability is itself changing
and, in this context, it should follow specific measures. So, measures of usability must be dependent
________________________________
Authors address: Mirjana Ivanovic, Faculty of Sciences, Trg Dositeja Obradovica 4, 21000 Novi Sad, Serbia; e-mail:
mira@dmi.uns.ac.rs; Aleksandra Klasnja Milicevic, Faculty of Sciences, Trg Dositeja Obradovica 4, 21000 Novi Sad, Serbia; e-
mail: akm@dmi.uns.ac.rs; Maria Ganzha, ul. Newelska 6, 01-447Warsaw, Poland; e-mail: maria.ganzha@ibspan.waw.pl;
Amelia Badica, Faculty of economics and business administration 13, A.I. Cuza Street, Craiova, 200585, Romania; e-
mail:ameliabd@yahoo.com; Marcin Paprzycki, ul. Newelska 6, 01-447 Warsaw, Poland; e-mail: paprzyck@ibspan.waw.pl;
Costin Badica, Department of computers and information technology, Blvd. Decebal nr. 107, RO-200440, Craiova, Romania; e-
mail: cbadica@software.ucv.ro;
Copyright©by the paper’s authors. Copying permitted only for private and academic purposes
In: Z. Budimac (ed.): Proceedings of the SQAMIA 2018: 7th Workshop of Software Quality, Analysis, Monitoring,
Improvement,and Applications, Novi Sad, Serbia, 27-30.8.2018, Also published online by CEUR Workshop Proceedings
(http://ceur-ws.org,ISSN 1613-0073)
6:2 • Mirjana Ivanović et al
on the way how usability is defined. Nevertheless, there exist some general classes of usability
measures. Specific standard ISO 9241-11 prescribes the following key elements (classes) of usability
measures:
- “effectiveness – the ability of users to complete tasks using the system, and the quality of the
output of those tasks,
- efficiency – the level of resource consumed in performing tasks,
- satisfaction – users’ subjective reactions to using the system.”
Of course, these general descriptions of classes can and must be fine-grained to offer more precise
measures and metrics, especially depending on context-specificity, area and domain of use of system.
As a consequence of the above mentioned, it is rather difficult to make comparisons of usability and
usability measures across different systems from different application domains. Another important
characteristic of e-learning systems, products and services is quality. Apart from existence of
different definitions it is usually defined as “fitness for purpose” related to the needs of the user of
such system or product. The results presented in [Bari and Djouab 2014] that were collected and
obtained from web–based survey (that included 450 participants) about the quality of e-learning
products and services, has shown interesting findings. In fact, the main conclusion was that quality
plays a key role in the success of e-learning. The most important conclusions and observations are
enlisted below [Bari and Djouab 2014]:
- “Learners must play a key part in determining the quality of e-learning services: a learner
orientation is imperative in the area of quality;
- Culture of quality in education and training: quality development must become a core process for
educational organizations;
- Quality must play a key role in education policy;
- Quality development as the norm in the educational landscape;
- Quality services might be created;
- Open quality standards must be widely implemented”.
Our study is mainly oriented towards usability issues in particular kind of interactive systems i.e.
e-learning environments, educational software and tools. So, we will concentrate on usability
questionnaires, as predominant instruments, widely used for evaluating usability and quality of
interactive systems. The main advantage regarding usability analysis and research is that a central
and essential effect and outcome of usability questionnaire is feedback from the end user. Usability
questionnaires usually clearly meet the requirements of inexpensiveness, they are easily applicable
and effective to administer and to score. In such a way, it is possible to collect significant amount of
data, representing reliable source to check if quantitative usability aims and issues have been met.
Positive aspects and effects of usability questionnaires are essentially important in e-learning
context. Inexpensive, rather short and easy to deploy usability evaluation method is needed for e-
learning practitioners and researchers [Zaharias 2004]. They usually prefer to select and use some
among already validated and well-known satisfaction and usability questionnaires. As well, some
authors try to propose and develop their own, new, and customized versions and forms of
questionnaires. From point of view of our study, and previous activities and experiences, such
approach is of special interest in area of technology enhanced learning [Ivanovic et al. 2017] [Putnik
et al. 2017] [Xinogalos et al. 2018].
In this paper we will present several aspects of measuring and comparing usability and quality of
wide range of e-learning systems, concerning Human-Computer Interaction (HCI) and
communication interfaces in such systems. Our approach is rather unique and consists of following
ideas and issues:
- Within the course “Educational Software” we perform specific exercise and ask the students to
take a role of teachers
Usability and Quality Parameters for E-Learning Environments and Systems • 6:3
- Students have to use and compare usability and quality of 3 educational systems (Moodle – as
widely used, ASQ-special in-house system developed at Lugano University, and the third one
they should freely select)
- Adequate results of comparison of these three systems, as a consequence of application of
particular questionnaire are reported in form of essay
- Finally we performed statistical analysis and obtained useful experiences and lessons learned.
Our initial experiments, case-study and some experiences will be emphasized in more details in
the rest of the paper. The rest of the paper is organized as follows. Related work is presented in
Section 2. Section 3 discusses some usability and quality parameters of HCI aspects in e-learning
systems. Section 4 offers State-of-the-Art analysis for usability of e-learning environments and
LMSs. Our approach and particular case study on usability and quality of e-learning systems is
presented in Section 5. Some concluding remarks are given in the last section.
2. RELATED WORK
As mentioned in the Introduction, according to ISO 9241 standard, usability is “The extent to which
a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and
satisfaction in a specified context of use”. Some authors [Alelaiwi 2015] expand this definition by
adding other characteristics as: memorability, learnability, error tolerance, and so on. Human-
Computer Interaction that is essential part of e-learning environments and systems is tightly
connected to usability. According to IEEE Std.610.12-1990 [Al-Khalifa 2010] usability is defined as
"the ease with which a user can learn to operate, prepare inputs for, and interpret outputs of a
system or component". E-learning environments and systems provide significant contribution to the
efficiencies and effectiveness of the educational processes. In last decade such systems are becoming
more and more complex and it is necessary to be designed for ease use and constrained by the
human’s perceptual and cognitive abilities. “The better human computer interaction that offers the
LMSs users, the easier of use and greater satisfaction users will have with in systems or tools they
involved.” [Onacan and Erturk 2016] Usability as an important factor in improving the learning
experience and sensible design of Human-Computer Interaction is one of the key issues in the design
and development of e-learning environments.
Our study is concentrated on comparison of different types of e-learning systems, including LMSs
but also some web-based systems and generally educational environments. Therefore, in this section
we will present several other researchers’ studies that evaluate e-learning systems from the point of
view of usability. Usability testing can be seen as observation of typical users “performing tasks with
a product, conducted for the purpose of determining what changes need to be made to the content,
presentation or user interface for that product“ [Alelaiwi and Hossain 2015]. Furthermore, usability
evaluation is oriented towards effectiveness of e-learning systems and environments, while usability
evaluation encompasses assessments of how well such technology supported tools and systems are
working for users. [Al-Khalifa et al. 2010] in their study concentrated on particular e-learning
systems – LMSs. They concluded that LMSs can benefit from usability research especially
concerning satisfaction of LMSs users. In [Melton 2006] a preliminary study is reported, where
usability of some simple Moodle services is considered by asking users to complete selected tasks.
Comparative usability study on four LMSs (two open source and two commercial) is presented in
[Inversini et al. 2006]. Authors use particular technique MiLE+ that balances between task-driven
techniques and heuristic evaluation. Heuristics evaluation of usability has also been conducted in
[Martin, 2008] where three main e-learning open source platforms are considered: Moodle, dotLRN,
and Sakai. In the paper [Al-Khalifa, 2010] authors employed “user satisfaction surveys” to measure
the usability of particular LMS - JUSUR. A little bit different approach is presented in [Cakmak
2011]. It is concentrated on usability level of an LMS web site. They developed special usability
6:4 • Mirjana Ivanović et al
scale called “User perception-based web site usability scale” that consists of four factors with 25
items. Agariya and Singh’s [Agariya and Singh 2012] proposed (in Indian context) a valid and
reliable e-learning quality measurement scales, from both perspectives: learner and faculty.
Thuseethan, on the other hand, gave an overview of LMSs, i.e. evaluation of the usability of LMSs,
using pre-defined usability standards has been performed in Sri Lanka universities [Thuseethan et
al. 2014]. An overview of used LMSs is given together with the measure of their effectiveness.
Authors of paper [Masood 2015] performed usability testing in order to identify the effectiveness of
LMS - eLearn@USM. They used TAM - Technology Acceptance Model in order to study the ease of
use and usefulness of eLearn@USM. Study was based on students’ participation in the forum
discussion, blogs, and messaging the system. In fact, TAM is a particular information systems
theory. It models how users accept and use a new technology. The expectation is that numerous
factors influence user’s decision how and when they will use a new technology. TAM has been
effective in consideration and assessment of lot of types of systems: e-learning, LMS, web portals,
and so on. This approach attracted attention of research community and several updates have been
proposed, in the meantime. On the other hand, some alternative models appeared, like HMSAM
[Fathema et al. 2015]. As TAM was not ideal to explain adoption of purely hedonic systems (like:
learning for pleasure and entertaining, online games/shopping, gamified systems, virtual worlds,
social networking and similar) HMSAM was proposed as hedonic-motivation system adoption model.
Moreover, HMSAM could be particularly useful in understanding gamification elements of systems
use.
Comparing abovementioned studies and research results, and to the best of our knowledge we can
say that our approach is rather unique. So it is not easy to compare other researchers’ approaches as
they are not similar to ours. While other authors either compare several e-learning systems or some
others ask students to assess one particular system, we apply rather complex approach: we asked
our students to take a role of teachers; then students have to compare usability and quality of 3
educational systems (Moodle and ASQ fixed in advance, and the third one they should freely
selected); after that they have to prepare essays and present results of comparison of these three
systems applying particular usability questionnaire. In this way we managed to collect results of
comparison of more than 30 different educational systems with Moodle and ASQ. Statistical analysis
of these findings provides good insight into the usability and quality, important factor for e-learning
and can be used by all e-learning stakeholders.
3. USABILITY AND QUALITY PARAMETERS OF HCI ASPECTS IN E-LEARNING SYSTEMS
Human-Computer Interaction is crucial part and component of any seriously designed interactive
systems. It is especially important, and could be even motivational, in e-learning environments and
systems. On the other hand, developers of such systems and teachers, who consider using them in
educational processes and activities (as preparation of educational materials), are interested in high
standard of usability and quality of such systems. In the rest of this section we will consider several
aspects of usability and quality and ways of their measuring in domain of e-learning and e-learning
environments and systems.
3.1. Cultural aspects of Usability and Quality Parameters in e-learning Environments
Essential educational processes (e-teaching, e-learning, e-assessment) still have not been that easy
to implement, as they are influenced by different complex parameters. Usually design of such
systems and especially usability and quality of human-system interaction are influenced by the
cultures of both key stakeholders: the users and designers. Interesting considerations on this topic
are presented in the thesis [Ogunbase 2016]. Author considered Web-based learning environments
(WBLE) and put attention on cultural differences and their impact on pedagogical design and
Usability and Quality Parameters for E-Learning Environments and Systems • 6:5
usability. In his thesis the following important questions on culture, usability, and design, have been
addressed and evaluated:
-“Is there a relationship between usabity attributes and learners’ learning culture?
-Is there a relationship among pedagogical design, pedagogical usability, and learners’ learning
culture?
-Is there a relationship between WBLE usability and learners’ learning culture?
-Is there a relationship between learner’s culture and learners’ learning styles; that is, any influence
of a learner’s culture on the learner’s learning style?
-What are the cultural issues influencing WBLE approaches?
-What are the key strategies for designing educational websites or WBLEs taking into
consideration the learner’s culture?”
3.2. SUS – System Usability Scale in Human-Computer Interaction
Often, for global assessment of usability and quality of a computer system, authors of such
evaluation process need quick, low-cost method. Also, to evaluate systems and compare usability
across a range of contexts, authors look for broad general measures. System Usability Scale, known
as SUS [Brooke 1996], is a popular, reliable, and low-cost usability scale, frequently used for global
assessments of different systems usability. SUS is rather simple Likert scale consisting of ten items.
It is based on subjective assessments of usability and gives general view of systems’ functionalities.
Likert scale is based on forced-choice questions, and the respondent has to indicate for each
statement the degree of (dis)agreement usually on a 5 (or 7) point scale. In the System Usability
Scale prescribed statements in fact cover range of aspects of system usability.
The SUS questions include following important aspects:
(1) I think that I would like to use this system frequently
(2) I found the system unnecessarily complex
(3) I thought the system was easy to use
(4) I think that I would need the support of a technical person to be able to use this system
(5) I found the various functions in this system were well integrated
(6) I thought there was too much inconsistency in this system
(7) I would imagine that most people would learn to use this system very quickly
(8) I found the system very cumbersome to use
(9) I felt very confident using the system
(10) I needed to learn a lot of things before I could get going with this system.
SUS produces a single number that represent a composite measure of the overall usability of the
studied system as scores for individual items are not meaningful on their own. Final SUS scores
range from 0 to 100 and the algorithm of calculation the score is following:
- sum the score contributions from each item and it ranges from 0 to 4
- for items 1,3,5,7, and 9 the score contribution is equal to scale position minus 1
- for items 2,4,6,8 and 10, the score contribution is equal to 5 minus the scale position
- finally, multiply achieved sum by 2.5 to come to the overall value (single number) of system
usability.
Usual way to use SUS is after the respondent use some time the system that will be evaluated,
but necessarily before any discussion about system with other peers. Respondents have to record, for
each item, their immediate response and opinion and not to think for a long time about
items/encountered issues. First feeling and impression about system functionality is extremely
important in this evaluation process. Nevertheless, to such immediate approach SUS has proved as
very valuable, reliable and robust evaluation instrument. Its popularity is a consequence of the fact
that it is a freely available instrument for usability assessment. From its invention until now it has
been frequently used for a range of research projects even industrial systems evaluations. SUS
6:6 • Mirjana Ivanović et al
correlates well with other subjective instruments, tools and measures of usability (e.g., the general
usability subscale of the SUMI inventory developed in the MUSiC project:
https://www.cordis.europa.eu/result/rcn/23029_en.html).
In our study, we also used SUS questionnaire as quick and good instrument to assess and compare
usability of different e-learning environments and systems.
4. STATE-OF-THE-ART ANALYSIS FOR USABILITY OF E-LEARNING ENVIRONMENTS AND
LEARNING MANAGEMENT SYSTEMS
E-learning environments and different LMSs are software tools designed to facilitate e-learning
activities and processes. In the last decade, such systems are becoming more and more complex,
diverse but used at different levels of education (from kindergarten up to high-level education). They
incorporate more and more different components and services needed for handling online teaching
activities. It is the “infrastructure that delivers and manages the instructional content, identifies
and assesses individual and organizational learning or goals, tracks the progress towards the goals,
collects and presents data for managing the learning process of an organization” [William 2007].
The main focus of e-learning research nowadays is oriented towards LMS technology to assist in
delivery of wide range of courses equally for formal and informal education. In the last decade,
numerous LMS have been implemented and deployed, by considering primary the
instructional/pedagogical design but equally also the human-computer interface and interaction
design concepts. The authors of the paper [Aydin 2016], focused on usability as one among essential
design concepts important for HCI part/component of e-learning environments and LMS. Usability in
contemporary e-learning systems plays extremely important role and can directly affect their
acceptance and success. Ease-of-use is a key issue in adaptation of students to such systems, as
primary goal of students is to be concentrated on educational material and not to try to learn how to
use the LMS. Aydin addressed and initiated the specific research questions as result of the study:
(1) “What are the trends for framework development for usability evaluation of LMS?
(2) Which usability evaluation methods are utilized for the evaluation of LMS in the literature?
(3) Have researchers covered all user population of the LMS or have focused particularly on a certain
group?
(4) Which LMS types have been investigated most frequently?
(5) What are the key areas researchers have emphasized so far? What are the gaps in the literature?”
The available publications and literature in the area of e-learning environments and usability
feature can be classified into case-specific categories i.e. papers that: (1) Evaluate the usability of an
LMS, (2) compare the usability of multiple LMSs, (3) propose a framework to evaluate any LMS, (4)
modify or extend previously developed framework, (5) review the literature of usability of LMS, and
(6) provide insights/perspectives on usability of LMS. Based on the State-of-the-Art analysis
presented in [Aydin 2016] it can be concluded that questionnaire (about 50%) and heuristics (about
20%) are the two main methods that are used or recommended in the usability evaluation of LMS.
Finally, as a general consequence of previous discussion, challenging future research task must be
undertaken to determine, which new usability evaluation methods, in assessing e-learning
environments, should be proposed. Further, based on comprehensive analysis, it can be expected
that those methods that are most reliable and robust will be, finally, recommended .
5. USABILITY AND QUALITY OF E-LEARNING ENVIRONMENTS – OUR CASE STUDY
Almost 20 years ago, Squires (1999) emphasized that it is necessary to integrate usability and
learning but also noticed that there was no collaboration of researchers and practitioners in
educational and HCI areas. Things changed a little bit meanwhile, but still there are a lot of open
Usability and Quality Parameters for E-Learning Environments and Systems • 6:7
questions and space for much better cooperation. Usability of e-learning designs is connected to their
pedagogical value. Therefore, it is of crucial importance that usability practitioners get familiar with
the key educational activities like: learning styles, knowledge testing and assessment, and some
elements of learning theory. However not too much has been done to critically examine the usability
of e-learning environments and systems, specially to consider cognitive and affective learning factors
leading learners to achieve better learning outcomes. In spite the fact that educational institutions
have been trying to improve educational processes by employing technological achievements, most of
e-learning systems show higher dropout rates comparing with traditional teaching methodologies.
Some of reasons could be: comfort level with technology, relevancy of content, availability of
technical support and so on. But most of instructors, teachers as well as learners believe that major
influential factor is the poor usability of e-learning systems.
5.1 Applied Methodology
Having experiences in using, in everyday practice, different aspects of technology-enhanced learning
and e-learning systems, recently we conducted a study with idea to compare usability, quality and
general functionalities of different e-learning educational systems and tools. We performed following
procedure. Within “Educational software” course (Elective course for students of 2nd and 3rd study
year) students have to perform and complete several practical tasks using three different e-learning
systems, two predefined in advance: Moodle, and ASQ, and as a third the particular educational
software, they selected individually. Moodle is selected as widely used and recognized LMS. On the
other hand, ASQ is very specific e-learning environment. ASQ (http://asq.inf.usi.ch/) is, in fact, a Web
application for creating and delivering interactive HTML5 presentations. It is designed to support
teachers who like to gather real-time feedback from the students while delivering their lectures and
has been developing at University of Lugano. Our students had to prepare an essay containing brief
description of these three selected e-learning systems and results of their comparison and
assessment. While using these learning systems, students have to play the role of teachers and try to
assess these systems from the point of view of teachers and process of preparation of educational
material. Their practical tasks included use of Moodle, ASQ and individually selected educational
software and in each of them they have to perform following activities:
- the development of electronic teaching material for one among lessons from the selected courses:
Software Engineering, Advanced Topics in Software Engineering and Human-Computer
Interaction,
- preparation of tests and quizzes for selected lesson,
- preparation of adequate Glossary,
- preparation of adequate choice and survey.
All educational materials are prepared and completed during the semester and graded by the
teacher of the “Educational software” course, according to the quality of prepared lessons (tests,
quizzes, glossary and choice) as well as oral presentation and public defense of produced essay.
Essays were presented to the other students who also evaluated educational software, filling out
SUS questionnaires. In the essay it was necessary to present functionalities of Moodle, ASQ and
individually selected educational system, i.e. specially selected educational software for the
development of educational content/material. As ASQ is a very unique educational software and
tool, it was necessary to make work for the students easier. Therefore, we prepared and arranged
sample presentations for educational material. Such presentations have been used as patterns that
students need to change and modify (using any html editor) in order to:
- develop their own educational material and electronic lessons for selected topic: Software
Engineering, Advanced Topics in Software Engineering and Human-Computer Interaction
courses,
6:8 • Mirjana Ivanović et al
- use different forms of ASQ questions to prepare adequate questions, connected to prepared
lesson, for assessment of gained knowledge.
The System Usability Scale (SUS), as a reliable tool was used for measuring usability of the three
educational software platforms: Moodle, ASQ and an individually selected educational system. SUS
includes 10 questions which students were asked to evaluate the perceived usability of selected
educational software. Students rank each question from 1 to 5 based on how much they agree with
the statement they are reading. 5 means they agree completely, 1 means they disagree passionately.
ASQ as an open platform is the educational solution used in the University of Lugano and it was
selected because: a) we had access to a wide variety of courses, and b) we wanted to have benchmark
data for its perceived usability. In all studies, participation was voluntary. Perceived usability affects
greatly student’s learning effectiveness and overall learning experience, and thus is an important
requirement of educational software.
5.2 Experimental results
During last 2 years we conducted experiments with students within the “Educational software”
course using previously described procedure and activities. Last year 46 students participated in the
experiment and have to perform above specified tasks and compare three educational systems.
Luckily in that year, as a third e-learning system, they selected a wide range of available educational
systems: Kornukopia, Civilization, Eliademy, Blackboard, Fun School, Jeliot, Encarta, Schoology,
CourseSites, ATutor, Simcity, Dugolingo, BlueJ, Joomla, Edmondo, JFlap, Mind Genius, WebAssign,
WorldWide, Capitalism, Civilization, Google classroom, NEO-LMS, Wolfram Mathematica, Scratch,
Google Earth, Knewton, Quizlet, studyX, Geogebra. Students had 4 weeks to examine each of the
three educational software. Moodle and ASQ were presented and used during lectures and exercises,
while for the third software, students got homework to study it and analyze the possibilities for
making different functionalities, activities and resources, e.g.: question bank, glossary, surveys, chat,
forums and other opportunities that the software offers. This year 50 students participated in the
experiment also with the same above explained tasks, to compare Moodle, ASQ and one more,
individually selected LMS. This year students worked in teams: 10 teams of 4/5 students.
Consequently, they selected ten different and popular LMS: Chamilo, Sakai, Schoology,
TalentLMS, LoveMySkool LMS, Claroline, Google Classroom, CourseSites by Blackboard LMS,
Canvas, JoomlaLMS, Geenio. After using selected three software platforms, students filled out the
SUS questionnaire from their own point of view, as a professor who used software for the
preparation of educational materials. Then, they came to oral defense of other papers and filled out
the SUS questionnaire on the usability of educational software presented to them by their
colleagues. We used the collected surveys to evaluate the usability of educational software from the
perspective of students, who had the opportunity to independently create educational material.
Figure 1 shows the overall response of the 96 students to the 32 selected educational software from
the usability aspect, based on SUS score. The average SUS scores for the 32 platforms were 74,7051.
The best way to interpret the score is to convert it to a percentile rank through a process called
normalizing, which takes raw SUS scores and generates percentile ranks and letter-grades (from A+
to F). The analyzed software is estimated with C or D letter grades, which according to the
interpretation of the SUS scale means that software is ‘good’. The results also showed that no
software received the highest score, rated by A+, A or B letter grades, which implies that there is
still a lot of room for all analyzed software to improve and meet expectations in terms of usability.
Reliability refers to the extent to which a questionnaire yields the same results under consistent
conditions. It is most commonly measured using Cronbach's alpha, which is a measure of internal
consistency. For our dataset of 156 completed surveys, the 10-item SUS questionnaire has a good
internal consistency; Cronbach's alpha=0.808.
Usability and Quality Parameters for E-Learning Environments and Systems • 6:9
6. DISCUSSION AND CONCLUSION
It is the clear evidence that usability evaluation of e-learning environments and systems is crucial
for achieving better, easy, and more user-friendly interfaces and communication in these systems,
from both points of view: learners and teachers.
Next quality step, ahead in designing usability questionnaires for evaluation of e-learning
environments should include additional very important aspects of general learning theories. Such
new evaluation instruments should for example focus on emotional and affective states of learners. It
is also unavoidable to take into account deeper understanding of the design issues. Furthermore,
significant input to e-learning design comes from assessment of learners. New forms of questions and
quizzes (similar to different forms presented in ASQ system) are necessary to be included in e-
learning systems in order to increase learners’ motivation. Currently, the proposed usability
evaluation instruments can point towards specific usability problems with an e-learning system.
Fig. 1. SUS score for 32 educational software
However, essential advancement in this area may be achieved through the realization of unique
standardized database, where the results of different usability evaluations and studies can be stored.
Correspondingly, it could contain the usability profiles of existing e-learning systems and can
“facilitate designers compare the usability of one application with a series of other e-learning
applications” [Aydin 2016]. Such standardized database could be used as starting point in new
research enterprises and studies as it can offer initial knowledge and can highly support re-usability
and accordingly innovative approaches.
As educational system in the context of learning and teaching at universities are more and more
commonly used, ensuring usability of such systems becomes an increasingly important issue in
design, development and maintenance. Only if a system supports the requirements of heterogeneous
user groups in an effective and satisfying way, will it achieve the desired degree of user acceptance
and satisfaction. The kind and extent of offered functions and services need to be carefully balanced
6:10 • Mirjana Ivanović et al
with users’ interests and requirements. In our study, we present findings related to various SUS
attributes in the context of educational system usability evaluation, using a dataset produced by 96
university students. Students were asked to evaluate the perceived usability of their course’s
Moodle-based, ASQ and more than 30 different educational software by completing SUS and
providing a 5-point scale adjective rating. The average SUS scores for the all 32 platforms were
74,7051. Regarding the contribution of this study, the collected data can help the designers of LMSs,
educational software and other different kind of educational system since they can be used as
benchmarks for the SUS score in the context of perceived usability evaluation. This can provide
significant feedback for their development, indicating the extent of the possible need for
improvement. In addition, the SUS score might provide a common code of communication between
manufacturers and their customers, who may not have the appropriate expertise in software
usability. The SUS score provides good insight into the perceived usability, an important factor for e-
learning and can be understood by all the stakeholders. Furthermore, the findings of this study can
help students, professors and educational organizations, who will be able to compare different
systems in order to choose the right one and reject a non-usable system for the course they wish to
follow or create. Further investigation in different educational institutions and levels of education is
required. The present study explores only students’ views. In future work additional user groups
could be studied, such as educators who may have quite diverging perceptions compared to. Future
work, as well, includes investigating relationships (if any) between SUS score and a) the attitudes of
participants towards the professor of the course who uses the educational software and b) the
education delivery method used, such as blended learning or distance education. Also one new
research direction could be focused on different emotional states of learners like anxiety, uneasiness,
embarrassment, enthusiasm, but also pride can provide adequate explanation of learners’ behavior.
ACKNOWLEDGMENT
This paper is a part of the Serbia-Romania-Poland collaboration within multilateral agreement on “Agent systems and
applications.” and Romania-Poland collaboration within bilateral project “Semantic foundation of the Internet of Things”.
First two authors are also thankful to the project SCOPES JRP/IP, No. 160480/20 as some initial ideas are presented here.
REFERENCES
Agariya, A.K., and Singh, D. 2012. e-Learning quality: Scale development and validation in Indian context, Knowledge
Management & E-Learning: An International Journal (KM&EL), Vol. 4, No 4.
Alelaiwi, A., and Hossain, M.S. 2015. Evaluating and Testing User Interfaces for E-Learning System: Blackboard Usability
Testing, Journal of Information Engineering and Applications, Vol.5, No.1, 23-30.
Al-Khalifa, H.S. 2010. A First Step in Evaluating the Usability of JUSUR Learning Management System. The 3rd Annual
Forum on e-Learning Excellence Bringing Global Quality to a Local Context. 1-3 February 2010. Dubai, U.A.E.
Aydin, B., Mukaddes Darwish M., and Selvi E. (2016). State-of-the-Art-Matrix Analysis for Usability of Learning
Management Systems. In Proceedings ASEE’s 123rdAnnual Conference and Exposition, New Orleans, LA, June 26-29,
2016, Paper 14949, p. 17.
Bari M. And Djouab R., 2014. Quality Frameworks and Standards in E-Learning Systems, International Journal of the
Computer, the Internet and Management, Vol. 22, No.3 (September-December, 2014, 1-7.
Brooke J. (1996), SUS-A quick and dirty usability scale, Usability evaluation in industry 189 (194), 4-7.
Cakmak, E.K., Gunes, E., Ciftci, S., and Ustundag, M.T. 2011. Developing a Web Site Usability Scale: The Validity and
Reliability Analysis & Implementation Results. Pegem Eğitimve Öğretim, I (II), 31-40.
Davis, F. D. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS Quarterly,
Vol. 13, No. 3, 319–340, doi:10.2307/249008
Fathema, N., Shannon, D., and Ross, M. 2015. Expanding the Technology Acceptance Model (TAM) to examine faculty use of
Learning Management Systems (LMS), Journal of Online Learning and Teaching, Vol. 11, No. 2, 210-233.
Ivanovic M., Xinogalos S., Pitner T., and Savic M. 2017. Technology enhanced learning in programming courses -international
perspective. EAIT Vol. 22, No. 6, 2981-3003.
Inversini, A., Botturi, L., and Triacca, L. 2006. Evaluating LMS Usability for Enhanced eLearning Experience. In Proceedings
of World Conference on EducationalMultimedia, Hypermedia and Telecommunications, Pearson, E. and Bohman, P.
(Eds.), Chesapeake, VA, 595-601.
Kirakowski, J. and Corbett, M. 1990. Effective Methodology for the Study of HCI. Elsevier.
Usability and Quality Parameters for E-Learning Environments and Systems • 6:11
Melton, J. 2006. The LMS Moodle: A Usability Evaluation. Languages Issues, Vol. 11/12, No.1, 1-24.
https://www.researchgate.net/publication/241339520 (Access Date: 26.06.2018).
Masood, M., and Musman, A. 2015. The Usability and its Influence of an e-Learning System on Student Participation.
Procedia-Social and Behavioral Sciences,197 (February), 2325–2330.
Ogunbase, A. O. 2016. Pedagogical design and pedagogical usability of web – based learning environments: Comparative
cultural implications from Africa and Europe. Dissertations in Interactive Technology, 23:13.
Onacan, M.B.K. and Erturk, A. 2016 Usability evaluation of learning management system in higher education institution: A
scale development study, International Conference on Leadership, Technology, Innovation and Business Management
(ICLTIBM-2016), Antalya, Turkey
Putnik Z., Stajner-Papuga I., Ivanovic M., Budimac Z., and Zdravkova K. 2017. Gender related correlations of computer
science students. Computers in Human Behavior 69: 91-97.
Rekkedal, T. 2006. Criteria for evaluating quality in e-Learning (EU Leonardo project),
. Accessed March 2014.
Sandoval, Zoroayka V. 2016. The Re-design Process of an Instrument to Evaluate Usability in Online Courses.
Squires, D. 1999. Usability and Educational Software Design: Special Issue of Interacting with Computers. Interacting with
Computers. Vol. 11, No. 5, 463-466.
Thuseethan, S., Achchuthan, S., and Kuhanesan, S. 2014. Usability Evaluation of Learning Management Systems in Sri
Lankan Universities. https://arxiv.org/ftp/arxiv/papers/1412/1412.0197.pdf, (Access Date: 26.06.2018).
Venkatesh, V., Morris, M. G., Davis G. B. and Davis, F. D. 2003. User acceptance of information technology: Toward a unified
view, MIS Quarterly, Vol. 27 No. 3, 425–478.
Watson W. R. and Watson S. L. 2007. An argument for clarity: what are learning management systems, what are they not,
and what should they become?. TechTrends, Springer Verlag, Vol. 51, No. 2, 28-34.
Xinogalos S., Pitner T., Ivanovic M. and Savic M. 2018. Students' perspective on the first programming language: C-like or
Pascal-like languages? EAIT, Vol. 23, No 1, 287-302.
Zaharias, P. 2004. Usability and e-Learning: The road towards integration. ACM eLearn Magazine, Vol. 6.
Zaharias, P. 2009. Developing a usability evaluation method for e-learning applications: From functional usability to
motivation to learn. International Journal of Human-Computer Interaction, Vol. 25, No. 1, 75-98.