=Paper=
{{Paper
|id=Vol-2508/paper-iva
|storemode=property
|title=On Measuring Learning Success of Students with Disabilities in Virtual Environments
|pdfUrl=https://ceur-ws.org/Vol-2508/paper-iva.pdf
|volume=Vol-2508
|authors=Mirjana Ivanović,Amelia Bădică,Maria Ganzha,Marcin Paprzycki,Costin Bădică,Aleksandra Klašnja-Milićević
|dblpUrl=https://dblp.org/rec/conf/sqamia/IvanovicBGPBK19
}}
==On Measuring Learning Success of Students with Disabilities in Virtual Environments==
https://ceur-ws.org/Vol-2508/paper-iva.pdf
6
On Measuring Learning Success of Students with
Disabilities in Virtual Environments
MIRJANA IVANOVIĆ, University of Novi Sad
AMELIA BĂDICĂ, University of Craiova
MARIA GANZHA, Warsaw University of Technology and Systems Research Institute Polish
Academy of Sciences
MARCIN PAPRZYCKI, Warsaw Management Academy and Systems Research Institute Polish
Academy of Sciences
COSTIN BĂDICĂ, University of Craiova
ALEKSANDRA KLAŠNJA-MILIĆEVIĆ, University of Novi Sad1
During more than the last three decades, the interest in educational arena supported by technological advancements has been
growing systematically. New, different, concepts and tools including: Innovative Educational Environments, Future
Classrooms, and Virtual Laboratories emerged constantly, introducing innovative ways of e-learning. On the other hand,
Inclusive Education is getting more attention and importance, as contemporary classrooms must include students of diverse
abilities, to learn and socialize together. The modern classroom should not discriminate between students with disabilities
and students without disabilities by offering them equal attention and opportunities. In this position paper, our intention is to
consider possible technological influences on Virtual Environments/classrooms having in mind Science, Technology,
Engineering and Mathematics (i.e. STEM) education and, accordingly, propose some possible measures for learning success of
students with disabilities. The key issue in these activities is to make all students feel welcomed and properly supported in
their efforts to gain adequate knowledge and skills, while collaborating with their peers and interacting with e-learning
environments.
1 INTRODUCTION
During the last several decades (As a very basic example of considerations concerning use of
technology in education that have been formulated almost 30 years ago see, for instance, this
references list 2 ), the interest in education supported by technological advancements has grown
enormously. New, creative learning environments emerged constantly, exhibiting highly promising
features, for advancing the educational arena. Virtual, multi-functional environments and
classrooms determine the more active involvement of teachers and students, supported by advanced
pedagogical approaches enabled by modern digital technologies. Challenging technological,
pedagogical and methodological approaches, in educational processes, promote positive impact on
students’ academic knowledge, skills, interaction and levels of technological literacy. Technology
Authors address: Mirjana Ivanovic, Faculty of Sciences, Trg Dositeja Obradovica 4, 21000 Novi Sad, Serbia; e-mail:
mira@dmi.uns.ac.rs; Amelia Badica, Faculty of economics and business administration 13, A.I. Cuza Street, Craiova, 200585,
Romania; e-mail:ameliabd@yahoo.com; Maria Ganzha, ul. Newelska 6, 01-447 Warsaw, Poland; e-mail:
maria.ganzha@ibspan.waw.pl; Marcin Paprzycki, ul. Newelska 6, 01-447 Warsaw, Poland; e-mail: paprzyck@ibspan.waw.pl;
Costin Badica, Department of computers and information technology, Blvd. Decebal nr. 107, RO-200440, Craiova, Romania; e-
mail: cbadica@software.ucv.ro; Aleksandra Klasnja Milicevic, Faculty of Sciences, Trg Dositeja Obradovica 4, 21000 Novi Sad,
Serbia; e-mail: akm@dmi.uns.ac.rs;
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International
(CC BY 4.0).
In: Z. Budimac and B. Koteska (eds.): Proceedings of the SQAMIA 2019: 8th Workshop on Software Quality, Analysis,
Monitoring, Improvement, and Applications, Ohrid, North Macedonia, 22–25. September 2019. Also published online by
CEUR Workshop Proceedings (http://ceur-ws.org, ISSN 1613-0073)
2
http://www.ibspan.waw.pl/~paprzyck/mp/cvr/education/neted.html
6:2 · Mirjana Ivanović et al.
enhanced learning assure flexible, responsive, and effective use of digital technology [Patrícia and
Neuza 2018].
Technological advancements are also directly influencing Inclusive Education. It is important
that students of diverse abilities and backgrounds get learning and socializing together in the same
classroom. However, such new way of learning and socializing needs significant educational
reforms, at least in the area of adequate teacher and staff training, as well as the availability of
technical support and adequate learning materials tailored to specific students’ needs. Inclusive
education, in essence, means that students with diverse abilities and backgrounds learn together, in
the same classroom (real or virtual/distance) by receiving high quality support that is needed to
achieve success in the core elements of the proposed curriculum. The modern classroom should treat
the students with disabilities (SwD) as being, fundamentally, as competent as students without
disabilities. Technological advancement in ICT, especially in domains like robotics, mechatronics,
and artificial intelligence, together with innovative instructional design and novel pedagogical
approaches, are essential premises for the successful inclusive education, regardless of student
differences and of their diversity in cognitive, academic, physical, social, and emotional traits [Savin-
Baden 2015].
Students with disabilities have specific, individual learning needs, as well as restricted learning
abilities. So the pedagogical methods have to be oriented towards developing unconventional
teaching practices, adequate educational resources, and they should apply specific assessments
supported by adjusted measures of learning success. Distance learning, as well as mobile learning
supported by the use of tablets, smartphones, and similar devices, may offer innovative solutions for
adequate education of SwD. Mobile applications and special educational tools with speech-to-text
and text-to-speech functionalities can highly support SwD by enabling their fair engagement in the
learning/teaching process, regardless if they are in real or virtual classrooms.
In their early days, distance learning and e-learning were concentrated predominantly on
narrative disciplines, without the need to use laboratories and hands-on activities that characterize
(Science, Technologies, Engineering and Mathematics) STEM disciplines. Modern technologies
strongly contributed to the inclusion of these disciplines into the focus of distance and e-learning
education, by providing adequate technological support, like for example the development and
application of online virtual laboratories for students of science and engineering disciplines
(mathematics, informatics, physics, robotics, mechatronics, control systems and so on). In such
specialized virtual laboratories a student can exercise specific practical tasks regardless of time or
location boundaries, fear of improper handling of equipment, requirement for a live instructor (who
will be replaced with Pedagogical Agents and Chatbots) [Terracina et al. 2016].Here, the good news
is that important early issues related to, broadly understood, computer literacy (see, for instance
[Paprzycki 1992]) have almost disappeared across the “developed world”.
Additionally, great potential lies in use of Internet-of-Things (IoT) and Internet-of-Everything
(IoE) that enable virtual use of various smart and specialized devices, thus making learning easier,
faster, and safer. These innovative approaches bring about incredible potential for students with
disabilities as well, by enabling them to learn from homes and, at the same time, to interact and
collaborate with their peers and teachers.
In this position paper we concentrate our attention on possible technological influences on Virtual
Environments and on some possibilities to measure the performance of such systems, in order to
increase learning success of students with disabilities.
The remainder of the paper is organized as follows. Different contemporary technologies and their
influences on learning of students with disabilities are presented in Section 2. Deeper considerations
of measures for learning success of SwD and proposition of most appropriate characteristics of
On Measuring Learning Success of Students with Disabilities in Virtual Environments·6:3
Pedagogical agents in social aspects of learning, are given in Section 3. Last Section brings
concluding remarks.
2 TECHNOLOGICAL INFLUENCES ON LEARNING OF STUDENTS WITH DISABILITIES
Continuous technological influences on improvement of classroom flexibility, development of modern
teaching methodologies, usage of e-services, mobile/smart devices and social media should be
adapted for students with disabilities. Current worldwide revolution that is happening in the
education (in both real and virtual classrooms) is initiated by numerous applications of Internet of
Things (IoT) and Internet-of-Everything (IoE).For example, the paper [Farhan et al. 2018]shows that
activities of students in an e-learning environment can be effectively measured using an attention-
scoring model (ASM). The model is based on the observation of students’ faces and eyes in order to
discover their attention and emotions. The IoT can have implications on the overall delivery of the
educational material in highly innovative manner in all aspects of students’ activities. IoT,
intelligent technologies and new concepts, such as cloud computing, educational and learning
analytics, wearable technology, etc. promote the materialization of smart education.
Availability of a wide range of multimodal educational resources that are at students’ disposal
and that can enhance teaching and learning rapidly increases in the emerging world of the IoE.
While IoT represents the networked connection of different physical objects, IoE represents a
network of smart objects, i.e. interconnected things where the difference between the physical object
and the digital information augmenting them is blurred [Selinger et al. 2013]. The huge number of
connections of people (including students and teachers), processes, data, and other things brings
about a completely novel concept of Internet of Learning Things [Selinger et al. 2013]. Inclusion of
IoE in educational activities, with the aim to improve learning and assessment capacities is seen as
highly promising, more sustainable, and challenging future direction. Different universities and
companies, all over the world, strive in development of IoE-based smart classrooms that include
numerous highly heterogeneous devices: smart tables, interactive whiteboards, 3D printers, sensor
gloves, eye-trackers, headsets attention monitoring systems, Human-Computer interfaces, and other
digital laboratory devices. Such devices support reducing different obstacles and barriers faced by
students with disabilities, such as physical, cognitive, social and organizational barriers. Human-
Computer interfaces, supported by suitable technical devices, are an essential element to support all
students in virtual educational environments. The developed IoE-based smart educational
environments can enable a completely revolutionized learning and teaching practice, for students
with disabilities, in STEM disciplines.
2.1 Role of E-Learning Environments in Educational Processes
During the last decade, a wide range of software systems, enriched by including numerous and
diverse aspects of multimedia and Web technologies, and seamless, multimodal, user-friendly
Human-Computer Interaction, have been developed for promoting innovative and smart learning.
One such general purpose Learning Management System that is probably mostly used nowadays
is Moodle. Its development was based on sound pedagogical principles, and it can support diverse
e-learning approaches, including distance education, flipped classroom and blended learning. Since
its introduction, about two decades ago, many improvements and enhancements of Moodle have been
developed [Open Source Technology 2014], [Link1] to follow requirements of ever-changing
educational demands. Constantly innovated Moodle environment helps educators to build
multimodal, multifunctional, and interactive e-lessons. There are a lot of additional educational
services and components that can enhance Moodle in order to improve classroom flexibility and offer
pleasant learning atmosphere, for all diverse categories of students.
6:4 · Mirjana Ivanović et al.
Apart from technological advancements, pedagogical aspects are of paramount importance for
modern distance educational environments. The preparation of educational resources, for Virtual
Environments and students with disabilities, to support accessibility with wide range of
functionalities, must be based on non-conventional hardware and software components such as
upgraded keyboards, speech-to-text and text-to-speech functionalities, scalable fonts, and so on, by
augmenting the Virtual Environment with:
- multiple formats (HTML, RTF, PDF, etc.), to ensure acceptability by a wider range of students,
- multiple modalities (visual, auditory, kinesthetic, or tactile)to increase students’ motivation and
interest,
- more flexible course materials to accommodate students’ differences and diversity,
- innovative use of modern multimedia technology supported by educational data mining and
learning analytics to obtain high level of personalization and tailored recommendations.
Additionally, today’s innovative learning models and concepts, like flipped classrooms [Lage et al.
2000], serious games, and massive open online courses (MOOCs), are constantly progressing and
enable more students (especially for students with disabilities) to engage from more and diverse
locations at a wider scale in higher quality education using adequately prepared educational
resources.
Regardless of these general purpose systems, worldwide research and academic institutions have
been developing their own, specific-purpose, usually intelligent and personalized educational
systems/environments. They are extremely important in facilitating higher-quality STEM e-learning
[Klašnja-Milićević et al. 2017]. An essential feature of such systems is personalization that provides
open, flexible, and tailored learning to students with diverse abilities.
Key desirable features of a wide range of e-learning systems include new models for intelligent
personalized interaction and teaching material recommendation. These particular intelligent
personalized interactions are usually required to address the specific and personal needs of each
student, including: learning style [Klašnja-Milićević et al. 2011], personal learning characteristics of
the learning style (like: Discussion forums, Simulations, Roles and serious games, Case studies and
so on), as well as the most suitable electronic media for representation of educational resources
(e-book, Forum, Wiki, Weblog, Podcast, and so on).These numerous possibilities to enhance
traditional learning provide students with disabilities better access to information based on visual,
auditory and kinesthetic means. At the same time, the teacher and peer’s interaction and
collaboration with SwD can be significantly improved. With the availability of numerous online
collaboration tools, existing today on the market, student-teacher and student-student interaction
has never been easier. Innovative ICT technologies (cutting-edge multimedia, speech and mobile
technologies, IoT and IoE) improve accessibility and contribute to the transition of teaching from a
traditional “one size fits all” approach to more individualized “one size fits one” learning solutions
that are more appropriate for students with disabilities.
In order to provide accessibility to literature for students with sensory, physical, cognitive and
psychosocial disabilities, the development of multimedia libraries and enhanced virtual laboratories
based on speech technologies is a challenging task [Lynch and Ghergulescu 2017]. Virtual
laboratories offer new way to students’ participation and interaction in inquiry-based classes. In
such classes students can perform their own experiments, learn from anywhere (by using virtual
objects).As another opportunity, multimedia libraries can enhance standard lecture presentations,
with accompanying explanations in both textual and audio forms (using, for example, Pedagogical
agents – see, the next sub-section).With the help of IoT, the audio form, in general, can increase the
accessibility of the educational resources to the visually impaired, while the textual and visual parts
On Measuring Learning Success of Students with Disabilities in Virtual Environments·6:5
of educational resources will make it more accessible to the hearing impaired students. The
utilization of speech-enabled mobile applications, for example, helps students with reading related
disabilities to access educational resources and assists students with writing difficulties to finish
their writing tasks. IoE-enabled devices and technology help students to access their courses at any
time, from anywhere in the manner most appropriate to them.
A very challenging issue, in this area that was recently taken into consideration is the
appropriate capture, stimulation and use of human senses. It is regarded as one of the prominent
practices of the educational process. Multi-sensory instruction is described as teaching that involves
all the senses: seeing, hearing, tasting, touching and smelling [Aleksandra Klašnja-Milićević et al
2018a]. Multi-sensory instructions, together with abovementioned innovative approaches, enhanced
by augmented reality technology, offer highly promising elements that can obtain new functionalities
in virtual learning environments for students with disabilities. For example, NEWTON project
[Lynch and Ghergulescu 2018] encompasses augmented reality with personalized learning and
virtual reality with gamification and emphasis on developing virtual laboratories tailored to the
specific needs of students with disabilities.
2.2 Role of Pedagogical Agents in E-Learning Environments
Research in Inclusive Education area show sthat the presence of SwD gives non-SwD students new
kinds of learning opportunities[Savin-Baden 2015],[Savin-Baden et al. 2019]. In such learning
organization, one significant opportunity occurs when non-SwD serve as peer-coaches. In fact, by
trying to help another student, the helper peer can improve his/her own performances (see, also
[Paprzycki and Vidakovic 1993] and references collected there).
Another challenge is connected to the teachers’ duties. To take care of their more diverse
audience, including students with disabilities, teachers must be able to provide instruction in a
wider range of learning modalities (visual, auditory, and kinesthetic) bringing benefits also to their
non-SwD. This task is definitely not easy for majority of teachers. However, using contemporary
technological advancements this problem could be successfully solved in near future using intelligent
software agents.
Autonomous, intelligent software agents, used in a learning context, are usually known as
Pedagogical Agents. Their aim is to support learners across a wide range of subjects. Pedagogical
agents are especially valuable to guide students through multimedia, multimodal learning
environments, by exploring their motivations and by assessing the learning effects and outcomes.
The use of Pedagogical Agents ranges from supplementing existing human-driven instruction with
expert features, to entirely replacing human teachers. “What is required is the use of such agents in
places of widening access, increasing diversity, and spaces that work against standardized models of
learning.” [Savin-Baden et al. 2019]. In the paper [Rickel et al. 2002] authors presented Autonomous
(Pedagogical) Agent as a kind of software that conveniently interacts with the user, possibly using
natural language, in form of: conversation, coaching to achieve solution of particular task or posing
questions to assess acquired knowledge. The desirable form of these agents is to be realized as virtual
visual assistants. Additionally, Pedagogical Agents have been found to improve motivation and
reduce cognitive load among the students [Bowman 2012], [Ivanović et al. 2015].
Pedagogical Agents are used for learning purposes in different domains and courses. However, in
the context of this paper, our intention is to present their advantages in STEM domains, with
emphasis on students with disabilities. For this purpose, interesting experiments were performed in
subjects spanning STEM with the AutoTutor system [Graesser et al. 2014], and for learning
programming with the PROTUS system [Ivanović et al. 2015]. In these systems, students have to
gain adequate knowledge and develop their problem-solving skills. Pedagogical Agents used in these
systems are aimed at providing students with a virtual tutor that can respond intelligently to their
6:6 · Mirjana Ivanović et al.
inquiries and that has the ability to emulate the teacher by offering students immediate feedback and
hints, thus helping them to improve their knowledge and skills.
Virtual Pedagogical Agents/Tutors raise difficult challenges and offer great opportunities to be
faced and exploited in virtual e-learning environments and laboratories that provide technologically
supported inclusive education. One of very important aspects, during learning for students with
disabilities, is attitude – a complex psychological concept that characterizes the mental and/or
emotional state of a person [Perloff 2016]. Therefore, the use of Pedagogical Agent as a motivator
that demonstrates positive attitudes towards the task and the desired levels of performance, helps
students to cope with situations where they feel as novices or with some level of anxiety.
One-to-one communication between student and system is valuable, especially for students with
disabilities, and it can help enormously in acquiring curricular knowledge and skills. However, other
personal diversities (like social and emotional) also play important role in multiple educational
activities. Moreover, recent approaches that target the realization of mixed group interactions and
conversations between agent(s) and students are getting more and more attractive for students with
disabilities. For example, a notable approach presented in [Graesser et al. 2014] concentrates on
trialogues, i.e. some simple way of group conversations. In fact “The incremental value of multiple
agents is that the student can learn by observing how the agents interact. A student can learn
vicariously by observing one agent communicating with another agent, showing how actions are
performed, and reasoning collaboratively with the other agent.”[Graesser et al. 2014].This paper
highlights situations where two agents can behave in different circumstances such as: disagreement,
contradiction, and holding an argument, thus providing the students’ with the opportunity to face
different situations that can appear in real classrooms.
This very interesting approach can represent an excellent starting point for further investigation
in this area, aiming at bringing useful consequences for students with disabilities, in supporting
their social and emotional aspects of learning. The development of future e-learning environments
must investigate if such virtual multi-conversational Pedagogical Agents actually encourage or
discourage the development of problem solving, reasoning, and, in-depth learning of students with
disabilities. Future investigations and experiments also have to include different configurations of
students in both situations: (1) real classroom with blended learning style or in (2) Virtual
Environments, where the use of chat forums is highly encouraged in both situations. For example, it
will be interesting to consider the following learning scenarios in which mixtures of both types of
students (SwD and non-SwD) are included in the same group:
- Scenario1: Real classroom with a blended style of teaching/learning with the possibility to use
chat forums.
o group of several students (including students with disabilities) communicate with one
Pedagogical Agent;
o one student with disabilities communicates with several Pedagogical Agents;
o group of several students (including students with disabilities) communicate with several
Pedagogical Agents;
- Scenario2: Virtual Environment with direct communication or by using chat forums:
o group of students can communicate with one Pedagogical Agent
o group of students can communicate with several Pedagogical Agents.
Students can either communicate directly or exchange messages and then expect the opinions and
suggestions of their peers, as well as of Pedagogical Agents.
Such scenarios and experiments, with different mixed groups of students, could bring valuable
insights into the pedagogical, methodological and motivational aspects of inclusive education
On Measuring Learning Success of Students with Disabilities in Virtual Environments·6:7
supported by contemporary technologies. However, currently this task is not easy, in spite the fact
that different frameworks for communication using natural languages offer great opportunities, like
for example Amazon Alexa and IBM Watson.
3 MEASURING LEARNING SUCCESS – STUDENTS WITH DISABILITIES
Measuring students’ learning success is one of key activities in all educational processes. One can
find a lot of different definitions (on the Internet, in books, and research papers) of learning success.
Some of them are rather complex and comprehensive, so in the paper we will concentrate only on
several key aspects important for students with disabilities. We focus on learning success that
considers the level and quality of acquired curricular knowledge, including social aspects i.e.
interaction and collaboration between students and teacher (real or virtual). Diverse measures of
learning success have been developed constantly, with main motivation to help in improving
methods, pedagogies and adjust learning activities to specific and particular students’ needs [Davis
1993], [Lorenzo el al. 2013], [Byers 2017]. These measures are oriented towards obtaining subjective,
or objective, evaluation of educational practices. Triggered by the contemporary necessity to take
special care of inclusive education, existing measures have to be re-considered and re-evaluated
under new circumstances, such that updated, as well as novel instruments must be proposed.
Modern trends and technologies like educational data mining (EDM) and learning analytics (LA)
offer instruments to answer increasingly important, but very complex, questions: what is the current
student knowledge level and whether a student is actively engaged in the learning process together
with her peers. Scientists, from different disciplines, connected to educational processes have
actively considered and experimented with new techniques, based on machine learning and data
mining from system-generated data that have shown promise for predicting students’ learning
achievements and outcomes. They analyzed students’ behavior in learning environments, trying to
recognize their different learning patterns, for possible later use in predicting further students’
learning activities and achievements, in order to increase quality of learning [Klašnja-Milićević and
Ivanović 2018b].
3.1 Measures for Learning Success of Students with Disabilities
In this sub-section we will consider some, possible, “general-purpose/standard” measures that can
contribute to the advancement of technology enhanced learning of students with disabilities.
MEASURING LEARNING SUCCESS - CASE 1: Rather than a standard way to measure
learning success and effects of learning, considering 3 specific learning situations is presented in
[Savin-Baden et al. 2019]. Authors considered students with different abilities and background and
they allocated them into one of three conditions. First group consisted of students that used support
of Pedagogical Agent. Second group consisted of students that used on-line teaching material. Third
group consisted of students that have traditional face-to-face session. Qualitative data was collected
through semi-structured interviews, while quantitative data was collected through both objective
(target subject attainment) and subjective (technology acceptance and learning approaches) self-
reporting measures. Technology Acceptance Model form (TAM) [Davis 1993] [Lorenzo et al. 2013]
was used to assess the usability and perceived usefulness of the Pedagogical Agent. The ASSIST
Questionnaire form [Tait et al. 1998] was used to evaluate students’ learning approach, i.e. to check
if the approach with the agent engagement is more effective. Analysis and final drawn conclusions
showed that, at the moment, students from the three groups prefer more the Online and traditional
F2F approach as compared to the Pedagogical Agent approach. Specifically, “for the Pedagogic Agent
groups, scores on the measure of technology assessment, the TAM, were highest for computer
playfulness and lowest for computer anxiety. For the Online group, scores were highest for the
perception of external control and also lowest for computer anxiety.”
6:8 · Mirjana Ivanović et al.
As Pedagogical Agents can offer additional methodological advancements and support in variety
of educational settings, they must be carefully considered and especially used in inclusive education.
Accordingly, we can suggest and conclude: to use Pedagogical Agents in inclusive education, students
need to be supported in understanding their preferred learning strategies, as well as to be able to
build on individual self efficacy to promote more effective engagement.
MEASURING LEARNING SUCCESS - CASE 2: In this case, a rather innovative approach of
measuring students’ learning success is discussed. A specific kind of learning observation metric
entitled Linking Pedagogy, Technology and Space (LPTS) was developed by Terry Byers [Byers
2017]. It supports “real-time empirical evidence of spatial interventions by teachers through their
practices and subsequent impact on students.” This, rather comprehensive, measure covers five
aspects of learning: Pedagogy; Learning Experiences; Communities of Learning; and Student and
Teacher Use of Technology. The measure was designed to determine the duration of each activity
and its behaviors associated with 36 indicators. The time spent in each activity and associated
behaviors are recorded as they occur during the learning process [Patrícia and Neuza 2018].
For Learning Experiences, the following indicators are assigned: formative assessment, receive
instruction, remember/recall, understand, apply, analyze, evaluate, creation/practical activity,
students disengaged. For Communities of Learning, the following indicators are assigned:
individual, small groups (the same number), whole class, mixed groups (different numbers), mixed-
class/year-level. For Student and Teacher Use of Technology, the following indicators are
assigned and they are the same for both participants: mode1: teacher-centered, mode2: student-
centered, mode3: informal, outside classroom, substitution, augmentation, modification, redefinition,
pen and paper, tablet/laptop (typing), tablet/laptop (touch or stylus), front data projector, additional
visual display/screens, whiteboards (writeable walls), camera or recording equipment, equipment or
tools. Obviously, it is possible to find indicators in each category that could be considered for use in
inclusive education. However, because of limited space of the paper, from the point of view of
inclusive education and success of students with disabilities, we focused here only on the Pedagogy
Aspect of the measure. This aspect includes the following indicators [Patrícia and Neuza 2018]:”
- Didactic Instruction- when the teacher is engaged in presenting/disseminating content,
concepts or information to students through a didactic/direct instruction mode;
- Interactive Instruction- when the teacher is engaged in demonstrating a process/ability or skill
through an interactive/dynamic instruction mode (using equipment and/or tools through a series
of interactive steps);
- Facilitating- when the teacher is moving about the room to observe/monitor/regulate students'
progress and behaviors;
- Providing Feedback - when the teacher provides feedback (advice, direction or suggestions) on
an individual, pair or small groups progress in a particular learning activity;
- Class Discussion- when the teacher promotes the instruction/discussion with the
students/between the class to provide input to a particular topic of discussion that they or the
whole class are participating in; when students interact/discuss with each other
- Questioning- when the teacher asks the student(s) (individual, pair, small groups, whole class)to
answer or respond to either closed or open questions about the thematic contents/ activities.”
Original definitions of indicators have to be adapted depending on specific characteristics of
experiments and educational circumstances. Concerning the role of Pedagogical Agents and the
possibilities to use them in different communication scenarios, during learning processes of students
with disabilities, the last three indicators can play a significant role from our point of view.
The social aspect of learning is generally very important, especially for majority of STEM
disciplines. It seems additionally valuable for students with disabilities. Concerning the indicators
On Measuring Learning Success of Students with Disabilities in Virtual Environments·6:9
mentioned above: Providing Feedback, Class Discussion and Questioning, the value of
student-teacher dialogue for students with disabilities has to be considered with special attention.
Concerning Scenarios proposed in Section 3.1, all explanations of Pedagogical Agents depend on two
basic processes that students must be engaged in, especially when on-line discussions are in use:
- “speaking”- externalizing ideas/opinions by posting messagesto the discussion forum;
- “listening” - consuming the externalizations of others by accessing existing posts.
Speaking, in on-line discussions, is visible to other participants, while listening is invisible, and it, is
in fact, the critical issue for discussions here. The different kinds of listening behaviors represent
step ahead in productive use of discussion forums. It is important to motivate students to be actively
engaged and support better connections in student-student and student-teacher listening and
speaking behaviors [Wise et al. 2013].Pedagogical Agents engaged in virtual learning environments
can highly motivate students to actively participate and regulate/decide how they will speak and
listen in online discussions. Having appropriate communication skills, Pedagogical Agents can
positively influence students with disabilities for their active and productive participation in Class
Discussion and Questioning.
Analysis of data collected from discussion forums (especially if we use educational data mining
and learning analytics) could be extremely useful in Providing Feedback from Pedagogical Agents
and tailoring and personalizing actions suggested for each individual student with disabilities.
For achieving this constructive supervision and tutoring of Pedagogical Agents, some basic
measures can be considered in data collection phase like: Range of participation, Number of sessions,
Average session length, Number of sessions with posts, Number of posts made, Average post length,
Number of posts read, Number of reviews of own posts, Number of reviews of other’s posts.
Additionally, as interaction/conversation has visual and audio nature, we can valorize the
additional power in the identification of frequently used words/phrases and the basic elements of text
and speech analysis that can significantly improve the personalized feedback provided by
Pedagogical Agents.
3.2 Measures for Empowering Students’ Interaction and Motivation
Different methodologies for qualitative and quantitative measurement of learning success are
necessary in order to increase the quality of learning. They are especially important in technology
enhanced learning and, in particular, when employing Pedagogical Agents in Virtual
Environments/classrooms. Recent investigations in the area of inclusive education show that, when
mixing both students with and without disabilities, both groups have the opportunity to learn more.
Many studies carried out over the past three decades have found that students with disabilities
obtain higher achievement and improved skills through inclusive education, while their peers
without challenges can benefit, as well[Bui et al. 2010], [Alquraini and Dianne Gut 2012].
Adequate measures (regardless if they are objective or subjective)must be used, to help in
empowering personal communication, to provide better recommendations of appropriate educational
resources and to increase the motivation of students. To summarize previously presented
possibilities to measure students learning success, we can suggest several possible main domains for
measuring learning success of students with disabilities in virtual learning environments.
Measure1: Learner-centeredness– provides the students with the opportunity to actively
participate in the teaching and learning process; supports learning; students are regarded as
contributors to their own learning [Makoelle 2014].
Measure2: Learning preferences–students’ learning approaches and preferences highly affect
their engagement with the Pedagogical Agent(s); such preferences influence the process of teaching
material tailoring and personalizing, in order to serve the special needs of students with disabilities.
6:10 · Mirjana Ivanović et al.
Measure3: Virtual interaction–these measures should offer opinions and suggestions for
further learning steps given by Pedagogical Agent and they are crucial for participation of students
with disabilities in virtual classrooms. For SwD, additional important measures should be oriented
towards assessment of visual appearance and interaction with Pedagogical Agent like:
- General characteristic - age, gender, clothing, weight, etc.;
- Quality of voice- high, medium, treble, etc.;
- Emotion expressions - compassionate, pleasant, strict, polite, etc.;
- Additional emotional factors- boredom, pride, pleasure, shame, etc.;
- Monitoring and directing motivation - arouse interest, highlight the relevance of the topic,
strengthen the student’s confidence, etc.;
- Capabilities- behave as expert (strict), motivator (friendly), and mentor (supportive);
- Human vs. non-human characters- appearing to be as static or animated;
- Communication mode - students can freely choose if and when to chat with theirPedagogical
Agents.
Measure4: Team dynamics– Teamwork is an important way of organizing manpower in
activities leading to producing solutions, in majority of STEM disciplines. Grouping students in
Virtual learning environments is an additional motivational factor for all group members regardless
of how diverse they are.
Measure5: Cognitive abilities–characteristics of the students who interact with the
Pedagogical Agent(s) include several cognitive factors like: prior knowledge, ability to integrate the
new information into the existing cognitive structure, ability to share knowledge and so on.
Measure6: Information processing– provides explicit information about prerequisites,
conditions, relationships or outcomes of the learning content, enables students to decompose new
information into smaller units, synthesizes them and is able to extract similarities and differences.
Measure7: Transfer of information–the ability of students to apply the new knowledge, to
transfer it to other topics, and to use it for solving new problems.
Contemporary learning, usually represents a challenging and unique mash-up of home-school-work-
media-peer-collaboration in both real and virtual classrooms/environments. It includes also the
following significant practices (based on [Savin-Baden 2015]) that are applicable to students with
disabilities, granting them equal opportunity in educational processes:
- Mentorship -using mobile devices to keep in touch with various educational players through
different means of communication including ubiquitous social media.
- Co-operative online learning - cooperation with peers and virtual agents to guide and support
completing homework, assignments, tests. Similar measures as abovementioned could be applied
here.
- Gaming: isolated or combined in order to share, teach, learn, offer advice, negotiate, and give and
receive hints, tips and solutions.
- Teaching technology: teaching and sharing experiences with peers and virtual agents about
applications, services, new devices, and helpful sources of information.
- Emotional learning: using digital media for peer to peer support to manage personal issues and
difficulties, and to receive hints and advice.
- Playful learning: trying things out and fiddling around, in order to experiment and discover.
All these practices additionally attract research community attention, influence further research
and raise a lot of interesting research questions in order to find and propose adequate new measures
On Measuring Learning Success of Students with Disabilities in Virtual Environments·6:11
or evaluate and adjust existing measures to meet requirements of inclusive education and higher
learning effects of students with disabilities.
In this position paper we pointed out initial considerations and proposals for establishing
measures for virtual learning environments with Pedagogical Agents that can help students with
disabilities to achieve better learning success and interaction with peers and teachers (real and/or
virtual).
4 CONCLUDING REMARKS
Continuous and rapid technological advancement essentially changes our traditional perceptions of
education. Numerous emergent technologies appear “on the monthly basis”. Impressive growth of
availability of IoT and IoE smart devices with sensing / actuating capabilities and applications that
can take advantage of them bring enormous potential in education and can significantly change its
pedagogical and methodological aspects. They are excellent facilitators for contextual, personalized
and seamless learning in smart environments, suitable for students with disabilities.
Having SwD and non-SwD in the same classrooms (real and/or virtual) is challenging for the wide
range of educational stakeholders. Henceforth, investment in designing and establishing appropriate
success measures of such systems can benefit students learning success and thus is an important
task. Moreover, active participation of students in these efforts is crucial for full success. Involving
students, asking for their opinion and suggestions is unavoidable and highly relevant. Brookfield and
Preskill [Brookfield and Preskill 2012] suggested and interesting approach and method for helping
students to create their own ground rules. These suggestions also can represent a good starting point
in preparation of specific measures of virtual learning environments (based on Pedagogical Agents)
for students with disabilities, allowing them to assume a more active role towards increasing their
learning success.
Acknowledgement. This paper is a part of the Serbia-Romania-Poland collaboration within multilateral agreement on “Agent
systems and applications” and Romania-Poland collaboration within bilateral project “Semantic foundation of the Internet of
Things”.
REFERENCES
T. Alquraini, D. Dianne Gut. (2012). Critical Components of Successful Inclusion of Students with Severe Disabilities:
Literature Review. International Journal of Special Education, 27, 42-59.
C. D. D. Bowman (2012). Student use of animated pedagogical agents in a middle school science inquiry program. British
Journal of Educational Technology, 43(3), 359-375.
Stephen D. Brookfield, Stephen Preskill. (2012). Discussion as a Way of Teaching: Tools and Techniques for Democratic
Classrooms, John Wiley & Sons, 336 pages
X. Bui, C. Quirk, S.Almazan, M. Valenti. (2010). Inclusive education research and practice. Maryland Coalition for Inclusive
Education. Retrieved from http://www.mcie.org
Terry Byers. (2017). “Development of an Observation Metric for Linking Pedagogy, Technology and Space”, In Ben Cleveland,
Heather Mitcheltree, Wes Imms (Eds.), What's Working? Informing Education Theory, Design and Practice Through
Learning Environmental Evaluation, 77-87.
F. D. Davis (1993). User acceptance of information technology: system characteristics, userperceptions and behavioral impacts.
International journal of man-machinestudies,38(3), 475-487.
Farhan Muhammad, Jabbar Sohail, Aslam Muhammad, Hammoudeh Mohammad, AhmadMudassar, KhalidShehzad, Khan
Murad, Han Kijun. (2018). IoT-based students interaction framework using attention-scoring assessment in eLearning.
Future Generation Comp. Syst. 79: 909-919
Arthur C. Graesser, Haiying Li, and Carol Forsyth. (2014). Learning by Communicating in Natural Language With
Conversational Agents, Current Directions in Psychological Science , Vol. 23(5) 374–380
Mirjana Ivanović, Dejan Mitrović, Zoran Budimac, Ljubomir Jerinić, Costin Bădică (2015) HAPA: Harvester and Pedagogical
Agents in E-learning Environments, International Journal of Computers Communications & Control, Volume 10, Issue 2,
April, 2015, pp. 200-210. ISSN 1841-9836
Aleksandra Klašnja-Milićević, BobanVesin, Mirjana Ivanovic, Zoran Budimac. (2011). E-Learning personalization based on
6:12 · Mirjana Ivanović et al.
hybrid recommendation strategy and learning style identification. Computers & Education 56(3): 885-899
Aleksandra Klašnja-Milićević, BobanVesin, Mirjana Ivanovic, Zoran Budimac, Lakhmi C. Jain. (2017). E-Learning Systems -
Intelligent Techniques for Personalization. Intelligent Systems Reference Library 112, Springer, ISBN 978-3-319-41161-3,
pp. 3-294
Aleksandra Klašnja-Milićević, Zoran Marošan, Mirjana Ivanović, NinoslavaSavić, BobanVesin. (2018a).The Future of
Learning Multisensory Experiences: Visual, Audio, Smell and Taste Senses.MIS4TEL 2018: 213-221
Aleksandra Klašnja-Milićević , Mirjana Ivanović. (2018b). Learning Analytics - New Flavor and Benefits for Educational
Environments. Informatics in Education 17(2): 285-300
T. Lynch. I.Ghergulescu. (2017). REVIEW OF VIRTUAL LABS AS THE EMERGING TECHNOLOGIES FOR TEACHING
STEM SUBJECTS, 11th International Technology, Education and Development Conference –INTED2017 Proceedings, 6-
8 March, 2017, Spain, pp. 6082-6091.
T. Lynch, I. Ghergulescu. (2018). “Innovative pedagogies and personalisation in STEM education with NEWTON Atomic
Structure Virtual Lab,” presented at the World Conference on Educational Media and Technology (EdMedia 2018),
Amsterdam, The Netherlands.
C. M. Lorenzo, L. Lezcano, S. S.Alonso. (2013). Language Learning in EducationalVirtual Worlds-a TAM Based Assessment. J.
UCS, 19(11), 1615-1637.
T.M. Makoelle (2014). Pedagogy of Inclusion: A Quest for Inclusive Teaching and Learning, Mediterranean Journal of Social
Sciences, Vol 5 No 20, pp. 1259-1267
Marcin Paprzycki. (1992) See publications capturing various aspects of computer literacy, as understood in the 1992-1996
time period, available at: http://www.ibspan.waw.pl/~paprzyck/mp/cvr/education/complit.html
Marcin Paprzycki, Draga Vidakovic, Using Computers in Calculus Teaching, Journal of Computing in Small Colleges, Vol. 8,
No. 5, 1993, 34-45 (http://www.ibspan.waw.pl/~paprzyck/mp/cvr/education/papers/SCSCCC_93.pdf)
Baeta Patrícia, Pedro Neuza. (2018). Future Classrooms vs Regular Classrooms: comparative analysis of established
pedagogical dynamics, 'EDULEARN18-10th annual International Conference on Education and New Learning
Technologies', Palma de Mallorca (Spain).
Richard M. Perloff. (2016) The Dynamics of Persuasion: Communication and Attitudes in the Twenty-First Century,
Routledge.
J. Rickel, S. Marsella, J. Gratch, R. Hill, D. Traum, W. Swartout. (2002). Toward a New Generation of Virtual Humans for
Interactive Experiences. IEEE Intelligent Systems. 17(4). 32-38.
Maggi Savin-Baden. (2015). Rethinking Learning in an Age of Digital Fluency Is being Digitally Tethered a New Learning
Nexus? London: Routledge.
Maggi Savin-Baden, Roy Bhakta, Victoria Mason-Robbie, and David Burden. (2019).An Evaluation of the Effectiveness of
Using Pedagogical Agents for Teaching in Inclusive Ways, In: Jeremy Knox, Yuchen Wang, Michael Gallagher (Eds.),
Artificial Intelligence and Inclusive Education Perspectives on Rethinking and Reforming Education, 117-134
H. Tait, N. J. Entwistle, and V. McCune. (1998). ASSIST: a re-conceptualisation of the Approaches to Studying Inventory. In
C. Rust (Ed.), Improving students as learners (pp.262-271). Oxford: Oxford Brookes University.
Terracina Annalisa, Berta Riccardo, Bordini F., Damilano R., Mecella Massimo. (2016). Teaching STEM through a Role-
Playing Serious Game and Intelligent Pedagogical Agents, ICALT 2016: 148-152
Alyssa Friend Wise, Yuting Zhao, Simone Nicole Hausknecht. (2013).Learning Analytics for Online Discussions: A
Pedagogical Model for Intervention with Embedded and Extracted Analytics, LAK '13, Leuven, Belgium
Selinger Michelle, Sepulveda Ana, Buchan Jim. (2013)Education and the Internet of Everything, How Ubiquitous
Connectedness Can Help Transform Pedagogy,
https://www.cisco.com/c/dam/en_us/solutions/.../education/education_internet.pdf, Accessed June 2019.
Open Source Technology: Concepts, Methodologies, Tools, and Applications, Management Association, Information Resources,
IGI Global, Nov 30, 2014, 2100 pages
[Link1] https://nextcloud.com/fr_FR/blog/moodle-3.6-is-here-with-nextcloud-integration/, Accessed June 2019.
Maureen J. Lage, Glenn J. Platt, Michael Treglia. (2000). Inverting the Classroom: A Gateway to Creating an Inclusive
Learning Environment. The Journal of Economic Education 31(1), 30-43 https://doi.org/10.2307/1183338