=Paper=
{{Paper
|id=Vol-3265/paper_9861
|storemode=property
|title=Improving undergraduate students’ performance through a Situation Aware e-learning system
|pdfUrl=https://ceur-ws.org/Vol-3265/paper_9861.pdf
|volume=Vol-3265
|authors=Roberto Capone,Massimo De Falco,Mario Lepore
|dblpUrl=https://dblp.org/rec/conf/telexbe/CaponeFL22
}}
==Improving undergraduate students’ performance through a Situation Aware e-learning system==
https://ceur-ws.org/Vol-3265/paper_9861.pdf
Improving undergraduate students’ performance through a
Situation Aware e-learning system
Roberto Capone 1, Massimo De Falco2 and Mario Lepore 2
1
University of Bari “A. Moro”, E. Orabona street, Bari, 70125, Italy
2
University of Salerno, Giovanni Paolo II street, Salerno, 84042, Italy
Abstract
This paper focuses on an adaptive learning system based on the principles of Situation
Awareness and Goal Directed Task Analysis to improve students’ performance by increasing
awareness of their status as defined by the parameters of engagement, motivation, and
participation. A technique based on a Fuzzy Cognitive Map (FCM) has been defined to
identify the current situation by tracking the learner's behavior and interactions with the
system. The FCM drives the feedback generation process to improve the situation awareness
of the learner and their motivation, engagement, and participation. The system has been
evaluated using the Situation Awareness Global Assessment Technique, involving students
during the academic years ranging from 2017 to 2022. The experimental results demonstrate
that the system, thanks to the FCM, can significantly improve the situation awareness of the
learner, even in an emergency.
Keywords 1
Fuzzy cognitive map, situation awareness, e-learning
1. Introduction
There has always been a close interdependence between communication and education since the
educational activity is a relational and communicative one that takes place between subjects operating
in space and time [1]. Sharing space and time between educators and students has been a constant point
of reference in education. It is still part of the common feeling that education, in the true sense of the
term, should take place through physical presence even if thanks to the network and modern information
technologies, educational dialogues and distance learning are possible. In fact, online learning, known
as e-learning, refers to the use of multimedia technologies and the Internet to improve the quality of
learning by facilitating access to resources and services, as well as remote exchanges and remote
collaboration. There are many advantages to preferring online learning to traditional education [2]: the
learner no longer has to attend the classroom physically but can connect to e-learning platforms to learn
and update at any time of the day and any place. This particular mode of training has proved very useful
over time, and its flexibility is often adopted not only by students of all ages but also by companies and
professionals, the former to provide training courses for their employees, the latter to attend refresher
courses in which they are obliged to participate periodically.
Moreover, they offer high usability of courses, which have no limits of place and time, a high rate
of interactivity among learners (communications in forums, open discussions for each course, email),
containment of course costs, ease of distribution of teaching materials (including interactive ones, which
are often uploaded on the platform or sent by email) and increased personalization of training and
learning, designed based on each learner and finally, unlike traditional methods, distance courses are
just in time. However, there are some negative aspects to be considered in e-learning. In particular,
reference is made to the difficulty of empathic interaction between teacher and learner, drops in
concentration, lack of sociability with one's peers, and certainly the difficulty for those who are not very
familiar with technology. In fact, even if it seemed that society and the school world were ready for a
Proccedings of the Third Workshop on Technology Enhanced Learning Environments for Blended Education, June 10–11, 2022,
Foggia,taly
EMAIL: roberto.capone@uniba.it (A. 1); mdefalco@unisa.it (A. 2); marlepore@unisa.it (A. 3)
ORCID: 0000-0001-9454-8453 (A.1); 0000-0001-9858-9420 (A. 3)
©️ 2020 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
radical technological change, rejecting the old forms of knowledge transmission, and extending learning
environments to virtual ones, when the school system was challenged severely during the COVID-19
pandemic period, i.e., when it came to conducting a full-scale test of distance learning, the results were
not as hoped for [3].
All over the world, for the entire duration of suspension of teaching activities in the physical venues,
Distance-Learning (DL) methods supported by technologies were activated. The problem was inventing
a new pedagogy and a new way to organize didactics concretely, since people were deeply impacted at
the emotional and affective levels. As a matter of fact, students have experienced moments of strong
emotional stress, with the risk of generating a state of frustration and discouraging them from studying.
It was not just a question of using technologies in teaching but of an actual adaptive process to the
exclusive use of technologies as the only way to teach. The compulsory use of distance learning as the
only means of teaching in schools and university courses has brought a variety of doubts and questions
to be addressed; some of them were: An example of bulleted list is as follows.
• What didactic strategies should the teacher adopt to ensure students acquire the same skills as
face-to-face teaching?
• How can students' interests be kept alive? How do stimulate them to follow the lessons at a
distance?
• Which aspects of face-to-face teaching are compromised with distance learning?
Answering these and other questions was crucial for the teachers to avoid leaving the students adrift
in their learning process [4]. Specifically, many students are not uncommon to drop out of their studies
due to the first obstacles encountered. This becomes even more pronounced if we refer exclusively to
online courses. It is not rare to find many students who leave the online learning course shortly after
the beginning; such a phenomenon, called dropout, is always more frequent among students who are
not sufficiently engaged and motivated with the learning experience [5]. The root causes of students
dropping out are a lack of motivation, engagement, and participation [6, 7]. The motivation [8] takes
into account the level of interest in the course, the engagement represents the level of involvement in
the learning experience [9], whereas participation [10] refers to the action of taking part in activities
and projects, the act of sharing in the activities of a group. For these reasons, a modern course cannot
be limited to the simple learning content delivery task. Still, it should support the learners in their whole
learning experience, leading them to reach their learning objectives successfully. To do so, a learning
platform could be used. Still, it should be adaptive, providing each learner with the contents, feedback,
suggestions, and experience tailored to her current learning state [11] and not simply being a tool to
access training material for self-instruction. As a famous aphorism of Benjamin Franklin says: "Tell
me, and I forget, teach me, and I remember, involve me, and I learn". Creating an engaging and
interactive virtual environment is necessary to achieve the following benefits: to make the learners more
interested in the training, to keep their concentration level high and consequently to achieve the results
of the full learning process.
This paper is part of research started in 2017 by the authors on the use of educational technologies,
especially through a custom-developed e-learning platform, based on Situation Awareness principles,
capable of providing adaptive feedback to students, and innovative teaching methodologies in STEM
courses to support learners in their learning process. In previous works, the importance of using
educational technologies in the educational dialogue to prevent drop out improves students'
engagement, motivation, and participation linked to the Situation Awareness (SA) model, has been
highlighted [12, 13, 14]. In the academic year 2017/18, blended teaching was tested using the Just in
Time Teaching and Peer-Led Team Learning methodologies integrated with a social platform. In the
academic year 2018/19 [15], the experimentation went on using Augmented Reality to address some
crucial topics of the mathematics course and evaluating student interaction and participation with Fuzzy
Cognitive Maps as a systemic structure model for analyzing critical success factors of the learning
system. Augmented Reality has been used to overcome some of these difficulties, using some
technological tools (3D glasses, computers, tablets) and innovative methodologies. Furthermore, in the
same year, studies were conducted on how the adaptive e-learning platform and its feedback generation
system influenced the students' Situation Awareness level. During the academic year 2019/20, during
COVID-19 pandemic, the authors [16] analyzed the impact of completely distance learning on student
motivation, participation, engagement and performance. Several important findings emerged from this
series of work:
• motivation, engagement, and participation are three good indexes for understanding the level
of situation awareness of the learner; therefore, they represent a good learner situation model.
• A learner with a high level of SA is more conscious of the current signs of progress, difficulties,
and objectives and could make better decisions regarding her learning process. This has beneficial
consequences on their performance.
One of the main objectives of this work, in continuity with the previous ones, is to present in detail
the e-learning platform that was designed and implemented following the principles of Situation
Awareness and Goal Directed Task Analysis (GDTA) [17], how it has evolved to respond to changes
in the context (Covid-19 emergency) and the situation model that was used to describe the learner's
status.
Situation Awareness Global Assessment Technique (SAGAT) [17] was adopted to assess how the
e-learning system has impacted students' awareness of the situation over the years. Since it has been
explored previously [12], the relationships with the learner's motivation, engagement, and participation
levels. Based on blended learning, after the end of the Covid-19 emergency, the obtained results in the
academic year 2021/2022 are compared with those obtained in previous experiments in which the
courses were conducted in a blended mode in the academic year 2018/2019 and completely remote in
the academic years 2019/2020 and 2020/2021. The comparison was also made in parallel with a group
of users who did not use the platform with full functionalities oriented to Situation Awareness to
understand the impact of the proposed solution on the students during the years was considered for the
experimentation. The data collected seem to show that the use of the e-learning platform developed
according to the principles of situation awareness and GDTA allows students to have a greater
awareness of what is happening in their learning path than those who used a standard platform.
Specifically, these students have a fairly clear idea of their status according to the parameters of
engagement motivation and participation; they know how to use it to make decisions, which has
beneficial consequences on their performance, as discussed in other authors' works. On the other hand,
in the long run, distance learning, in addition to other social factors, has a negative impact on students’
situation awareness modeled through motivation, engagement, and participation, despite the countless
teaching strategies implemented and the use of an adaptive e-learning platform. This has also negatively
affected the level of competencies and has accentuated the drop-out phenomenon, but those aspects will
be covered in detail in another work.
2. Background knowledge
2.1. Situation Awareness
Situation Awareness (SA) is a faceted concept encompassing many different elements ranging from
cognitive mechanisms and decision-making processes to information processing and human factors.
Intuitively, SA means to understand what is happening around us in a specific moment in order to be
able to perform a correct action or make a coherent decision with respect to our goal. Consequently,
providing a universal definition of SA fitting for different contexts is not an easy task. One of the best
definitions is the one provided by Endsley [20]:
"Situation awareness is the perception of the elements in the environment within a volume of time
and space, the comprehension of their meaning, and the projection of their status in the near future".
In this definition, the three levels which concur to the formation of the SA can be identified:
perception, comprehension, and projection. The first level of SA (level 1 SA) is the perception of the
status of the elements in the environment. Although it may seem easy to perceive the elements related
to a specific task, it can be quite challenging in many application domains just to detect all the necessary
data. Moreover, the amount of data often exceeds the operator's ability to perceive it all correctly.
The second level to achieve good SA (level 2 SA) is understanding what the data perceived at
level 1 means in relation to goals. The data acquired at level 1 are synthesized and aggregated
so they can subsequently be assigned a meaning related to the objective to be achieved.
Understanding what the available data means requires good knowledge and a mental model to integrate
and interpret the different pieces of information. Level 3 SA means to predict what the perceived
elements will do in the future with respect to the goal. In order to have an adequate level 3 of situation
awareness it is essential to have a correct understanding of the situation (SA level 2), knowing the
system's dynamics and the environment. It is usually quite demanding, as it requires a good
understanding of the domain, the situation and a great ability to project the state of many elements into
the future. Experience plays an important role in this level because it gives the ability to anticipate
future situations and to be proactive with respect to them.
These three levels of Situation Awareness should be placed in the context of dynamic decision-
making. Starting from the well-defined and separate situation awareness process, it can be extended to
a broader dynamic decision-making model as defined by Endsley herself. The extended model is shown
in figure 1. According to this representation, the SA is the operator's internal model of the state of the
environment: the operator makes a decision based on what is happening in the environment (i.e. the
identified situation) and what he thinks will happen in the near future and then performs the necessary
actions. These actions will affect the environment, thus creating a new situation and leading to a new
cycle of situation awareness and decision. Beyond this, there is an important distinction to be made,
namely that between situational awareness understood as an internal model of the world, i.e., a state of
knowledge of what is happening at a given time, and the process of acquiring and maintaining
situational awareness by processing and understanding new information. We refer to the process of
acquiring situation awareness as situation assessment.
Figure 1. Endsley's Model of Situation Awareness in dynamic decision making
To support the process of SA formation, a proper design of the system and its graphic interface,
which explicitly took into account the human factors related to SA, should be needed. This aspect is
also fundamental for addressing the SA demons, which are a set of causes that lead to typical errors in
SA due to lack of situation awareness. Endsley defined in [19] the eight common causes for lack of SA,
called the Demons of Situation Awareness (SA demons). These eight causes are attentional tunneling,
data overload, complexity, memory trap, workload and stressors, wrong mental models, misplaced
salience, and out-of-the-loop. They occur as a result of individual factors as well as system or
environmental factors. Therefore, one of our main goals was to design and implement a system based
on Situation Awareness that could improve students and teachers' and teachers' SA levels by softening
the issues with SA demons.
2.2. Goal Directed Task Analysis
In some domains, the information an actor has to process to achieve the set objectives is beyond his
cognitive capacity. This is due to the high dynamism of the environment in which one operates, which
presupposes constant attention on the part of the user, who, therefore, will maintain a high level of
awareness with difficulty, thus threatening the achievement of the pre-set objectives. To support SA,
Endsley has developed a cognitive task analysis methodology that provides valuable support in system
design and promotes situational awareness, thereby improving decision making. This methodology is
known as GDTA, or Goal-Directed Task Analysis [17]; according to this technique, goals are a key
element from which to determine the key information needed to perform specific tasks and how these
should be used and combined to achieve good SA, but not only that, this technique also provides a
systematic approach to derive and analyze system requirements based on the goals the user wants to
achieve.
Thus, this methodology aims to obtain precise knowledge of the goals that users need to pursue and
the decisions to be implemented to achieve these goals.
Identifying goals is key to success for the correct system design, as it offers valuable help in
organizing information about it. In fact, this information will be shown to the user taking into account
the current goal to be achieved, thus limiting the problem of information overload: in fact, providing
only the necessary information at the right time and in the right way, it avoids unnecessary cognitive
overload for the user, allowing him to focus only on what matters with respect to the current task.
In order to define the information requirements and complete the decision-making process, the
GDTA approach uses different requirement elicitation techniques, such as interviews, ethnographies or
the detailed analysis of technical documentation.
The GDTA approach follows several steps, starting by identifying the actors actively involved in
the application domain of interest and then defining for each type of actor the goals to be achieved and
the most appropriate decisions to achieve them; at the end of these phases, it will be possible to obtain
all the information necessary to support the decision-making process.
The GDTA, therefore, consists of three main elements, first of all there are the goals, which represent
what the user wants to achieve in the function of a particular task that he has to perform; the goals can
be distinguished in overall goal, main goal and sub-goal, the overall goal represents the highest-level
goal which the user aims at, to this are associated main goals and sub-goals, preparatory to its
achievement.
The general scheme of the goal hierarchy is shown in figure 2, at the top of which is the overall goal,
which represents the user's maximum objective in using the platform, to then pass, through successive
refinements, to a decomposition of the latter: first into main goals and then, for each of them, into sub-
goals.
Figure 2: Overview of the goal hierarchy defined through GDTA
2.3. Fuzzy cognitive map
This section describes a soft computing technique for modeling and controlling systems: Fuzzy
Cognitive Map (FCM). This technique is one of the core points of our research, in fact it was used for
the situation identification technique we defined and implemented in our system. Specifically,
parameters related to interaction, participation and motivation, and their causal relationships, have been
identified and analysed through a Fuzzy Cognitive Map to describe the student's state during the course.
In addition to this analysis purpose, they were also used as a tool for generating feedback to the students
to try to maintain an adequate level of student engagement, participation and motivation and thus
mitigate the drop-out phenomenon.
A Fuzzy Cognitive Map (FCM), as introduced by Kosko [20], is a symbolic representation based on
a fuzzy graph useful for representing causal relationships. It can be used to describe complex
systems/environments symbolically, highlighting events, processes and states. An FCM consists of an
interconnection of nodes through weighted edges: a graph node is called concept and an edge is called
weight. The edge allows for implementing a causal relationship between two concepts, and the weight
represents the strength of the influence of the relationship, described with a fuzzy linguistic term (e.g.,
low, high, very high, etc.).
An FCM can be formalized through a 4-tuple (N, W, A, f), where:
1. N = {N1, N2, …, Nn} is the set of n concepts which are represented by the nodes of the graph.
2. W: (Ni, Nj) → wi,j is a function (NxN→[-1,1]) which associates the weight wi,j to the edge
between the pair of concepts (Ni , Nj).
3. Ni → Ai is the activation function which associates to each concept Ni a sequence of activation
values, one for each time instant t: ∀t, Ai(t) ∈ [0,1] is the activation value of the concept Ni at
time t.
4. A(0)∈[0,1] is the initial activation vector containing the initial values of all the concepts; A(t)
∈[0,1] is the state vector at a certain time instant t.
5. f: R → [0,1] is a transformation function with a recursive relation t≥0 between A(t+1) and
A(t):
𝑛
∀𝑖 ∈ {1, … , 𝑛}, 𝐴(𝑡 + 1) = 𝑓 ∑ 𝑤𝑗𝑖 𝐴𝑗 (𝑡) (1)
𝑖=1
( 𝑗≠𝑖 )
Different types of functions can be used as f (x), such as the sigmoid function, the bivalent function
or the linear function. FCM can be used to make a what-if inference, starting from a given initial
activation vector A(0), to understand what will happen next to the modelled system/environment.
Basically, a Fuzzy Cognitive Map is developed by integrating existing experience and knowledge
related to a system. This can be achieved by using a group of experts to describe the structure and
behavior of the system under different conditions. With FCM it is usually easy to find which factor
needs to be changed and, being dynamic modelling tools, the resolution of the system representation
can be increased by applying further mapping. According to Codara [21], FCMs can be used for various
purposes, including: underline the behavior of agents, understand the reasons for their decisions and
actions taken, highlighting any distortions and limits in their representation of the situation (explanatory
function). They can also be useful for predicting future decisions and actions (forecasting function) and
for helping decision-makers reflect on a given situation's representation (reflexive function).
Several applications of fuzzy cognitive maps in different domains, such as control, multiagent
systems, dynamical characteristics, learning procedures, etc., are realized to improve these systems'
performance [22]. The main reasons why FCMs are used are: easy to build and parameterize, easy to
use, flexible in representation, easily understandable even to non-expert users, convenient for managing
complex problems related to the processing and management of knowledge in a structured way,
convenient for managing the feedback structure of the modelled system with dynamic effects.
3. Situation identification model
One of the cornerstones of our research work is the analysis of the student's current status to support
her in her educational journey in the best possible way, preventing her from dropping out of the courses
and achieving her educational goals. In order to do this, it is necessary to be able to describe the state
the student is in formally. For this purpose, we used the theory of situation awareness, presented in
section 2.1, to define the student's status in the three high-level concepts of engagement, motivation,
and participation. The motivation [8] takes into account the level of interest in the course, the
engagement represents the level of involvement in the learning experience [9, 23], whereas participation
[10] refers to the action of taking part in activities and projects, the act of sharing in the activities of a
group. An in-depth discussion of these parameters and why they have been chosen to describe the
student's status can be found in the authors’ previous works [13, 24]. This section describes in detail
the situation awareness technique we defined and implemented based on a Fuzzy Cognitive Map
(FCM). The objective of the FCM is to consider all the effects that the variables identified in the Status
Model have on the learner's engagement, motivation, and participation, which are the three high-level
concepts representing the learner's current status.
Before choosing FCMs, an in-depth study of the scientific literature was conducted, evaluating the
different fuzz-type intelligence computing approaches proposed by the community to represent the
situation model (like, for instance, Fuzzy Inference Systems with if-then rules). With respect to other
fuzzy approaches, the use of FCM provides us with these advantages:
• FCMs are based on causal cognitive mapping, which provides an efficient way to elicit and
capture the knowledge of the experts of the domain and provide an intuitive way to represent
such knowledge which can be easily managed and updated by such experts Kokar and Endsley
[25];
• maps can be based on interviews, text analysis or group discussions and can be easily modified
or extended by adding new concepts and/or relations or changing the weights assigned to causal
links Kosko [18], Kokar and Endsley [25];
• FCMs have been extensively used as a way to support situation identification and decision
making, helping decision-makers in gaining a better understanding of the domain of the
situation and improving their mental models Jung and Lee [26];
• traditional FIS could require many rules to represent complex relations, especially when a high
number of inputs needs to be considered Abeer and Miri [27].
The designed FCM, shown in Fig. 3, resulted from a consensus process in which a team of four
education experts worked. Each expert, starting from the status model we have defined, has proposed
her FCM to identify the causal relationships and weights between the available concepts, such as a
process described in [26]. The weights are represented by seven linguistic terms: no impact = 0.00, very
low = 0.165, low = 0.335, medium = 0.50, almost high = 0.665, high = 0.835, very high = 1.00. Then,
we aggregate the different maps proposed by the experts to obtain one FCM. When some differences
arise between the relationships and weights proposed by the experts, we asked them to discuss these
differences and try to find an agreement until they achieve a sufficient degree of consensus.
The FCM can be considered as organized in different layers. The low layer contains the concepts of
the FCM representing the variables discussed in [13, 24], which are partially listed in Fig. 3 for legibility
reasons. The activation levels of these "leaves" concepts represent the value of each variable. When the
value of these variables changes (due to the actions performed by the student), the other concepts of the
FCM are influenced according to the causal relationships between them. The middle layer contains the
concepts composing the engagement-motivation (Interaction, Assignment and Forum Activities) and
participation (Emotion, Social Activity).
The final layer contains the concepts of Engagement, Motivation, and Participation representing the
current learner status. These concepts are influenced directly by the concepts of the middle layer and
indirectly by the concepts of the low layer.
Figure 1. Fuzzy Cognitive Map for situation identification
4. Situation-aware e-learning system
This project aims to adopt a motivational approach to support learning by creating an engaging
experience to reduce student dropouts. The main objective of this section is to present the e-learning
system we have worked on both in its design and development during the Research & Development
Project "MOLIERE". The system has been developed following the best design practices and principles
of Situation Awareness Theory [17] and exploiting the domain experts' knowledge (teachers and
educators). The overall design process that was adopted is shown in figure 4.
Figure 4. The three phases of the design process of a system based on Situation Awareness
The first step (SA requirements) involved studying the systems, techniques and approaches of SA,
which was useful in outlining the main areas of research in computer science in situational awareness,
to understand what are the most common categories of approaches that scholars and researchers usually
adopt to overcome the errors and demons of SA [19]. Thus, the most promising works were categorized
according to the underlying technique or approach they proposed or adopted (e.g., data mining, logic
and formal theory, machine learning, computational intelligence, etc.). Furthermore, considering the
functional view of the SA system, it was possible to classify each of the research areas identified in SA
with respect to its most adopted system functionalities. After this initial study, the appropriate
techniques and methodologies functional to the proposed system were chosen. Furthermore, GDTA
approach in this phase allowed a clear classification of the objectives the user should achieve; it started
by defining the high-level overall objectives, and then identified the related lower-level objectives. The
teacher, for example, has the general aim of making the learning experience productive, and to achieve
this objective, she will necessarily have to pursue other lower-level objectives, such as: carefully
planning the training course, involving the course participants in each phase, constantly monitoring the
results, etc. In the same way, the hierarchy of the student's goals is defined. The student's overall goal
is to successfully acquire the end-of-course certificate by passing the exam but to achieve this goal she
will have to: acquire all the course content, interact with the learning environment, constantly monitor
her learning path, etc.
Then, the development of the system was carried out according to the different stages of the Endsley
model, taking into account the opinion of domain experts and the users’ goals previously identified (SA
design). Some preliminary evaluations were necessary in order to understand if the chosen
methodologies were compatible, giving good comfort (SA measurement). In fact, some modifications
and improvements were made with respect to the first version of the system [16] in order to better
respond to the users' needs during the Covid-19 emergency period.
In order to better present the main features of this system, reference should be made to figure 5 in
which the final conceptual architecture of the system is sketched.
Figure 5. Conceptual architecture of situation aware learning system
The architecture was organized into tiered subsystems; from bottom to top, we have: i) Data
acquisition and preparation; ii) Data storage and management system; iii) Behaviour tracker; iv)
Situation identification module v) Learning Management system; vi) Recommendation module; vii)
Data analytics module viii) Presentation module.
At the lowest level of the system, directly in contact with the users, there are both real and virtual
sensors to record students' interactions with the e-learning platform and to monitor their facial
expressions used to identify the situation. Data acquisition and preparation deal with acquiring the raw
data provided by the sensors and analyzing and organizing them. Such data could then be elaborated
through some integration techniques and semantically enriched before being transferred to the Data
storage and management system. In this way, it is possible to generate new knowledge from the
aggregated data that is not evident from the individual data. Data storage and management system
includes relational databases and triple stores capable of storing the data and models that are used by
all the higher-level subsystems, which are: the Learning Management System (LMS), Situation
Identification Module, and the Behavior tracker.
The LMS, divided into E-Learning module, Collaborative Module and Communication System,
includes the modules necessary for the classic services of an e-learning system such as Course
Management, User Management, Blog, Forum, Chat, etc. Situation Identification Module is certainly
one of the most important modules of the system because it allows, through the use of the Fuzzy
Cognitive Map, the identification of the student's status in terms of motivation, engagement and
participation useful both to generate feedback and to provide the basis of information for in-depth
analysis by teachers in order to make important decisions, for example about the organization of the
course and teaching content. Whereas, the Behavior Tracker subsystem acquires data, observations and
models which are examined and processed in order to produce useful knowledge to derive user
behavior, through Formal Concept Analysis (FCA) processes and reasoning on semantic models.
Specifically, it is useful for extracting some behavioral patterns to identify dropout risks. To limit drop-
out, the recommendation module produces and forwards adaptive and personalized feedback to students
to help them maintain adequate levels of engagement, motivation and participation. The modules of
this subsystem are Feedback Generation, Feedback Representation and Feedback Delivery. The
feedback generation module generates feedback based on the learner's current situation identified by
the situation identification module with FCM. The process of feedback generation has been discussed
in detail in one of the authors’ papers [26, 27] but it is considered appropriate to point out what changes
have been implemented in this module to better cope with the emergency situation during the Covid-
19 period. The main motivation behind the changes that were made to this module are described in
authors' paper [12], which reports on the analysis of the students' situation in terms of engagement,
motivation and participation during the first year of lockdown. Specifically, what was observed in that
period was a clear decline in student participation due mainly to the lack of attention and frustration
caused by the impossibility of living a normal life. This could lead to a wide dropout. Therefore, it was
considered necessary to act mainly by sending personalized feedback related to the participation
parameter, through the Cognitive Emotion-Individual Emotion-Social Activity values, if this was below
a threshold. Otherwise, the generation process remains linked to the Engagement and Motivation
parameters, as done in the past. Another important change was to make the presence of the professor
marginal in the selection phase of the feedback, mainly of supervision, covering this role with an expert
system able to make the choice autonomously. It was considered appropriate to act in this way to lighten
the cognitive load of the professor during a period of enormous stress. The new feedback generation
and selection process is shown in figure 6.
Figure 6. The new feedback generation and selection process
Whereas, the Feedback representation module has the task of visually constructing the feedback
based on the target device, for example by appropriately choosing icons, colours, and text to enhance
its meaning and to facilitate the understanding. The third module, Feedback delivery, has the function
of transferring the feedback generated to the Presentation module, towards the device of the user (e.g.,
a mobile device, an augmented reality device, or a classic PC).
At high level, decision making requires appropriate and representative information and data to be
analyzed. Typically, more effective decisions can be made using a meaningful set of data, which has an
appropriate amount of information with respect to the intended goal. Data analytics module combines
data entry and manipulation capabilities with report production, graphical display and statistical
analysis facilities. The student's status and behavior, cross-referenced with lower-level data such as
answers to questionnaires, are used by this module. We have massively exploited this module during
the experimentation phases of our scientific work. The Presentation module is the subsystem that deals
with the display of information to users according to the models and principles of Situation Awareness
and Goal-Directed Task Analysis [17]. This module includes views for student and teacher dashboards,
which can be displayed according to the device used.
5. Evaluation
In the evaluations that have been done in other works of the authors [13, 14, 15, 16], it has been
verified that the situation identification technique, used in the process of feedback generation and based
on the Fuzzy Cognitive Map, is useful to increase the level of situation awareness (SA) of the learner.
Furthermore, it has been noticed that a learner with a high level of SA is more aware of current signs
of progress, difficulties, and goals and can make better decisions about his learning process and achieve
better results. Furthermore, it was verified that the level of motivation, engagement, and participation
are three good indices to understand the level of the learner's situation awareness and therefore represent
a good model of the learner's situation. So, the purpose of the evaluation of this paper is to assess for
four academic years, i.e., from 2018/2019 to 2021/2022, the impact of the proposed e-learning platform
on students' Situation Awareness, comparing the results with those obtained from the use of a standard
platform not oriented to Situation Awareness.
5.1. Method
The experimentation involved the Calculus II students in the first year of Mechanical Engineering
and Management Engineering at the University of Salerno. The experimentation results are obtained by
comparing the collected data in the academic years 2018/2019, 2019/2020, 2020/2021, and 2021/2022.
It should be noted that in the academic years 2018/2019 and 2021/2022 it has been used a, blended
learning (students attended classroom lessons and used the reference e-learning platform as an
integration), whereas in the academic years 2019/2020 and 2020/2021, a full distance learning was used
(students followed the course completely online). After the first year of distance learning in which
professors suddenly had to change their teaching due to the sudden closure of the university due to the
pandemic, classes in the 2020/2021 academic year were also conducted completely online. During
2019/2020, one of the teacher's challenges was quickly adapting the didactic content plan with
completely remote teaching. In order to motivate students, encourage participation and engagement,
keep students' degree of attention alive, and stimulate their epistemic curiosity, interest, optimism, and
passion, it was necessary to readjust the disciplinary methodological knowledge and teaching contents
through the use of the proposed e-learning platform. Teachers and students were left disoriented due to
the extended pandemic emergency and still forced into completely distance learning for the second year
in a row. Once again, teachers were faced with the challenge of maintaining high levels of engagement,
motivation, and participation, thus promoting students' educational success. In fact, students during the
first year of the pandemic have accepted, even with difficulty, completely distance learning as a means
to ensure their education.
The four classes, one per year, in which the experimentation was conducted were made up of 131,
112, 98, and 104 students, respectively. Cochran's formula was used to calculate the sample size for the
experimentation:
𝑍 2 𝑝𝑞
𝑛0 = 2
𝑒 (2)
Where: e is the desired level of precision (i.e., the margin of error); p is the (estimated) proportion
of the population with the attribute in question; q is 1 – p; the z-value is found in a Z table. It is s the
abscissa of the normal curve cuts off an area α at the tails (1 - α equals the desired confidence level,
e.g., 95%); n0 is the sample size.
In our experimentation the chosen parameters were: {Z = 1.65; p = 0.75; e = 0.15} that led to a
sample of 30 people.
In order to conduct the experimentation, the SAGAT methodology was adopted [17]. Specifically,
it was useful to assess how the proposed e-learning system impacted the student's situation awareness
over the years. SAGAT relies on the knowledge of domain experts to develop a questionnaire to assess
the users' level of situation awareness. The user is involved in simulations of one or more realistic
scenarios with the implemented system. At some point, the simulation freezes, according to SAGAT
guidelines, and a series of questions are asked to the user to probe SA. The questions are chosen to
assess the degree of awareness in the three levels of perception, comprehension, and projection. The
two scenarios identified in the first authors' experimentation [16] were re-used in order to perform a
comparative analysis with previous years. These scenarios were simulated using the adaptive learning
system. The participants in this experiment were students who would have to figure out what the next
activity should be to improve their learning processes and achieve their learning goals. The questions
asked to assess perception (Level 1 SA) in the two scenarios are related to identifying specific items or
parameters. For example, the student should identify the completion percentage of course activities. To
test level 2 SA (comprehension), questions were asked about the student's status to be assessed through
the activities performed and the results obtained. Finally, for Level 3 SA (projection), the questions ask
what action should be taken: the student should choose the next activity. Each scenario was run two
times in random order to test two different system modes. In the first mode, the system does not have
the situation-aware elements such as feedback, i.e., It lacks the notification section and the widget with
the list of received feedback; in the second, it has full functionality developed following situation
awareness principles; for example, it provides students with feedback using the FCM approach. In this
way, by comparing the difference in the percentage of correct answers given by the participants, it was
possible to understand whether the proposed system is useful for increasing situation awareness, even
in a changing environment such as that experienced during the Covid-19 pandemic.
5.2. Results and discussion
This subsection reports the results, specifying the percentage of correct answers the evaluation
participants gave using the SAGAT methodology. Specifically, Figure 6 shows the levels of Situation
Awareness of the users obtained in the four years ranging from the academic year 2018/2019 to the
academic year 2021/2022, considering both scenarios and compared with each other. This comparison
was made considering that for the years 2020/2021 and 2021/2022, the results obtained by users who
used the system with feedback in the updated version were used in contrast with the other two years in
which the previous version of the system was used. The figure shows the average rate of correct answers
given by participants.
Figure 7. Comparative results of SAGAT evaluation for academic years 2018/2019, 2019/2020,
2020/2021, and 2021/2022
In detail, the results obtained from comparing the levels of Situation Awareness recorded in the four
years under observation are shown in Figure 6 (in blue 2018/2019, in orange 2019/2020, in grey
2019/2020, and yellow 2021/2022), reveal how they went down during the two years of Covid-19
pandemic both for users who used the system in a standard version and for users who used the system
with full functionalities. Situation Awareness values in all three levels (level 1-perception, level 2-
comprehension, level 3-projection) and overall peaked in the 2018/2019 year when teaching was held
in person. The first significant decay occurs in 2019/2020, i.e., the first of completely distance teaching
due to the emergence of Covid-19. The declining trend is also confirmed in 2020/2021, which seems
roughly linear. On the one hand, there is a certain amount of gratification because the updated system
has been able to contrast the inexorable decline in the levels of Situation Awareness. The changes made
have made it possible to obtain, in the second consecutive year of completely distance learning, a total
level of SA equal to 72%, which is acceptable if we consider that under the same conditions, the system
without SA elements went down to 54%.
On the other hand, there is a good deal of concern because, despite the efforts made and the targeted
changes implemented according to the principles of Situation Awareness design, the values in the three
levels still decreased. One plausible reason for this phenomenon is that context played a key role. The
anxieties, stress, and lack of leading a normal life emphasized by the absence of sociability and distance
from their colleagues combined with an exaggerated cognitive load of online information have taken
over the students' concentration, attention, and a clear head. Although the system was objectively valid,
it was not as effective as in a condition of normality and tranquility as seen in the 2018/2019 year. An
encouraging revival occurred in the 2021/2022 academic year when lectures returned to face-to-face
mode. The platform was a complementary tool to teach. In fact, the results obtained are high and
comparable with those of the 2018/2019 year, which is the best one among the fourth years analyzed:
the return to normality, the reduction of the cognitive load, and above all, the regain in social interaction
have played a fundamental role.
Finally, two of the most important results of this research are:
• the use of the e-learning platform developed according to the principles of situation awareness
and GDTA allows students to have a greater awareness of what is happening in their learning path
than those who used a standard platform. Specifically, these students have a fairly clear idea of their
status according to the parameters of engagement motivation and participation; they know how to
use it to make decisions, which has beneficial consequences on their performance, as discussed in
other authors' works.
• Technologies are a tool to support teaching action but cannot completely replace the social
action of face-to-face teaching. Although teaching activities have been remodeled and technologies
have been used adaptively to optimize teaching, cognitive processes have been influenced by
external factors that have partly compromised the effectiveness of the educational action. The
authors hope that some aspects appreciated by the students can be integrated with traditional face-
to-face teaching in the future.
6. Conclusions
An adaptive e-learning system based on situation awareness has been discussed. The system has
been designed and developed according to the design principles of SA and GDTA. To improve the
awareness of the students' defined through their motivation, engagement, and participation levels,
adaptive feedback is sent to learners. The feedback selection process is driven by a Fuzzy Cognitive
Map, implemented to identify the learner's situation by analyzing her activities on the platform.
Furthermore, this work conducted a comparative analysis of the Situation Awareness of undergraduate
students in the academic years 2018/2019, 2019/2020, 2020/2021 and 2021/2022. In 2018/2019 and
2021/2022, a blended type of teaching was adopted (in-person lectures and support of supplementary
activities through an e-learning platform), while in the other two years, a completely distance teaching
due to the COVID-19 pandemic. In the context of the emergency of COVID-19, students have
experienced moments of strong emotional stress, with the risk of generating a state of frustration and
discouraging them from studying. The use of the proposed adaptive e-learning system based on situation
awareness and remodeled teaching has been essential in limiting this phenomenon. The experimentation
was conducted using the SAGAT methodology, involving students participating in classes during the
courses held over the past four years. The results show that the situation identification technique and
the situation model can increase the level of situation awareness of the students, even in an emergency.
On the other hand, however, there has been a significant decline in Situation Awareness in pandemic
years due to the marked presence of some of the eight demons identified by Endsley: the anxieties,
stress, lack of leading a normal life emphasized by the absence of sociability and distance from their
colleagues combined with an exaggerated cognitive load of information available only online have
taken over the concentration, attention and clear head of the students. Technologies are a tool to support
teaching action but cannot completely replace the social action of face-to-face teaching. Although
teaching activities have been remodeled and technologies have been used in an adaptive way to optimize
teaching, cognitive processes have been influenced by external factors that have partly compromised
the effectiveness of the educational action. This work is part of research conducted by the authors over
the years; other findings related to the impact of the e-learning platform on engagement, motivation,
participation, drop-out, and students' skills will be reported in other works.
7. Acknowledgments
This research was supported in part by the Italian Ministry of Economic Development (MISE) under
the Project “MOLIERE (MOtivational Learning and Interactive Education Revolution)” – PON I&C
2014-2020. The authors would like to thank Italdata S.p.A, the CORISA, and the DISA-MIS of the
University of Salerno, who participated in the project, for the interesting discussions and suggestions.
8. References
[1] Nichols, M. (2003). A theory for eLearning. Journal of Educational Technology & Society, 6(2),
1-10.
[2] Cross, J. (2004). An informal history of eLearning. On the Horizon, 12(3), 103-110.
[3] Müller, A. M., Goh, C., Lim, L. Z., & Gao, X. (2021). Covid-19 emergency elearning and
beyond: experiences and perspectives of university educators. Education Sciences, 11(1), 19.
[4] M Alhumaid, K., Ali, S., Waheed, A., Zahid, E., & Habes, M. (2020). COVID-19 & Elearning:
Perceptions & Attitudes Of Teachers Towards E-Learning Acceptancein The Developing
Countries. Multicultural Education, 6(2), 100-115.
[5] Y. Levy, "Comparing dropouts and persistence in e-Learning courses", Computers & Education.
48. 185-204, 2007.
[6] J. Keller, K. Suzuki, "Learner motivation and E-learning design: A multinationally validated
process", Learning, Media and Technology. 29. 229-239, 2004.
[7] R. Carini, G. Kuh, S. Klein, "Student Engagement and Student Learning: Testing the Linkages",
Research in Higher Education. 47. 1-32, 2006.
[8] Lepper, M. R. (1988). Motivational considerations in the study of instruction. Cognition and
Instruction, 5(4), 289–309.
[9] Kuh, G. D. (2003). What were learning about student engagement from NSSE: Benchmarks for
effective educational practices. Change: The Magazine of Higher Learning, 35(2), 24–32.
[10] Dominguez, R. G. (2012). Participatory learning. Encyclopedia of the Sciences of Learning,
2556–2560.
[11] F. Gasparetti, A. Micarelli, G. Sansonetti, "Exploiting web browsing activities for user needs
identification" in Proceedings - 2014 Int Conf on Computational Science and Computational
Intelligence, pp. 86-89, 2014.
[12] Capone, R., & Lepore, M. (2021). From Distance Learning to Integrated Digital Learning: A
Fuzzy Cognitive Analysis Focused on Engagement, Motivation, and Participation During
COVID-19 Pandemic. Technology, Knowledge and Learning, 1-31.
[13] Capone, R., & Lepore, M. (2020). Augmented Reality to Increase Interaction and Participation:
A Case Study of Undergraduate Students in Mathematics Class. In International Conference on
Augmented Reality, Virtual Reality and Computer Graphics (pp. 185-204). Springer, Cham.
[14] D’Aniello, G., De Falco, M., Gaeta, M., & Lepore, M. (2020). A situation-aware learning system
based on fuzzy cognitive maps to increase learner motivation and engagement. In: Proceedings
of the 2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) (pp. 1–8).
[15] Capone, R. (2022). Blended Learning and Student-centered Active Learning Environment: a
Case Study with STEM Undergraduate Students. Can. J. Sci. Math. Techn. Educ. 22, 210–236
https://doi.org/10.1007/s42330-022-00195-5
[16] Branchetti, L., Capone, R., & Rossi, M. L. (2021). Distance – Learning Goes Viral: Redefining
the Teaching Boundaries in the Transformative Pedagogy Perspective. Journal of E-Learning
and Knowledge Society, 17(2), 32-44. https://doi.org/10.20368/1971-8829/1135418
[17] M. Endsley, “Designing for situation awareness: An approach to user- centered design”, CRC
press, 2016.
[18] M.Endsley,“Toward a theory of situation awareness in dynamic systems: Situation awareness”,
Human factors. 37. 32-64, 1995.
[19] M.R.Endsley, "A taxonomy of situation awareness errors," Human factors in aviation
operations,vol.3,no.2,pp.287–292,1995.
[20] B.Kosko,“Fuzzy cognitive maps,”International journal of man-machine studies, vol. 24, no. 1,
pp. 65–75, 1986.
[21] Codara, L. (1998). Le mappe cognitive: uno strumento di analisi per la ricerca sociale e per
l’intervento organizzativo. Carocci
[22] Aguilar, J. (2005). A survey about fuzzy cognitive maps papers. International journal of
computational cognition, 3(2), 27-33.
[23] Y. Xiong, H. Li, M. Kornhaber, H. K. Suen, B. Pursel, D. Goins, , "Examining the Relations
among Student Motivation, Engagement, and Retention in a MOOC: A Structural Equation
Modeling Approach". Global Education Review. 2. 23-33, 2015.
[24] D'Aniello, G., de Falco, M., Gaeta, M., & Lepore, M. (2020). Feedback generation using Fuzzy
Cognitive Maps to reduce dropout in situation-aware e-Learning systems. In 2020 IEEE
Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA) (pp.
195-199). IEEE.
[25] M. M. Kokar and M. R. Endsley, "Situation Awareness and Cognitive Modeling," in IEEE
Intelligent Systems, vol. 27, no. 3, pp. 91-96, 2012.
[26] Jung, Y., & Lee, J. (2018). Learning engagement and persistence in massive open online courses
(MOOCS). Computers and Education, 122, 9–2
[27] Abeer, W., & Miri, B. (2014). Students’ preferences and views about learning in a
MOOC. Procedia-Social and Behavioral Sciences, 152, 318-323.
[28] D’Aniello, G., Gaeta, M., Lepore, M., & Perone, M. (2021). Knowledge-driven fuzzy consensus
model for team formation. Expert Systems with Applications, 184, 115522.
[29] Lepore, M., & Petruzziello, A. (2021). zA Situation-Aware DSS to Support Assisted
Reproductive Technology Outcome Prediction. In 2021 IEEE Conference on Cognitive and
Computational Aspects of Situation Management (CogSIMA)(pp. 103-107). IEEE.