E-learning systems are available since decades, they became widely accessible and ofered by most collages and universities. It has been recognized in recent years that the future of e-learning lies in crowd-sourced learning, where multiple users can contribute to the global knowledge base. [6, 15] However, in such systems, the question of quality arises. In the development of our e-learning system, one aim was to assure the high quality of the created contents. To achieve this goal, we have incorporated a workflow-engine, that is commonly used in business intelligence software, to manage business processes. We created a system that provides user interfaces for the definitions of state transitions and the corresponding user permissions. Thus, enabling versatile validation processes for the different types of materials, so that a newly created content can become an approved and verified learning object. When we are talking about e-learning quality, we must not forget that this includes the aspects of accessibility. When we enable users to create new content in our system, we want to be sure that it is accessible. On the other hand, we do not wish to put the burden of image description entirely on the users. Therefore, we have incorporated a pre-trained “Show and tell” model [24] to generate natural sentences describing an image, to annotate our image database. This not only improves the accessibility of the system, but also makes it possible for our Elasticsearch search engine to provide relevant search results, even when it comes to images. Deep learning emerged recently as a universal tool for machine learning tasks. The used “Show and tell” model is one of such, a combination of a convolutional neural network (CNN) and a long short-term memory (LSTM) network.
In this paper we present the architecture of our e-learning system, highlighting the concepts towards quality assurance.
E-learning systems has been around since decades. By today, they became the part of the everyday life of colleges and universities. It has started as the adaptation of the traditional distance learning model, such that online courses were organized in a standard way, following a specified curriculum in a predetermined pace. With the advances of web-technologies, new trends evolved in the field of e-learning as well. State of the art systems enable the user to conduct an individual learning procedure. Moreover, learners can communicate, collaborate and share content with each-other over the e-learning software.
In this study, we focus on a Learning Management System (LMS), more
precisely a Learning Content Management System (LCMS). An LMS is used for
delivering, tracking and managing training and education. An LCMS is more about the
creation of contents and their sharing. These can be hosted in an LMS system as
learning objects. It has been pointed out in the literature, that the future of such
content creation and management and the development of learning objects lies in
crowd-sourcing and machine learning. [
Crowd-sourcing makes it possible to utilize the intelligence and wisdom of large
groups of people in solving problems and creating and validating content. Despite
all the advantages, it raises new challenges when it comes to quality control. This
is due to the fact that the crowd can be composed of people with varying skills,
intellects and objectives [
We have recently entered the era of Big Data, which means that way more data
is produced than what can be processed by human operators. This is also true in the
case of LCMS applications. Machine learning ofers a wide range of tools that can
help in this matter, especially when it comes to automatically annotating data.
This not only aids the searchability of learning objects, but also increases their
accessibility. For instance, by helping visually impaired people better understand
the content of images. Inclusive education is also gaining attention nowadays and
has been addressed in the literature [
The goal of this paper is to propose an architectural design for an LCMS system that incorporates building blocks and methodologies responsible for the quality control and accessibility of learning objects.
In the past years, we developed a national e-learning platform called Ekernel in Hungary for elementary schools. In this software, besides the basic LMS functionalities, it is possible to create content in the form of online books, interactive exercises, uploaded media, etc. Furthermore, users can search for all types of contents and share them with each-other.
To enable search engines to find relevant information, we must focus our
attention on the annotation of our learning objects, even when they are in the form of
images, videos or soundtracks. This not only helps the search, but if done well,
helps the accessibility of our educational portal. Since recent advances in the field of
deep neural networks, the quality of automated image caption generation increased
considerably.[
As we opened towards the direction of crowd-sourced content creation, we soon realized that we need to set up a sophisticated protocol to publish the produced data. Our primary target audience is elementary school students, therefore it is of high importance to filter the content according to quality and relevance. We had observed the possibilities and found that we can take examples from the industry where quality control normally plays a crucial role. This lead to the decision to adapt a Workflow Management System (WfMS) [7] to our needs. Such a WfMS enables the description of validation stages, such as professional and linguistic proofreading, plagiarism check etc. In the Ekernel system, we do not only have standalone learning objects, but we allow the creation of lecture notes and complete books. In these cases a validation hierarchy is required, when the check of subsequent chapters and lessons is the prerequisite of the final publication. In the proceeding sections, we aim to give an overview of how a WfMS can be integrated in an LCMS architecture in a customizable way.
In designing the Ekernel system, we aim to give an easily understandable, accessible and intuitive user interface. We put high emphasis on the search capabilities of our system: the global search page is available from the main menu and in addition, we included additional search fields inside all pages of the interactive books.
To increase the user satisfaction with usability, all the created pages are responsive, thus all content is accessible from mobile devices. This is crucial, knowing that it is highly probable that the students will work on tablets or smartphones during class. Therefore, we expect that students will find the webpage easy to use and easy to understand. We expect that the teachers will be more confident in using this tool during course work, and will feel motivated to contribute to the global knowledge base with their own creations. We also suppose that both teachers and students will find the search functionalities easy to use and will be able to find relevant contents in our Ekernel system. We expect that the quantity and quality of learning objects will make the users more motivated to use our software.
In Fig. 1. the core architecture of our Ekernel system is illustrated. In designing
the system, we put emphasis on using the most recent technologies that came to
existence by the innovations of Big Data and semantic web. Due to the versatility
of data types used in an LCMS application, we decided to use a scalable noSQL
database besides our relational database, that can support large scale and
highconcurrency applications with heterogeneous data.[
To follow the trends of semantic web, we have incorporated an Elasticsearch
engine, that is a distributed, scalable, real-time search and analysis engine [
Text based data, and the metadata of all objects is stored in a PostgreSQL relational database.
As shown in Fig. 1. we created a service over our MongoDb, for image annotation.
This service is for two main purpose, one is two give keywords for an image, and
one is to describe it in a complete, natural language sentence. We have found
that several publications have addressed the problem of image captioning before
[
We chose to use two pre-trained model, one for keyword generation and one
for the whole sentence image captioning. For the former, we selected the
InceptionV3 model [
We created a service over our MongoDb that processes the uploaded images and
creates the captions for them. This data is stored in the PostgreSQL database, and
is indexed by the Elasticsearch engine in the background.
3.2. WfMS
A workflow management system (WfMS) is a software system that helps the setup,
execution and monitoring of processes and tasks. The goal of such a system is to
increase productivity, agility, improve the information exchange within an
organization and assure high quality.[
During the design of a workflow, one has to take into consideration the followings: 1. execution order of activities, 2. permission-based availability of activities, and 3. permission-based availability of information. In our scenario, we found that we need a rooting type system, that conducts the routing of the flow of information and documents.
The database architecture of the proposed highly configurable WfMS is presented in Fig. 3. We start with the creation of a meta table (meta.table info) describing all the existing tables in our PostgreSQL database and assigning id-s to them. We store the types of possible workflows in the workflow type table. Users with admin permissions can define separate workflows for documents, images, book chapters but even for the admission of students to an online course in the system. For each workflow type, one can create stations and define transitions connecting them. Such stations can be: under editing, under professional proofreading, rejected etc. Both stations and transitions can be constrained, such that only users in a specified permission group can access the data when it is in a certain state or can perform a transition to a new state. All of the executed transitions are stored in the database, so one can trace back the history of any item in the workflow queue.
In our system, we enable the users to visualize a specific the workflow as directed graph and configure them via an extensible user interface. Fig. 4 show parts of (a) State transition constraint matrix (b) Definition of sub-processes the user interfaces. One can define the possible states, then configure the possible transitions via a matrix representation (see Fig. 4a) and set the corresponding privileges. It is possible to create dependent workflows, when the state of one item defines the possible states and transitions of a related item. For instance, if we have a lesson that contains an interactive exercise, we can specify that the lesson can only move to a published state, when the contained exercises are published as well. This particular setup is shown in Fig. 4b.
The Ekernel system was first tested from January to June, 2019 by 310 Hungarian public schools. During this test, 446 teachers and 3706 fifth grade students used our Ekernel system for a period of 5 months. At the end of the test run, we ofered the participants to fill out a voluntary questionnaire that addressed their satisfaction with our created LCMS.
The basis of the construction of our questionnaire was the theoretical
framework of user experience. This research framework distinguishes between perceived
ergonomic quality, perceived hedonic quality and perceived attractiveness of a
product. [
In the context of the questionnaire it is necessary to consider both pragmatic
and hedonic aspects if we want to measure how satisfied users are with a our system.
It should allow the users to express feelings, impressions, and attitudes that arise
when using our product.[
The questionnaire consisted of 18 questions for both students and teachers for
our user experience assessment. Out of these, 12-12 used a 5 level, and 5-5 used
3-level Likert scale [
The questions in the case of teachers can be organized in 3 main categories: 1. satisfaction and confidence (e.g. “ How confident did you feel during teaching?’ ’) 2. perceived student satisfaction (e.g. “ Did the students understand the material better with the interactive books than with the paper ones? ” , 3. platform usability (e.g. “ Did you find what you were searching for easily?
Could you navigate where you wanted with ease? ”) The students had another set of questions that we grouped to the following 3 categories: 1. understanding (e.g. “ Did you understand the material better with the interactive book than with the paper books? ”) 2. motivation and mood, (e.g. “ Did you find the materials interesting and exciting? ”) 3. platform usability (e.g. “ Did you find what you were searching for easily? ”) (a) Teacher satisfaction
(b) Student satisfaction
To be able to filter out those who gave random answers, we included opposing questions as well, one in the 5-level part and its pair in the textual answer part.
Until the time of our final survey, our editorial community of 250 users has written and validated 83 books, containing 4 568 lessons. Furthermore, 13 490 interactive exercises were created and 147 612 images were uploaded to our servers.
Overall 41 teachers and 258 students participated in our anonymous survey. The answers of 8 participant were filtered out due to inconsistency. The results of the survey are presented in Fig. 7. It shows that 66% of the students claimed that they had a better understanding of the topics with the use of the online, validated materials, than with printed books. Furthermore, 74% of the students and 91% of the teachers answered to the platform usability related questions positively.
As a hypothesis, we stated that teachers will be motivated to participate in content creation. To the question ‘‘How much would you like to contribute to the global knowledge base with your own content? ” 59% of them responded positively.
In the textual questions part, we asked the participants whether they find this software better than the digital tools they have used so far. In this case, the possible answers were: “no”, “the same” and “yes”. 56% of the teachers and 61% of the students answered “yes”.
In the future, we plan to extend our editorial user-base. We would like to allow all the registered users to submit their creations (books, lesson plans, interactive exercises, etc.) for a reviewing process.
The image caption also leaves some things to be desired, e.g. higher classification accuracy has been reported in the literature recently. Fig. 8. shows to examples of complex scenes, where the annotations have low accuracy.
The currently used image annotating deep learning models can only operate on natural images, however our image database contains a great number of illustrations as well, thus, we continue our search for networks that can handle such data. Furthermore, we plan to use our image database to fine-tune a classification model. We would also like to continue our progress towards accessibility and create machine learning supported video sub-titling services.
The research was supported by the grant EFOP-3.6.1-16-2016-00001 (“Complex improvement of research capacities and services at Eszterhazy Karoly University)”. [7] Du, W., and Elmagarmid, A. Workflow management: State of the art vs. state of the products. HP LABORATORIES TECHNICAL REPORT HPL (1997).