This paper presents the design and implementation of a Multidomain Learning Environment that integrates and interconnects different heterogeneous educational applications. The learning environment has three main aims: 1) to interconnect web-based educational applications regardless of their development programming language, operating system and physical location; 2) to allow students access to the set of educational applications by using a unique and secure user account; and 3) to share the students' data generated by each application, in order to be used as an integrated learning analytic approach. The implemented system is both robust and scalable; based on federated identity architecture enables institutional nodes to share single educational applications, allowing their users to obtain access to the whole applications network using the same identification data. We extended the conventional implementation of identity federation by allowing the capture, integration and analysis of students learning traces from different sources.
Computer-based educational systems are intended to support the further development of the learning process and address the need to personalize instruction in a massive way. Educational software is supporting a diversity of domains at all the educational levels. However, there is still a lack of effective integration or communication among those systems, triggering three main problems: 1) single users with multiple systems accounts; 2) redundancy and inconsistency of users’ information; and 3) minimal or Copyright © 2017 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. null use of data generated by the same user while using different applications (e.g. access time preference, hints inclination, typing speed, cognitive level, and so forth).
Nowadays, there exist some ways to avoid the use of multiple user accounts, by
implementing authentication protocols such as Open standard for Authorization
(OAuth) [
There are a number of web-based educational systems and learning platforms (e.g.
Moodle, Open edX, Blackboar, etc.) available for students and teachers. Through
these applications, a wide variety of learning activities are conducted by students, and
educators can analyze students learning performance and behavior [
This paper presents the design and implementation of a Multidomain Learning Environment (MDLE). First, we describe related work and the use of the SAML authentication standard, as a useful and effective tool to integrate and interconnect heterogeneous educational software systems. Then, we show the learning analytics conducted by different application within MDLE. Finally, we discuss how the analysis of data about learners can be enhanced by implementing a multidomain learning analytic approach. 2
Efforts to simplify the use of multiple computer-based systems are intended to reduce the overwhelming task to manipulate different user accounts and passwords. At the same time, users, in their own benefit, may require the integration of data generated from multiples applications; over the complete set of communication, entertainment or educational systems they use.
Nowadays, the variety of web applications and social networking tools has
fomented the use of authentication protocols, such as SSO, SAML and OAuth [
Even though it is possible to share users’ data between interconnected applications,
there is not enough research work describing the use of this capability to implement
learning analytics functionalities. Dyckhoff and colleagues [
A similar work is presented by Brusilovsky [10]. In it, the author describes the implementation of KnowledgeTree, an adaptive e-Learning architecture that integrates distributed servers hosting educational services. This educational environment, by using a SSO approach, allows users to authenticate through a unique service named learning portal. Similar to our work, KnowledgeTree includes three additional servers: activity servers, which include reusable educational content and services; student model server, representing the current competencies and needs of students in order to adapt instructional materials available in several courses; and finally, the valueadding service, used to add a higher level adaptive functionality, such as content integration and sequencing. The author considers that multiple instances of KnowledgeTree can collaborate and interchange students’ data with each other, however, there is no specific functionality intended to conduct learning analytics.
In 2010, Arnold [11] conducted an analysis emphasizing how academic analytics, at institutional level, have the potential to improve students’ performance. By institutional scope they refer to data from different educational systems. More recently, the Report on Building the Field of Learning Analytics for Personalized Learning at Scale, published in 2014 by The Learning Analytics Workgroup, in its section on priorities for research, describes three grand challenges as main focus areas for early research in LA. The third grand challenge refers to creating multimodal learning analytics by “Expanding education to capture contextual features of learning environments…” [12, pp. 45]. Particularly, this challenge emphasizes the importance of developing SSO infrastructure to integrate data from heterogeneous platforms with multimodal data sources. In 2016, the LAK community organized the Cross-LAK workshop, aimed at encouraging participants to explore blended learning by researching and implementing LA across physical and digital spaces. Being the first theme addressed: learning analytics across digital spaces, where the main objective was to discuss about applications of learning analytics using multiple educational learning environments to facilitate learning activities [13].
In this work we describe the implementation of an educational environment using the enterprise federation protocol supported by SAML. Emphasizing, the importance of users’ behavior and performance data sharing, such as self-esteem, learning preferences, study habits and cognitive level, in order to provide a more comprehensive learning analytics functionality. 3
This section details the design and implementation of the MLE system, as well as educational services currently available in the environment. We start describing the architecture that supports the integration and communication within the proposed environment. Then, we present the mechanics of the interaction process and the learning analytics features of two educational applications integrated within the environment. 3.1
We implemented and configured a Federated Identity (FI) architecture that allows secure access to the integrated MDLE system, and serves as a repository of educational computer-based applications. The implemented architecture is both robust and scalable environment that allows the integration of heterogeneous applications in relation to its development environment and format of educational material. Based on a Full Mesh Federation (FMF) principle, applications could be hosted by an academic institution and users from other institution can benefit from such applications by establishing a circle of trust.
Full mesh federation is one of the most common and frequently used federation architecture. Also, the FMF principle is the simpler to implement, since federation activity is distributed and there is no need for a central hub or component that requires to be specifically protected by administrators. Instead, in this category of federation, the responsibility of users’ administration is distributed across the different nodes. In this work, in order to obtain an independent and scalable environment, the FMF architecture was chosen to implement our federated architecture.
In Figure 1, we can observe the implemented architecture for the MDLE system. Three academic institutions (UABC, UCol and IPN), current nodes participating in this federated network, exemplify the interconnectivity of the environment. Two of these nodes (UABC and UCol) were configured with their own identity Provider (idP), connected to a local database storing information about its own users, and an arbitrary number of Service Providers (SP) were installed. The idP is in charge of the access control to the MDLE and the SP manage specific educational applications. The role of the IPN node is only as consumer of the available federated services.
All these entities (idP and SP), are typically listed in a SAML metadata file, which is consumed by all of them. The metadata file basically describes all the shared services and information available in the environment. In the federation, each institution decides which services would be shared within the circle of trust, and also determines which attributes of the users are going to be available, such as name, age, email, among others. Using the circle of trust, configured among the environment nodes, information about the academic performance of students could be shared between the educational applications. Specifically, we have a database containing information about the educational background of users. This information is obtained, firstly, from the Preliminary Evaluation system, based on a set of diagnostic instruments (hosted in the UCol service provider); as is explained later. Then, this information, optionally, can be used or complemented by any other system on the network. Our aim is to use this shared database to conduct more comprehensive learning analytics functionality and personalize instructional content.
Regarding the entire MDLE functionality, there are three different user profiles: administrator, teachers and students. Administrator can manage (accept, activate and deactivate) user accounts. Teachers manage the content and learning activities and each of them is able to view his/her students’ progress. Finally, students are able to attend their learning activities and view their generated learning analytics information. Figure 2a shows the registration or login webpage, where users can choose which identity provider they want to use, based on their educational institution. In Figure 2b we can see six different services available within the MDLE environment, including the Preliminary Evaluation (Diagnostic) and the PreMath systems described below. First experience with the MDLE system takes place using the Preliminary Evaluation system. This module is intended to identify educational gaps of freshman students in the fields of mathematics and reading and comprehension, as well as evaluates aspects regarding study habits and self-esteem; which are themes considered significant on the success of the education of university students [14].
This system, through the use of specific assessments, consisting of multiple choice questions, evaluates students’ knowledge and academic behavior (see Figure 3). Students are asked to answer all of the assessments in order to gain access to the rest of the educational services available in the MDLE environment. The information obtained is processed and used as a starting point for the assignment of activities to each student; based on his/her specific knowledge gaps and learning habits. Detailed analysis can be conducted locally on the data generated. In addition, this data can be used to complement learning analytics of other educational systems in the MDLE. Regarding the technical characteristics of the system, this computer application was implemented using the .NET Framework developed by Microsoft. Particularly, the system was built using the Visual C# language and the SQL Server database management system for the data layer. 3.3
Another computer-based educational application, integrated within the MDLE environment, is an Intelligent Tutoring System (ITS) supporting high-school and university students to enhance the understanding of mathematics. This system is intended to help freshman students to address gaps in their math current knowledge. The Preuniversity Mathematics system (PreMath system), embedded in a Moodle environment, includes a set of instructional content (instruction and practice), considering a set of 20 math topics such as: multiply and divide monomials, monomial with an integer exponent, fractions, decimals, percentages and other topics learned in previous educational levels (see Figure 4). The PreMath system aims to reduce the university failing grades and drop-outs rates. The set of topics included in PreMath was defined by a group of university math professors, which were invited to make specific inputs about the most complicated math topics causing difficulties for students to succeed in their freshman year courses [15].
When the students interact with the system, they are provided with theoretical content (see Figure 5a), and then requested to solve a minimum of three math exercises for each topic. Each topic consists of 50 exercises with different level of complexity. The system provides feedback in a proactive (when a student commits a mistake) and reactive (under student request) way (see Figure 5b). The inputs of students and feedback provided by the system are used to conduct learning analytics; providing information on the performance of students and quality of the educational content. Details about the PreMath learning analytics features are described in the next section.
As an example of the heterogeneity supported by the implemented MDLE environment, different to the Preliminary Evaluation system, the intelligent tutor core of PreMath was made using the PHP language, XML and MySQL. In addition, the instructional content was built using the Adobe Flash platform. 4 4.1
Considering the four key elements included in the learning analytics definition: measurement, collection, analysis and reporting of data about learners [16], the preliminary evaluation module mainly works as an early alert system for the whole MDLE environment. First, by using the domain specific assessments (see Figure 6a) the system evaluates, collects and stores the students’ data. Second, the system measures students’ background knowledge and learning behavior. This information is stored in the MDLE shared database, and is available to be used for any other application on the network. Then, a deep analysis is conducted to determine those students that require leveling courses, and in what particular domains. Based on this information, the MDLE core runs a process by which the students are referred to use the rest of the available educational modules in the environment (math, reading and comprehension, study habits and self-esteem). Finally, this module generates and displays information about students’ performance, as described next.
In order to provide adequate visual analytics for students and teachers, it was considered the four user-interface design criteria used in [17]: Adding a simple interface for the learning analytics visualization charts.
The student view is intended to provide a graphical representation of their current knowledge (for all the considered domains), that motivate and enable them to make flexible and adequate decisions about what material they want to review. The level of academic performance of students is displayed by using color-code. As shown in Figure 6b, the green color indicates a good or very good student performance. Average or acceptable result is depicted in yellow color, domain intervention is recommended in this case. Finally, mandatory domain intervention is displayed in red, in order to ensure that students that have difficulties receive the help they need before the regular courses start.
For each particular domain, topics performance is presented in specific detail by using the same color-code. Figure 6c exemplifies the six topics used for determining the student performance regarding the study habits domain. This figure illustrates the scenario in which the student performed properly in three topics (peer-social relationships, study motivation and time management) and acceptable in the other three (concentration, memory and test taking strategies). This type of graph is used as a guide that may be used in helping the student select appropriate support. At the same time, the core system of the preliminary evaluation application uses this information to determine and habilitate the specific sections in which the students need additional assistance; turning the topics name into links to the precise instruction section that could be available as a service provided by a different system hosted in a different institutional node.
Since the MDLE has the capability to assign specific learning activities to students, the teacher view is mainly intended to provide both individual and group view of knowledge and academic behavior. An example of the group information a teacher can see is presented in Figure 7. In this case, this graph corresponds to the self-esteem evaluation of a group of about 100 freshman students that used the system last summer. Observations on the upper side are students identified with a very good selfesteem level. On the other hand, those examinees on the lower side of the graph were identified as students with a low self-esteem. This is interesting information that can be used by teachers to pay special attention to some specific students. At the same time, this information can be used by other systems within the MDLE environment, to automatically implement motivational strategies to encourage the participation of students with low self-esteem. The Mathematics module is responsible for characterizing the student knowledge level and the different exercises complexity using a series of mathematical formulas based on the Item Response Theory (IRT) model [18]. Then, both the student and the teacher are offered the possibility of graphically accessing the information generated. The exercises are categorized by their difficulty. The complexity of an exercise will be greater when the student needs more capacity to solve it [19]. This parameter is calculated using the following formula: _ = _ _ ___ __ℎ_ = = _ _ = 1+exepx(p −(−))
Students are characterized by their skill, efficiency and likelihood of success in an exercise. The skill is the ability of a student to solve an exercise successfully and efficiency is the ability of a student to solve an exercise successfully in the shortest possible time. The efficiency and the skill are calculated as shown in formulas below: (1) (2) (3) (4)
For the likelihood of success in an exercise the IRT model is used. The IRT is a probability that will tell us what possibilities a student has to successfully complete a specific exercise [18]. This information will be very useful to know what exercises should be shown to a student and increasing their difficulty as long as the student's skill is growing. For the calculation of the IRT the Rasch model was used, where the probability is defined as shown in formula 4 (where θ represents the student’s skill and β is the level of difficulty of the exercise) [20]:
In order to display the information visually, a web module has been developed where users can navigate between the different tabs and access different configurable graphs. In addition, has been developed a system of credentials (teacher and student roles), so that, depending on the person accessing the analytics module, different privileges are available. In this way, the information shown to a teacher is different than that for students. While the student can visualize only his own information, teachers are able to analyze individual and group learning performance.
Some examples of the graphs that students or teachers can see are shown below. On the left side of Figure 8, the graph compares the difficulty of an exercise with all the exercises of the same topic (available for the teacher role). And on the right side of Figure 8, the graph compares the ability of a particular student with the rest of the people enrolled in a particular course (available for the student and teacher roles). At the same time, dashboards can be generated by combining graphical information from this educational system with external visual aids, such as the one presented in Figure 7.
In this paper, we have presented MDLE, a multidomain computer-based learning environment as a proposal for students’ data sharing among interconnected educational software systems. The main idea is to combine a set of educational systems, hosted in different locations, and their generated users-data to implement a comprehensive learning analytics service. Each educational system integrated in the proposed environment could be a provider and/or consumer of preprocessed students’ information. Enhancing nodes collaboration will provide a more comprehensive understanding of the students learning activity and facilitate service and data reuse. As a demonstration of the MDLE interoperability, we described two educational computerbased systems that were integrated and evaluated in this environment.
The presented version of the proposed environment will be subsequently supplemented based on two main factors: recommendations of users and technical improvements. First, stakeholders’ opinion (students and teachers) is critical in order to understand the type of learning analytics views that this environment can facilitate to enhance the learning process. We are considering conducting further investigation about the benefits of this environment interoperability, generating dashboards by integrating learning analytics views from systems attending multiple domains. Regarding the technical implementation of the proposed environment, we are working on the design and evaluation of MDLE using the OAuth standard as communication and authentication protocol. We are intended to apply the OAuth protocol to use a complementary method for data transfer between nodes, instead of using a centralized database; this will require the implementation of a communication protocol used for data transfer.
Work partially funded by the CONACYT project under grant no. PDCPN201401/247698; and by the PRODEP project, supported by the 2015 call for the Integration of Thematic Networks of Academic Collaboration. This work has been also partially funded by the eMadrid project, funded by the Madrid Regional Government with grant No. S2013/ICE-2715, and the RESET project (grant No. TIN2014-53199-C3-1-R) and the SNOLA project (grant No. TIN2015-71669-REDT), projects funded by the Spanish Ministry of Economy and Competitiveness. 10. Brusilovsky, P.: KnowledgeTree: a distributed architecture for adaptive e-learning.
In: Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters. ACM, New York, NY, USA, pp. 104-113. (2004) doi: 10.1145/1013367.1013386 11. Arnold, K. E.: Signals: Applying academic analytics. Educause Quarterly, 33(1), (2010).
Retrieved April, 2017 from http://er.educause.edu/articles/2010/3/signals-applyingacademic-analytics 12. Pea, R., Oxon, D.P., Jacks, D.: The learning analytics workgroup: A report on building the field of learning analytics for personalized learning at scale. Stanford University (2014). 13. Martínez-Maldonado R., Hernandez-Leo, D., Pardo, A., Suthers, D., Kitto, K., Charleer, S., Radi Aljohani, N., and Ogata, H.: Cross-LAK: Learning Analytics Across Physical and Digital Spaces, LAK '16, Edinburgh, United Kingdom, ACM, (2016) doi: 10.1145/2883851.2883855 14. Cobián-Alvarado, M., Damián-Reyes, P., Andrade-Aréchiga, M., Pulido, JRG. AcostaDíaz, R.: Combatiendo la deserción de estudiantes universitarios con el uso de tecnologías de información. In: Memorias del XXIX Congreso Internacional y XV Congreso Internacional de Informática y Computación. ANIEI, Toluca, México (2016). 15. Álvarez-Xochihua, O., González-Fraga, J.A. Valencia-Moreno, J.M., Jordán-Arámburo, A.: Sistema de Asesoría Inteligente en Apoyo a la Nivelación Académica Universitaria. In: Memorias del Congreso Internacional de Investigación. Academia Journals, 5(3), pp. 142147. Celaya, México (2013). 16. Siemens, G., Long, P.: Penetrating the fog: Analytics in learning and education.
EDUCAUSE review, 46(5), pp. 30-40 (2011). Retrieved April, 2017 from http://er.educause.edu/articles/2011/9/penetrating-the-fog-analytics-in-learning-andeducation 17. Ruipérez-Valiente, J. A., Muñoz-Merino, P. J., Leony, D., Kloos, C. D. ALAS-KA: A learning analytics extension for better understanding the learning process in the Khan Academy platform. Computers in Human Behavior, 47, pp. 139-148. doi: https://doi.org/10.1016/j.chb.2014.07.002 18. Johns, J., Mahadevan, S., Woolf, B.: Estimating student proficiency using an item response theory model. In: Ikeda M., Ashley K.D., Chan TW. (eds) Intelligent Tutoring Systems. ITS 2006. Lecture Notes in Computer Science, vol 4053. Springer, Berlin, Heidelberg, pp. 473-480 (2006). doi: 10.1007/11774303_47 19. Ho Yu, C.: A Simple Guide to the Item Response Theory (IRT) and Rasch Modeling. Creative Wisdom, (2010) Retrieved April, 2017 from http://www.creativewisdom.com/computer/sas/IRT.pdf 20. Wauters, K., Desmet, P., Van Den Noortgate, W.: Item difficulty estimation: An auspicious collaboration between data and judgment. Computers & Education, 58(4), 11831193 (2012). doi https://doi.org/10.1016/j.compedu.2011.11.020