The introduction of course management systems such as WebCT at a university influences a variety of processes, both in terms of learning and of administration. That is why it is important to understand the way in which this influence can be assessed. This paper offers a solution: combine WebCT log files with the university management information system and the WebCT data. A data warehouse model is proposed for the storage of these integrated data. The model supports various levels of e-learning views. These are for senior management, faculty management, instructors who are course designers, and the departments responsible for the quality and planning of the learning process. Measurements typical for business functions are proposed for view definition, and analysis of the measurement results is offered in terms of the WebCT usage at the university during one term of study.
E-learning is the network-enabled transfer of skills and knowledge [
One group of CMS uses a standardised approach to development and usage. A
second uses artificial intelligence methods to support the individualised learning
process. Their architecture does not satisfy the needs of learning and administration at
the same time [
The standardised approach is typified by WebCT [
The introduction of an e-learning environment such as WebCT at a university
influences a variety of processes– management processes such as registration of
students for courses and the workload of instructors, as well as learning processes,
e.g., how e-learning is conducted, whether it replaces the teaching of some course at
all or is combined with traditional learning methods, how knowledge acquired during
such a course is assessed, etc. It is usually difficult to select suitable indices and
methods for the assessment of e-learning and its influence. Quantitative indices or
1 This work was supported by the European Social Fund (ESF).
qualitative aspects can be analysed [
In the traditional learning process, students are evaluated through formal
evaluations– tests, assignments, exams, and by informal evaluations based on
teacher’s observations of participation, interest, body language. In distance learning,
instead of observations, e-learning systems allow to evaluate student’s interaction
with the e-learning environment. For instance, the interest and participation can be
expressed by the number of times the student accesses the course and the number of
messages he sends [
Course management environments usually have a built-in student tracking tool that
enable the instructor to view statistical data such as a student’s first and last login, the
number of accesses, etc. [
The assessment opportunities supported by these tools are often inadequate, because the visualisation of data is insufficient or absent. These data are usually oriented toward the instructor’s view, because statistics are represented for one particular course. There is no management view of the overall use of CMS.
In this paper, we offer a solution in which WebCT log files are combined with the university management information system and the WebCT internal data.
In Section 2, we present related work. In Section 3, we introduce our own approach and characterise the research environment. Section 4 discusses a data warehouse model which supports the data analysis objectives described in Section 3. WebCT as a data warehouse data source of multiple formats, including Web log files and XML files, is described in Section 5. In Section 6, we define the objectives of data analysis, identify business processes influenced by the introduction of WebCT at a university, and define four views with analysed indices. We conclude with directions for future work in Section 7.
Various studies have focused on the evaluation of e-learning. In [
Another study [
Web log analysis was used in [
Many researchers are devoted to the idea of using data mining for the evaluation of
e-learning. Log files are also used as data sources in [
A data warehouse is usually used for management information analysis of various
kinds [
There is only one study of e-learning which proposes the use of data warehouse
facilities. The developed e-learning evaluation model described in [
The primary goal in this research has been to present an environment which supports analysis of the influence of e-learning on various university processes. Four factors characterise the proposed method: 1. The objectives of the analysis: • Estimating the usage of the system to determine whether further course development is necessary and to recognise the problems which users of the course experience; • Estimating the activities of instructors to know how their workload increases; • Estimating the usage of tools– which resources are used more, whether CMSspecific resources or just ‘content delivery’ features are used; • Usage of courses by students– how this influences grades. Students can be divided into groups in accordance with their grades. Analysis can be conducted to determine which group’s students are using the courses and tools to a greater extent. 2. An orientation toward a broadly used and standard CMS such as WebCT. 3. A data warehouse as the environment– the choice was based on the need to integrate many WebCT data sources with MIS data to achieve the goals of the analysis. There is also the need to analyse data at the scale of the entire university. 4. The approach to data analysis– the definition of views with indices that have different granularity of dimension hierarchies, which illustrate the different needs for information in accordance with the business functions of the data users.
The data for the research were collected with reference to all course instructors and students for fall of 2004 (22 weeks). During this time, 274 of the 402 courses developed in the WebCT environment were taught in various subject areas. The courses were developed and taught by 213 course designers. A total of 4,171 students took the courses. There are 29,090 students in all at the university.
Data for the data warehouse were obtained from the two source systems– WebCT and the university MIS, from which data about courses, persons, study programmes and student grades were extracted. WebCT as a data source is discussed in detail in Section 5.
The data warehouse was implemented in the Oracle RDBMS. Oracle Discoverer is used for data access. Data are loaded into the data warehouse on a weekly basis.
The data warehouse star schemas represent information about the structure and usage of the courses. These consist of three fact tables: Structure Fact, Usage Fact, Activity Fact, which contain measurements, and of dimensions, composed of descriptive data.
This description will make reference to the concept of tools, which are separate
WebCT resources, e.g. mail, discussions, etc. A static tool is produced once and
rarely or never updated; a dynamic tool is modified or developed over the teaching
period [
Time Time Hour Date Month Year Term Study year
Tool registered students are active (see Section 4.2). The attribute Structure level can also have two values. The value ‘content delivery’ is assigned to courses which involve only study materials, use CMS as an ordinary Web page. The value ‘advanced’ is set to other courses, which operate with additional WebCT resources (quizzes, assignments, etc.). • The dimension table Teaching contributes one attribute, which states whether a course is taught during the term. Fact table records are also collected for courses not taught during the term.
The hierarchies were created in several dimensions to make it possible to analyse
data about the course structure and usage at different levels. The Tool hierarchy of the
Tool dimension, for instance, consists of three levels– the type of tool
(static/dynamic), the category, and the sub-category. Tools were classified by
category and sub-category in accordance with the WebCT resource classification [
The data warehouse usage star schema is shown in Fig. 2. The following concepts are used in this description: Registered students are users registered in WebCT as students. Active students are registered students who have accessed a course at least once.
Time Time Hour Date Month Year Term Study year
Usage Fact Number of registered students Number of active students
The fact table Usage Fact contains two measurements– Number of registered students, Number of active students. This star schema uses the dimensions Course, Time described above.
The activity star schema (Fig. 3) includes information about student activities in WebCT during course acquisition.
Person Name Last name Personal code Gender Year of birth
Role Role
Time
The fact table Activity Fact incorporates usage information: Hit number (see Section 5.1), Data amount and Time, which records the duration of a student’s usage of a course tool. The dimensions Time, Course, Tool, Teaching are described above, but the schema also involves new dimensions: • Data about all course users are stored in the Person dimension. • The Grade table contains the grades students received taking WebCT courses, and the average grade for all courses. Satisfaction describes whether a grade is satisfactory or unsatisfactory. Part characterises the course for which the grade was earned. It has the values compulsory, partly elective, free choice. The Weighted average grade is included for the purpose of student classification. • The Role dimension stores user roles in courses– student, designer, teaching assistant, guest. Instructors in this case take on the role of designer or teaching assistant. • The Study Program dimension was introduced for university study programmes. • Records in the Session dimension correspond to a single connection to WebCT.
Access place is identified by an IP address. The following classifications were used in the Length category: Short (0-1 min.), Average (1-10 min.), Long (10-60 min.), Very long (more than 60 minutes).
The data sources of the data warehouse implemented include the university
management information system (MIS), the WebCT Web server log files, and the
WebCT’s internal database. MIS is a relational database, and data are extracted by
methods well known in data warehousing [
After partial processing, URI does not contain parameters, because these are unnecessary in identifying course tools. Records corresponding to WebCT standard icons and pictures are eliminated. This is possible because they include no usernames. The table with the processed data from the whole analysed period contains a total of 5,107,197 records.
Tool Identification. The WebCT log file URI consists of a course code, activity or accessed file name. All potential WebCT activities with courses (scripts) which can appear in a log file were analysed, and each activity was matched with a record from the Tool dimension. The activity-to-tool matching example for the log file records from Table 1 is shown in Fig. 4.
serve_home serve_marks.pl
Course opening page
Student grades
Session Identification. Session is defined as a sequence of user activities from login
to logout or moving to other Web site and not returning [
The maximum time during which a user remains logged in without performing any activity with WebCT is set at 60 minutes. If the period of time between a user’s consecutive records in a log file is less than 60 minutes, all such records are considered to be a single session. The following information is obtained about sessions: • User– because WebCT authorisation is required for all users, and all usernames are stored in the university MIS, it is possible to extract user information by integrating data from these sources. • Session length in seconds from the first activity to the last. • The IP address which allows for identification of the place where the system was accessed (faculty, computer lab, etc.). • The session category, determined by session length.
Activity Identification. Activity here is defined as a single hit by a user on a WebCT tool. Many log file records can occur for a single hit. A hit on an html file with multiple images, for instance, generates a log file record containing an accessed file name and additional records for each image. Counting log file records, therefore, does not result in the true number of activities. Due to this problem, all records with the same session, tool and time are considered to be a single activity. The number of activities is stored in the fact Hit number.
The transferred data amount and time, defined in our case as the difference between the current and the next record, are calculated for each activity. When there are several log file records for an activity, their data amount is summarised.
To enable separate analysis of the activities of students and instructors, user roles are extracted from the WebCT internal database, which is not a relational database. WebCT API is used for this purpose. It transfers all course users, their roles, and other data into an XML file.
Fig. 5 illustrates the structure of the XML file created in this way. Ovals denote elements, rectangles represent attributes. Elements and attributes used for the data warehouse are outlined with thick lines. The file contains a tree with a root group for each course. That is where course information is stored. It includes a unique course code (the element id), a WebCT category (the element orgunit) which identifies the faculty, etc. There is also a tree with a root membership for each course. It stores data about registered course users. The id element represents the course code. A sub-tree with the root member is created for each course user. It includes a username (the element userid), a role (the attribute roletype), etc.
...
properties ...
person source
id orgunit
enterprise sourcedid ... org group
membership source id ... role sourcedid member roletype userid ...
Fig. 5. XML file structure Identifying the User Role and WebCT Course Category. Initially, the necessary data are extracted from the XML file and loaded into two tables. The first table is comprised of course codes and the corresponding WebCT categories. It is used to update the Course dimension. The second table stores data about users and their roles in courses. Course designers, teaching assistants and students are all identified from this table.
Finally, the amount of transferred data, the number of hits, and the time are aggregated for each course, user, session and tool. Corresponding identifiers from other dimensions are attached, and fact table records are obtained. Table 2 shows several columns of processed data from Table 1. The total number of records in the Activity Fact table which refer to data about courses taught during the analysed term was 511,261.
Data about the course structure (the number and file size of tools) and the number of registered and active students are extracted from the WebCT internal database. The data are summarised and loaded into a table via the specially developed script. The resulting table contains the following columns: timestamp, course code, indices.
Indices related to each course are merged into data strings composed of records separated by a semicolon. Each record contains a value and a special index code assigned to every tool, and to the number of registered and active students. Each record in the table is processed, and a set of fact records is obtained. It is loaded into the data warehouse tables Structure Fact and Usage Fact. The total number of records in these fact tables which refer to data about courses taught during the analysed term was 110,624 and 4,774 respectively. 5.4 The Summarized WebCT Data Source Usage Process
WebCT Data
Sources Log file XML file
WebCT database
Staging Area
The goal of this study and the data warehouse development was to examine the influence of e-learning on university processes. So the subset of functions interesting for management was distinguished. Interaction among these functions is shown in Fig. 7.
Four data analysis views were defined in accordance with the performers of the relevant processes: university management, faculty dean, department management, instructor. The definition of each view includes: • Analysed indices; • The granularity of the dimension hierarchies, at which the particular index remains interesting.
University management process (Management) Learning organization (Faculty and Department) ...
t ce g ru in o n seR laP uanHm rscoeueR nenaagem
M
In this paper, the designation Time(month) means that the granularity of the Time dimension is intended until the level ‘month’. The notation Role(role=student) implies that only those records with the corresponding attribute value (in this case– ‘student’) are selected. The notation Course(course=Value) means that the index is calculated separately for each possible value of the attribute.
University management are interested in an assessment of e-learning from the viewpoint of course usage. Indices shown in Table 3 characterise course usage for the management view.
These indices can be compared to the financing for WebCT purchase and maintenance and the amount of money invested in course development. The analysis covers the entire university; the granularity is until the faculty level; the time dimension covers the entire reporting period or offers monthly data.
For language comprehensibility, samples of analysis results are represented as tables. Two original screenshots of reports are included in this paper as examples of the department management and instructor view. Data analysis results from the management view indices are shown in Table 4. The definitions of registered and active students are given in Section 4.2.
The goal of the faculty dean data analysis view is to evaluate course usage indices within the framework of the faculty: ‘How much’ students are taught. Also of importance is the assessment of those indices which describe the learning process– ‘How are students taught’, for instance, which course tools (static or dynamic) are used. It is also interesting to see which instructors are most active in terms of usage time and number of accesses, the aim there being to know how the WebCT usage affects the workload of instructors.
The faculty dean view is defined by the indices shown in Table 5.
When top-management view indices are used, the analysed dimension hierarchy level is changed, e.g., index MV2 from the management view (number of sessions) uses the following level of detail of the hierarchy: Time(month), Course(course), Session(category). Besides, the data are analysed within a particular faculty (Course(faculty=value)).
The analysis in Table 6 contains data from index FV3, for the period from September 1, 2004, until January 30, 2005.
This view refers to department managers who organise the learning process and ensure its’ quality. Here, again, the activities of instructors are of interest, except that this time, as opposed to the faculty view, the entire university is analysed. Department management are also interested in the way in which WebCT is used– the tools used more often, the final grades students achieve through the use of dynamic and/or static tools, etc.
The department management view is defined by new indices shown in Table 7.
The results of the department management view index DV3 are shown in Figure 8, which is the original screenshot from the report. The report demonstrates the average time, in minutes, during which a student was working with different course tools, broken down on the basis of student grade satisfaction (i.e., students with satisfactory and unsatisfactory final grades).
Page items in the order they appear in the figure: Study year, Term=’Fall’, Teaching=’yes’, WebCT category, Course.
X axis labels in the order they appear in the figure: Interface, Course content, Communication tools, Evaluation tools, Activity tools, Student tools, Other tools.
Legend:
Satisfactory,
Unsatisfactory
The instructor view related to course assessment is associated with analysis of the usage of taught courses, i.e., the activity of particular students (hits, time) with tools, the activity among all students with course tools (time, hits), the number of sessions in one month, as well as analysis of session length. The instructor view is defined by the indices shown in Table 8. Code Index TV1 Particular student activity (time in
minutes, hits) TV2 Course structure– tools, file size TV3 Number of sessions per hour of the day
Star Analyzed dimensions and hierarchy schema granularity Activity Role(role=Student);Tool(tool)
Course(course=Value);Person(name) Structure Tool(tool);Course(course=Value)
This paper discusses the use of the data warehouse in analysing the WebCT usage at a university. The potential data warehouse data sources and methods usable for the extraction and integration of interesting data were identified.
Four views were defined to identify the indices necessary to evaluate course usage at different management levels, as well as at the level of course instructors. The results were demonstrated through one index from each view. All index results were obtained, but these were not displayed in the paper, because analysis and interpretation of results have not yet been finished. The demonstrated results were not commented upon, although they were included to illustrate certain views.
Integrated WebCT usage analysis at the university level would not be possible without the data warehouse. At this stage, not all data stored in the data warehouse and interesting for the evaluation of e-learning are actually used for the discussed definition of views– e.g., the analysis of quantitative indices by IP address, etc. A survey on e-learning quality assessment is also necessary. The survey results can be analysed together with the indices which characterise course usage. These topics should be the subject of further research.