=Paper=
{{Paper
|id=Vol-152/paper-14
|storemode=property
|title=Using Data Warehouse Resources for Assessment of E-Learning Influence on University Processes
|pdfUrl=https://ceur-ws.org/Vol-152/paper15.pdf
|volume=Vol-152
|dblpUrl=https://dblp.org/rec/conf/adbis/SolodovnikovaN05
}}
==Using Data Warehouse Resources for Assessment of E-Learning Influence on University Processes==
https://ceur-ws.org/Vol-152/paper15.pdf
Using Data Warehouse Resources for Assessment of
E-Learning Influence on University Processes1
Darja Solodovņikova and Laila Niedrīte
Department of Computer Science,
University of Latvia, Raina bulv. 19, Riga, Latvia.
{sd00028, Laila.Niedrite}@lu.lv
Abstract. The introduction of course management systems such as WebCT at a
university influences a variety of processes, both in terms of learning and of
administration. That is why it is important to understand the way in which this
influence can be assessed. This paper offers a solution: combine WebCT log
files with the university management information system and the WebCT data.
A data warehouse model is proposed for the storage of these integrated data.
The model supports various levels of e-learning views. These are for senior
management, faculty management, instructors who are course designers, and
the departments responsible for the quality and planning of the learning process.
Measurements typical for business functions are proposed for view definition,
and analysis of the measurement results is offered in terms of the WebCT usage
at the university during one term of study.
1 Introduction
E-learning is the network-enabled transfer of skills and knowledge [23]. There are
many Course Management Systems (CMS) which support e-learning [3,10,12,21].
One group of CMS uses a standardised approach to development and usage. A
second uses artificial intelligence methods to support the individualised learning
process. Their architecture does not satisfy the needs of learning and administration at
the same time [2].
The standardised approach is typified by WebCT [22], one of the most popular e-
learning environments in the world. The number of parties potentially concerned with
the described problems and solutions can, therefore, be very large.
The introduction of an e-learning environment such as WebCT at a university
influences a variety of processes– management processes such as registration of
students for courses and the workload of instructors, as well as learning processes,
e.g., how e-learning is conducted, whether it replaces the teaching of some course at
all or is combined with traditional learning methods, how knowledge acquired during
such a course is assessed, etc. It is usually difficult to select suitable indices and
methods for the assessment of e-learning and its influence. Quantitative indices or
1 This work was supported by the European Social Fund (ESF).
233
234
qualitative aspects can be analysed [1,18]. The attitude of students toward their
experience with courses is the most important factor when it comes to the quality of
courses [4,14].
In the traditional learning process, students are evaluated through formal
evaluations– tests, assignments, exams, and by informal evaluations based on
teacher’s observations of participation, interest, body language. In distance learning,
instead of observations, e-learning systems allow to evaluate student’s interaction
with the e-learning environment. For instance, the interest and participation can be
expressed by the number of times the student accesses the course and the number of
messages he sends [16]. E-learning environments allow logging, that is, gathering
information and analyzing user’s actions [13].
Course management environments usually have a built-in student tracking tool that
enable the instructor to view statistical data such as a student’s first and last login, the
number of accesses, etc. [17].
The assessment opportunities supported by these tools are often inadequate,
because the visualisation of data is insufficient or absent. These data are usually
oriented toward the instructor’s view, because statistics are represented for one
particular course. There is no management view of the overall use of CMS.
In this paper, we offer a solution in which WebCT log files are combined with the
university management information system and the WebCT internal data.
In Section 2, we present related work. In Section 3, we introduce our own approach
and characterise the research environment. Section 4 discusses a data warehouse
model which supports the data analysis objectives described in Section 3. WebCT as a
data warehouse data source of multiple formats, including Web log files and XML
files, is described in Section 5. In Section 6, we define the objectives of data analysis,
identify business processes influenced by the introduction of WebCT at a university,
and define four views with analysed indices. We conclude with directions for future
work in Section 7.
2 Related Work
Various studies have focused on the evaluation of e-learning. In [11], two objectives
were discussed for e-learning assessment– the recognition of problems among
students in e-learning (e.g., they are not reading the materials, they are spending too
much time in discussions), as well as the evaluation of e-learning to improve the
quality of courses (recognising those course materials that are not being used).
WebCT was used in this research as the CMS. To facilitate the analysis of results, a
new, extra tool was proposed, which enables the visualisation of analysis results.
Another study [13] performed evaluation of e-learning to assess and control the
study process. Log file analysis (time and number of accesses) was used. Log files
used in the study were not Web server log files, although the authors spoke to the
possibility of using Web server log files in research about student activity.
Web log analysis was used in [15] with the purpose of assessing the effectiveness
of course usage, defined as course usage intensiveness, manner and usefulness. The
author of the study asserts that three sources can be combined to obtain an overall
235
insight into a students’ online learning process– Web logs, student demographics, and
survey results. Statistical methods were used for the data analysis.
Many researchers are devoted to the idea of using data mining for the evaluation of
e-learning. Log files are also used as data sources in [9,19].
A data warehouse is usually used for management information analysis of various
kinds [5,6,8], as well as for Web data analysis [7].
There is only one study of e-learning which proposes the use of data warehouse
facilities. The developed e-learning evaluation model described in [16,17], among
others, supports three levels of evaluation – user communication tracking, evaluation
of foreseen activities, and application of data mining. The data warehouse model is
proposed for storage of data, which satisfies the needs of the aforementioned three-
level evaluation. It has not, however, been specified which CMS are compatible with
this model.
3 Research Profile
The primary goal in this research has been to present an environment which supports
analysis of the influence of e-learning on various university processes. Four factors
characterise the proposed method:
1. The objectives of the analysis:
• Estimating the usage of the system to determine whether further course
development is necessary and to recognise the problems which users of the course
experience;
• Estimating the activities of instructors to know how their workload increases;
• Estimating the usage of tools– which resources are used more, whether CMS-
specific resources or just ‘content delivery’ features are used;
• Usage of courses by students– how this influences grades. Students can be divided
into groups in accordance with their grades. Analysis can be conducted to
determine which group’s students are using the courses and tools to a greater
extent.
2. An orientation toward a broadly used and standard CMS such as WebCT.
3. A data warehouse as the environment– the choice was based on the need to
integrate many WebCT data sources with MIS data to achieve the goals of the
analysis. There is also the need to analyse data at the scale of the entire university.
4. The approach to data analysis– the definition of views with indices that have
different granularity of dimension hierarchies, which illustrate the different needs
for information in accordance with the business functions of the data users.
The data for the research were collected with reference to all course instructors and
students for fall of 2004 (22 weeks). During this time, 274 of the 402 courses
developed in the WebCT environment were taught in various subject areas. The
courses were developed and taught by 213 course designers. A total of 4,171 students
took the courses. There are 29,090 students in all at the university.
Data for the data warehouse were obtained from the two source systems– WebCT
and the university MIS, from which data about courses, persons, study programmes
236
and student grades were extracted. WebCT as a data source is discussed in detail in
Section 5.
The data warehouse was implemented in the Oracle RDBMS. Oracle Discoverer is
used for data access. Data are loaded into the data warehouse on a weekly basis.
4 The Data Warehouse Model
The data warehouse star schemas represent information about the structure and usage
of the courses. These consist of three fact tables: Structure Fact, Usage Fact, Activity
Fact, which contain measurements, and of dimensions, composed of descriptive data.
4.1 The Structure Star Schema
This description will make reference to the concept of tools, which are separate
WebCT resources, e.g. mail, discussions, etc. A static tool is produced once and
rarely or never updated; a dynamic tool is modified or developed over the teaching
period [15]. Fig. 1 illustrates the structure star schema– a fact table with the
corresponding dimensions.
Time Course
Structure Fact
Time WebCT course code
Number WebCT category
Hour Tool File size
Date Level
Month Name
Credits
Year
Name Scientific branch
Term
Study year Sub-category Teaching Teaching
Category Last student access date
Requirement Last designer access date
Type Activity level
Teaching Structure level
Fig. 1. The Structure star schema
The fact table Structure Fact incorporates data about the course structure, i.e., the
Number and File size of tools in a course. This fact table has the following
dimensions:
• Time is a standard dimension in a data warehouse;
• Tool contains information about WebCT resources. The attribute Requirement
states whether the tool is or is not required for the course structure (the tool
requirement is defined by the course development rules of our project). Type has
two values: static or dynamic.
• Data about the courses are stored in the dimension table Course. Level refers to the
study year when the course is taught. Credits stores the number of credit points
earned. Teaching determines whether the course is taught during the current term.
Activity and Structure level are course classification attributes. Two activity levels
were identified– active and passive. A course is active when more than 5% of
237
registered students are active (see Section 4.2). The attribute Structure level can
also have two values. The value ‘content delivery’ is assigned to courses which
involve only study materials, use CMS as an ordinary Web page. The value
‘advanced’ is set to other courses, which operate with additional WebCT resources
(quizzes, assignments, etc.).
• The dimension table Teaching contributes one attribute, which states whether a
course is taught during the term. Fact table records are also collected for courses
not taught during the term.
The hierarchies were created in several dimensions to make it possible to analyse
data about the course structure and usage at different levels. The Tool hierarchy of the
Tool dimension, for instance, consists of three levels– the type of tool
(static/dynamic), the category, and the sub-category. Tools were classified by
category and sub-category in accordance with the WebCT resource classification [22].
4.2 The Usage Star Schema
The data warehouse usage star schema is shown in Fig. 2. The following concepts are
used in this description: Registered students are users registered in WebCT as
students. Active students are registered students who have accessed a course at least
once.
Course
Time
WebCT course code
Usage Fact WebCT category
Level
Time Name
Hour Credits
Date Number of registered students Scientific branch
Month Number of active students Teaching
Year Last student access date
Term Last designer access date
Study year Activity level
Structure level
Fig. 2. The Usage star schema
The fact table Usage Fact contains two measurements– Number of registered
students, Number of active students. This star schema uses the dimensions Course,
Time described above.
4.3 The Activity Star Schema
The activity star schema (Fig. 3) includes information about student activities in
WebCT during course acquisition.
238
Session Grade
Role Length category Course
Person Grade
Length Satisfaction
Access place Weighted avg grade
Role Activity Fact Part
Name WebCT course code
Last name WebCT category
Personal code Level
Gender Time Hit number Name
Year of birth Data amount Tool Credits
Time Scientific branch
Teaching
Time Last student access date
Hour Study Program Name Last designer access date
Date Teaching Sub-category Activity level
Month Category Structure level
Year Name Requirement
Term Level Type
Study year Teaching
Subject area
Fig. 3. The Activity star schema
The fact table Activity Fact incorporates usage information: Hit number (see
Section 5.1), Data amount and Time, which records the duration of a student’s usage
of a course tool. The dimensions Time, Course, Tool, Teaching are described above,
but the schema also involves new dimensions:
• Data about all course users are stored in the Person dimension.
• The Grade table contains the grades students received taking WebCT courses, and
the average grade for all courses. Satisfaction describes whether a grade is
satisfactory or unsatisfactory. Part characterises the course for which the grade was
earned. It has the values compulsory, partly elective, free choice. The Weighted
average grade is included for the purpose of student classification.
• The Role dimension stores user roles in courses– student, designer, teaching
assistant, guest. Instructors in this case take on the role of designer or teaching
assistant.
• The Study Program dimension was introduced for university study programmes.
• Records in the Session dimension correspond to a single connection to WebCT.
Access place is identified by an IP address. The following classifications were used
in the Length category: Short (0-1 min.), Average (1-10 min.), Long (10-60 min.),
Very long (more than 60 minutes).
5 Data Sources
The data sources of the data warehouse implemented include the university
management information system (MIS), the WebCT Web server log files, and the
WebCT’s internal database. MIS is a relational database, and data are extracted by
methods well known in data warehousing [8]. This process, therefore, is not analysed
in detail. The structure and usage of other data sources are considered to a further
extent.
239
5.1 WebCT Web Server Log Files
WebCT log files are consistent with the Common Log Format (CLF) [20]. Data from
a log file are first partially processed and loaded into a database table. An excerpt
from the table is shown in Table 1.
Table 1. Processed data from log file (excerpt)
Date/time URI Username IP address Bytes transferred
11/05/2004 GET/SCRIPT/2DAT5080/
12:22:38 scripts/student/serve_home sd00028 213.175.115.2 7134
11/05/2004 GET/SCRIPT/2DAT5080/
12:22:43 scripts/student/serve_marks.pl sd00028 213.175.115.2 15795
11/05/2004 GET/SCRIPT/2DAT5080/
12:22:46 scripts/student/serve_home sd00028 213.175.115.2 7134
After partial processing, URI does not contain parameters, because these are
unnecessary in identifying course tools. Records corresponding to WebCT standard
icons and pictures are eliminated. This is possible because they include no usernames.
The table with the processed data from the whole analysed period contains a total of
5,107,197 records.
Tool Identification. The WebCT log file URI consists of a course code, activity or
accessed file name. All potential WebCT activities with courses (scripts) which can
appear in a log file were analysed, and each activity was matched with a record from
the Tool dimension. The activity-to-tool matching example for the log file records
from Table 1 is shown in Fig. 4.
Log file Matching tool
serve_home Course opening page
serve_marks.pl Student grades
Fig. 4. Activity-to-tool matching
Session Identification. Session is defined as a sequence of user activities from login
to logout or moving to other Web site and not returning [15].
The maximum time during which a user remains logged in without performing any
activity with WebCT is set at 60 minutes. If the period of time between a user’s
consecutive records in a log file is less than 60 minutes, all such records are
considered to be a single session. The following information is obtained about
sessions:
• User– because WebCT authorisation is required for all users, and all usernames are
stored in the university MIS, it is possible to extract user information by integrating
data from these sources.
• Session length in seconds from the first activity to the last.
240
• The IP address which allows for identification of the place where the system was
accessed (faculty, computer lab, etc.).
• The session category, determined by session length.
Activity Identification. Activity here is defined as a single hit by a user on a WebCT
tool. Many log file records can occur for a single hit. A hit on an html file with
multiple images, for instance, generates a log file record containing an accessed file
name and additional records for each image. Counting log file records, therefore, does
not result in the true number of activities. Due to this problem, all records with the
same session, tool and time are considered to be a single activity. The number of
activities is stored in the fact Hit number.
The transferred data amount and time, defined in our case as the difference
between the current and the next record, are calculated for each activity. When there
are several log file records for an activity, their data amount is summarised.
5.2 Data Export to XML Format
To enable separate analysis of the activities of students and instructors, user roles are
extracted from the WebCT internal database, which is not a relational database.
WebCT API is used for this purpose. It transfers all course users, their roles, and other
data into an XML file.
Fig. 5 illustrates the structure of the XML file created in this way. Ovals denote
elements, rectangles represent attributes. Elements and attributes used for the data
warehouse are outlined with thick lines. The file contains a tree with a root group for
each course. That is where course information is stored. It includes a unique course
code (the element id), a WebCT category (the element orgunit) which identifies the
faculty, etc. There is also a tree with a root membership for each course. It stores data
about registered course users. The id element represents the course code. A sub-tree
with the root member is created for each course user. It includes a username (the
element userid), a role (the attribute roletype), etc.
enterprise
properties person group membership
... ... sourcedid sourcedid
source source
id id
... member
org
...
orgunit role
roletype
userid
...
Fig. 5. XML file structure
241
Identifying the User Role and WebCT Course Category. Initially, the necessary
data are extracted from the XML file and loaded into two tables. The first table is
comprised of course codes and the corresponding WebCT categories. It is used to
update the Course dimension. The second table stores data about users and their roles
in courses. Course designers, teaching assistants and students are all identified from
this table.
Finally, the amount of transferred data, the number of hits, and the time are
aggregated for each course, user, session and tool. Corresponding identifiers from
other dimensions are attached, and fact table records are obtained. Table 2 shows
several columns of processed data from Table 1. The total number of records in the
Activity Fact table which refer to data about courses taught during the analysed term
was 511,261.
Table 2. Summarized records of Table 1
Session Course Session Hit Data
Tool Person Role Time
start code ID number amount
11/05/2004 Course Darja
2DAT5080 Student 327878 8 3 57
12:22:36 opening page Solodovnikova
11/05/2004 Darja
2DAT5080 Student grades Student 327878 3 1 15
12:22:43 Solodovnikova
5.3 Data Extraction for Structure and Usage Facts
Data about the course structure (the number and file size of tools) and the number of
registered and active students are extracted from the WebCT internal database. The
data are summarised and loaded into a table via the specially developed script. The
resulting table contains the following columns: timestamp, course code, indices.
Indices related to each course are merged into data strings composed of records
separated by a semicolon. Each record contains a value and a special index code
assigned to every tool, and to the number of registered and active students. Each
record in the table is processed, and a set of fact records is obtained. It is loaded into
the data warehouse tables Structure Fact and Usage Fact. The total number of records
in these fact tables which refer to data about courses taught during the analysed term
was 110,624 and 4,774 respectively.
5.4 The Summarized WebCT Data Source Usage Process
Fig. 6 summarises the whole data extraction, transformation and loading process from
the WebCT sources described in Section 5.1-5.3. The full data warehouse loading
process also uses data from MIS.
242
WebCT Data Staging Area Data Warehouse
Sources
Log file Table with processed activity Session Dimension
information (Table 1)
Activity Fact Table
XML file Table with data about courses,
users and their roles
Structure Fact Table
WebCT Table with course data and indices
(number and file size of tools) Usage Fact Table
database
Fig. 6. The summarized loading process from WebCT data sources
6 Analysis Views
6.1 Definition of Processes and Data Analysis Views
The goal of this study and the data warehouse development was to examine the
influence of e-learning on university processes. So the subset of functions interesting
for management was distinguished. Interaction among these functions is shown in
Fig. 7.
Four data analysis views were defined in accordance with the performers of the
relevant processes: university management, faculty dean, department management,
instructor. The definition of each view includes:
• Analysed indices;
• The granularity of the dimension hierarchies, at which the particular index remains
interesting.
University management process (Management) Learning organization (Faculty and Department) ...
processes
Management
Support
workload of
registration
academic
Resource
Resource
Student
Planing
Human
staff
The
...
...
Business processes
E-learning (Academic staff)
Traditional learning process (Academic staff)
Research (Academic staff)
Fig. 7. E-learning interaction with university processes
In this paper, the designation Time(month) means that the granularity of the Time
dimension is intended until the level ‘month’. The notation Role(role=student)
243
implies that only those records with the corresponding attribute value (in this case–
‘student’) are selected. The notation Course(course=Value) means that the index is
calculated separately for each possible value of the attribute.
6.2 The University Management View
University management are interested in an assessment of e-learning from the
viewpoint of course usage. Indices shown in Table 3 characterise course usage for the
management view.
These indices can be compared to the financing for WebCT purchase and
maintenance and the amount of money invested in course development. The analysis
covers the entire university; the granularity is until the faculty level; the time
dimension covers the entire reporting period or offers monthly data.
Table 3. Management view definition
Code Indices Star Analyzed dimensions and hierarchy
schema granularity
MV1 Average activity (hits, time in minutes) Usage Course(faculty);Time(month);
of registered and active students Activity Role(role=Student)
MV2 Number of sessions Activity Time(month);Session(category)
MV3 Average session length Activity Time(month)
MV4 Number of courses in the term Usage Teaching(teaching=Yes)
MV5 Number of registered students in the Usage Teaching(teaching=Yes)
term
MV6 Number of active students in the term Usage Teaching (teaching=Yes)
MV7 Number of active instructors in the term Activity Role(role=designer or role=teaching
assistant)
For language comprehensibility, samples of analysis results are represented as
tables. Two original screenshots of reports are included in this paper as examples of
the department management and instructor view. Data analysis results from the
management view indices are shown in Table 4. The definitions of registered and
active students are given in Section 4.2.
Table 4. Results of the management view indices
WebCT Registered Active Student Courses Active Course
category students students activity ratio courses activity ratio
LU-PPF 1,057 337 31.88% 21 19 90.48%
. . .
LU-TF 101 72 71.29% 13 5 38.46%
6.3 The Faculty Dean View
The goal of the faculty dean data analysis view is to evaluate course usage indices
within the framework of the faculty: ‘How much’ students are taught. Also of
244
importance is the assessment of those indices which describe the learning process–
‘How are students taught’, for instance, which course tools (static or dynamic) are
used. It is also interesting to see which instructors are most active in terms of usage
time and number of accesses, the aim there being to know how the WebCT usage
affects the workload of instructors.
The faculty dean view is defined by the indices shown in Table 5.
Table 5. Faculty dean view definition
Code Index Star Analyzed dimensions and hierarchy
schema granularity
FV1 Total activity of particular Activity Course(Course);Role(role=Designer)
designers in a course (hits, time in Course(faculty=value);Person(name)
minutes)
FV2 Activity of particular designers in Activity Course(Course);Role(role=Designer)
course tools (hits, time in minutes) Course(faculty=value);Tool(Tool)
Person(name)
FV3 Activity of the most active faculty Activity Course(Course);Role(role=Designer)
designers in tools by total hits, time Course(faculty=Value);Tool(Tool)
Person(name)
When top-management view indices are used, the analysed dimension hierarchy
level is changed, e.g., index MV2 from the management view (number of sessions)
uses the following level of detail of the hierarchy: Time(month), Course(course),
Session(category). Besides, the data are analysed within a particular faculty
(Course(faculty=value)).
The analysis in Table 6 contains data from index FV3, for the period from
September 1, 2004, until January 30, 2005.
Table 6. Results of the faculty dean view indices
Designer Hit number
Last name Name Dynamic tool Static tool Total
L1 N1 54,633 5,525 60,158
. . .
L10 N10 2,925 2,668 5,593
6.4 The Department Management View
This view refers to department managers who organise the learning process and
ensure its’ quality. Here, again, the activities of instructors are of interest, except that
this time, as opposed to the faculty view, the entire university is analysed. Department
management are also interested in the way in which WebCT is used– the tools used
more often, the final grades students achieve through the use of dynamic and/or static
tools, etc.
The department management view is defined by new indices shown in Table 7.
245
Table 7. Department management view definition
Code Index Star Analyzed dimensions and hierarchy
schema granularity
DV1 Total designers’ activity (time in Activity Role(role=Designer);Tool(tool)
minutes, hits) drill-down by tools
DV2 Total designers’ activity (time in Activity Role(role=Designer);Tool(type)
minutes, hits) drill-down by tool
types
DV3 Students’ grade level and tool usage Activity Role(role=Student);Course(Course)
The results of the department management view index DV3 are shown in Figure 8,
which is the original screenshot from the report. The report demonstrates the average
time, in minutes, during which a student was working with different course tools,
broken down on the basis of student grade satisfaction (i.e., students with satisfactory
and unsatisfactory final grades).
Page items in the order they appear in the
figure: Study year, Term=’Fall’,
Teaching=’yes’, WebCT category, Course.
X axis labels in the order they appear in the
figure: Interface, Course content,
Communication tools, Evaluation tools,
Activity tools, Student tools, Other tools.
Legend: Satisfactory, Unsatisfactory
Fig. 8. Results of the department management view indices
6.5 The Instructor View
The instructor view related to course assessment is associated with analysis of the
usage of taught courses, i.e., the activity of particular students (hits, time) with tools,
the activity among all students with course tools (time, hits), the number of sessions
in one month, as well as analysis of session length. The instructor view is defined by
the indices shown in Table 8.
246
Table 8. Instructor view definition
Code Index Star Analyzed dimensions and hierarchy
schema granularity
TV1 Particular student activity (time in Activity Role(role=Student);Tool(tool)
minutes, hits) Course(course=Value);Person(name)
TV2 Course structure– tools, file size Structure Tool(tool);Course(course=Value)
TV3 Number of sessions per hour of the Activity Time(hour);Course(course=Value)
day
Fig. 9 shows the results of the TV3 index- the number of students’ sessions per
hour of the day.
Page items in the order they
appear in the figure: Study
year, Term=’Fall’,
Teaching=’yes’, WebCT
category=’Faculty of Physics
and Mathematics’,
Course=’Operating systems’.
Fig. 9. Results of the instructor view indices
7 Conclusions and Future Work
This paper discusses the use of the data warehouse in analysing the WebCT usage at a
university. The potential data warehouse data sources and methods usable for the
extraction and integration of interesting data were identified.
Four views were defined to identify the indices necessary to evaluate course usage
at different management levels, as well as at the level of course instructors. The
results were demonstrated through one index from each view. All index results were
obtained, but these were not displayed in the paper, because analysis and
interpretation of results have not yet been finished. The demonstrated results were not
commented upon, although they were included to illustrate certain views.
Integrated WebCT usage analysis at the university level would not be possible
without the data warehouse. At this stage, not all data stored in the data warehouse
and interesting for the evaluation of e-learning are actually used for the discussed
definition of views– e.g., the analysis of quantitative indices by IP address, etc. A
survey on e-learning quality assessment is also necessary. The survey results can be
247
analysed together with the indices which characterise course usage. These topics
should be the subject of further research.
References
1. Berk, J.: Learning Measurement: It’s Not How Much You Train, But How Well. In: The
Elearning Developers Journal, Vol. 3 (2003) 1-8
2. Brusilovsky, P.: KnowledgeTree: A Distributed Architecture for Adaptive E-Learning. In:
Proc. of the 13th Intl. World Wide Web Conference (WWW 2004), New York: ACM Press
(2004) 104-113
3. Brusilovsky, P., Miller, P.: Course Delivery Systems for the Virtual University. In: Tschang,
T., Della Senta, T. (eds.): Access to Knowledge: New Information Technologies and the
Emergence of the Virtual University. Elsevier Science, Amsterdam (2001) 167-206
4. Cheung, D.: Developing a student evaluation instrument for distance teaching. In: The
American Journal of Distance Education, Vol. 19(1) (1998) 23-42
5. Imhoff, C., Galemmo, N., Geiger, J.G.: Mastering Data Warehouse Design. Relational and
Dimensional techniques. Wiley Publishing (2003)
6. Inmon, W. H.: Building the Data Warehouse. QED Press/John Wiley (1992) (Last Edition:
3rd edn., Wiley & Sons (2002)
7. Kimball, R., Merz, R.: The Data Webhouse Toolkit: Building the Web-Enabled Data
Warehouse, John Wiley and Sons, Inc. (2000)
8. Kimball, R., Reeves, L., Ross, M., Thornthwaite, W.: The Data Warehouse Lifecycle
Toolkit, Wiley Computer Publishings (1998)
9. Lei, X., Pahl, C., Donnellan, D.: An Evaluation Technique for Content Interaction in Web-
based Teaching and Learning Environments. In: Proc. of the 3rd IEEE Intl. Conf. on
Advanced Learning Technologies (ICALT’03) (2003) 294-295
10. Mamoukaris, K., Economides, A., Delchanidou, N.: Evaluation of web-based educational
systems. Internet: JABE Proceedings(2002) http://www.abe.villanova.edu/proceeding.html
11. Mazza, R., Dimitrova, V.: Visualising Student Tracking Data to Support Instructors in
Web-Based Distance Education. In: Proc. of the 13th Intl. World Wide Web conference on
Alternate track papers & posters, ACM, USA (2004) 154-161
12. Plugge, L.A., Schoenmakers, S., Kirschner, P.A.: Electronic environment for management
of learning systems. In: The International Journal of Technologies for the Advancement
Knowledge and Learning, 3(1) (2001)
13. Rahkila, M., Karjalainen, M.: Evaluation of learning in computer based education using log
systems, Proc. of 29th ASEE/IEEE Frontiers in Education Conf., San Juan, Puerto Rico
(1999) 16-21
14. Roberts T.G., Irani T., Lundy L.K., Telg, R.: Practices in Student Evaluation of Distance
Education Courses Among Land Grant Institutions. In: Journal of Agricultural Education,
Vol. 45(3) (2004) 1-10 http://pubs.aged.tamu.edu/jae/toc45.html
15. Sheard, J., Ceddia, J., Hurst, J., Tuovinen, J.: Inferring Student Learning Behaviour From
Website Interactions: A Usage Analysis. In: Education and Information Technologies, Vol.
8(3), Kluwer Academic Publishers (2003) 245-266
16. Silva, D.R., Vieira, M.T.P.: An ongoing assessment model for distance learning. In: Proc.
of the Intl. Conf. on Internet and Multimedia Systems and Applications (IMSA2001),
Hawaii, USA (2001) 17-21
17. Silva, D.R., Vieira, M.T.P.: Using Data Warehouse And Data Mining Resources For
Ongoing Assessment Of Distance Learning. In: Proceedings of IEEE Intl. Conf. on
Advanced Learning Technologies (ICALT 2002) (2002) 40-45
248
18. Smith-Gratto, K.: Best Practices and Problems North Carolina A & T State University.
Report to the Distance Education Evaluation Task Force Distance Education (1999)
http://qed.ncat.edu/ir&p/report.htm
19. Zaiane, O.R., Luo, J.: Towards Evaluating Learners’ Behaviour in a Web-Based Distance
Learning Environment. In: Proc. of Intl. Conf. on Advanced Learning Technologies
ICALT’01, IEEE (2001).
20. Appache Http Serwer Log Files http://httpd.apache.org/docs/logs.html
21. EDUTOOLS: Course Management Systems http://www.edutools.info/course/compare
22. WebCT http://www.webct.com
23. Webopedia: On-line encyclopedia dedicated to computer technology
http://www.webopedia.com/TERM/e/e_learning.html