The Organization for Economic Cooperation and Development defined LMS technology as a technology used by instructors to build and maintain courses. It features personal communication via email, group communication via chatting and forums, posting content including syllabi, papers, presentations and lesson summaries, performance evaluation via question and answer repositories; selfassessment tests, assignments, quizzes and exams, instruction management via messaging, grade posting and surveys, and more.
There are many LMS systems on the market that can be obtained for free and are Open Source (i.e. Moodle, Sakai, Claroline, ATutor, etc.) or through payment (i.e. Blackboard, WebCT, Clix, and many others). All of them support many different features which can be used as evaluation criteria and analyzed from different aspects [6]:
1. Pedagogical aspect 2. Learner environment 3. Instructor tools 4. Course and curriculum design 5. Administrator tools and 6. Technical specification.
Pedagogical criteria can, for example, include [15]: Learner control, Learner activity, Cooperative/ Collaborative learning, Goal orientation, Applicability, Added value, Motivation, Valuation of previous knowledge, Flexibility and Feedback.
On the other hand, Kurilovas [12] groups technical criteria as follows: It is obvious that selection of the most suitable LMS is a complex task that involves defining the evaluation criteria and selecting a method for criteria evaluation that will be systematic, comprehensive, easy to use, etc.
Once defined, the criteria can be evaluated using a selfevaluation questionnaire that employs a 7-point Likert scale 1 (strongly disagree) -5 (strongly agree), 6 (not applicable), 7 (don't know) [7,13,14,15]. Other evaluation tools include MS-Excel spreadsheets application [1], fuzzy logic [6], an expert system shell for multi-attribute decision support DEXi [2], a hybrid Multi-criteria decision-making (MCDM) model based on factor analysis and DEMATEL [21] etc. The number of features for evaluation is usually very high in all these applications (e.g. 57 in Pipan [16]; 52 offered in Cavus [6]). To evaluate such a great number of features, a significant amount of time and effort is required of the evaluator.
We believe that reliable results can be obtained with fewer criteria if the problem is decomposed in order to more easily comprehended sub-problems that can be analyzed independently, i.e. presented as a hierarchy. One of the most popular methods that deal with decision hierarchies is Analytic Hierarchy Process (AHP) [17] and we propose this method for the evaluation of selected LMS products because: (1) it supplies management in both education and industry with a less complex and more appropriate and flexible way to effectively analyze LMSs, (2) it supports their selections of an appropriate product, and (3) achievement of a higher level of e-learner satisfaction [18]. Other advantages of AHP that should be emphasized are that AHP provides a measure of consistency of the evaluator and that it can be used for participative evaluation of LMS product.
To verify AHP applicability, an expert is asked to use AHP on an originally developed hierarchy of the problem of selecting the most appropriate LMS for the student target group. Consistency of the expert is checked throughout the process. At the end, results of the evaluation are compared with results presented in [16].
One of the key issues in decision making is eliciting judgments from the decision maker (DM) about the importance of a given set of decision elements. If a problem can be structured hierarchically, then a certain ratio scale can serve as an efficient tool to enable this hierarchy by performing pair-wise comparisons. The core of AHP [17] lies in presenting the problem as a hierarchy and comparing the hierarchical elements in a pair-wise manner using Saaty's 9-point scale, Table 1. This way, the importance of one element over another is expressed in regards to the element in the higher level. The AHP is a multi criteria optimization method which creates so-called local comparison matrices at all levels of a hierarchy and performs logical syntheses of their (local) priority vectors. The major feature of AHP is that it involves a variety of tangible and intangible goals, attributes, and other decision elements. In addition, it reduces complex decisions to a series of pair-wise comparisons; implements a structured, repeatable, and justifiable decision-making approach; and builds consensus.
In standard AHP, an eigenvector (EV) method is used for deriving weights from local matrices; the EV is called the prioritization method, and the computational procedure is consequently called prioritization. After local weights are calculated at all levels of the hierarchy, a synthesis consists of multiplying the criterion-specific weight of the alternative with the corresponding criterion weight and summing up the results to obtain composite weights of the alternative with respect to the goal; this procedure is unique for all alternatives and all criteria.
AHP is aimed at supporting decision-making processes in both individual and group contexts. In later cases various aggregation schemes are applicable, e.g. AIJ and AIP [9], as well as various consensus reaching procedures are easy to implement. This issue is out of scope here; namely, the paper deals strictly with an individual application of AHP.
The DM makes judgments more or less consistently depending not only on his knowledge of the decision problem itself, but also on his ability to remain focused and to ensure that his understanding of the cardinal preferences between elements will always, or as much as possible, be formalized properly while using a verbal scale or related numerical ratios [20]. For example, if the Saaty's 9-point ratio scale is used, the question could be: will the DM put a ij = 3, or a ij = 2, if he considers element E i slightly more important than E j ? Or, if there are seven elements to be compared, then matrix A is of size 7x7, and the question could be: is the DM really capable to preserve consistency while comparing head-to-head 21 times all pairs of elements? How is the DM to override the imposed difficulty with Saaty's scale when he compares elements E i and E k , after he has judged the elements E i and E j , and E j and E k ? If he has already made the judgments a ij = 3 and a jk
Absolute preference (element i over element j)
Very strong preference (i over j) 7
Strong preference (i over j) 5
Weak preference (i over j) 3
Indifference of i and j 1 Weak preference (j over i) Consistency analysis of the individual DM can be based on the consistency ratio (CR) defined by Saaty [17], and the total L 2 ED for each comparison matrix. Whichever method is used to derive the priority vector from the given local AHP matrix [19], if it already has all the entries elicited from the DM, measuring consistency is necessary in order to ensure the integrity of the outcomes.
Standard AHP uses EV, the prioritization method, and the consistency coefficient CR to indicate the inconsistency of the DM [17]. The other commonly used consistency measures are the total Euclidean distance, and minimum violations measure.
The CR is calculated as a part of the standard AHP procedure. First, the consistency index (CI) is calculated using the following equation:
where max λ is the principal eigenvalue of the given comparison matrix. Knowing the consistency index and random consistency index (RI) defined also by Saaty [17], the consistency ratio is obtained:
(2)
Saaty [17] suggested considering the maximum level of the DM's inconsistency to be 0.10; that is, CR should be less or equal to 0.10.
The problem is stated so as to assess and rank by applicability the three e-Learning Management Systems based on three typical qualitative criteria and a number of qualitative sub criteria. An expert is asked to perform the decision making processes by applying the AHP model.
An original hierarchy of the problem [16] consists of five levels: goal -criteria set -sub criteria set (4+4+3 per criterions in upper level) represented by specific groups of attributes -sub sub criterions (24 in total under sub criterions), represented by groups of more detailed attributes -and three alternatives (LSMs). In order to reduce the number of decision elements, the fourth level in the hierarchy (sub sub attributes) is avoided and thus the reduced hierarchy of the problem is created as shown in Figure 1.
Identify LSM with the best applicability characteristics
The set of criteria is the key component of the decisionmaking model. In creating the model [16], an attempt is made to meet the requirements set by Bohanec & Rajkovič [5] by taking into account the principle of criteria integrity (inclusion of all relevant criteria), appropriate structure, non-redundancy, comprehensiveness and measurability [4].
Comprehensiveness means that all the data about the subject are actually present in the database. Nonredundancy means that each individual piece of data exists only once in the database. Appropriate structure means that the data are stored in such a way as to minimize the cost of expected processing and storage [3].
The criteria set is stated under three main scopes: Student's learning environment, System, technology & standards, and Tutoring & didactics. These three scopes represent the global skeleton of the multi-attribute model with attributes (considered as sub-criterions) associated with each criterion.
(
The first scope is adopted as the first criterion and declared as the Student's learning environment. It is composed of four basic attributes:
• (EASE) Ease of use [11]. In the context of e-learning technology, standards are generally developed to be used in system design and implementation for the purposes of ensuring interoperability, portability and reusability, especially for learning resources as they require for their preparation qualified professionals and are very time [10].
(3) T&D (Tutoring & didactics): Third group of criteria is merged into Tutoring & didactics. The tutor's quality of environment is assessed using the: Activity tracking undoubtedly provides important support to the tutor in the learning process. Here we have focused on monitoring students in the process of learning and the possibility of displaying students' progress, analysis of presence data, sign-in data and time analysis.
The multi-attribute decision making model was completed with three learning management systems (LMS):
A1. Blackboard 6 (www.blackboard.com): Blackboard is among the most perfected and complex LMSs on the market. The system offers various communication options (both synchronous and asynchronous) within the learning environment. The Blackboard LMS is designed for institutions dedicated to teaching and learning. Blackboard technology and resources power the online, web-enhanced, and hybrid education programs at more than 2000 academic institutions (research university, community college, high school, virtual MBA programs etc.). Blackboard has 5,500 clients representing 200 million users (2.5 million from its largest, hosted client; 100,000 from its largest, self-hosted client) in 60 countries [8].
A2. CLIX 5.0 (www.im-c.de): CLIX is targeted most of all at big corporations because it provides efficient, manageable, connected and expandable internet-based learning solutions. This scalable, multilingual and customizable software aims at providing process excellence for educational institutions. For educational administrators, CLIX offers powerful features for course management and distribution. Additionally, it provides personalized learning paths for students, a tutoring centre for lectures and a whole bunch of innovative collaboration tools for both user groups, e.g. a virtual classroom. Altogether, CLIX makes planning, organizing, distributing, tracking and analyzing of learning and teaching a smooth and efficient process.
A3. Moodle 1.5.2 (www.moodle.org). Moodle is a free, open source PHP application for producing internet-based educational courses and web sites on any major platform (Linux, UNIX, Windows and Mac OS X). The fact that it is free of charge is especially attractive for schools and companies which always lack resources for the introduction of new learning technologies. Furthermore, the Moodle system is not only price-efficient -it can easily be compared to costly commercial solutions on all aspects.
Courses are easily built up using modules such as forums, chats, journals, quizzes, surveys, assignments, workshops, resources, choices and more. Moodle supports localization, and has so far been translated into 34 languages. Moodle has been designed to support modern pedagogies based on social constructionism, and focuses on providing an environment to support collaboration, connected knowing and a meaningful exchange of ideas. It has nearly 54,000 registered sites (over 9,800 from the U.S.) representing over 200 countries, 44.3 million users, and 4.6 million courses. Moodle's wide spread international use, coupled with its continued growth over the past six years, has made it the leading open source LMS solution.
After a brief explanation of basics and concepts of AHP, the expert compared in pairs first criteria versus goal, then sub criteria versus criteria, and finally alternatives with respect to each of the sub criteria. Comparison matrices and related calculated local weights of decision elements are presented in Figures 2-3. The alternative with the highest final weight is CLIX 5.0 (0.590) and can be considered as the most applicable LMS for the students. The second ranked alternative is Blackboard, while Moodle 1.5.2 is the least applicable LMS.
It is worthy to mention that the expert was very consistent during the whole evaluation process. Overall HCR is 0.059.
One of the important problems in the field of e-learning is the selection of an appropriate LMS that will satisfy most of the users' preferences and requirements. The complexity of the problem is increased due to the growing number of LMS each year and also due to the number of features that should be taken into account while evaluating each LMS. To reduce that complexity and facilitate selection of an appropriate LMS, we propose a decomposition of the problem to more easily comprehended sub-problems that the evaluator can analyze independently. The AHP methodology based on pair-wise comparison of decision elements on one hierarchy level was found to be appropriate for such analysis. Also, the final result of AHP application, which found CLIX 5.0 to be the most applicable LMS, proved that the proposed approach was justified: the reduced hierarchy and use of AHP led to the same result as the one provided by the DeXi evaluation of 57 criteria.