The lifelong competence development and transfer of competencies acquisition is a tendency of global world and specifically for e-learning.. Generally, an instructional based-competence process consists in four single sub-processes, the competence definition, competence development, competence assessment process and certification. This paper presents a competence-based adaptive assessment process for judging competence of learners in the context of a virtual learning environment. The goal of our research is build repositories of items linked to competencies definitions and rules specifications in order to the generation of adaptive evaluation. Assessment process cover different evaluation types in the virtual learning environment, linking the repositories with the correct assessment engine tools. The approach provides more accurate estimations of student's competencies level and a stronger relation between knowledge, activities, learning resources and type of evaluation tools, supporting in this way the automatic assessment and learning design generation. The process is supported by usage of educational standards and specifications and for an integral user modeling.
The lifelong competence development is a global tendency and e-learning process is used with the purpose to eliminate the space and time barriers. In this context, new pedagogical models supported for new assessment process models are necessary.
In order to integrate properly assessment within learning process, some proposals claim as main ideas: 1) Introduce assessment as another key element of leaning process and 2) Link each learning objective or competence with one or many kind of assessments. In this way, assessment becomes a way of spiral measuring for student's learning achievement. Consequently, assessment turns into a good source for feedback to learners, for generation of recommendations and for drive adaptations in the learning environment.
In this context we have analysed some proposals of new competence assessment process models and software tools. In this paper, a characterization of these models and software tools is presented.
We propose two different approaches in order to improve competence eassessment process. The first approach is the generation of adaptive assessment structure in the learning design based in the competence element definition. The second
Competencies are complex processes that people put into play in order to solve problems and to carry out activities (both at everyday life and at the workplace) [1]. Users and their characteristics are key elements in a competence-based learning process development, especially considering that the very evolution over time of those characteristics reflect the expected acquisition of users' competencies.
There are different standard and specifications to support competence definitions; some of them include elements about the competence development process and the associated actors. Table 1 describes three of the most important approaches. Minimalist, but extensible competence and educational objectives description. It considers basic elements such as competence title, description and also it offer the possibility of extend the competence information adding a general element <statement> in which can be added specific elements in the competence definition. The RDCEO Schema can be used in both academic and business contexts. It focus is to offer.
The objective of this project is the creation of an XML schema to provide trading partners standardized and practical means to exchange information about competencies within a variety of business contexts [2]. Additionally to the general information in the RDECO specification, this approach define explicitly two specifics elements in the competence definition, the evidence used to capture information to substantiate the existence, sufficiency, or level of a Competence and Weight element to capture of information on the relative importance of the Competency in different aspects. Ontology-Based Competency Management: Infrastructures for the Knowledge Intensive Learning Organization.
Above approaches are focused only in the information about the competence definition. This approach was created to support competence management, for this reason take into account elements such as the actors in the business process and job situations in which competence should be demonstrated. It principal purpose is to offer a complete framework to support decisions in human management in the business context.
In some countries, the trend is build and evaluates higher education academic curriculums driving by competencies definition. For example, in Colombia the National Education Ministry has been develop, through the Superior Education Foment Colombian Institute (ICFES), a quality standard measuring reference for higher education checking the degree of competencies development in students attending the final year of undergraduate. This standard is called ECAES (for his Spanish acronym, Exámenes de Calidad de Educación Superior en Colombia) [3]. In ECAES for each academic curriculum a series of competencies are defined within his specific knowledge context.
Interoperability, reusability, efficiency and abstract modeling have always been the main characteristics in e-learning design and e-assessment standards and specifications. In particular IMS Question and Test Interoperability (QTI) [4] is an open technical elearning specification to support the interoperability of systems and reusability of assessment resources. With QTI assessment items and test can be expressed and interchanged. IMS Learning Design (LD) [5] is a specification for a meta-language which enables the modeling of learning processes, designed to express many different pedagogies. The activities to develop in a learning design can be expressed with LD.
The current need of evaluation for competencies in long life learning process have exhibited some shortcomings of these standards and specifications mentioned. IMS QTI is just a specification about question definitions and response processing, and has nothing to do with teaching and learning activities [6]. Conversely, LD is used to support teaching-learning processes, but cannot explicitly support assessment [6].
In order to support the measuring of competencies development within an eassessment process new assessment types are required. Table 2 presents taxonomy of new assessment types.
After a period of work, the learner takes a test and then the teacher marks the test and assigns a score. The test aims to summarize learning up to that point.
Consider an assessment 'formative' when the feedback from learning activities is used to adapt the teaching to meet the learner's needs or to students take control of their own learning.
Portfolio assessment is that it emphasizes and evidences the learning process as an active demonstration of knowledge. It is used for evaluating learning processes and learning outcomes. It is used to encourage student involvement in their assessment, their interaction with other students, teachers, parents and the larger community.
Assessment where students making judgments about their own work. Students critique their own work, and form judgments about its strengths and weaknesses.
Student assessment of other students' work, both formative and summative. 360 Degree Feedback Is feedback that comes from all around the student. The name refers to the 360 degrees in a circle, with the student in the center of the circle. Feedback is provided by subordinates, peers, and teachers. It also includes a self assessment and, in some cases, feedback from external sources. Specific Competencies Assessment Specific competencies are directly related to a specific occupation and are focused on the "know" and "do". The individual competencies are a particular type of specific competencies.
These affect various fields and are transferable to a multitude of functions or training programs. They are focused on the "to be". Special types of transversal competencies are the collaborative competencies. They allow a group of individuals to carry out a job as the result of joint effort and cohesion towards achieving a common goal. Some researches have been produced software tools to support new specific types of assessment in e-learning as [7] - [16] and so on. Table 3 shows a summary of some tools with support for new types of assessment in e-learning environments. Nevertheless his data models are not based on standards producing loss of interoperability and reusability.
Therefore, in order to support a competence e-assessment process preserving interoperability, reusability, efficiency and abstract modeling, new models to extend the current specifications are required. First approaches in this sense [17] [18] propose to realize extensions providing insight into gaps between these different specifications, a UML model is proposed to extend and to combine QTI and LD specifications. Then, other research clarify the technical mechanism to do it, for example, it is possible to combine QTI and LD specifying how an outcome variable of QTI can be coupled to an LD property and integrating assessment applications tools to LD as services [19] - [24].
The most recent proposal over the initial idea of extend the current specifications promotes to create a new layer over QTI an LD establishing a new specification although building high-level assessment process modeling meta-language [3] [25].
Other kind of proposal has arrived with the LAMS project [26] in which LD and QTI specifications are the basis, but a totally new specification is being built in order to support whole range of possibilities in e-assessment.
Table 4 shows a summary of most important new models for e-assessment process focused in use of specifications and performing of traditional and new types of assessment.
Type of assessment Peers [7] Peer Assessment Peer Grader [8] Peer Assessment Net Peas [9] Peer Assessment eSPARK [10] Peer Assessment Espace [11] Peer Assessment Turnitin Peer Review [12] Peer Assessment SEUV [13] ECAES [3] TELOS [14] Portfolio Assessment, Specific Competencies Assessment Coala [15] Specific Competencies Assessment in programming Middleware to connect APIS QTI engine and Google Maps [16] Specific Competencies Assessment in manage of maps
Table 5 shows a summary of new tools for e-assessment based on these new models. Tools have been developed only for TENCompetence Assessment Model and LAMS Model. For OUNL/CITO assessment model were not developed Tools perhaps because TENCompetence Assessment Model is a reduce version of it and the research is concentrated in this small version.
According with the analysis of the state of the art in the competence e-assessment process there are different open questions in this research area such as: How it can express all types of assessment task in a standard learning design? What types of assessment are more appropriate for the educational objectives of a learning experience? How can these types of assessments to be customized to a specific learning context and to the expected benefits of a particular learning experience? What are the strategies for monitoring, assessment and evaluation? What are the adaptive strategies to provide in e-assessment process?. In particular, we are interesting in the automatic generation of adaptive assessment structures in a learning design, the support of whole e-assessment process and support of all kind of assessment types.
We propose two different ways to address the problem, the first use the Competence Element Definition, specifically, the evidence definition to decide how assessment structure can be generated. Our second approach describes a Data Model for Competencies Assessment based on item's Meta-data which are the input to an adaptive retrieval process. Finally the AEEA proposal incorporates the two approach mentioned above in a competence e-assessment process.
Our interest in this first approach was define a particular model for competence definition that permit us to specify the necessary elements in order to generate adaptive learning designs in the context of learning management systems [27].
IMS-RDCEO was the specification selected because it offers the possibility to define completely, the necessary elements for the learning design generation, which were identified by analyzing different curricular design methodologies.
Competence definition consists of elements such as learning results, essential knowledge, evidences, and competence context. Each of them has a specific identifier in the definition.
The proposal to extend [27] is to make that each learning resource have a competence element associated, in particular, those which their type is assessment, have a reference to the identifier of the evidence element in the competence definition. This association support the assessment structure generation in the leaning design. Figure 1 shows an example of a learning design generated, performance evidence activities are the assessment structures generated. This first approach is based on the existence of repositories of different types of evidence and in the association of this evidence to the competence definition.
In this second approach we begin proposing a modification on dotLRN Assessment Package. This Package is an implementation of QTI Light Specification.
Assessment offers to users the possibilities to add general information about the items such as the item description, if the item is required or not, the feedback for the student, associated points and the description of question type.
Our interest is to improve the assessment package in order to support retrieval process based in the item meta-data.
In this way, we propose to associate information about the competence definition in the item meta-data. In the Table 6, the information proposed to add is described.
With this extra information and the data existing now we are testing some vectorial algorithms to support assessment construction step.
Describes the main needed content to be addressed in order to be included in the adapted learning design for supporting competence acquisition.
Implement retrieval process based in the knowledge domain.
Environment in which the competence should be demonstrated.
Implement retrieval process based in the business associated context.
The proposal of adaptive assessment process is based on e-assessment process model proposed on [17] and [29] which define six steps. We group the steps in two big stages: Design time and Run time. Design time involves the first three process steps and Run time involves the last three process steps. We also propose that the adaptive decision could affect not only the feedback of the first step but also the feedback of the fourth step. Figure 2 show the e-assessment process model adopted.
Design Time Process Steps: In our previous work [28] a first approach of AEEA was proposal. In accord of the new e-assessment process model adopted, a second version has been produced. The AEEA is composed of two packages: Author Assessment Package and Monitoring Assessment Package. Figure 3 shows the new AEEA proposal. The Author Assessment Package supports the three first steps of e-assessment process in design time. The Monitoring Assessment Package supports the three last steps of e-assessment process in run time.
In the first step, Competencies Assessment Plan Design, an LD Editor Software Tool is used for configure the LD assessment plan where outcome variables of QTI can be coupled to LD properties. The result is the Competencies Assessment Plan supported over LD specification and XML meta-data for competencies information. Additionally, the Competencies Data Model and the Student's User Model are design too inside this step.
In the second and third steps, Items Construction and Tests Construction, items and tests are designed using Assessment Editor Software Tools and communication with External Test/Item Repositories. The complete result is the Competencies Assessment Data Model which is composed by four elements: Teacher's Formative Assessment Model, Summative Assessment Model, Self Assessment Model and a Peer Assessment Model. This data model is based in specifications as QTI and XML metadata to keep relation between competencies and assessment items.
The Monitoring Assessment Package provides Assessment Software Tools as services to LD for monitoring user's assessment tasks and update Student's User Model, executes adaptive transformations according the LD assessment plan and deliver recommendations. In order to produce adaptive transformations, Competencies Assessment Plan rules are checked and Student's User Model is modified.
The AEEA has been conceived to support new types of assessment, in particular: Summative Assessment, Self Assessment, Teacher's Formative Assessment and Peer Assessment. Also, the most important, assessment objectives are integrated with the other key elements of learning design through the Competencies Assessment Plan and the monitoring process for delivering feedback to learners in all assessment tasks.
Assessments play a significant role in the competence development process, and consequently there is a clear need for run interoperable and adaptive assessment test in the e-learning systems.
In this paper, we have looked at the problems associated with adaptive eassessment systems. Through an analysis of QTI and LD, we found that a combination and extension of both and service-oriented approach can meet technical requirements for supporting new forms of e-assessment.
We have proposed two different approaches in order to support competence eassessment process. First, the generation of adaptive assessment structure in the learning design based on the competence element definition. Second, the concept of new meta-data on the evaluation items for maintains information of competencies. The AEEA proposal is based on new models for e-assessment process which extend LD and QTI specifications. The AEEA proposed can give direction to the use of the LD and QTI specification to align teaching, learning and assessment. This educational model has been constructed to match the new approach of assessment, and can be used to describe new assessment types. Our approach has advantages in supporting interoperability, flexibility, and seamless integration with learning activities.
Our working now is focused on the first part of the AEEA implementation, in particular, develop of the Assessment Editor Software Tools and also prepare items of evaluation in different repositories and testing some vectorial algorithms.
As future work, the implementation of the Assessment Software Tools as services of dotLRN and proof of the architecture for design time and run time are projected.
Authors would like to thank to LASPAU Program, The Academic and Professional Programs for the Americas, (scholarship No. 20080847), to the Alban Program and the European Union Program of High Level Scholarships for Latin America (scholarship No. E06D103680CO). Also, thanks to Universidad del Valle, Colombia for its support and for the Spanish Science and Education Ministry for the financial support of A2UN@ project (TIN2008-06862-C04-02/TSI).