The lifelong competence development and transfer of competencies acquisition is a tendency of global world and specifically for e-learning.. Generally, an instructional based-competence process consists in four single sub-processes, the competence definition, competence development, competence assessment process and certification. This paper presents a competence-based adaptive assessment process for judging competence of learners in the context of a virtual learning environment. The goal of our research is build repositories of items linked to competencies definitions and rules specifications in order to the generation of adaptive evaluation. Assessment process cover different evaluation types in the virtual learning environment, linking the repositories with the correct assessment engine tools. The approach provides more accurate estimations of student's competencies level and a stronger relation between knowledge, activities, learning resources and type of evaluation tools, supporting in this way the automatic assessment and learning design generation. The process is supported by usage of educational standards and specifications and for an integral user modeling.
The lifelong competence development is a global tendency and e-learning process is used with the purpose to eliminate the space and time barriers. In this context, new pedagogical models supported for new assessment process models are necessary.
In order to integrate properly assessment within learning process, some proposals claim as main ideas: 1) Introduce assessment as another key element of leaning process and 2) Link each learning objective or competence with one or many kind of assessments. In this way, assessment becomes a way of spiral measuring for student’s learning achievement. Consequently, assessment turns into a good source for feedback to learners, for generation of recommendations and for drive adaptations in the learning environment.
In this context we have analysed some proposals of new competence assessment process models and software tools. In this paper, a characterization of these models and software tools is presented.
We propose two different approaches in order to improve competence eassessment process. The first approach is the generation of adaptive assessment structure in the learning design based in the competence element definition. The second is to introduce the concept of new meta-information on the evaluation items in order to support information of competencies within the assessment repositories.
An Adaptive Evaluation Engine Architecture (AEEA) is proposed too to supports the improved competence e-assessment process completely.
Both approaches and the AEEA take into account different methods of assessment for monitor the student’s competencies knowledge evolution and produce adaptive changes in assessment and learning design, and also it is our goal to integrate both approaches upon the open source learning management system dotLRN.
This paper is structured as follows: In section 1, the context and background of the proposal is described. In section 2, an extension for assessment process based on competencies evidence definition is proposed. In section 3, the data model for competencies assessment based on item’s meta-data is exposed. In section 4, the assessment process model adopted and the AEEA proposal is presented. In section 5, we outline some concluding remarks and future work.
Competencies are complex processes that people put into play in order to solve
problems and to carry out activities (both at everyday life and at the workplace) [
There are different standard and specifications to support competence definitions;
some of them include elements about the competence development process and the
associated actors. Table 1 describes three of the most important approaches.
National Education Ministry has been develop, through the Superior Education Foment
Colombian Institute (ICFES), a quality standard measuring reference for higher
education checking the degree of competencies development in students attending the
final year of undergraduate. This standard is called ECAES (for his Spanish acronym,
Exámenes de Calidad de Educación Superior en Colombia) [
Interoperability, reusability, efficiency and abstract modeling have always been the
main characteristics in e-learning design and e-assessment standards and specifications.
In particular IMS Question and Test Interoperability (QTI) [
The current need of evaluation for competencies in long life learning process have
exhibited some shortcomings of these standards and specifications mentioned. IMS
QTI is just a specification about question definitions and response processing, and has
nothing to do with teaching and learning activities [
In order to support the measuring of competencies development within an eassessment process new assessment types are required. Table 2 presents taxonomy of new assessment types. tools with support for new types of assessment in e-learning environments. Nevertheless his data models are not based on standards producing loss of interoperability and reusability.
Therefore, in order to support a competence e-assessment process preserving
interoperability, reusability, efficiency and abstract modeling, new models to extend
the current specifications are required. First approaches in this sense [
The most recent proposal over the initial idea of extend the current specifications
promotes to create a new layer over QTI an LD establishing a new specification
although building high-level assessment process modeling meta-language [
Other kind of proposal has arrived with the LAMS project [
Table 4 shows a summary of most important new models for e-assessment process focused in use of specifications and performing of traditional and new types of assessment.
Table 5 shows a summary of new tools for e-assessment based on these new models. Tools have been developed only for TENCompetence Assessment Model and LAMS Model. For OUNL/CITO assessment model were not developed Tools perhaps because TENCompetence Assessment Model is a reduce version of it and the research is concentrated in this small version.
According with the analysis of the state of the art in the competence e-assessment process there are different open questions in this research area such as: How it can express all types of assessment task in a standard learning design? What types of assessment are more appropriate for the educational objectives of a learning experience? How can these types of assessments to be customized to a specific learning context and to the expected benefits of a particular learning experience? What are the strategies for monitoring, assessment and evaluation? What are the adaptive strategies to provide in e-assessment process?. In particular, we are interesting in the automatic generation of adaptive assessment structures in a learning design, the support of whole e-assessment process and support of all kind of assessment types.
We propose two different ways to address the problem, the first use the Competence Element Definition, specifically, the evidence definition to decide how assessment structure can be generated. Our second approach describes a Data Model for Competencies Assessment based on item’s Meta-data which are the input to an adaptive retrieval process. Finally the AEEA proposal incorporates the two approach mentioned above in a competence e-assessment process.
Our interest in this first approach was define a particular model for competence
definition that permit us to specify the necessary elements in order to generate adaptive
learning designs in the context of learning management systems [
IMS-RDCEO was the specification selected because it offers the possibility to define completely, the necessary elements for the learning design generation, which were identified by analyzing different curricular design methodologies.
Competence definition consists of elements such as learning results, essential knowledge, evidences, and competence context. Each of them has a specific identifier in the definition.
The proposal to extend [
Figure 1 shows an example of a learning design generated, performance evidence activities are the assessment structures generated.
This first approach is based on the existence of repositories of different types of evidence and in the association of this evidence to the competence definition.
In this second approach we begin proposing a modification on dotLRN Assessment Package. This Package is an implementation of QTI Light Specification.
Assessment offers to users the possibilities to add general information about the items such as the item description, if the item is required or not, the feedback for the student, associated points and the description of question type.
Our interest is to improve the assessment package in order to support retrieval process based in the item meta-data.
In this way, we propose to associate information about the competence definition in the item meta-data. In the Table 6, the information proposed to add is described.
With this extra information and the data existing now we are testing some vectorial algorithms to support assessment construction step.
Element of Information Competencies knowledge Competence Context
The proposal of adaptive assessment process is based on e-assessment process model
proposed on [
Design Time Process Steps: 1. Competencies Assessment Plan Design: To select the sequence of assessment types that are appropriate for yield student’s competencies. Construction and definition of decision rules and assessment policies for adaptation. 2. Items Construction: To prepare items of evaluation in different assessment authoring software tools. 3. Tests Construction: To build units of assessment for each type of assessment propose in the assessment plan. The unit must assure the type and value of expected response in the plan.
Run Time Steps: 4. Assessment Execution: To display tests according to assessment plan and manage the student’s answers. 5. Qualification, Classification and Response: To calculate rubric score for tests and calculate the indicator score of competence assessment for each student. 6. Adaptive Decision Making: To follow the assessment plan rules for adaptive changes for each student. In some cases adaptations impact the execution of next tests, in other cases implies actualizations of the assessment plan.
In our previous work [
The Author Assessment Package supports the three first steps of e-assessment process in design time. The Monitoring Assessment Package supports the three last steps of e-assessment process in run time.
In the first step, Competencies Assessment Plan Design, an LD Editor Software Tool is used for configure the LD assessment plan where outcome variables of QTI can be coupled to LD properties. The result is the Competencies Assessment Plan supported over LD specification and XML meta-data for competencies information. Additionally, the Competencies Data Model and the Student’s User Model are design too inside this step.
In the second and third steps, Items Construction and Tests Construction, items and tests are designed using Assessment Editor Software Tools and communication with External Test/Item Repositories. The complete result is the Competencies Assessment Data Model which is composed by four elements: Teacher’s Formative Assessment Model, Summative Assessment Model, Self Assessment Model and a Peer Assessment Model. This data model is based in specifications as QTI and XML metadata to keep relation between competencies and assessment items.
The Monitoring Assessment Package provides Assessment Software Tools as services to LD for monitoring user’s assessment tasks and update Student’s User Model, executes adaptive transformations according the LD assessment plan and deliver recommendations. In order to produce adaptive transformations, Competencies Assessment Plan rules are checked and Student’s User Model is modified.
The AEEA has been conceived to support new types of assessment, in particular: Summative Assessment, Self Assessment, Teacher’s Formative Assessment and Peer Assessment. Also, the most important, assessment objectives are integrated with the other key elements of learning design through the Competencies Assessment Plan and the monitoring process for delivering feedback to learners in all assessment tasks.
Assessments play a significant role in the competence development process, and consequently there is a clear need for run interoperable and adaptive assessment test in the e-learning systems.
In this paper, we have looked at the problems associated with adaptive eassessment systems. Through an analysis of QTI and LD, we found that a combination and extension of both and service-oriented approach can meet technical requirements for supporting new forms of e-assessment.
We have proposed two different approaches in order to support competence eassessment process. First, the generation of adaptive assessment structure in the learning design based on the competence element definition. Second, the concept of new meta-data on the evaluation items for maintains information of competencies. The AEEA proposal is based on new models for e-assessment process which extend LD and QTI specifications. The AEEA proposed can give direction to the use of the LD and QTI specification to align teaching, learning and assessment. This educational model has been constructed to match the new approach of assessment, and can be used to describe new assessment types. Our approach has advantages in supporting interoperability, flexibility, and seamless integration with learning activities.
Our working now is focused on the first part of the AEEA implementation, in particular, develop of the Assessment Editor Software Tools and also prepare items of evaluation in different repositories and testing some vectorial algorithms.
As future work, the implementation of the Assessment Software Tools as services of dotLRN and proof of the architecture for design time and run time are projected.
Authors would like to thank to LASPAU Program, The Academic and Professional Programs for the Americas, (scholarship No. 20080847), to the Alban Program and the European Union Program of High Level Scholarships for Latin America (scholarship No. E06D103680CO). Also, thanks to Universidad del Valle, Colombia for its support and for the Spanish Science and Education Ministry for the financial support of A2UN@ project (TIN2008-06862-C04-02/TSI).