Computer methods for evaluating student's knowledge have traditionally been based on Multiple Choice Questions (MCQs) or llin-the-blank exercises, which do not provide a reliable basis upon which to assess student's underlying misconceptions. Because of this lack, we have devised and implemented a procedure for automatically deriving clickable students' conceptual models from their free-text answers. A student's conceptual model can be de ned as a network of interrelated concepts associated with a con dence value that indicates how well each student knows a concept. Several knowledge representation formats are used to show the generated conceptual model to the student. Furthermore, students can click on the concepts to get more information about them. 22 English Studies students are taking advantage of this new resource to review their Pragmatics course. Initial results show that they have found it very useful and claim that it is a good support for their review of the subject.
According to the theory of constructivism [
Therefore, it is necessary to have some reliable strategy to model the
student's conceptual knowledge. Currently, there are systems such as ConceptLab
[
In previous work [
The procedure has been implemented in the Will Tools1, which are a set of web-based applications that consist of: Willow, an automatic and adaptive freetext students' answers scorer; Willov, a conceptual model viewer; Willed, an authoring tool; and, Willoc, a con guration tool. In this paper, we present the next step of the procedure: to give the student more control over the generated model, with the consequence that it can be used not only for evaluation but also for tutoring. In order to achieve this goal, the students are no longer presented all the domain concepts in the conceptual model. Instead, only concepts with a con dence value higher than a certain threshold are shown. In this way, students can see how they construct their knowledge at their own particular rhythm from a blank conceptual model to a conceptual model with all domain concepts. Each domain concept will appear as it is correctly used in the answers provided to Willow but only if its con dence-value is higher than the threshold (e.g. 0.1).
Furthermore, the conceptual model is not only inspectable but clickable. Students can click on each concept of their conceptual model and learn more about it. This is useful to orient the study towards the concepts that are least understood, and guide the student to the questions that involve these concepts. It is also important to observe that since students can look at the conceptual model of the whole class, they can click on a concept that does not appear in his or her particular conceptual model, but that appears in the class conceptual model, and which may be important to assimilate if the concept is a precondition for assimilating other concepts.
A study is being undertaken in the 2007-2008 academic year, with 22 English Studies students using the Will Tools to review their Pragmatics course. Initial results show that students have found this new resource useful and they claim that it is a good support for their review of the course.
This paper is organized as follows: Section 2 describes the domain and student's conceptual models; Section 3 depicts some clickable and evolving representation formats in which the students' conceptual models are shown; Section 4 reports the results of the experiment performed with a group of English Studies students; and, nally Section 5 provides the main conclusions of the paper. 1 The systems are available on-line at http://www.eps.uam.es/~dperez/index1.html
The domain model contains the reference information of the course or area-ofknowledge under assessment. The information is provided by the teachers using the authoring tool called Willed. There may be one or more teachers using Willed to describe a course. In particular, it would be convenient that there are more than just one teacher as, in this way, the creation of the domain model is less dependent on a particular individual.
Firstly, teachers are asked the name of the course to model. Secondly, they are asked the name of the lessons of the course, and thirdly, they have to provide a set of questions per topic. The minimum information that should be given per question is: its statement in natural language; its maximum numerical score; its numerical score to pass the question; its di culty level in the range low (0), medium (1) or high (2); the topic to which the question is related to and, nally, a set of correct answers or references in natural language.
In order to organize this information provided by the teacher in the domain
model, we have devised a hierarchical structure of knowledge into three di erent
types of concepts. The reason for using this structure is to follow the organization
of the course provided by the teachers as much as possible. The three types of
concepts devised are:
{ Area-of-knowledge-concepts (AC): It is the name of the course to assess
as indicated by the teachers.
{ Topic-concepts (TCs): They are the name of the lessons of the course as
indicated by the teachers.
{ Basic-concepts (BCs): They are the key concepts of the area of
knowledge under study. BCs are automatically extracted from the correct answers
provided by the teachers to each question of the course using an automatic
Term Identi cation module [
For instance, for an \Operating Systems" course, the AC would be \Operating
Systems", one TC could be the \Concurrency" lesson and, and one BC could be
\thread ". Moreover, given that the goal is to nd out the level of assimilation of
each concept per student, all concepts are associated to a con dence-value (CV)
that re ects how well the system estimates that the student knows them. The
CV of a concept is between 0 and 1. A lower value means that the student does
not know the concept as s/he does not use it, while a higher value means that
the student con dently uses that concept. The CV is automatically updated as
the student answers questions according to a set of metrics [
Regarding the relationships between the concepts, we have devised three types of links between them according to the type of concepts that they relate (and following the criterion of adjusting the model as much as possible to the traditional course provided by the teacher): { Type 1, between ACs and TCs: Given that a course is usually structured into lessons, type 1 links relate the concept representing the whole course (the AC) with each lesson (each TC). A topic-concept may belong to di erent area-of-knowledge concepts, but as the model only represents one course, each TC can only be related to the AC. Type 1 links are automatically extracted from the information provided by the teachers (i.e. which lessons correspond to each course). { Type 2, between TC and BC: Given that each lesson has a set of questions with correct answers, type 2 links relate the concept representing the lesson (each TC) with each concept treated in that lesson (each BC). A basic-concept belongs to one or more topic-concepts. These relationships are important because they give us information about how the basic-concepts are grouped into topic-concepts and, how the students are able to use the BC in the di erent questions of the topics of the course. TCs are not linked among themselves, as the relationships between the topics are already captured by the type 3 links. Type 2 links are automatically extracted from the relationships between the topics and, the concepts found in the reference answers of the questions of the topic. { Type 3, between two BCs: A basic-concept can be related to one or more basic-concepts. These links are very important as they re ect how BCs are related in the student's cognitive structure as extracted from the students' answers. Therefore, unlike type 1 and type 2 links that are automatically extracted from the information provided by the teachers, type 3 links are automatically extracted from the information provided by the students.
We de ne a student's conceptual model as a simpli ed representation of the concepts and relationships among them that each student keeps in his or her mind about an area of knowledge at a given point of time. Conceptual models are useful both as a data model to guide the system's assessment of the student, and also as a form of feedback to both student and teacher, indicating the current state of progress of the student. As a resource to the system, the order and content of questions can be selected to focus on the misconceptions or erroneous links detected. In terms of feedback to the teacher and student, the presentation of a student's conceptual model makes evident the student's strengths and weaknesses. The teacher can also view the conceptual model of the class as a whole to see the strengths and weaknesses of the class, which may suggest that they need to spend more time teaching certain topics.
The student's conceptual model is not introduced by the teacher or by the
student, but generated from the answers provided by the students to the Willow
system [
Initially, each student's conceptual model has only the area-of-knowledge
concept (AC) and the topic-concepts (TCs) as indicated by the teacher and stored
in the domain model. Both AC and TCs have been associated a 0 con
dencevalue indicating that the student has never used them. Similarly, only type 1
and 2 links are represented as extracted from the domain model. Next, when the
students start using Willow to answer the questions indicated by the teachers,
they will start providing free-text answers, and from these answers, Willow
automatically identi es the basic-concepts used. Moreover, Willow calculates the
con dence-value associated to each concept according to the frequency metrics
[
The conceptual model can be represented in several knowledge representation
formats: a concept map, a conceptual diagram, a table, a bar chart and a
textual summary. The conceptual model is always updated with the information
gathered from the students' answers. This permits the capture of the
conceptual evolution of the students, since the conceptual models generated at di erent
times can be stored and reviewed later. In our previous work [
Therefore, we have changed the way the student and class models are accessed. In particular, students can now look at their own conceptual model and the class conceptual model in the Willow system, whereas teachers can look at the conceptual model of any student or group of students in a new conceptual model viewer for teachers (Willov). In this way, both students and teachers can keep track of the evolution of the models by looking at them several times during the course. The di erence now is that students can only see the concepts with a CV higher than a certain threshold (e.g. 0.1, that is, the concepts that have been mentioned at least once in their answers), while the teachers' representation is the same as in the previous version, showing all the concepts irrespectively of their CV. Additionally, students and teachers can see the conceptual model for each topic under review independently of other topics, and also a global view for all topics.
Furthermore, in order to help students understand the concepts that they have used wrongly, they can now click on each concept and be presented with an automatically generated explanation page. That is, the models are now not only inspectable but also clickable, and thus more power has been given to the student to control his or her learning. This does not give more work to the teacher. In fact, the teacher does not have to write the explanation page. It is generated from the information provided when the course was created. In particular, the explanation page shows all questions and the correct answers in which the concept has been used. The concept is marked with a color background so that the student can extract the meaning of the concept from the di erent contexts in which it appears.
Regarding the possible representation formats of the automatically generated student's conceptual models, two will be described in this paper: concept maps and conceptual diagrams. Concept maps are particularly useful for displaying networks of concepts. Each node represents a concept and the links between the nodes represent the relationships between the concepts. A web-like organization of the map has been chosen, as it is one of the most suitable formats for the hierarchy of concepts (BC, TC, AC) proposed. The type of node is indicated by the size and place in the concept map: the AC is bigger and it is always at the center, the TCs are medium-size and are placed in the second radial line, while the BCs are smaller and are placed in the outer radials lines; and, the links have been reorganized in an e ort to avoid crossings. The conceptual model can also be presented as a hierarchical diagram, with the most important concept at the top and less relevant concepts below. In this format, the focus is just on the concepts and, the relationships among them are not explicitly represented. Figure 1 shows a concept map and conceptual diagram representations of the student's conceptual model for one topic. 4
In the 2007-2008 academic year, Willow was used by 22 students out of 45 studying a \Pragmatics" course within the Department of English. Teachers provided material for Willow, consisting of 49 questions, each with 3 correct answers and covering four topics of the \Pragmatics" course. The use of the system was completely voluntary and did not a ect the grade given in the subject. The goal of the experiment was to nd out whether the students nd the new utilities in Willow useful for reviewing their course. It is important to highlight that since Willow is a Blended Learning tool, we do not aim to replace the teacher, but to support both the teachers and students by providing an alternative knowledge acquisition, assessment and representation format.
The only technical knowledge needed to use Willow is the ability to use a web browser. However, as it was the rst time the students used computers as a support for their studies, we gave them a short tutorial on the main features of Willow, and we organized a rst day of using Willow in class (in contrast with the normal intention of using the system after class). As we did not want to interfere with their manner of interaction with Willow (just the opposite, we wanted the students to explore the system by themselves), we did not explain some new features such as how to get more information about concepts by clicking on the display of the conceptual model, or how to follow their progress by looking at their conceptual model several times during the semester.
Rather than basing our evaluation on user questionnaires, which requires more work from the student, we set Willow to log each action the student performs within the system. In this way, at the end of the rst day of using Willow in class we had 22 logs (49% of the students volunteered to use the system in class). These logs revealed that even though they had not been told that they could check their progress by looking several times at the model after having answered questions, 14 students looked at the conceptual model 44 times, as gathered in Table 1.
Regarding how the conceptual model was viewed, the concept map format was most popular (32% of views). The conceptual diagram form was second in popularity (27%), while the bar chart and the textual summary were the least popular formats (possibly because they were the last options on the menu). Regarding the use of the individual versus class conceptual model, in 52% of the cases, students looked at only their own conceptual model, while in 48% of the cases they looked at both their own and the class conceptual models. When tabular presentation was used, the students were more concerned with their own results rather than the global results of the class. It is also interesting to observe that the number of students who looked rst their individual model and secondly, the class conceptual model is similar to the number of students who looked at the models in the reverse order. 5
The use of automatically generated students' conceptual models from the freetext answers provided to Willow has been extended not only for evaluating purposes but also for tutoring. Only concepts with a con dence value higher than a certain threshold are shown in the representation of the generated conceptual model, so that concepts that have never been used by the student do not appear in his or her own model. The student can still see these concepts in the class conceptual model and click on them to generate an immediate explanation page to nd out what information is lacking in his or her answers and to improve them. In this way, the next time that s/he answers the questions failed in Willow, if the student uses the new information provided by the explanation page, s/he will be able not only to pass the question but to generate a conceptual model with more concepts marked as correctly known, indicating that s/he has achieved a better knowledge of the subject.
A study is being undertaken in the 2007-2008 academic year, with 22 English Studies students using Willow to review their Pragmatics course. From the logs of the use of Willow, it can be stated that one of the most popular representation formats is the individual concept map. 6
This work has been sponsored by Spanish Ministry of Science and Technology, project number TIN2007-64718.