In computer aided assessment (CAA) and automated tutoring the feedback to a submitted solution is very important for the learning outcome of students. It should help them to improve their performances, which means it is designed to go beyond a classication into right or wrong. In this paper we consider the domain of mathematics and introduce a general framework that allows to localise mistakes precisely and oers many options in exercise and feedback design. We present rst experiences with the tool and discuss problems, we would like to work on in the future.
Whenever large numbers of students apply for a single university course, it is
often not possible to provide a suitable number of face-to-face tutoring sessions.
Experiences show that training sessions and feedback to exercises are very useful
to promote learning success [
Four major requirements were dened for the system: First, the system is intended to not only judge solutions in terms of right or wrong, but also to provide feedback as detailed as possible. Second, it must allow and encourage students to work on the same exercises more than once, e.g. by permitting dierent ways of solving tasks or by oering variations of tasks. Third, it must provide exibility in the design of exercises so that exercises can be added and changed easily. This implies the need for a generic way of representing mathematical content independent of the means of input specic to an exercise (e.g. plain text input or 1 Work on this project was funded by the Federal Ministry of Education and Research under grant number 01PL11075 input supported by formula editors) as well as of the means specic to grading and feedback generation (e.g. connections to computer algebra systems). Finally, the system is intended to not only cover school mathematics, but to provide all necessary means for input and evaluation of exercises in higher mathematics to cover a wide range of bachelor degree courses.
As a consequence of these requirements, the e-assessment system JACK is
used as a baseline for the project since it is designed as a exible framework
for computer supported exercises [
Details on the current progress of the project are presented in this paper in the following order: Section 2 reviews existing work both on e-learning systems for mathematics and generic representation of mathematical content. Section 3 elaborates on the concept of path-based exercises as well as on the techniques of feedback generation used within the JACK implementation. Section 4 digs into the details of the implementation and the integration of dierent parts of the system by using OpenMath. Section 5 describes the use of JACK in university classes. And Section 6 presents preliminary conclusions and next steps. 2
Recently, many e-learning systems have been developed. The e-learning systems
ActiveMath[
MathCoach[
Another e-learning system is Lon-Capa[
As a general tool for e-assessment and automated tutoring, exercises in JACK are not limited to the domain of mathematics. Instead, exercises may contain multiple choice, ll-in, and drop-down elements for receiving input from the students. Thus a minimal exercise consists of some sort of task description, at least one element that receives input from students, and at least one feedback message. However, JACK oers some options for a more sophisticated exercise design and more detailed feedback messages, in particular for exercises with mathematical content, including the support of LATEX, input with a formula editor and an evaluation of solutions with a computer algebra system. We will discuss the options in the following subsection. 3.1
As a basic feature of JACK, tasks may be generic by using variables as placeholder. These placeholders are lled in dynamically, so the exercise presents dierent content to the student every time it is attended. There is no limitation in where to use these variables, so the task description may vary, the number or content of multiple choice or drop-down options may vary, the expected correct results may vary, and so on. While this is of no immediate use for a single visit on a single exercise, it is very helpful when students work with the tutoring system for a longer time. In this case, they can work on the same exercise more than once, receiving dierent values within that exercise. Thus exercises remain challenging for a longer period of time. Moreover, it possibly helps students to understand the abstract concepts and encourages them to talk to each other about solution strategies instead of plain solutions.
In addition, JACK not only allows for single tasks within an exercise, but also for so-called path-based exercises. A path-based exercise in JACK consists of an arbitrary number of steps, where each step denes a single task. Usually, these tasks are related to each other in some way, e.g. by sharing some context or by reusing values or even inputs from a previous step. A good example of such an exercise is the sketching of a curve, where the determination of each feature of the curve can be seen as a step in the sense of path-based exercises.
Path-based exercises in JACK are not limited to linear sequences of steps, but can be considered as a directed graph. In this graph, each node represents a step of the exercise, and edges are allowed between any two steps. Each step (except for a pre-dened one, called end-step, that terminates the exercise) has at least one outgoing edge for the next step. If there is more than one outgoing edge, conditions must be provided under which the system takes the student to the particular step. In this way, not only the values inside a task can vary, but also the sequence of steps can be dierent every time a student attends the exercise. A sample graph for a path-based exercise is shown in gure 1.
Conditions for choosing an edge can depend on the following parameters: Variables: As we discussed earlier, exercises in JACK can be generic, e.g. the coecients of a polynomial can vary every time a student attends the exercise.
Hence it might depend on the variables, whether the polynomial has or hasn’t got zeroes. In case the function has got zeroes, the student might have to compute these, while it wouldn’t make sense to confront the student with this task when there are no zeroes.
Student’s result: The system can react dierently to correct and incorrect solutions. For example the student might have to redo the step if he gave the wrong answer, while he may continue otherwise. This approach enables teachers to simulate a strategy often used in oral exams.
Student’s input: The student’s input from previous steps can be used by the system in the following ones. This enables students to dictate their own pace of work. For example, when students have to solve a linear equation system, JACK is able to accept every correct transformation of the system and provide a new step, where the student has to continue with his input from the previous step.
For students who are not capable of solving a task, authors can dene a special succeeding step in case a student wants to "skip" the step. In this case, the student has the option to get to the next step without submitting a solution. In addition to that, the author has to dene a message, that is shown to the student instead of the usual feedback message. The possibility of skipping a step can help to prevent students from losing their motivation and thus giving up during an exercise. In summary, path-based generic exercises oer a big variety of possibilities for authors. Using these features, many aspects of an exercise from an oral or a written exam can be simulated. 3.2
consists of a score and a feedback message. According to the typology of Tunstall
and Gsipps[
We consider the latter kind of feedback as one of the central features of a tutoring system. It is intended to help the students to comprehend where they made mistakes or where they were correct. To be able to do so, JACK has to understand the semantics of a solution. For this reason, a computer algebra system (CAS) is used to evaluate solutions. On one hand, the CAS can verify the correctness of a solution, even if there are innitely many correct ones. On the other hand, it is able to locate errors precisely and to compare a student’s solution with a standard solution. This enables authors to provide granular feedback, evaluative feedback as well as descriptive feedback. 3.3
1. Conceptual mistakes: In case a student submits a solution, that does not have the desired form, JACK provides a special feedback for this purpose: (1) If the submitted equation does not depend on x, the system generates a feedback message, telling the student to use the variable x. (2) In case the student uses x and another variable to describe the line, the feedback message recapitulates how a line can be described through the equation y = mx + n. (3) If the submitted equation depends only on x, the degree in x of the term on the right-hand side still might be wrong. This means, the student has not computed a line. 2. Mistakes in computing: When the student has understood the concept of describing a line by its slope-intercept-form, he might still make mistakes during the computation of this form. Feedback cases were provided for the following mistakes: (1) If the slope was computed correctly but the y-intercept is wrong, the system generates a feedback message for this case. (2) Inverse slope: When students mix-up numerator and denominator in the formula that returns the slope for two given points, a feedback message is generated for this case. (3) If the line only passes through one of the given points, the feedback message that is generated can be seen in gure 3. (4) When none of the above mentioned feedback cases match, the feedback shows the submitted solution and the points P and Q as shown in gure 3, but with a dierent text. So the student can see, that the line passes through none of the given points. 3. Formal mistakes: Even students who have understood the concepts of computing a line from two given points and make no errors in computing may fail in this step because they do not use the desired form. They could for example use the form n + mx instead or extract some factor. While this is possible from the mathematical point of view, it may miss a didactical point of the exercise. Thus a special feedback and an additional step is dened for this case (see gure 1). 4
In exercises where a solution has to be entered into a text input eld, there can be virtually innite many possibilities to enter a correct solution. The expression 2a + 2b could be written as 2a + b + b or a + b + a + b, for example.
This is why a computer algebra system (CAS) needs to be used for the evaluation of submitted solutions. A CAS is able to compare the student’s input with the correct solution and can tell whether they are semantically equal or not. However, most CAS focus on one specic mathematical subdomain. Examples are Maxima 3 for analysis and Magma 4 for algebra.
In contrast to that, a web-based assessment system for mathematics should be able to cover a big variety of mathematical subdomains, such as algebra, analysis, numerical analysis, statistics and many more. Hence it should be considered to use more than one CAS for the evaluation of solutions. At the same time we cannot expect students to submit their solution in the syntax of the CAS, responsible for the evaluation. In fact, students do not even need to know which CAS is used to evaluate their particular input. Hence it is desirable to have a standardised input syntax. For complex mathematical expressions it even makes sense to support the input with a formula editor, which uses its own representation for expressions. Consequently, an e-assessment system needs to be able to translate between one or more input syntaxes and several CAS syntaxes.
This observation is not limited to the input side of systems, but exists in a similar way on the output side: It should be possible to include the output produced by a CAS into the feedback messages displayed to the students. Since the students may not know the output syntax used by a particular CAS, it again has to be translated into a readable syntax like LATEX. Moreover, the feedback may contain dynamically created graphics as shown in gure 3, so the output from the CAS needs to be translated into a syntax readable by the plotting tool.
For this reason, JACK uses the well-known approach of using OpenMath as
a standard representation for mathematical expressions [
Until now, two phrasebooks have been developed for the integration of external services to JACK, a Symja5 phrasebook and a LATEX-phrasebook6. Both phrasebooks are based on OpenMath Version 2.0 and they make use of the ocial OpenMath schema7. The two phrasebooks implement all the content dictionaries (CD) with status "ocial" 8 but they can easily be extended because
of their structure. To ensure a smooth integration into JACK, both phrasebooks were implemented in Java.
The CAS Symja provides round about 300 functions and shows a certain strength in the eld of parsing and comparing expressions. Of course the relatively small number of functions already shows, that Symja is limited in its abilities. For instance, there are no implementations of the inverse hyperbolic functions, which are part of the ocial OpenMath content dictionaries. This again shows the need for extensibility in terms of computer algebra systems, which is why we have started to work on the integration of Sage 9 into JACK. In addition to that, variables can be determined with the programming language R10. This method is already in use in some lectures that work with JACK. 5
JACK is already successfully in use in some courses at the University of DuisburgEssen. We started in spring 2013 with microeconomics I, a lecture attended by 722 students. 28 exercises were produced for JACK, which students could work on during the semester. Additionally, students could achieve bonus points in ve mini exams, held in JACK. The bonus points were then added to the students’ results in the nal exam, but only if they had passed the exam. Before using JACK only 48% of the students passed the exam (2011), while this semester (2013) 76% of the students were successful. After the promising start with microeconomics I, we continued using JACK in the lectures microeconomics II in the winter semester 2013/14 and microeconomics III in the summer semester 2014.
Since autumn 2013 JACK is used in classes for statistics, beginning with the lecture descriptive statistics in the winter semester 2013/14. JACK was used in the same way as in the courses for microeconomics. After a successful run with descriptive statistics, JACK is now used in the course inductive statistics as well. We gave workshops for both groups on how to write exercises for JACK, which
10 http://www.r-project.org/ allows them to provide the exercises for JACK on their own. As you can see in table 1 both groups have produced a signicant amount of exercises for JACK until now, which shows that the eort for creating an exercise is not too high.
In addition to that, JACK is used for a preperation course in mathematics as a blended learning concept for the department of economics. Students work on exercises in JACK to refresh their knowledge of school mathematics. Additionally they can ask questions in a forum or in face-to-face sessions. In table 1 you can see the number of exercises and number of students for all courses that used JACK. In this paper, we introduced a web-based tutoring system for mathematics in which exercises can be represented as directed graphs. Each node of the graph is a subtask, that needs to be fullled to solve the exercise. Authors of exercises have many options for connecting the nodes of such an exercise, which adds a great didactical value to the system. Furthermore the system supports authors with many feedback options, which enables them to respond very precisely to student’s mistakes. Feedback options include computations using the input as well as dynamic plotting of functions. Combining both, exercise design and feedback options, the system oers a big variety of tutoring skills, which can serve to improve the student’s learning outcome. We have pointed out with the help of some examples, how this can be done.
From the technical perspective, we have worked out that computer algebra systems play a central role for the purpose of providing feedback. Hence we need a semantic representation of mathematical expressions. A rst step has been taken with the connection to Symja, but we have seen that a more general solution for translations between various syntaxes is required. In this context, the use of OpenMath has proven its worth for the system, because an easy integration of further CAS is possible by implementing further phrasebooks. Our experiences from the implementation of the Symja phrasebook should be useful for this purpose and we will actually try to reuse some parts of it.
The fact that JACK is already in use in some courses at the University of Duisburg-Essen indicates that our approach is heading into the right direction. The pool of exercises as well as the number of participating groups is constantly growing. However, an analysis has shown that the abilities of JACK are not enough for some courses. This is particularly the case for classes from the eld of classical mathematics, such as Linear Algebra or Analysis where exercises go beyond computations and include proof strategies. It is our aim to extend our framework with such exercise types, which requires further research in the eld of automated theorem proving and reasoning as well as in other elds like mathematical user interfaces and representation formats for mathematical knowledge.