This paper presents an innovation that includes modalities of generating and providing automated formative feedback to Computer Science undergraduate students. The paper addresses a challenge posed by the increasing numbers of students and amount of information teachers must deal with. A software solution providing automated formative feedback has been developed and used in this study in order to both support the students learning process and to use a criteria-based approach for assessment and peer-feedback. The automated data allowed for employing an algorithms-based analysis. Such analyses and findings provided the lecturer with input for new ways and tools to improve the students' learning gains, while the logs from the software program can further be used through machine learning and AI to teach the program able to self-determine the grade.
This paper presents the development and use of a software program and criteria developed for providing automatic feedback in a Computer Sciences undergraduate program. A user experience study was conducted to identify the students’ experiences both with the use of the system and with receiving feedback in this manner.
Feedback is viewed as a pedagogical strategy in teaching-learning environments that has potential to
facilitate the students’ learning in a meaningful manner (Jansson, 2006). Generally, evidence supports the idea
that feedback to students plays a significant role for student learning
Tools for generating automated feedback have been developed in the last years, but there is a new bust
in the field, with technologies of increased sophistication providing more realistic solutions. Common for all
these applications are that answers can be expressed in a formal way, which makes it feasible to automatically
analyze them. Automatic assessment of answers composed of free textual sentences, however, is evidently much
harder. Usually, the automatic feedback in an assessment system is given by a short indication of whether the
answer is wrong or right, some systems also allow a resubmission. Malmi and Korhonen (2004) conducted a
study that allows the students to resubmit assignments, but the exercise randomly changed the initial data, thus
giving a positive effect on the learning outcome.
Our innovation addresses these issues and this study aims to provide a better understanding of: how a software program developed for providing automatic formative feedback in a Computer Sciences undergraduate course can be implemented, how the automated feedback contributes to improve learning and how the students experience both the use of the system and receiving feedback in this manner. Whether students engage productively with feedback, whether it enhances their learning and performance, and whether automated fee feedback can have meaningful role in these processes are questions that require later empirical examination.
The software program
How does this software program work? The evaluators tick the checkboxes for which "mistakes" the student has made and each of these arguments have a weight which impacts the grading. The degree of impact for a given mistake is predefined by the teacher and closely linked to both the criteria and the Impression scale bar (see top-right box in Figure 1). Nevertheless, if the evaluator finds the impact of a certain mistake to be weighted too much, s/he can override the weight by manually adjusting the scrollbar at the Impression scale. The program will constantly follow and log all the choices made. All data is then processed so that the system can calibrate each evaluator 's discretionary assessments of the assignments. Through the quantified criteria, the assessment program will perform a variety of calculations. These calculations lead to an aggregation of several predefined text phrases that compile to a comprehensive textual feedback. This individual feedback is automatically generated based on different thresholds for the criteria determined in advance. Each criterion has an underlying text phrases and depending on the weighted accumulated value of the criterion, a specific text phrase will be picked out. Thus, all criteria are put together for a comprehensive individual feedback. The program can provide both the arguments for the grade as well as an individual formative feedback. If the evaluator or the teacher chooses to provide students with both arguments for their grade and an individual formative feedback, the first part of the feedback will consist of the arguments for each assignment and grade. The second part will be an individual formative feedback based on feedforward principles.
The user-interface has been designed primarily for the evaluator/teacher. Each assignment has its own tab at the top, but they all have some parts in common. The parts all assignments have in common are the blocks called Criteria, Arguments and Impression. For the Criteria, these are the main criteria which the teacher has predefined and are closely derived from the course taxonomy. For the Arguments, they are also predefined by the teacher. The arguments are “errors/mistakes” the students usually do. The Impression scale is a continuous scale with the only labeling of “Failed” – to – “Excellent”, not divided by grade or points. The formative assessment criteria The main component of the assessment program is the generation of the criteria, their weight in measurement and the textual phrases linked together. A taxonomic model (inspired roughly by Bloom’s taxonomy) was used to develop a set of criteria that focused on the students’ learning understanding of abstract knowledge and the way to employ this in solving computing problems. The set of developed criteria that were implemented in the system and were attached particular types of feedback is presented below: - Understanding (F) - the students’ ability to demonstrate good understanding; - Critical Abilities (K) – the capacity to be critical to their answers when asked to solve a task in different ways; - Neatness (R) - structure and tidiness for the entire answer; - Writing and Language (S) - mainly the clarity of the answer, using a professional language and terminology; - Blank answers / missed assignments (O) - number of missed assignments in intervals of (0), (1-3), (46), (7->); - General (G) - general points that are not captured by the above criteria. These can be grouped into: o Explanation of mind (FT) - assumptions or mistakes that make it difficult to follow the thought o Time for important tasks (TVO) - part of the most important tasks (the highest weighting) is not answered or that the student o Time allocation (TD)- use of the exam time o Exam Training (E) - unstructured and unprioritized solution to the weighted assignments o Time to Control (TK) – time management, finding alternative ways to solve the same task to see if the answers become the same o Presentation (P) - writing and language is not clear and comprehensive enough.
There is a trade-off for choosing the number of criteria. Too many criteria would make the assessment process long and with too many parameters which may lead to misunderstanding between the reviewers. On the other hand, too few criteria would restrict the nuances of assessing. The golden mean would be to have a hierarchical construction of criteria. On the top level, the main goals of the course, i.e. course taxonomy, can act as a grouping tag/name. Within each group there can be a few and specialized criteria, called sub-criteria. At this level of sub-criteria there can be different ways of putting in a weight, e.g. number scale, grades and so on. The criteria, sub-criteria and their representative weight generated a multidimensional matrix. This matrix is used to obtain a rich nuance for giving individual feedback for an assessment. As an example, one teacher had the following hierarchy shown in Figure 2. The teacher had chosen mostly to denote the weights as ´1´, ´0´ or ´-1´, meaning positive, neutral and negative impact, respectively. For the criterion “knowledge”, the sub-criteria had a different weighting scheme. Under the “grade” the scale was A-F, while for the “Arguments” there were letter-tags corresponding to a “predicted” error. Any given assessment requires a set of rules, criteria or guidelines for the evaluator to assess a task. When a teacher authors an exam or an assignment s/he has a prior knowledge of what the students may fail on. These predictions of possible errors or mistakes are collected and tagged in the program. The list of possible errors and mistakes are shown to the evaluator when assessing the exam/assignment. It is this list that is displayed in the Arguments box in Figure 1.
The back-end of the program is constantly monitoring and analyzing the evaluator’s choice and overruns. The results of these analysis lead to individual feedback to each student. The feedback consists of three main parts; (i) academic feedback and discipline-based justification of grade, (ii) personal feedforward and finally (iii) a profiling for the learning outcome. The length of the feedback is entirely dependent on the amount of choices the evaluator has made and the accumulated sum of the weights of the criteria throughout the whole assignment. The accumulated sum for each criterion is normalized to the classes and based on predefined thresholds groups the results. Each criteria result will give a corresponding match in one cell in the matrix holding text phrases. The number of cells to be selected depends on the significance, i.e. the normalized sum for a criterion and formed as a sequence of text.
All the students’ hand-in assignments were scanned and automatically uploaded into this assessment program. After each iteration, the students were asked to answer an online questionnaire. The average response rate has been 77%. Different questionnaires have been used to collect in answer regarding one or more of these topics: questions about the assignment; perception of the feedback received; evaluation of the technical aspects of the assessment program (computer program, usability, and time usage); development in relation to the professional domain; their experience of being a peer reviewer (evaluator); learning outcome for the students as a participant and a peer-reviewers. The results from the questionnaires were also discussed in the qualitative interviews. Since the questionnaire was anonymous, there was no opportunity to connect the questionnaires with the interviews. We have conducted qualitative interviews with 15% of the enrolled students for each course.
At the beginning of developing the assessment tool, one of main challenge was establishing the criteria for assessment. The most complicated aspect was how to define their descriptive scope as well as the nuance of weighting. The criteria were adjusted during the process, some becoming more detailed and others being removed. The process and several iterations resulted in grouping the criteria in a more general group, which can be linked to the course taxonomy. Based on the work done in this paper, new courses can easily employ these criteria with minimal configuration. Working with the criteria has also given the lecturer better understanding in how to develop better assignments, i.e. assignments with a clear distinction of levels.
From the user/students’side, the responses regarding to the usability and stability of the software program are displayed in Figure 1 below. We notice a high number of positive responses regarding both variables, with more than 60% of the students being content or very content with the stability of the tool and about half of them with the usability. Considering this was the first iteration of the innovation, we consider these experiences as being satisfactory and good input for proceeding with further adjustments.
The questionnaire data, from which we extracted answer to a number of core questions (Figure 4), to pin down students’ experiences with the feedback provided through the innovation we implemented. The majority of the students were generally positive both to the way the feedback was given as well as the content. More that 90% of the students liked the form in which feedback was provided, namely, the way the feedback is structured and the way different elements are described. The responses indicate overwhelming positive engagement from the students with the feedback as a report or tool. They responded in the open comments field that this feedback helped them to understand their shortcomings regarding to what they are assessed for. An expectation was that students would use this information and deploy it to advance their knowledge and learning, but that appears not to be the case. We also mapped the students’ prior expectations and how they felt the feedback is tailored for their assignment. For the extent of the feedback matching their own expectations on how they performed on the assignment the majority, 74%, responded good and very good. This could mean that the students are well informed about their own knowledge level, but not sure of what they are being measured, given their answers that they appreciate guidance about what is expected of them.
Based on the questionnaire answers, the students appear to have a good understanding of their own level of knowledge and how they performed on the assignment, self-efficacy, but lack the context of what and how they are to be assessed. Especially, it seems to be of great interest for the students how their assignment is assessed in term of / in context of the learning goals of the course. Finally, with regard to the tailoring of the feedback to their assignment and performance, almost 60% of the student rated it as very good and good, which indicated that the automated assessment program is able to provide feedback that is perceived as individualized and appropriate.
This paper presented a case study that involved the development and testing of a software program and criteria for formative individualized assessment of exams and automated feedback in computer science education. The innovative software collects data within several dimensions, by means of the hand-in assignments, the evaluators’ response and the formative feedback. The automated data allows for employing an algorithms-based analysis, which provide the lecturer with input regarding new ways and tools to improve the learning gain of the students. Furthermore, the collected data and the logs from the software program can be used through machine learning and AI to teach the program able to self-determine the grade. This innovative assessment program gives a unique and efficient way to collect the much-needed dataset to potentially perform and verify new algorithms within Big Data and AI.
In addition, the students’ perspectives to feedback on their exams and the experiences with the software
program and the feedback provided were explored. The findings indicate positive experiences and students
benefiting from the feedback, showing that providing the students with explicit feedback in addition to regular
summative evaluation (grades and point sum) stimulated their understanding and awareness of their knowledge
level and performance
At the level of the educational practice, the method employed provides an innovative tool available for the teacher and the evaluator to provide the students with feedback. The developed system involves teacher’s work to define in writing (phrase) what given values of criteria mean, with several criteria leading a matrix of possible feedback. The different feedback will make it easier to calibrate the evaluators across the disciplines. That way, not only the students benefit from this approach, but also the teachers/evaluators gain better understanding of what is required from the students in the various assignments and future learning. Furthermore, the system provides possibility to perform reliable assessment, i.e., calculate degrees of agreement between evaluators, which can represent a considerable achievement.