In a blended learning environment, paper-based evaluation has still been the preferred assessment method of student's learning because of the convenience it offers. In programming learning, several automatic programming evaluation tools have been developed. However, they are less focused on grading paper-based programming problems. In this paper, we discuss a homegrown educational technology called Web-based Programming Grading Assistant (WPGA). It is a system that facilitates grading and feedback delivery of paperbased assessments. A classroom study was designed and feedback from the users were collected. This paper reports a subjective evaluation of the role of the system to their learning, the ease of using the system, and their feedback on some specific features. Results show that students generally gave positive feedback on the WPGA system. Furthermore, the inclusion of analytics and social/peer learning features were some of the recommendations solicited to improve the system.
In a blended learning environment, paper-based evaluation has still been one of the
most preferred assessment methods of student’s learning because of its convenience in
terms of preparation. Furthermore, it is easier to reduce the risk of academic dishonesty,
which is prevalent in its online counterpart. However, it has its own drawbacks. For
instance, the inconsistency among or even within graders [
To address the above-mentioned issues, specifically in programming learning domain, a web-based platform was developed called Web-based Programming Grading Assistant (WPGA). WPGA serves as a digital-and-physical conjunction platform, which supports traditional blended classroom conduct as well as harnesses modern digital footprints. This system is capable of capturing students’ learning behaviors by tracing their learning activities. WPGA is designed such that it would stimulate the students to review and reflect on their graded assessments. Also, students will be able to access a digital copy of their graded physical test paper. It is hypothesized that student’s learning will be positively impacted by their ability to review and reflect on their graded test papers online. 2
Positive feedback does not necessarily produce positive results on the growth of the
students [
Aside from assessing what the students know, the assessments should also be able
to allow the students to reflect and evaluate their performance. It enables them to
understand their reasoning processes. There are two types of reflection: in action (during
study and practice) and on action (after being assessed) [
In the system developed, students will receive feedback in electronic form after their works are graded. This includes both formative and summative feedback. This will allow the students to react on the graded items and to take some notes for future reference. The objective of this work is to make it possible for the prompt delivery of feedback to students and to capture how students attend to it. Also, it seeks to understand students’ reviewing and reflecting behaviors and how they impact on their learning. 3
Several systems have already been developed to automate the generation of feedback
in STEM courses. Examples of which include WEB-CAT [
A few studies have attempted to address the problem. One proposed solution was to
digitize the giving of feedback and provide a centralized grading interface for the
graders. This is made possible by tablet grading system [
A web-based system was developed to help in grading paper-based exams and in providing direct feedback online. The name of the system is Web-based Programming Grading Assistant (WPGA)1. It connects paper-based assessments to the cyber world and ensures instructors that they can still have paper-based exams without having to learn new tools. It has three key features: (1) digitalization of paper-based assessments, (2) augmented grading platform, and (3) reflective feedback delivery. 4.1
WPGA utilizes quick response (QR) codes to label and identify a hard copy of an exam of a student. Instructors upload their student rosters, which contain a list of student identification and names. An automatic document feeder is used to scan all the paper exams of the students. Afterwards, these images are uploaded to the online system. All the scanned exams will be stored as images and will be recognized and indexed using the QR code. 4.2
After digitizing and labeling an exam, instructors can partition it into multiple sections. These sections can be assigned to different graders as WPGA allows multiple graders to grade them simultaneously. In effect, the system reduces the turnaround time of the distribution of grades. Also, graders’ grading coherence will improve.
The grading interface is shown in Figure 1. The system allows multiple types of
feedback to be given through its interactive buttons, which are located on the upper
right corner. Every rubric default to a perfect score, which translates to a full
understanding of the concept. For every click of the button, the color of the button changes:
red (partial understanding) or grey (missing the concept). Also, the grade is
decremented and the overall score is recalculated. Free form feedback can also be provided
in the comment section. The grader can even write or mark on the question image (the
1 http://cidsewpga.fulton.ad.asu.edu/gradingHelper/
paper exam image in the middle of interface). According to a previous study [
The collection of feedback was administered during the last week of the Fall semester in 2016. All the students who were using the system in any of their CSE courses were invited to fill the feedback form. They were informed that this survey will not affect their academic standing. In this manner, they can be honest about their answers. A total of 199 respondents answered the WPGA survey. 42.7% responded positively while 26.9% were uncertain or undecided. Since the system has just been recently introduced to the students, and was offered to support returned paper delivery, (we aimed to capture students’ natural reviewing and reflecting behaviors, reported in the main conference paper), this might explain why less than half were using it to help them in their learning. Students were asked how they would normally review for a programming exam. The following were the popular choices: review the slideshows from lecture (78.2%), review assignments (68%), and create a study guide (45.7%). Based on this result, majority of the students identify their study preparation process involves with a range of review activities. It led us to believe that capturing and modeling the corresponding reviewing behaviors can predict learning outcome. In terms of using the WPGA system, 56% of the students found it easy to use. Majority of the students felt comfortable using the system after taking 1-2 quizzes. A quick survey of the awareness of functionalities revealed that majority of the users already knew what the color-coding means for a given problem. Unfortunately, only few students know how to use the advanced features, which includes filtering important/bookmarked/unknown questions, bookmarking a question, or making a note. The possible explanations could be because they are using a new system and it was introduced only in the latter part of the semester. Also, during that time, there was no user manual that could have helped them familiarize the system. Another possible explanation could be students are just mainly focused on examining their exam scores and they do not bother exploring other features offered to them by the system. With WPGA, it is easier for them to access their exam scores virtually anywhere. However, this raises an issue on whether the other features of WPGA are indeed visible to the users.
The students were asked to provide some feedback on some specific features of WPGA. When asked about the usefulness of the bookmark feature, majority (65.8%) had no opinion about it. This could be attributed to the fact that only few users know it or are aware on how to use it. The same goes for the “SHOW: Unknown” feature. Majority (68.2%) had no opinion about it because many them do not know that the feature exists.
The following were the features suggested to be included to improve the system: (1) make available to the students the analytics showing the overall performance of everyone in the class; and (2) include social and peer learning features which will allow them to communicate not only with the professor but with other students as well.
Students would like the ability to communicate with the professor or TA through WPGA because they wanted to ask for help for them to understand a specific question (63.4%). The next reason would be they wanted WPGA to be able to help suggest on what content to focus on (51.2%), which suggests more advanced predictive models (i.e. recommenders, intelligent tutors). Lastly, they wanted a facility on how to rebut about points assigned on a specific question on an exam (43%). Overall, based on the feedback provided by the students, 53.6% would like to use WPGA for another CSE course. 6
We designed and deployed a system called Web-based Programming Grading Assignment (WPGA), which is a new educational technology that facilitates grading and feedback delivery of paper-based assessments. The system is designed to bridge the learning analytics from physical papers to digital electronic formats. From the classroom deployment and students’ survey results, we learned that most of students use WPGA only to view test results and questions but not use advanced functions like bookmark and note taking that could have helped them in reviewing. In addition, the system was just deployed during the latter part of the Fall semester in 2016 which limited its usage. Despite this, students generally gave positive feedback on the WPGA system. They even wanted to use it in their other CSE courses.
Using the feedback provided by the students, another version of WPGA will be developed to help improve the interaction of the users and the user interface. Through this, the students would become more aware of the existence of several important features. Furthermore, new functionalities, such as giving personalized feedback and recommendations to students based on their mistakes, will be explored to increase the impact of WPGA on learning.