This paper reports about a pedagogical experiment at Zaporozhye National University (ZNU) aiming at improving motivation and learning quality in Computer Science Bachelor programme. The major novelty in teaching and learning practice introduced in the experiment was the use of peer evaluation for the assessment of coursework reports in two disciplines - one in the II-nd and the other in the IV-th year of study. The results were compared to the historical data collected in the previous 3-4 years. Our experiment proved that exploiting students' aspirations for informal leadership and incurred competition constructively is effective and yields some increase in motivation to learn and learning quality. The assessments were also subjectively regarded as more clear and better justified by the students involved in the experiment. A good side effect is also that the students learn the working patterns of the professionals in their field broadly used in academia and industry for making qualitative and unbiased peer evaluations.
Recent higher education experience reveals a substantial decrease of the popularity of
University education and degrees reflected for instance in the decrease in degree
completion rates [
Academia can not remove or relax demographic or economic factors unfortunately. Hence, the only feasible way of keeping academic performance at a competitive level is focusing on the stimuli for their students based on more social than purely pragmatic basics. For example, exploiting the value of informally assessed professional capability and leadership in student groups may be an effective way of stimulating spending more effort in learning.
The research presented in this paper aims at finding out such stimuli for Computer Science students based on their attitude to informal leadership grounded in professional competencies. The idea behind our pedagogical experiment was to place the subjects in an environment which is similar to professional and offer them to peerevaluate their individual work. Hence, the higher the grades a person gets from his or her peers in such an evaluation – the higher becomes the professional reputation of the person in the group, making him or her informal leader in the group of the peers.
In fact the approach we have taken is not new and has been effectively exploited in
the academic world as a peer evaluation mechanism as well as in social networks for
forming communities of interest and building social reputation for the individuals in
these communities. Such stimuli are qualified as solidary (in contrast to material)
incentives [
We have observed that being an evaluator for the peers’ work proved to be a noticeable incentive for the subjects who took part in our pedagogical experiment. Consequently the degree of active involvement and the quality of individual assignment results have increased substantially, in particular for the group in the last year of our Bachelor programme in Computer Science. This observation is backed up by the results presented in Section 4.
The rest of the paper is structured as follows. Section 2 gives a brief overview of the related work in higher education students’ motivation. Section 3 presents the setup of our pedagogical experiment. Section 4 discusses experimental results. Conclusions and plans for the future work are given in Section 5. 2
Motivations are denoted as “…reasons individuals have for behaving in a given
manner in a given situation” (c.f. [
Many authors stress the importance of a skill to maintain and enhance students’
motivation as one of the core capabilities of a University lecturer. “A wide variety of
theories of learning and teaching recognises motivation as an essential prerequisite for
successful learning. The ability to maintain and enhance student motivation is
therefore one of the most important skills …, and many publications and training
programmes devote considerable space and time to this matter. Applying this
theoretical knowledge in practice, however, remains difficult due to the complexity of
the concept and the number of different models of motivation available” (c.f. [
The mainstream of experimental studies in higher education teaching and learning
is centred around using the methodologies of individual subjective assessment by
subjects – based on interviews, questionnaires, etc (e.g. [
Stimulated by the necessity to seek for a remedy confronting the decrease of interest for quality learning at Universities, we have planned and further conducted a pedagogical experiment at ZNU. We have focused on the individual coursework as one of the important kinds of students’ creative activities in which motivation plays a very important role.
Our major objective was to prove a pedagogical hypothesis:
If students are given an opportunity to act as peer-evaluators of the other students reports, their extrinsic motivation to: (a) deliver the coursework; and (b) to perform as good as they can – will be higher than among those who do individual work in a traditional way and are graded by their instructor only.
Furthermore, the quality of submitted reports is expected to be better, as students informally compete and cross-evaluate their quality. Finally, the objectivity of the assessments will be higher; those will be perceived as fair by the subjects.
Chosen the disciplines: (a) for which the historical data on the coursework grades existed for several years; (b) for which the complexities of doing coursework assignments were comparable; and (c) the coverage of all four years of our Bachelor programme was even Developed detailed assessment forms for inexperienced evaluators offering a clear procedure and set of explicit metrics for coursework report assessment per each involved discipline Chosen the student groups comprising the cases when the same group acted as an experimental sample and formerly – a control sample; briefed the experimental groups Configured the set of software tools to support the experiment and developed written methodological recommendations for the subjects Adopted and adapted simple and effective metrics that allowed measuring the proofs of our research hypothesis 3.1
Pedagogical and Methodological Set-up The pedagogical set-up of our experiment covers: the choice of disciplines; the preparation of the evaluation forms; and subjects’ briefing about the evaluation procedure and tools.
First, we have chosen the disciplines with historical data and good coverage of out Bachelor programme. The choice is summarized in Table 1 – showing that: 3+ year historical data on the coursework assignment grades is available The disciplines cover all 4 years of study within the programme evenly
The complexity of the assignments, though different per discipline, is comparable as shown in Table 1. Table 1. Choice of disciplines and complexities of related coursework assignments.
Discipline Programming Algorithms and Data Structures DataBases and Information Systems Intro to Logical Programming and AI
Year
I II III IV 2008 --
Grades data for the coursework assignments in Programming (year I) and Databases and Information Systems (year III) form our first and second baseline control datasets respectively.
The complexity of the I-st year coursework assignment in Programming has been chosen as basic – represented by 100 abstract points. This coursework contains a survey part on a particular topic and a practical assignment to develop a program solving a given simple problem. The complexity of the coursework in this discipline remains without change for all the 3 years of our observations.
The complexity of the 3-d year coursework in Databases and Information Systems is also static within the period of observation. However, it is 1.5 times more complex as contains several interrelated practical problems in database and IS development using SQL Server software. Another difference is that the subjects for this assignment were the III-d year students whose motivation from one hand and experience from the other hand differ from the ones of I-st year students.
Observations in Algorithms and Data Structures and Introduction to Logical Programming and Artificial Intelligence contain both control and experimental (shaded gray in Table 1) data.
The complexity of the coursework assignment in Algorithms and Data Structures increases from 100 points in 2008 to 250 points in 2011. In 2008 it was very similar in structure to the coursework in Programming – a detailed written presentation of a sorting algorithm studied individually and its practical implementation in a computer program. In 2009 the task of analytically evaluating the computational complexity of the algorithm was added – raising the complexity up to 150 points. In 2010 the task of experimental measurement of the computational complexity and comparing it to the analytical estimation was added – the complexity has therefore increased to 200 points. In 2011 the coursework has been complicated (up to 250 points) by offering a comparative evaluation exercise – the students were tasked to measure the performance of their program and compare to the performance of a program developed by a fellow based on several common datasets containing records of different types.
Secondly, we have developed the evaluation forms for coursework reports in both disciplines. An example of a fragment of an evaluation form is pictured in Fig. 1.
X.Y.Zzzz
DD.MM.2011 Reviewer: Date: Report No:
The forms are in fact structured questionnaires covering all the sections of the
report and suggesting several weighted Likert scale [
Algorithms and Data Structures
It has been decided that the overall grade for a coursework report of maximum 20 points is divided in two parts: The coursework grade (0 – 15 points) computed as a mean of the three assessments done by two peers and one instructor The evaluation grade (0 – 5 points) computed as 5 minus the mean of deviations of the subject’s evaluation scores from the mean scores. So, the closer an individual scoring is to the mean scoring in all the evaluation assignments – the higher the resulting evaluation grade is. 3.2
Experimental and Control Groups Two experimental groups in the II-nd and IV-th year of study have been selected so that the historical coursework grade data was available for them. For comparison, the control data about the grades in the other groups of different years of study and in all four chosen disciplines have been taken into account. The groups for which the control data was accounted for have been further treated as control groups. Table 3 depicts the distribution of the control and experimental groups over the years of study. As could be seen in Table 3, the experimental groups are also control groups but in different disciplines and years of study. So, different ways of comparing the activity and performance in doing coursework assignments arise: the same group in different years; the same group in different disciplines; the same group as experimental and doing the work in a traditional way; etc.
At the beginning of the experiment the subjects of our two experimental groups were briefed about: the deadlines; the objectives of peer evaluation; the structure and the content of the evaluation forms; the grades that will be assigned for the reports and for the reviews; the tools they will use in the peer evaluation process. 3.3
Instrumental Set-up Two procedures have been chosen for evaluation that differed in the used tools. For the experiment with the II-nd year students the workflow based on e-mail exchange and manual supervision has been adopted. For the IV-th year students we have introduced the EasyChair Conference Management System1 as a tool to manage the process, final ranking and grading. In both cases the structured evaluation forms have been offered to the subjects to be filled out using MS Excel.
Evaluation process has been organized and executed similarly to peer evaluation of conference papers by programme committee members. Students were invited to serve on the programme committee and the review assignments have been made by the instructors who acted as programme chairs. The results of evaluation have been collected and processed using two different patterns: For II-nd year students – collected by e-mail and processed manually using Excel spread sheet as shown in Fig.2 in an anonymized way <Author 1> 7.40 <Author 2> <Author 3> <Author 4> <Author 5> <Author 6> <Author 7> <Author 8> <Author 9> 6.20 <Author 10> <Author 11> <Author 12> <Author 13> 1 re w e i ev R 2 re w e i ev
R
to the II-nd year students have been manually communicated by e-mail; and the IV-th year students have been notified by the EasyChair. 4.1
Additional Effort for Tutors As experienced, the additional instructors’ effort for organizing and managing the peer review process was substantial.
The major part of their additional work could be qualified as the set-up effort: developing review forms; creating review environments; compiling briefing manuals for the student reviewers; preparing management tables; and configuring the software tools. The result of this effort may however be re-used quite substantially – so the start-up effort may be regarded as an initial investment and neglected in further considerations.
Following two different workflows for the II-nd and IV-th year students implied different management efforts because of using different toolsets. Overall, using EasyChair Conference Management System appeared to be about 3 times less effort consuming than using just e-mail and MS Excel. 4.2 Interpretation of Experimental Results
Avg Submission
Avg Ratio GrNooup (S0c-o2r0e) SuNbmois TSotuta-l Ratio F(a0c-2tu0a)l sions dents
Avg Score among Submitted
Aligned by Complexity 4329
5,00 Broad horizontal sections correspond to the data related to one discipline. Two of them are baseline (as explained in Section 3.1) – Programming and Databases and Information Systems. The other two contain both control and experimental data – Algoritms and Data Structures (II-nd year) and Introduction to Logic Programming and AI (IV-th year). The Year column informs about the timing attribution of data (years of study and calendar years); The Group No column associates the rows to the academic groups. Group numbers may be found similar in several cases – reflecting the availability of both control and experimental measurements for several groups in different years and disciplines. The average scores are in fact based on the total number of students in a group which makes it different to the scores in the last two columns computed based on the number of submitted reports. Average Submission Ratio is in fact the measure that reflects the motivation of our students to submit their work The Factual Average Scores are the averages for the submitted reports, but without balancing them by coursework complexity Finally, the rightmost column contains the score averages multiplied by the complexity scaling factors provided in Table 1
Let us explain now how the results given in Table 4 and further interpreted graphically in Fig. 4 prove our research hypothesis.
Firstly, we expected that the introduction of peer reviews as an untraditional way of teaching will increase students’ extrinsic motivation. This expectation was valid as pictured by the values of submission ratio. Indeed, the ratio of coursework submission in our experiment with the II-nd year students reached the global maximum of 0.87 across all the disciplines. The next lower value was 0.72 which is 15 per cent lower. For the IV-th year subjects the increase in motivation was not that significantly high overall, though very substantial within their year of study. Indeed the reached submission ratio of 0.54 is 1.86 times better than the next lower value of 0.29 in 2010.
Secondly, the quality of submitted reports may have been interpreted as quite average in our experiments: 11.69 in the II-nd year and 13.33 in the IV-th. The registered decrease in scores, compared to the previous year, is: 23.96 per cent for the the II-nd year; and 20.03 per cent for the IV-th year. A compensation for that decrease in quality is twofold: (i) As the ratio of submissions increased the proportion of the best students (who always submit their work) decreased – so did the average scores. For the II-nd year the ratio increase was 15 percent versus a 23.96 decrease in scores. However, for the IV-th year the increase in submission ratio (86 per cent) substantially outperformed the decrease in average score (20.03 per cent). So, it could be concluded that our approach proved to be effective for the final year students of our Bachelor programme. (ii) The observed decrease in scores is to some extent explained by the increase of coursework complexity. Indeed, the maximal values of the average scores have been reached in the cases with substantially less complicated coursework assignments – as explained in Table 1. For example, the global maximum of 19.21 corresponds to the assignment weighted 150 points. It is ‘outperformed’ by the score of 13.33 in our IV-th year experiment because the complexity of the experimental coursework is 250 points. This imbalance is corrected by the values shown in the Aligned by Complexity column of Table 4.
Figure 4 pictures the trends observed in our experiment graphically. The Y-values in Fig. 4(a) are the numbers from the Submission Ratio column of Table 4; while the Y-values in Fig. 4(b) are taken from the Aligned by Complexity column of this table. (a) Submission Ratio (Motivation) (b) Quality of Reports
Finally, we hypothesized that the objectivity of the assessments will be higher in our experiment. We did not elaborate a proof for that as we did not undertake an experiment for scientifically measuring the objectivity. However, as a very draft estimation, we interviewed our subjects informally. These interviews revealed that the students treat their scores as more clear and objective compared to the previous experience, even if the scores were lower both individually and on average (column Factual of Table 4). 5
This paper reported about our pedagogical experiment undertaken for seeking a way of improving extrinsic motivation and learning quality of Computer Science students in our Bachelor programme at ZNU. The increase in motivation has been proven convincingly. Exploiting students’ aspirations for informal leadership and incurred competition constructively is effective and attracts people to learning. A gain in the quality of learning was a little bit over-estimated. Indeed, having involved more students in a creative learning activity does not guarantee that the quality of their work increases dramatically by miracle. However, the increase in motivation helped increasing also the quality to some degree – as shown in the previous section.
A good side effect is also that the students learn the working patterns of the professionals in their field broadly used in academia and industry for making qualitative and unbiased peer evaluations.
The results discussed in Section 4 appeared to be positive also for the other colleagues at the department of IT at our University. So, we plan to extend the experiment by covering more disciplines and collecting a broader sample of results in the near future. Among other things, this will allow us basing our work on a statistically representative set of subjects and making our results statistically valid. Finally, we plan to undertake an evaluation of the objectivity of the scoring in our settings.