Can One ECTS Credit Make All the Difference? Comparisons of the Actual Student Workload versus the Credit Inflation Timo Hynninen Antti Knutas Jussi Kasurinen Department of Information LUT School of Engineering LUT School of Engineering Technology Science Science South-Eastern Finland University Lappeenranta University of Lappeenranta University of of Applied Sciences Technology Technology Mikkeli, Finland Lappeenranta, Finland Lappeenranta, Finland timo.hynninen@xamk.fi antti.knutas@lut.fi jussi.kasurinen@lut.fi ABSTRACT How large impact does one ECTS credit have on the student motivation and effort? In this paper, we compared the results ACM Reference format: from learning environment data and post-course questionnaires between three different years of the same introductory Timo Hynninen, Antti Knutas and Jussi Kasurinen. 2018. Can One ECTS Credit Make All the Difference? Comparisons of the Actual Student programming course to gain insight on the perceived workload Workload versus the Credit Inflation. In Proceedings of the 2018 Workshop from a student’s point of view. While the teaching material and on PhD Software Engineering Education: Challenges, Trends, and Programs required assignments for successfully completing the course (SWEPHD2018). St. Petersburg, Russia, 6 pages. stayed mostly the same apart from minor scheduling tweaks, the reward for completing the course was raised from 5 ECTS credits to 6 ECTS during the observation period. According to our 1 Introduction statistical analysis, the difference in student perception of the With increased student volumes, the teaching methods have course workload in relation to the reward was insignificant: to adjust towards self-oriented approaches, and the curricula in Even though the reward was higher in the latter years and general adapts to the needs and realities of the available teaching passing requirements were mostly the same, the students’ resources [1]. In some situations, this leads to the strategy of assessment of the workload and their course activity did not repackaging the taught subjects into larger mass-course change or did not lead to better results. This indicates that the modules, which if nothing else, streamlines the bureaucratic sixth credit may have been lost to the credit inflation caused by process behind the course management, since there are the revised curricula, and that the one extra credit does not numerically fewer courses to manage. However, this activity has increase the overall motivation towards the course to a large the unbeneficial side-effect of grade and degree inflation [2-3], degree. This also implies, that from the viewpoint of providing which have several root causes, but the oversimplification of the more educational content, offering several small courses might curricula is amongst of them. Also in general, the student be more efficient than offering few large course modules. workload has been reported to be on downward cycle in several universities and colleges [21]. CCS CONCEPTS In this paper, we study the effect of grade and study point • Social and professional topics → Professional topics; inflation against the actual workload of the students, with three mostly identical courses. The first one is a five credit course and Computing education; Student assessment second is a six credit course extending the five credit course. Additionally, we present the data from the same six credit course KEYWORDS arranged the following year with some changes in place, to put Curriculum, course workload, grade inflation the comparison of the two different ECTS rewards in better context. The research questions are 1) how to measure the incentive of ECTS reward to student effort, and 2) how a revised Copyright © 2018 for the individual papers by the papers' authors. Copying reward affects students’ perception of course workload. These permitted for private and academic purposes. This volume is published and questions are important since many European universities use copyrighted by its editors. The 2018 Workshop on PhD Software Engineering Education: Challenges, the ECTS system as a common quantifier for measuring the Trends, and Programs, September 17th, 2018, St. Petersburg, Russia volume of studies, and many times students plan their own study schedules based on factors, which include course sizes. Curricula sizing is therefore an important part of software engineering SWEPHD2018, September, 2018, St. Petersburg, Russia T. Hynninen et al. education on all levels, from undergraduate to graduate and backgrounds, the key indicators implied minor improvements in doctoral programs. the motivation, and major improvements in the final grades. To answer these questions, this work presents a case study In purely motivational aspects, the different technical where the students’ results, the effort required to complete solutions such as robots or game-like design for added coursework, and perceived course workload from an motivational push has been studied. For example McGill [9] introductory programming course are analyzed. The study was applied personal robots and robotics in the fundamental courses conducted by comparing student data from two iterations of the in programming. Although not novel concept (for example [11]), first programming course (CS1): The first group of respondents the robot programming was considered an important aspect for took the course in 2015 and were asked to evaluate their own the student motivation to learn programming. In the study, workload when they received 5 ECTS credits for completing the McGill observed that even though developing programs with an course. The same course was arranged in 2016, where the actual robot increased the attention of the students towards the passing requirements and course material stayed the same with 2 course, it had minimal or neutral impact on the satisfaction, more weeks to accomplish everything, this time awarding 6 confidence or relevance factors, which were the other measured ECTS credit points We also compare the differences of the 2015 attributes. This result is partially supported for example by and 2016 course versions with the latest iteration of the course in McWhorter and O’Connor [12]; if the motivational aspect fails, 2017. the robots do not provide meaningful amounts of other We accumulated data from three sources: The virtual learning improvement factors. environment (VLE) (see [4]) was used to gather the data on the Games as the motivation tactics was studied by Jiau et al. time used for individual assignments. Coursework grading [10]. Their approach was to include algorithm optimization provided the performance ratings, while the post-course self- exercises as a game development task and problem-solving assessment questionnaires were used to survey how well the challenge. In their study, they report significant improvement on students thought the courses ECTS volume and workload the student outcomes and course results by applying this estimate correspond to actual time used for completing the technique, a similar observation that was also applied in the coursework. development of the Alice learning tool for object-oriented This study is also a continuance work to our previous studies programming [13]. into the student motivation and activities during the The intrinsic student motivation is an important aspect of the programming courses. In our prior work, we have studied for course outcomes, but the other aspect of motivation is the example collaborative learning [5], student plagiarism networks incentive-based motivation [14]. As based on a meta-analysis [6] and the impact of online-enabled course content to the conducted on a large population of volunteers, Cerasoli et al. intrinsic motivation [7]. In this study, the objective is to assess [14] observed, that the intrinsic motivation is most effective on the effect of incentive-based motivation, by comparison of two the highly professional and specialized work, whereas the course implementations which share similar features, faculty, incentive-based motivation is more effective on repetitive and student population and content, but the other course offering low-level and straightforward assignments. Similar observations one additional study point because of minor changes to the on the effects of extrinsic rewards has been made also by for course curricula and the overall study program structures. example Gagné and Deci [15]. Following these concepts, in the The rest of the article is structured as follows: Section 2 fundamentals-level learning assignments where the level of discusses related existing studies, and the research process is customization and analyzation is low, and the personal interest explained in the Section 3. Section 4 has the results, which are towards the subject is not guaranteed, the students should discussed in the Section 5. Section 6 summarizes the paper with respond to the incentive-based motivational aspect positively, the conclusions. and it should have a meaningful impact. 2 Related studies 3 Research process It is an established fact that in software engineering in The course Introduction to Programming was used as a test general, and especially in education, motivation is the key for case in our experimental setup. The course spans the fall achieving progress and enhanced results (for example [8-10]). semester, consisting of 12 to 14 weeks of lectures, programming In a study by Forte and Guzdial [8], the introductory course exercises, a 50 hour programming project and midterm exams or for computer science was tailored for the audiences in an a separate final exam. The original course was designed to attempt to increase the student interest towards the topic, while minimize the amount of so-called hygiene problems (see [16]), simultaneously minimizing the number of withdrawals and the small annoyances which hinder the actually productive work failed final grades. In their work the first programming course by causing interruptions and unnecessarily rising the learning was offered in three versions: the traditional introductory curve, and to promote the student motivation over the course course, a course tailored for the engineering students, and a coverage, deferring advanced topics such as the memory course tailored for other non-computer science disciplines. The management or pointers to the following advanced courses. The results were reflective of the motivational aspects: in the courses original design and implementation work is documented in where the content was tailored towards the audience detail in the publication by Nikula et al. [17]. Can One ECTS Credit Make All the Difference? SWEPHD2018, September, 2018, St. Petersburg, Russia An opportunity to study the effect of the incentive-based Table 1. Outline and offline workload estimates for motivation came possible, when the course curricula was revised the case courses. to follow a standard of 6 ECTS/course structure in replacement 2015 2016 (6 2017 (6 of the 5 ECTS/course structure our university followed earlier. (5 ECTS) ECTS) The change on the amount of given credits required the course ECTS) syllabus to include 27 hours of extra work from the students, to Lectures (offline, not 24 h 28 h 28 h justify the addition of one ECTS course credit. In practice, the mandatory) added hours to the course plan caused main difference between Tutoring / recitation 24 h 28 h 28 h the two implementations to be that the 6-credit implementation groups (offline, not mandatory) ran two weeks longer, had two additional lectures on ancillary Reserved for completing 35 h 40 h 40 h topics, and two additional sets of exercises replacing the the programming exercises voluntary extra assignments from the 2015 implementation. The (online, mandatory 25% requirements for grades stayed the same in 2016, even though completion) the course lasted for an additional two weeks during which Reserved for completing 45 h 50 h 50 h previously extra credit only weekly exercises and additional the programming project lecture material was covered. (online, mandatory) Essential learning goals also remained the same throughout Exam and midterm 3+3h 3 + 3 h 3 + 3 h the comparable years. In 2017, the weekly assignments and (mandatory) (paper (online) (computer programming project were developed further to better fit the and classroom pen) exam, not ECTS sizing, and therefore we were unable to use the data from online) the latest iteration of the course for statistical analysis. However, Reserved for other - 10 h 10 h we can use the descriptive indicators from 2017 in comparison to independent self-study the last two years to establish context beyond examining the such as preparation to the difference between single years. exam Overall, on both 2015 and 2016 implementations the final Total expected work hours 80 h 96 h 90 h grade was based on the separate grades from the exam, exercises online and the project work. The courses organized in 2015 and 2016 Total expected work hours 135 h 162 h 162 h were identical in the expected minimum effort, and the effort for the course (1 ECTS = 27h of work) needed to receive the best possible grade. Additionally, even though the amount of students on the course rose from previous in 2016, the student body was still very homogenous, as all the students for whom the course is mandatory came from degree Table 2. Descriptive statistics from the studied cases programs in technology and engineering. 2015 (5 2016 (6 2017 (6 The two student populations from the courses were compared ECTS) ECTS) ECTS) ** against each other to test whether the groups were statistically Total students 325 527 545 similar. This was tested with chi-squared test [18] to establish (enrolled) that the two groups and their course performances were Survey 116 (36%) 182 103 (19 %) independent variables, ie. that the probability to pass the course respondents (N) (35%) was not affected by the participation year and that the groups Average online 61,8 78,6* 55.8 were similarly capable. The chi-squared test concluded that the time (hrs) per user groups were independent with results being independent Average online 61,8 75,7 55.8 time (hrs) per user variables with the confidence level below p<.01. Therefore, the without online student samples were not correlating with the participation year exams or possibility to pass the course, and the 2015 and 2016 metrics Median online 52,3 58, 8* 50.1 could be compared against each other. Table 1 presents the time (hrs) course activities and planned online and offline workloads, Median online 52,3 56,3 50.1 which are intended to represent the time and effort an average time (hrs) without student spend throughout the course, with the Table 2 online exams summarizing the student average effort. The other statistical data Course work 287 454 523 was collected from the learning environment used to collect and started Passing grades 249 (76.6%) 374 380 (69.7 %) autograde assignments (see [17]) and from the student surveys (Pass-%) (70.9%) conducted at the end of the course. From these data, the Average grade 3,86 (5) 3,09 (4) 3.49 (4) collected information was analyzed with the Mann-Whitney U from the project test to evaluate the difference in distributions between 2015 and (median) 2016. The test was selected because it is suitable for the *includes the online exam which was used in 2016 only. independent sample, non-parametric data [19]. **in 2017 the course assignments and project were different. SWEPHD2018, September, 2018, St. Petersburg, Russia T. Hynninen et al. 4 Results Table 3. Weekly activity of the students; the student In this section the different results and metrics collected from completed all exercises during this week the activity logs are presented. First of all, the Mann-Whitney U 2015 (5 ECTS) 2016 (6 ECTS) 2017 (6 test was applied to assess whether the two courses (2015 and ECTS) 2016) had significant differences between the reported time Week 262 (81% of all) 405 (77% of all) 477 (91% of 1 all) distribution and sessions activity based on the VLE data logs. Week 273 (84%) 440 (84%) 436 (83 % of After applying sanitation measurements of rejecting students 2 all) with less than 30 hours of recorded activities, and compensating Week 274 (84%) 434 (82%) 400 (76 % of the 2016 group for the online exams which weren’t available for 3 all) the 2015 course, the H0 hypothesis of “The distribution of total Week 249 (77%) 399 (76%) 316 (60 % of hours is the same across categories of course year” was retained, 4 all) despite the averages and medians being higher. Median time Week 216 (67%) 336 (64%) 329 (63 % of used in 2016 was 58.78 hrs, and 52.25 hrs in 2015; the 5 all) distributions in the two groups did not differ significantly Week 204 (63%) 339 (64%) 256 (49 % of (Mann-Whitney U=50591.5, n1=454, n2=287, P > 0.05). In 6 all) Week 167 (51%) 272 (52%) 385 (73 % of comparison, the median online time in 2017 was 50.12 hrs. 7 all) Similarly, the student activities were tracked based on their Week 227 (70%) 357 (68%) 280 (53 % of weekly submitted exercise assignments during the course. This 8 all) data indicates, that the activity patterns are almost identical, Week 210 (65%) 281 (53%) 83 (16 % of with most of the students working similarly on both courses 9 all) during the first half, and dropping off at the latter part of the Week 86 (27%) 169 (32%) 136 (26 % of course, with the larger 2016 course having a small increase in 10 all) activity during the weeks 10-14. This data is summarized in the Week 87 (27%) 182 (35%) 122 (23 % of Table 3.. 11 all) Finally, the student activity and the perceived workload was Week 95 (29%) 224 (42.5%) 185 (35 % of 12 all) assessed with the student feedback. The student feedback survey Week 4 (1%) (elective, 140 (27%) 90 (17 % of covered topics such as the perceived workload and difficulty of 13 same as week all) the course, grading of the different course components and also 14) open feedback on how the course could be improved. Overall, Week 4 (1%) (elective, 46 (9%) 31 (6 % of all) the results indicate that the sixth credit given on the 2016 course 14 same as week did not affect the student performance, workload or motivation 13) to a large degree. The results are summarized in Table 4. Overall, the student feedback did not differ to a large degree descriptive feedback, and the wording of the question has between the courses, although the trend was that the 5 ECTS changed over time. However, in all the feedback surveys course was considered better by the student feedback; the questions used the Likert scale of 1 to 5, and asked the students appropriateness and overall grades for the 2015 implementation to evaluate how the actual course workload compares to the were both over 4 (in scale 1-5, 5 best), whereas in the 2016 course ECTS sizing of the course. The average grade from these they were half a grade worse. Similarly, the amount of positive questions was not very different between the years (3.1 in 2015, feedback declined in the 6 ECTS course, although this can be 2.9 in 2016). Additionally, as the statistical analysis shows, the explained also with the technical problems concerning the online answers are not statistically significantly different from each exams and the learning environment. On the assessment of the other. amount of effort, the self-assessed perceived workload actually In all years the course workload and insufficient credits are declined somewhat, but the difference between the some of the most highlighted themes students gave feedback implementations (from 3.1 to 2.9) is not very meaningful. In about, especially if ignoring technical details such as bugs in the contrast, the workload was perceived as much higher the in learning environment or exercises. It should be noted though, 2017. The amount of student feedback about the workload and that in 2017 the survey included a separate open-ended question required effort increased between 2015 and 2016, and the 2017 about the perceived course workload, which may explain the course collected significantly more feedback about the amount of high number of negative workload related feedback in that year. work than either of the previous years. Additionally, even though our VLE system has an automatic 30 minute inactivity logout feature for the sessions left open, a handful of students recorded very unrealistic hundreds of hours 5 Discussion and implications of online activity. These clear outliers were sanitized, and due Obviously there are limitations to the collected data, and this issue the median values were applied in the overall analysis. elements presented in the results. For example, the survey The student body for whom the course is mandatory is very instrument changed between the years to prompt more heterogenic, as it covers almost all undergraduate engineering Can One ECTS Credit Make All the Difference? SWEPHD2018, September, 2018, St. Petersburg, Russia Table 4. Metrics collected from the course ending passing grade, instead of putting more effort into the course. This survey. observation is in line with the observations on the college-level 2015 (5 2016 (6 2017 (6 student time usage and effort reported in [21]. Even if we did not ECTS) ECTS) ECTS) observe the offline work hours, the student performance and Amount of survey 117/69 182/114 103/53 course outcomes does not imply that there would have been a responses / open (41%/24%) (40%/25%) (20%/10%) meaningful difference, especially since the populations and their feedback left (% of performance results were statistically comparable. In the grand total) picture, this also implies that the students receive 20 percent Appropriateness of 4.3 3.7 3.4 larger reward for their effort, since for the Master’s degree, the the teaching methods. (1-5 students are required to take in average 50 completed 6 ECTS grading, 5 best) modules instead of 60 completed 5 ECTS modules. As based on Overall grade for 4.1 3.5 3.4 the workload estimations from our case study, it could be argued the course (1-5 that the approach with 60 modules with 5 credits provides better grading, 5 best) learning outcomes, and the one ECTS course credit difference in How much time did 3.1 2.9 3.9 the default module content scaling imposes the risk of the you use on this difference of ten modules in the Master’s degree curricula. This course? (1 much translates to the issue that on the long run, almost one year less than estimated, worth of studies could be lost to the credit inflation, similar to 3 estimate, 5 much more than the grade [2], and college degree [3], inflation. estimated) Considering the entire curricula, the results here are an Amount of positive 24 (34.7%) 13 (11.4%) 14 (26%) interesting observation on that the larger, topic-spanning feedback courses are not as efficient as smaller topic-oriented courses, Amount of negative 11 (15.9%) 24 (21.0%) 34 (64%) especially if the course can be successfully completed with a feedback about the subset of information not covering the entire content. One workload suggestion on why this happens is that since the students can optimize their effort during the course, they can simply select to students in the university. Additionally, most students take the submit works where the assignments are relatively easier to course during their first year of studies. This is a limitation in the compensate on the added topics which tend to be more difficult. sense that the freshmen have few other courses to compare the In our case, the only more active weeks on the 2016 course were workload with, and may for that reason have difficulty during the weeks 10 to 12, where there were 10 to 15 percent estimating the real hours they have had to invest. more activity. However, it is worthwhile to observe that getting The usage statistics from the VLE platform provide some 25 percent of the mandatory assignments completed is possible insight to the actual working hours of students. The statistical within the first 5 weeks of the course. If the students are willing analysis indicated that the time usage was similar between the to accept worse final grades, like our students based on the years, even if the 2016 statistics exhibit about 7 hours more median grades did on the 2016 course, they are not actually median online working time, most likely caused by the online required to do extra work on the latter part of the course. exam, which was added to the 2016 course. Regardless of the reason, from a teacher’s perspective the same learning goals were achieved using the same amount of time or even slightly 6 Conclusions more. In 2017 the average and median online working time was In this paper we have presented the results of our study into again similar to 2015, this time in a course setting which had the incentive-based motivation in the student participation been altered for the specific reason of adding more content for activity and studied the effect on a course implementation, the new ECTS sizing. These numbers suggest at least partial where the only major differences were the two additional weeks credit inflation, as the students are passing the same course with of lectures, and one additional study credit. the same learning goals but getting a higher reward for their Based on our observations, the answer to the research efforts. In addition, since the relative amount of negative questions 1) how to measure the incentive of ECTS reward to feedback on the workload increased for the 2016 course, and also student effort, and 2) how a revised reward affects students’ in 2017, it can be said that the incentive of extra credit did not perception of course workload can be summarized as follows: provide much of a difference in our case. one credit difference does not translate to the student motivation In general, it seems that the incentive-based motivation of or workload in a meaningful manner. There were no indications one additional study credit probably does not translate into that the one-point difference, or the two extra weeks of the actual work effort of 27 hours. As based on our observations, the course had a meaningful impact since the only actual difference student workloads do not differ to a large degree between the was 7-hour increase in the median, which could also be five and six credit effort. The grade averages actually fell by one explained with minor changes in the course arrangements. Even grade when the course was revised to become a larger module; it though the 7-hour increase in online working time between 2015 seems that the students were content with getting a worse and 2016 could indicate that the students were in fact working SWEPHD2018, September, 2018, St. Petersburg, Russia T. Hynninen et al. more and the rest of their active work was completed offline [7] Herala, A., Knutas, A., Vanhala, E., and Kasurinen, J. (2017) ,”Experiences from Video Lectures in Software Engineering Education”, International Journal of (which we could not measure), we could see from the 2017 Modern Education and Computer Science (IJMECS), Vol.9, No.5, pp.17-26, course data that the average and median working times came 2017.DOI: 10.5815/ijmecs.2017.05.03 [8] Forte, A., & Guzdial, M. (2005). Motivation and nonmajors in computer down again. Additionally, it is worth noting that if the students’ science: identifying discrete audiences for introductory courses. IEEE working time was on the rise, the additional reward in credits Transactions on Education, 48(2), 248–253. would still be lost in inflation, as the course’s key learning goals https://doi.org/10.1109/TE.2004.842924 [9] McGill, M. M. (2012). Learning to Program with Personal Robots: Influences stayed the same throughout the years. on Student Motivation. Trans. Comput. Educ., 12(1), 4:1–4:32. Similarly, as reported elsewhere [20], the students usually https://doi.org/10.1145/2133797.2133801 [10] Jiau, H. C., Chen, J. C., & Ssu, K. F. (2009). Enhancing Self-Motivation in optimize their time usage to minimize the required effort. The Learning Programming Using Game-Based Simulation and Metrics. IEEE amount of mandatory work being a percentage of all Transactions on Education, 52(4), 555–562. assignments during the course translates to the problem where https://doi.org/10.1109/TE.2008.2010983 [11] Pattis, R. E. (1981). Karel the Robot: A Gentle Introduction to the Art of the amount of needed extra effort to get the sixth credit does not Programming (1st ed.). New York, NY, USA: John Wiley & Sons, Inc. require 27 hours of extra work. This problem can be summarized [12] McWhorter, W. I., & O’Connor, B. C. (2009). Do LEGO® Mindstorms® motivate students in CS1? In ACM SIGCSE Bulletin (Vol. 41, pp. 438–442). as follows: if the students can select the weeks and topics on ACM which they spend the needed extra time, completing two extra [13] Dann, W. P., Cooper, S., & Pausch, R. (2011). Learning to Program with Alice (W/ CD ROM) (3rd ed.). Upper Saddle River, NJ, USA: Prentice Hall Press. assignments in the earlier, easier part of the course allows them [14] Cerasoli, C. P., Nicklin, J. M., & Ford, M. T. (2014). Intrinsic motivation and to skip the two weeks’ worth of added content on the latter part extrinsic incentives jointly predict performance: A 40-year meta-analysis. of the course. Psychological Bulletin, 140(4), 980–1008. https://doi.org/10.1037/a0035661I [15] Gagné, M., & Deci, E. L. (2005). Self-determination theory and work In our observed cases, the sixth ECTS was lost to the credit motivation, Journal of Organizational Behavior, 26(4), 331–362. inflation, meaning that the extra credit was awarded with no https://doi.org/10.1002/job.322 [16] Herzberg, F. (1974). Motivation-hygiene profiles: Pinpointing what ails the additional learning activities required from the student, and no organization. Organizational Dynamics, 3(2), 18-29. extra learning objectives achieved. On a scale of a degree [17] Nikula, U, Gotel, O., and Kasurinen, J. (2011). A Motivation Guided Holistic program, this inflation-caused small difference would mean that Rehabilitation of the First Programming Course. Trans. Computer. Education. 11, 4, Article 24 (November 2011), 38 pages. the student with an average of fifty 6 ECST modules would be a DOI=http://dx.doi.org/10.1145/2048931.2048935 full semester worth of knowledge behind the student, who did [18] Chernoff H. and Lehmann E.L., 1953. The use of maximum likelihood estimates in χ2 test for goodness of fit, The Annals of Mathematical Statistics, sixty 5 ECST modules, while technically receiving the same Vol. 25(3), pages 579-586. doi: 10.1214/aoms/1177728726 degree from the same program. [19] Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B. and Wesslén, A. Obviously there are also risks in this study; the amount of 2012. Experimentation in software engineering. Springer Science & Business Media. independent self-study was not measured, and the activity logs [20] McCormick, A. C. (2011). It's about Time: What to Make of Reported Declines were inaccurate to the degree that some sanitation had to be in How Much College Students Study. Liberal Education, 97(1), 30-39. done. However, the statistical analyses indicated that the results between the courses were identical, as were the worktime and activity estimations. Even though there might have been issues with the data collection tools, the issues were the same for both datasets. As for future work in this topic, we have established that the effect of credit inflation exists and that one study credit is not very efficient motivator for students to put more effort into their work. Therefore it would be interesting to study this effect further, for example from the viewpoint of counter- measurements, or the effect of the knowledge deficit between the different curricula approaches. REFERENCES [1] Fischer, G. (2014). Beyond hype and underestimation: identifying research challenges for the future of MOOCs, Distance Education, 35:2, 149-158, DOI: 10.1080/01587919.2014.920752 [2] Johnson, V. E. (2006). Grade inflation: A crisis in college education. Springer Science & Business Media. [3] Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students (Vol. 2). San Francisco, CA: Jossey-Bass. [4] Viope Solutions Oy. available at http://www.viope.com. Referenced 20.12.2018. [5] Knutas, A., Ikonen, J., Ripamonti, L., Maggiorini, D., & Porras, J. (2014, November). A study of collaborative tool use in collaborative learning processes. In Proceedings of the 14th Koli Calling International Conference on Computing Education Research (pp. 175-176). ACM. [6] Hynninen, T., Knutas, A., and Kasurinen, J. (2017). Plagiarism networks: finding instances of copied answers in an online introductory programming environment. In Proceedings of the 17th Koli Calling Conference on Computing Education Research (Koli Calling ’17). ACM, New York, NY, USA, 187-188. DOI: https://doi.org/10.1145/3141880.3141906