Goals and Principles for the Redesign of a Programming Course Erno Vanhala Jussi Kasurinen University of Tampere Lappeenranta University of Technology erno.vanhala@uta.fi jussi.kasurinen@lut.fi ABSTRACT However, as the passing rates of case courses have improved, a more fundamental question becomes interesting: whether the In this study it is discussed how programming courses have been students actually learn programming skills in the courses. The redesigned. The article is based on three courses lectured during level of learning has not been studied much in the programming academic years from 2005 to 2016. During these years several education field but, for example, a Bayesian Knowledge Transfer modifications and revisions were made, such as moving a course algorithm has been proposed to fit to estimate the learning of from Python 2 to Python 3, another one from C++ to Java and in specific programming structures [2,14]. one occasions updating the course to include the latest web The redesign process started in 2006 with an attempt to development concepts and technologies. The first course remove observed problems in the Fundamentals of programming, included video lectures and the latter two were upgraded to CS1 (Case A). Similarly, teaching methods became interesting utilize flipped classroom teaching method. From these revisions when it was observed how students appreciated video lectures the student feedback was collected and examined to gain introduced in Case A. With Case B and C it was developed understanding on what ideas work and what do not. The overall further with flipped classroom method. results could be summarize in four key concepts: 1) provide The overall philosophy for revising the case courses has been easy-to-use working environment 2) give students freedom, 3) to make it easy for the students to download the programming fortify the frequency of key concepts and 4) separate theory to environment and start using it. We have a few aims in improving pre-classroom learning and action to in-classroom learning. programming courses: 1) reducing dissatisfaction, 2) increasing motivation and 3) estimating learning. In this article it is reported CCS CONCEPTS the goals of the course redesigns. K.3.2 [Computer and Education]: Computer and Information Science Education – computer science education. 2 RELATED RESEARCH KEYWORDS In this section related research is presented to introduce the topics that are relevant when discussing the reasons and the programming, object-oriented programming, flipped classroom, process of redesigning of a programming course. redesign principles ACM Reference Format: 2.1 Student motivation Erno Vanhala and Jussi Kasurinen. 2018. Goals and Principles for Student motivation has raised considerable interest among the the Redesign of a Programming Course. In Proceedings of The researchers [3,21]. Besides the hygiene factors – elements (e.g. air 2018 Workshop on PhD Software Engineering Education: conditioning or programming manual) that are not the actual key Challenges, Trends, and Programs (SWEPHD2018). St. Petersburg, components of the process, but can greatly affect the success of Russia, 6 pages. the task (e.g. office work or learning programming) – there are also motivating things on learning, which can greatly affect on 1 INTRODUCTION the actual course outcome [21]. Why bother changing a programming course unless you have to? For example, the assignments and lectures on the course have The question is, of course, what is a good enough reason to to be easy enough to be understandable for all, but they also introduce changes which are likely to cause change resistance, should challenge the most advanced students. Students with less increased effort, and worse short term results [7]? IT skills tend to frustrate near the end of the course when assignments get harder. On the other hand, students with existing programming skills are frustrated since they are not Copyright © 2018 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and challenged, and are just required to participate. There are, copyrighted by its editors. however, ways to detect frustration based on compilation logs The 2018 Workshop on PhD Software Engineering Education: Challenges, Trends, and and time spent on Virtual Learning Environment (VLE) [22]. Programs, September 17th, 2018, St. Petersburg, Russia When a students get overly frustrated they can easily lose their confidence [10] and will get a grade worse than their actual skill level would indicate. SWEPHD2018, September 2018, St. Petersburg RUS E. Vanhala et al. In principle, the students who have positive impression on a subject tend to be motivated [21]. Motivated students also have a positive perception of the subject and amount of practical work [21]. In this sense, it is obvious that these traits should be supported, by providing programming assignments, which the students find motivational [24]. 2.2 Student dissatisfaction Hygiene factors [11] are not widely studied in the area of introductory programming, but few studies can be found (e.g. Figure 1. Bayesian Knowledge Tracing algorithm [14]. [13,15]). Before a student can be motivated, the basic hygiene factors need to be present [11]. In programming courses these 2.4 Flipped classroom method Herzberg's hygiene factors include, for example, comfortable learning environment and learning conditions. The first one As technology develop it also opens new methods to improve contains the lecture halls and computer classes, and the latter teaching methods. The core idea of flipped class room is to let material supporting the learning and students' time schedules. students to study theory on their own outside the classroom and When the hygiene needs and requirements are met, the concentrate on actual doing in class with teacher. This method is motivational aspects become increasingly important to increase already used in computer science on various course [12,16]. the student satisfaction. Herzberg introduces 7 principles [11] Flipped classroom can also be used in other areas from primary from which three can be easily modified to be used in teaching school to university level education although it originated from programming: 1) In the learning environment it is quite easy to economics [4]. give the students access to the scores of one's programming tasks. 2) When giving tasks to the students it is possible to give 3 RESEARCH METHOD several tasks covering different degree of difficulty. 3) Access to The present study started with two research questions: 1) When additional tasks providing deeper understanding to engage should a programming course be revised? and 2) What should be advanced students in the course. taken into account when redesigning? In general the study had three objectives: First to demonstrate 2.3 Course success measurement how the collected data can be used to pinpoint problems and After the course implementation has been carried out, usually development needs in the course. Second, to report the goals and the final survey has been given to the students. Based on this principles for the course redesign. The third objective was to feedback it is possible to pinpoint difficult parts of the course assess how programming assignments could be developed to take topics and unsuccessful ideas or failed implementations. into account the developed principles. In this sense, the study has Yet, a course success measure should reflect the actual the characteristics of both natural and design science [17] but it objective of course participation, the learning of the course is reported as a case study since the case study methods [25] fit contents. The learning processes have been studied, for example, well the study and, in general, the courses referred to in the from the point of view how novice programmers understand study represent case studies with literal replication [25]. programs [23], how they structure their own code [5,20], and The data used in the study comes from three courses taught in how the programming knowledge can be measured – the a Finnish university, between 2005 and 2016. Each course has Bayesian Knowledge Tracing algorithm (BKT) [2,6,8] and been concluded with a final survey that has been distributed to Adaptive Control of Thought - Rational (ACT-R) [1]. The ACT-R all students enrolled in the course. The final surveys have is focused on long-term learning, skill acquisition, and included quantitative questions like the difficulty and usefulness deterioration while the BKT algorithm has been reported to have of the different course elements but also open questions to allow a history of success in programming and algebra, and thus the the students to express their feelings and concerns about the BKT has been reported sufficient for skill mastery estimation [8]. course. The principles of the BKT algorithm are outlined in Figure 1: The analysis has been based on both quantitative and the prior knowledge (Ln) is taken into account when the qualitative methods. In particular, the BKT analysis is based on probability of learning the concept in question is calculated quantitative measurements and statistical analysis while the (p(T)) to measure the student learning. As student repeats the problem analysis has been based more on the qualitative data process, the possibility of guessing the right answer (p(G)) and to acquired from multiple sources. For example, problematic weekly err even when the concept is learned (p(S)) are taken into assignments have been identified by analyzing the points each account. It has been estimated that repeating the process for six assignment got but deciding about the changes required by each times predicts the learning of the concepts in question in a assignments has been estimated case by case by the designer reasonable level. even though the student problem reports and feedback have helped a lot in some cases. Goals and Principles for the Redesign of a Programming Course SWEPHD2018, September 2018, St. Petersburg RUS 4 CASE STUDIES was selected as the programming language for Case A as it can be easily installed on Windows, Mac and Linux computers and it The next chapters describe three case courses, their descriptions already includes IDLE as IDE. IDLE itself is easy-to-use and does and upgrades done to them. not throw everything to students at once, and it has a minimalist and simple look, which replicates between the different 4.1 Course descriptions platforms, which suits CS1 where the students meet This study utilizes three programming courses. Table 1 lists key programming code for the very first time. features for the courses. All the courses have positioned in With Case B it was also thought how to give students the bachelors level, but also not computer science students from suitable IDE with minimal work to be done. As Case B had also masters level have taken part of the courses. GUI-programming and the JavaFX was chosen as the GUI-toolkit it was decided to use NetBeans as the IDE as both JavaFX and Table 1: Key features for the case courses NetBeans were bundled together from the developers of Java. Case Name Key changes Data As web developing can be done with pretty much every tool available, and the course required both backend and frontend A Fundamentals of Introduction of lecture 2006 – development, the Case C was a different scenario. The most programming videos, change from 2015 Python 2 to Python 3, pressing issue was the server side system; how this should be programming manual addressed in the university infrastructure, since the system had to be usable by the students, while still retaining a certain level B Object-oriented Change from C++ to Java 2010 – of cyber security. In this case, the solution was to use a cloud programming and change from lecture 2016 platform that enabled the option to run the student-generated videos to flipped classroom, programming manual code without the need of installing web server, database and various libraries. Additionally, some students decided to use their C Webbed Introduction of flipped 2015 own environments and followed guideline videos on how to applications classroom and updating all install the necessary packages to Linux or Mac. the materials Case A and half of the Case B utilized VLE for the assignment For Case A, a larger Python programming manual was written submission and grading, since the course had thousands of to replace a course book, since at the time there was no book student-submitted works. Since this VLE could not manage available in the local language. When the course was updated to graphical user interfaces, the second half of the Case B and Case Python 3, the number of weekly programming tasks was C applied the traditional teacher grading, but by allowing the increased and the format of the programming project was students to demonstrate their solutions with quick demonstration changed from GUI-based “motivational” project to text- and at the exercise event, or by sending a video link explaining what calculation based real-life problem. they had done. On latter implementations, a peer review Case B followed examples of Case A as it had video lectures, approach by other students was also applied. but in the end it was decided to change the programming Teaching materials were provided for every case course. With language of the course from C++ to Java and in the same time to Case A this meant the programming manual, style guide and drop the traditional lectures-exercises-exam paradigm and move Python installation manual with lecture examples and videos. to the flipped classroom. In addition, two programming manuals Case B had videos covering all the course topics, two written were written; one for Java in general and one for GUI- manuals and code examples available in a git repository. Case C programming in Java. had videos covering all the course topics, short manual Similarly, Case C was transformed to flipped classroom presenting the most important techniques and tools used in web method and all the course materials were updated to reflect the programming and code examples in a git repository. fast development of web programming techniques. One of the usually overlooked hygiene factors is the exam. All the case courses consisted of 12–14 weeks depending on When learning to program it is normal to ask students to write a the year it was lectured. All the courses followed the format short program in the exam, yet when the exam is pen and paper, where besides the introductory lecture, each week a new concept it does not reflect to the real world where the students would was introduced and the previous topics were utilized with the have compiler messages and manuals with them. To avoid this newly learned skills. issue Case B introduced online exam and Case C had no exam at all, but an essay to show how students had learned their new 4.2 Changes in the course skills. In the end Case A also introduced an online exam for programming as it would decrease the manpower needed to 4.2.1 Hygiene factors grade all the exams. The first major observation was the students were complaining 4.2.2 Changes done to the weekly programming about very basics on case courses, for example “Where do I get the needed software?” or “How do I return my assignments?”. assignments One of the key principles when revising and improving With all cases weekly programming assignments were courses was to remove these hygiene factor obstacles. Python constructed so that they had usually 5 smaller tasks thus students SWEPHD2018, September 2018, St. Petersburg RUS E. Vanhala et al. were able to gain 5 points from one week. With Case A example experience for the future. When Case B was transformed to Java solutions were provided and they were coded to show how style the course project was also redesigned. guide help to build easy-to-read programming code. The new project included parsing publicly available open data With all cases the number of weekly assignments were also sources and building a graphical user interface to visualize the increased – though some tasks were divined into two smaller data on a map overlay. This gave students a feel of working with tasks – thus students had to repeat the key structures and the real-world problems, tools and data. concepts at least six times as suggested. Although one has to With Case C the course project was not focusing on any note that techniques presented in the end of the course could not specific area. As the Case C is more advanced course than A or B, be covered such many times. it was decided to give students more creativity and responsibility The BKT analysis was done with the program source codes with the course project. There were elements that gave certain retrieved from the VLE database after the course, and the prior amount of points and combining several elements students could programming knowledge of the students was estimated with an get the number of points they wanted to get certain grade. For initial survey in the beginning of the course. When the example creating responsive design gave 5, utilization of cache programming assignments were developed in 2006 it was only gave 3 and front-controller design pattern gave 3 points. aimed at reasonable assignments, and only few programming structures met the BKT analysis requirements. The overall 4.2.4 Lectures vs. flipped classroom outcomes were up 10–20% on all measured categories. The Case A provided video lectures from the beginning of Other positive side of the increased number of assignments Python era. Students found the videos very useful. Also with was that average students did not need to do all the five tasks per Case B video lectures were provided from the beginning of data week, but students with advanced skills had more to do. The last collection period. When Case B was revised it started to use task is set to be the hardest one, so the students who were flipped classroom method. Similarly Case C was revised to interested in programming could get more challenge when they flipped classroom when data collection started with new wanted. administrating lecturer. Students focus on tasks that are beneficial to them, that is, Both video lectures and flipped classroom seem to be suitable tasks giving points to them. All the case courses had previously when teaching programming. Students have given feedback on weekly exercises that were voluntary and there were no points how they value the opportunity to watch theory when they have given from a successful solutions. When the case courses were time and repeat it as many times they need. revised all of them shift to a model where students could get With flipped classroom, although the initial cost of creating points from all the work they do for the course and the final new course format is high it later gives more man power to be grade was calculated from the points got from weekly used in classroom teaching then benefiting the students. assignments, course projects, exam, and other parts of the course. 5 DISCUSSION 4.2.3 Course project In the beginning two research questions where set: 1) when All of the cases have larger programming project in the end of should a programming course be revised? 2) what should be the course. The previous version of the Case A was taught with C taken into account when redesigning? Here the questions are programming language and its course project was transformed to discussed based on the experience gained from three cases the first implementation of Python course. The text-based project described in the previous chapter. was not seen up-to-date and it was then changed GUI-based Turtlet programming project. Although it was supposed to be motivating, students gave mixed feedback, similarly as observed 5.1 When should a programming course be for example by [18]. Turtlet had limitations on the usefulness and revised? student creativity as well as maintenance problems, so a new The present study explored the reasons for initiating the project was to developed for the course. programming assignment revision and found four key reasons With Python 3 the programming project used more for the revision: engineering approach. First theme on the project was to calculate Problems with existing assignments. The students reported district heating systems. The project was approximately double different kind of defects in the assignments over a number of the size of the old Turtlet version, yet it did not yield complaints years which indicated that there were problems with the more than the Turtlet. Nor was it completely success. The project assignments. Collected data suggests that problems in the required a lot of precise mathematical calculation that the programming assignments reduced the student motivation to students found hard to get right. The project was later moved to complete them, and thus such problems became the hygiene work with temperatures, data files and generating svg-graphics. problems in these courses. Thus, to avoid dissatisfaction – Case B had had various humorous – game-like – projects hygiene problems – among the students, the programming when it was implemented with C++. The projects required to assignments should not pose undue problems for the students. understand the concepts of C++ but they did not reflect to real Mismatch between the students and the assignments. Even problems or were not in an area, which would have been useful though the assignments would not have any problems, the Goals and Principles for the Redesign of a Programming Course SWEPHD2018, September 2018, St. Petersburg RUS students may not be motivated by them. Collected data suggests down it can now be discussed what parts of improvements were that the students were concerned about the usefulness of the success and what parts still require work. assignments they had to complete and especially the game-like With Case A the most difficult part has been developing a project with graphical user interface (Case A) had been criticized programming project that would be easy enough for beginners every year it was used. Thus, to keep the programming but would also let students to show their skills if one wants to. assignments motivating for the students they should be aligned This is an issue that has no real solutions, but it can be iterated with the student experiences and expectations as the student towards project that would have all the necessary parts required body evolves. This observation is in line with the earlier work and still be easy and useful. from for example [3]. Case B failed in grading for the first implementation after Technological development. Programming languages and revision. Points were provided too easily and students got high tools develop quickly. With Case A the change from Python 2 to grades – although they did good work. Some example solutions 3 required to check all the material and assignments in the were also not perfect and issues arose when JavaFX required course. In Case B, where the programming language changed, all special version of Java, which was not installed by default. These the assignments needed to be redone. And web programming hygiene factors were then hot-fixed, but would still require more (Case C) changes all the time so the revision of everything was a work to be done. mandatory task to be carried out at least once in five years. The biggest issue with Case C is the number of tools and Pedagogical development. The teaching methods and tools techniques web development has. In the course dozen of different are improved all the time. Where PowerPoint slides came in the techniques are introduced to students and then they do not have 90s, this millennium has given video lectures and flipped enough time to repeat newly learned skills. This problem would classroom. Teachers need to match the new ways of studying be solved by extending the course somehow or by splitting it into and thus courses need to be revised also from pedagogical point two separated courses. of view. The traditional lecture-exercise-exam model can be With all the cases teachers are required to do a lot of grading. replaced with modern ways where students get more individual With Case B it was changed so that students peer reviewed time from teachers. With Case A video lectures where provided programming project so that teachers could focus more on all the time with Python course and with Case B and C the teaching. This similar method could be considered also for Case transition to flipped classroom was carried out when courses A and C. where revised. Both the video lectures and flipped classroom method generated praises from students. 5.4 Limitations and validity of the study As this is a partly qualitative study, the observations presented 5.2 What should be taken into account when here are not strong, confirmatory results, but guidelines and redesigning? sophisticated suggestions on things that have been discussed in The response to the second research question of what should be this article [9]. Although we have triangulated the data against taken into account in the redesign is twofold. The problems quantitative data sources, this study is not free from possible bias leading to the revision should, of course, be fixed but a revision towards any direction that has been missed. Bias is addressed in provides also an opportunity to improve the course. In the the following ways. We have followed the three principles of present study the revisions made it possible to assure that the data collection [25]: we have used multiple sources of evidence, assignments were fit estimating the learning outcomes of the created a multicase study [19], and maintained a chain of course with the BKT algorithm. Overall the following design evidence. We have also triangulated our data from multiple principles for programming assignment revision can be pointed: sources (e.g., student surveys and feedback, VLE program • Motivating assignments reflecting the real world database, and grading data), both authors have participated in problems, useful for the studies as well as the future the analysis, we have used different theories, and we have used careers and assignments as engineers both qualitative and quantitative methods to analyze the data. • Give an option to do more assignments when the topic requires them 6 CONCLUSION • Repeat all the main programming structures at least six This article presented three university level case courses, which times in the weekly assignments to allow accurate had all been revised and improved. When revising a course one mapping of the learned topics with BKT or ACT-R should note, for example, to follow the latest technical and algorithm pedagogical tools and methods, make sure students repeat • Follow both the technological development and learned tasks at least six times and motivate students with real- pedagogical development to be able to utilize up-to- life assignments. date tools and methods This is a latest wrap up of the on-going improvement of these three courses. The shift from lecture room lectures to video 5.3 Retrospective lectures and flipped classroom method has already widely begun, The reported modifications to the programming courses have but there is still much to do. For example, only Case A was a been made during the years 2005 – 2016. After few years of cool mass course and only Cases B and C where flipped. In the future SWEPHD2018, September 2018, St. Petersburg RUS E. Vanhala et al. it is required to study whether same methods can be used when [13] Jussi Kasurinen and Uolevi Nikula. 2007. Revising the flipping a mass course. First Programming Course - The Second Round. 92–101. [14] Jussi Kasurinen and Uolevi Nikula. 2009. Estimating 7 REFERENCES programming knowledge with Bayesian knowledge [1] J. R. Anderson and C.D. Schunn. 2000. Implications of the tracing. In Proceedings of the 14th annual ACM SIGCSE ACT-R Learning Theory: No Magic Bullets. Advances in conference on Innovation and technology in computer instructional psychology 5. science education, 313–317. [2] Ryan Baker, Albert T. Corbett, and Vincent Aleven. 2008. [15] Jussi Kasurinen, Mika Purmonen, and Uolevi Nikula. 2008. More Accurate Student Modeling through Contextual A Study of Visualization in Introductory Programming. Estimation of Slip and Guess Probabilities in Bayesian 181–194. Knowledge Tracing. 406–415. [16] Mary Lou Maher, Celine Latulipe, Heather Lipford, and [3] Tiffany Barnes, Eve Powell, Amanda Chaffin, and Heather Audrey Rorrer. 2015. Flipped Classroom Strategies for CS Lipford. 2008. Game2Learn: improving the motivation of Education. In Proceedings of the 46th ACM Technical CS1 students. In Proceedings of the 3rd international Symposium on Computer Science Education (SIGCSE ’15), conference on Game development in computer science 218–223. DOI:https://doi.org/10.1145/2676723.2677252 education, 1–5. [17] S.T. March and G.F. Smith. 1995. Design and Natural [4] Jacob Lowell Bishop and Matthew A. Verleger. 2013. The Science Research on Information Technology. Decision flipped classroom: A survey of the research. In ASEE Support Systems 15, 4 (1995), 251–266. National Conference Proceedings, Atlanta, GA. [18] William Isaac McWhorter and Brian C. O’Connor. 2009. [5] Janet Carter, Kirsti Ala-Mutka, Ursula Fuller, Martin Dick, Do LEGO® Mindstorms® motivate students in CS1? John English, William Fone, and Judy Sheard. 2003. How SIGCSE Bull. 41, 1 (2009), 438–442. shall we assess this? In Working group reports from ITiCSE [19] Christine B. Meyer. 2001. A Case in Case Study on Innovation and technology in computer science Methodology. Field Methods 13, 4 (November 2001), 329– education, 107–123. 352. DOI:https://doi.org/10.1177/1525822X0101300402 [6] Cristina Conati, Abigail Gertner, and Kurt VanLehn. 2002. [20] Keir Mierle, Kevin Laven, Sam Roweis, and Greg Wilson. Using Bayesian Networks to Manage Uncertainty in 2005. Mining student CVS repositories for performance Student Modeling. User Modeling and User-Adapted indicators. In Proceedings of the 2005 international Interaction 12, 4 (November 2002), 371–417. workshop on Mining software repositories, 1–5. DOI:https://doi.org/10.1023/A:1021258506583 [21] Matthew Mitchell, Judy Sheard, and Selby Markham. 2000. [7] P.D.I. Elrod and D.D. Tippett. 2002. The “death valley” of Student motivation and positive impressions of change. Journal of Organizational Change Management 15, computing subjects. In Proceedings of the Australasian 3 (2002), 273–291. conference on Computing education, 189–194. [8] Kevin A. Gluck. 2004. Knowledge Tracing for Complex [22] Ma. Mercedes T. Rodrigo and Ryan S.J.d. Baker. 2009. Training Applications: Beyond Bayesian Mastery Coarse-grained detection of student frustration in an Estimates. 383–384. introductory programming course. In Proceedings of the [9] Nahid Golafshani. 2003. Understanding Reliability and fifth international workshop on Computing education Validity in Qualitative Research. The Qualitative Report 8, research workshop, 75–80. 4 (2003), 597–606. [23] B Simon, R Lister, and S Fincher. 2006. Multi-Institutional [10] Stuart Hansen and Erica Eddy. 2007. Engagement and Computer Science Education Research: A Review of frustration in programming projects. In Proceedings of the Recent Studies of Novice Understanding. 12–17. 38th SIGCSE technical symposium on Computer science [24] Alan L. Tharp. 1981. Getting more oomph from education, 271–275. programming exercises. In Proceedings of the twelfth [11] Frederick Herzberg. 1968. One more time: How do you SIGCSE technical symposium on Computer science motivate employees? Harvard Business Review 46, 1 (1968), education, 91–95. 53–62. [25] R.K. Yin. 2002. Case Study Research: Design and Methods [12] Diane Horton and Michelle Craig. 2015. Drop, Fail, Pass, (3rd edition ed.). SAGE Publications., Thousand Oaks, CA. Continue: Persistence in CS1 and Beyond in Traditional and Inverted Delivery. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (SIGCSE ’15), 235–240. DOI:https://doi.org/10.1145/2676723.2677273