The first programming course (CS1) at LUT University passed the 1000 student mark in the fall 2022. The continuously increasing number of students attending the course introduces new challenges for the course arrangements. The core of our course is a course specific programming guide, which is updated regularly to keep up with the course contents. For the past few years, we have been developing a style guide that defines the programming style to use in the course project, and in fall 2022 a new autograder was taken in use in all the LUT programming courses. We have also concluded that if we want to keep improving the course and how it is run, we need to develop ourselves small tools to support the course activities. In this paper we summarize three tools that we have developed over the past few years. The first tool is ASPA, which is used to detect style guide violations. Second, we have GradeTool to help in grading student projects and exams. And third, we have a tool, Mímir, to help manage the increasing number of programming assignments and their variations, and to generate assignment documents automatically. All these tools have the same basic goals to reduce the amount of manual work and time required by the tasks, at the same time they increase quality by producing more consistent outputs in a systematic way. All these tools are still in the development phase, but we have started their empirical testing. The initial feedback from both students and teaching assistants using the tools or their outputs suggests that these tools can speed up doing the tasks and improve the quality of the end products. Thus, we plan to continue their development and explore expanding their adoption.
Even after a major revision of the course in 2006, the interest in the course continued to fall. Only in 2011 the interest increased for the first time in seven years and the registration count reached 206 students. The interest has continued to increase thereafter, and in fall 2022 the course was ofered for 1114 students both in Finnish and in English. The course changes between 2005 and 2009 have been reported in detail in [1], and the improvement work has continued ever since. In this paper we look at the main ongoing course development topics.
is based on the Python programming language [2]. The course covers basic programming topics like variables,
(J. Kasurinen) and cultures, and the online course is open to any Finn who can complete the strong identification required by the university. Thus the need to support diferent kind of learners increases as students and their diferent backgrounds increase, and we also need to be able to handle large numbers of all kinds of study related events and activities.
In this paper we describe our ongoing tool development actions on the CS1 course. These actions started in the Finnish CS1 context, but all the tools can be adapted to suit for diferent courses and, for example, GradeTool and Mímir were used in our C-programming course in spring 2023. In particular, the tools have been designed so that they can be adapted to other programming courses independent of the programming language used, and serve courses ofered both in English and in Finnish. We are currently developing 3 tools: we do not evaluate all these programs manually but use an autograder to execute the programs and check that they work as instructed. As of now most student programs pass the autograder tests, but in manual review their structure is not always understandable, which makes both grading and code maintenance dificult. Thus we have moved our focus from a working program to understandable code, which we did by introducing a style guide. Since many software companies are using style guides today, we decided to use it in our course to show students how to write understandable code. One key reason to introduce the style guide was the use of global variables that is syntactically acceptable in typical programming languages, but from the good programming style point of view it has been deemed not acceptable since 1970’s. That is, even if a compiler/interpreter ac1. ASPA. ASPA is a static analyzer that can be used cepts global variables, in our CS1 it is forbidden to use to analyze the student programs and especially
them by the course style guide.
programming style problems in them. Introduction of a style guide in our CS1 course caused 2. GradeTool. Evaluating a large number of problems since many students aimed at a working proprojects is one thing, and another thing is to pro- gram and once it passed the autograder, no second vide students feedback that is consistently pre- thought was given to it. Thus requiring that programs sented for all the students and detailed enough follow a style guide raised a lot of objection, and to reduce so that students can understand their errors and this we introduced a tool to find style guide violations. ifx them. The exams need to be graded, too, even First we developed an abstract syntax tree (AST) anaif now students do not fix the errors but should lyzer in 2020 and called it ASPA. As of now the staf uses learn how to avoid them in the next exam. The ASPA to check to student projects and exams, and we also key requirements for the tool are to speed up the give it to the students so they can check their programs grading, and provide more systematic and helpful with it, too. Second, we adopted a new autograder in fall feedback to students. 2022 and we are currently using CodeGrade [5] which 3. Mímir. Mímir is a tool to manage weekly as- has a semgrep-feature supporting structural style guide signments and their variants. Using the same checks. Including the style guide checks as a part of auweekly assignments multiple times increases the tograder and rejecting submissions that do not pass them likelihood that students return a solution from a makes it easier for students to understand that the style friend without doing it him/herself. Thus we are guide needs to be followed. In this paper we describe developing a tool to support management of mul- our ASPA-tool in short. The tool was developed for the tiple variants of the assignments, and automatic ifrst programming course with Python starting with the generation of systematic weekly assignment doc- following research questions: uments with minimal efort.
in more detail, and look at the following three aspects: • research questions and motivation • tool developed • empirical observations and data.
programming assignments? RQ2: Does a usage of static analyzer correlate with grades?
gramming education point of view, and many such systems have been developed. For example, PyTA is a wrapper module for Pylint [6], PEDAL is a feedback system for Python 3 programs [7], and AutoStyle is a style tutor providing hints for student [8, 9]. On the other hand, Semgrep is an open-source based extended AST solution that detects patterns in source code. It supports over 30 programming languages, as well as generic option for
rules: First, the file must contain a function called main.
Second, the main level code should contain only one function call, which calls the main-function. Third, variables should not be defined in global namespace. Notice that the third rule focuses on global variables, and global constants are acceptable. Since Python does not support constants, ASPA needs to diferentiate global variables and constants by checking how often the identifier is set – if it is set once, it is considered constant, but if it is set more often, then it is considered a variable. In Figure 1 data_list is defined in the global namespace and is modified after definition, so it is a global variable and violates the style guide. While comparing to other solutions, such as PEDAL and AutoSyle which provide hint for solution, ASPA points the violation and style guide is student’s is source for correct style. This way student can also see correct solution before and after usage of the tool.
ASPA was first introduced in our CS1 in 2020, and currently it has 41 checks. Over time the rule base evolves as Python develops and the students get more creative.
For example, Figure 1 demonstrates the basic global variable, but we have also seen students using classes and functions as global variables. Namely, classes are defined ASPA was introduced in our CS1 course in the fall 2020 As of now we are using ASPA in our CS1 with Python, but and it is now a standard part of the course. Our standard we have also considered extending it to other programcourse feedback survey in fall 2022 was answered by 156 ming courses. Current ASPA is implemented with the or 26.1 % of the students and 92 % of respondents had used Python 3 abstract syntax tree, but since syntax trees are ASPA while doing the course project. The most common available for other languages, ASPA can be extended for reason for not using ASPA, 3 % of the respondents, was other languages. One key benefit of the AST approach that they did not want to use an extra tool. In general is that it allows measuring student learning based on with Likert scale of 1-5; 1 worst and 5 best, ASPA was their submissions. We have done studied learning earfound useful for the project with an average 4.4, selecting lier [15], and initially ASPA seems to fit this task well. the checks was easy 4.5, selecting the files was easy, 4.6, ASPA was initially developed with Tkinter to avoid need and the results were understandable 4.5. to any installations before it can be used. However, as
A more comprehensive analysis of ASPA and its use- seen in our more recent tool development eforts, other fulness was done for 2020 course [13]. Also then students user interface frameworks could help improve the usabilfound ASPA useful as seen in Figure 3. ity and make the tool even easier to use than now. A key consideration with ASPA development is synchronizing it with the autograder style checker. As of now a simple standalone tool seems to help students follow the style guide rules, but since our current autograder has a built-in style checker, it is faster and more powerful in enforcing the rules, and we need to study this balance closer.
Figure 3: Usefulness of ASPA in CS1 course. ASPA mentioned in Section 2, are assisting teaching assistants (TAs) in assessment process of programming
At the end of 2020 course, 205 students reported their assignments. The current aim in introductory courses is ASPA usage via survey. Weekly assignment data is an to synchronize assessment process between TAs and help integer number representing student’s answer to a ques- them to provide consistent feedback for such cases where tion In how many weekly assignment you used ASPA? and automated grading is limited or is not yet supported at course project data is an answer to a question Did you use all. Generation of feedback based on selected violations ASPA while doing the course project?, with options “No”, would both save time of TA, due to the decreased amount “Yes, once”, “Yes, 2-3 times”, or “Yes, multiple times”. Pear- of writing required, and standardize majority of given son’s correlation is used to analyse correlation between feedback. However, there are cases where general feedweekly assignments and performance on programming back is not enough, and content should be personalized assignments, while for the course project usage, which and therefore possibility to modify the feedback is reis ordinal data, Kendall’s Tau B is used instead. In both quired. In addition, the usability of grading tool should there is a clear positive correlation between ASPA use such simple that assessment process is easier and faster and grades, presented in Table 1. For correlation calcula- compared to more primitive methods such as spreadtions JASP program [14] was used. sheets and text editors.
We decided to develop a tool, called GradeTool, for to exist twice in the selected students’ submission. The this purpose. A development process started in summer graded students can be selected on the right side of the 2022 and currently the second version of the tool is in GUI, while below this student list, level of submission, use. Course staf uses GradeTool to keep track of pro- current grade, and current amount of errorpoints are gramming errors and style violations in assessed student shown. At the bottom right the generated skeleton for submission, and based on marked errors the tool cal- open feedback is shown, currently there are three diferculates grade, generates categorical feedback and skele- ent violations selected, aforementioned global variable ton for open feedback. Open feedback can be modified violation, file which is left open, and close command withby evaluator, if needed. In this study we describe our out parenthesis. Finally, at the top right there is WRITE GradeTool-tool in short. The tool was developed for the TO FILES-button which saves all the assessments files. ifrst programming course with Python starting with the Open feedback in Feedbacks section can be edited following research questions: by grader if needed, similarly, as in ALOHA tool [17].
The research questions are as follows: In some cases, it is also intended to give students more specific and personalized feedback, e.g., in simple case RQ3: Can a grading tool make evaluation process of giving name of global variable or in more complicated programming assignments easier and faster for case explain why implemented analysis option is not teaching assistants? suitable but is against the project instructions or style RQ4: What kind of programming violations are com- guide.
monly done by students?
such as Labtool, a website to given and share feedback from staf to students [ 16], ALOHA, a grading rubric based online tool to generate feedback based on selected errors [17, 18], GradeIT, a grading and program repair system for TAs [19].
In addition, automated grading systems, which enable feedback generation, have also been utilized in education. For example, an automatic grading system, which utilized Bash scripting, grep and regular expression [20], gdb and valgrind based spectral error localization solution to generate feedback report [21], and education platforms ViLLE [11] and A+[12] which allow both automated tests and manual feedback.
2022, and in addition to functionality, usability of the tool has been continuously improved by removing unnecessary steps and creating GUI responsive with flexible layout to allow grader to modify to it if desired. GradeTool is developed with Python and selected GUI module is Dear PyGui [22], which enables variety of layout flexibility out of the box. To store all possible violations and all marked violations, grades, feedback etc. metainformation for each graded submission, JSON files are utilized.
A grader can mark all the errors and violations via GUI, shown in Figure 4, as well as modify open feedback.
In Figure 4, list of all categorized violations can be seen on the left side of the GUI, with category Parameters and return values being open. Inside the open category a programming violation global variable is marked
To enable selection of common violations, these violations are related to a certain assignment and all related information, i.e., identifier, name visible for grader, category, errorpoints depending on the error amount, and generated feedback phrase, are stored to a JSON file before starting an assessment process. However, violation set can be updated and modified during the evaluation process, if needed and just by reopening GradeTool with updated violation file updated errors can be selected. A minimal example of violation set used is shown in Figure 5.
ID is unique identifier of a violation, text is label shown in GUI, feedback is generated skeleton phrase and category is heading under which the violation belongs to in GUI and in feedback. As same violation can be done multiple times in same submission there is possibility to scale grading based on how often certain violations occur, e.g., in Figure 5 close command without parenthesis has three error values, for occurrences 1, 3 and All. One time is considered as a careless mistake and therefore only 0.7 error points are given, while 3 times doing same violation is not anymore just a careless mistake and therefore more error points, 1 in this case, are given. The label All is used when student has never done this specific part correctly, and it leads to the most error points, in this case 2. In the GUI this can be marked with value of -1.
On the other end, Global variable is considered as a fatal error and therefore already a single occurrence leads to 2 error points, in this case all option is not very relevant but is stored to allow similar behaviour for every violation.
In addition to traditional grading, GradeTool enables grading of multiple levels of same assignment, meaning that the grade calculation is based on marked violations and the level of submitted programming project.
In LUT CS1 course project three levels were used successfully. When assessment is done, marked violations can be saved to a JSON file, which allows returning to and did not help students to locate their actual problem. assessment process and continuing later from exactly From graders perspective, three TAs, with experience same assessment state, if needed. In addition, generated from previous years courses, estimated the time used feedback also saved to a separate file which is used while to give these feedbacks being 33 % - 50 % less than with importing assessments to learning management system previous methods they were using. However, these val(LMS). The entire generated feedback contains three sec- ues are estimates and are not statistically significant. On tions in following order. First general information about the other hand, assessment results were published to stuwhat was graded and overall pass/fail grading for that dents 10 days earlier than estimated publication date. The submission. Second, category feedback for each category, estimation was based on previous assessment process. in CS1 course 11 categories were used and they can be Assessment of student submissions with GradeTool seen in Figure 4, and third, open feedback, which is the generates also quantitative data about violation occuronly part intended to be modified by graders. rences. In fall 2022 Finnish version of CS1 course with circa 600 students, the course project assessment resulted 3.3. Results 1418 violations for 76 diferent violations. As an example, subset of this dataset with only 10 violations is shown in GradeTool was used in LUT CS1 course during fall 2022 Table 2. The colouring is used to visually highlight higher and CS2 course during spring 2023. In the latest use dur- values, used coloring scale is from green, 0 occurrences, ing CS2 course with four graders and 169 graded course to red, the highest occurrence, in this example case 13. projects, assessment process with GradeTool resulted In the complete set, the most common violations were over 17 000 words and 2860 lines of feedback, while ex- programming practice violations which violated style cluding empty lines, which is almost 17 lines of feedback guide. The highest value of 196 occurrences was with per student. Empty lines were used to distinct mentioned Clearing data structures at the end of the program, the sections from each other and to make feedback more second highest value of 96 occurrences was initialization readable within sections. While the amount is large, also of values in analysis was not done according to guide. The consistency of the feedback has, at least so far, been con- style guide guides to initialize value with the first elesistent enough as there have not been any complains ment of analysed list to ensure that analysis will always about varying grading between similar violations, how- work regardless of dataset. However, many students iniever, few feedback comments were perceived too general tialize values with real numbers, e.g., for minimum and maximum search they use “big value” and zero (0), or zero for both. In cases with negative values in dataset zero works also for minimum, but often that is not the case.
cess of CS1 and CS2 courses with hundreds of students. With four graders, the produced 2860 lines of feedback, excluding empty spacer lines, is such high amount of text that it is easy to argue that GradeTool have saved huge efort of writing all that. In addition, the feedback generated by GradeTool is consistent enough, but feedback phrasing should be focused on more in the future to avoid unnecessary misunderstandings and confusion about feedback.
The data about time used by TAs is not statistically significant and it is based on TA estimates, not measured values. However, being able to publish course project assessments 10 days earlier in a mass course like LUT CS2, is a great success. Moreover, gradings can be also directly imported to LMS, which objectively saves time and removes one possibility for human errors. In addition, estimated diference is high and amount of generated feedback is high compared to fully manual feedback, therefore benefit of tool should be studied more to thoroughly answer RQ3 about faster and easier evaluation process. However, it is important to mention that amount itself is not our target but quality of the given feedback. As an answer to RQ4 the most common violations for CS1 course are clearing data structures at the end of the program, and initialization of values in analysis was not done according to guide. which are style guide violations. As exact occurrences for each marked violation are now available and in the future, these can be used to focus teaching more on topics which seem to create the most problems. Moreover, detection of clearly distinguishable assessments between TAs is also possible, which could be utilized to standardize grading and marking of violations even more to ensure fair and standardized assessment for every student.
assignments, moreover, automates generation of assignment instruction document provided to student, which ensures they are always done with consistent formatting While detection of plagiarism in normal academic writ- and layout. Mímir is developed with Python and selected ing is fairly straightforward, the detection of plagiarism GUI module is Dear PyGui [22], in addition, instruction in source code can be dificult [ 23, 24]. In addition, sim- document generation utilizes LaTeX, which features a rople plagiarism detection tools can be easily fooled by bust and programmatically easy way to compile uniform changing variable names, by changing the structure of and clear documents from diferent types of data, and the program, or by doing other types of obfuscations [25]. JavaScript Object Notation (JSON) files are used to store Moreover, when similar programming assignments, with assignment data. It is intended to be easy to use and to no major updates to them, are utilized multiple years in help in compiling the assignment instructions from the a row students start to do plagiarism even more [26]. To selected assignments, while keeping the visual style and mitigate these two plagiarism problems, i.e., lack of ad- readability consistent across each lecture week during vanced plagiarism detector and similarity of assignments one course implementation, as well as across multiple over the course implementations, in our programming course implementations. This aims to help students focus courses an autograder was changed and all the program- on the task itself and helps the course instructor to foming assignments in LUT CS1 and CS2 courses were cus on assignment creation and other duties rather than completely updated in fall 2022 and in spring 2023, respec- trying to make the visuals of the assignments readable tively. Current autograder used in LUT programming enough. In addition, Mímir enables the instructor an courses, CodeGrade, has integrated plagiarism detector easy method of editing the assignments based on precalled JPlag, which have detected many obfuscated stu- vious feedback or changes in teaching. This way the dent submissions already [5, 27]. An enormous efort edits are easy to implement and do not require fiddling of updating assignment included designing new assign- simultaneously with text editors and IDE. User can inments, programming example solutions, rewriting as- put all information needed, e.g., title, lecture number, signment instructions and setting up autograder tests for instructions, and example files, to generate assignment circa 100 assignments. instructions via GUI, shown in Figure 6, or by utilizing
Since the efort for recreating assignment base was existing files, e.g., source code files of an example solutime-consuming, we decided to develop a tool to manage tion. assignments and their variations for upcoming course implementations. Tool needed to be able to generate assignment instruction papers based on information inputted by user. The generated document would naturally need all the assignment related information which were available in handmade instructions, such as assignment instructions, example outputs, and possible example data from file(s). In addition, for course TAs an example solution is needed to be attached to the document. The developed tool is called Mímir, and in this study we describe our Mímir-tool in short. The tool was developed for the CS1 and CS2 courses with Python and C, respectively, starting with the following research questions:
assignments? RQ6: Are generated assignment instructions clear and understandable enough for students?
available solution, which would solve our assigment management problem, we created our own proof-of-concept solution for such tool. The tool, Mímir, helps a programming course instructor to combat plagiarism by automating selection of course assignments from a pool of similar
Course metadata is inserted in a main window, seen in the background in Figure 6, and for the new assignment a pop-window is opened, seen in foreground in Figure 6. In this demo example, the assignment is simple, with only single variation called “variation A”, and instructions part is extremely short. After insertion data is stored to a JSON file, which structure can be seen from Figure 7. in Figure 8. In normal course setting there would be 3-5 assignments per week, but now only one assignment was added to demo week 2.
CT00A0000 Demo Course L2 Assignments • Reading files Read course book chapter 2 and return the assignments to the autograder.
Contents L2E1: File reading Page 1 / 1 1 L2E1: File reading Make a C-program to read the contents of a file and print it to the terminal. Use fopen() to open the file and fgets() to read the contents.
Example from input file ’Inputfile.txt’: This is the first line.
And this is the 2nd line.
Example run 1 Inputs: Inputfile.txt Output: Input filename: Inputfile.txt File contents: This is the first line.
And this is the 2nd line.
Mímir v0.2.12
Week 2
language independent, so that it can be used on multiple courses that use diferent programming languages. However, as the tool is designed to be used on programming
The main level value pairs in JSON object contain the courses specifically, there are no features or options for general assignment information that are not specific to a other type of assignments. The major external depenvariation, or a single example run. As assignments are dencies which are not included in the program itself are not fixed to single week or order per week and therefore pdflatex program and programming language’s comalso these metainformation values are editable. The vari- piler or runtime. pdflatex is needed to compile the ations of the same assignment are then listed as a list of LaTeX documents into Portable Document Format (PDF) assignment objects, each having their own versions of in- files, and compiler or runtime are used to generate examstructions and paths to the files. Example runs, generated ple runs by executing an example solution. However, the to the instruction paper, are tied to a variation and there example solution compiling and running is voluntary, so can be multiple example runs per variation, therefore, if user chooses not to use the feature, external compilers they are stored inside a list that is inside the variation. or runtimes are not needed.
In addition, an example run object must store inputs and outputs for that specific run as they may difer. Finally, 4.2. Results each variation also stores implementations that it has been used in, such as ”Spring 2023”. Generated assign- A survey about generated assignment instructions was ment instructions for this demo assignment are shown conducted during CS2 course in spring 2023, in total 65 out of 240 students, i.e., 27.1 %, answered. Students were asked how they perceive visual clearness of generated assignment instructions, as well as, how the generated documents compared to handmade documents, used in the first four weeks of the course. In addition, there were four questions about how layout and formatting highlights help students while doing their assignments.
Answer distributions for these question sets are shown in Figures 9 and 10, respectively.
As seen in Figure 9, with the scale of 1-5; 1 worst and 5 best, documents generated by Mímir were considered slightly clearer than handmade ones, with an average 3.2, colored highlight of keyword was perceived better than only bolding the keyword, with an average 3.9. Visual Figure 10: Do generated layout and formatting help students clearness and readability as well as distinction of sep- understand the assignment instructions. arated instruction sections were considered good with averages 3.9 and 4.2, respectively. As seen in Figure 10, formatting and layout helping students with assignment were perceived very good for listing functions in bullet points, separating inputs as a distinct section, colouring keywords, and background highlighting, with averages 4.1, 4.4, 4.1 and 4.1, respectively. Mímir is a tool, developed with Python 3, to manage sets of programming assignments, with option to automate selection of programming assignments for new course implementations. It stores assignment data to JSON files and utilizes LaTeX to generate assignment instructions as PDF files. The tool is designed to be used by course staf either before the course implementation to generate dataset for the entire course at once, or when needed before each assignment set is released. While Mímir is not for students, they are a stakeholder as they the target group for generated assignment instruction papers, and therefore the survey was conducted to gather feedback about generated documents. Based on the survey students were satisfied with the generated content, which motivates us to develop the tool more and utilize it in upcoming courses. As Mímir is not limited to generate assignments for CS2 course and C programming language, it is planned to be used for fall 2023 CS1 course lectured with Python 3.
For the future development Mímir could be utilized to automatically generate autograder test cases, which could then be imported to autograder. However, this development option requires closer cooperation with the used autograder.
now. To improve the quality to the student materials as well as to reduce the time and efort required to complete all the course tasks, we are currently developing three new tools to support the course arrangements. In this paper we described three tools we are developing - ASPA, GradeTool and Mímir - and presented initial data that indicate the usefulness of these tools.
ASPA is an AST-based static analyzer tool to detect common programming style violations, and saves time both for students and teaching assistants by providing feedback anytime needed. Thus TA can focus on less common problems which require human to be detected or to explain reasons for required changes to student. Empirical data shows that students find the tool useful, and numerical data shows a statistically significant positive correlation between ASPA usage and student grades.
GradeTool is a tool to assist programming assignment grader in assessment process by providing convenient way to mark detected programming violations. Empirical data shows that the feedback generated by GradeTool is consistent enough, but feedback phrasing should be focused on more in the future, and numerical data shows otm-d92aa788-3b57-4580-bb82-60b836198ee9/ that with help of the tool, graders can generate a lot brochure. of feedback for students without need to write every- [3] E. Vanhala, U. Nikula, Python 3 – ohjelmointiopas thing. Furthermore, while using GradeTool TAs generate versio 1.2.1, LUT University, 2020. URL: https://urn. numerical data about violation occurrences which can fi/URN:ISBN:978-952-335-622-1. used to improve teaching and assessment processes even [4] The Finnish Institute of Technology, Fitech: Apply more. to summer courses from 4 april, 2023. URL: https:
Mímir is a programming assignment manager to han- //fitech.io/en/. dle assignment variations and generate assignment in- [5] CodeGrade, Streamline code learning and grading., struction documents for selected assignment variations. 2021. URL: https://www.codegrade.com/. Empirical survey data shows that generated assignment [6] D. Liu, A. Petersen, Static Analyses in Python Proinstruction documents are clear and consistent enough, gramming Courses, in: Proceedings of the 50th even a little bit better than handmade ones, which were ACM Technical Symposium on Computer Science used earlier. In addition, students perceived that, on the Education - SIGCSE ’19, ACM Press, Minneapoaverage, used layout and formatting highlights helped lis, MN, USA, 2019, pp. 666–671. doi:10.1145/ them to focus on the actual programming task and its 3287324.3287503. implementation instead of assignment instructions. [7] L. Gusukuma, A. C. Bart, D. Kafura, Pedal: An
Each of these tools has already been successfully used Infrastructure for Automated Feedback Systems, in: in programming courses and they suit for CS1 context, Proceedings of the 51st ACM Technical Symposium both in Finnish and in English, with Python program- on Computer Science Education, ACM, Portland OR ming language but are not limited to it, which is why USA, 2020, pp. 1061–1067. doi:10.1145/3328778. GradeTool and Mímir are already used also in CS2 course 3366913. lecture with C programming language. Due to the suc- [8] R. R. Choudhury, H. Yin, A. Fox, Scale-Driven Aucess, development is continued, and aim is moved for tomatic Hint Generation for Coding Style, in: Prointegrating these tools together more closely not just use ceedings of the 13th International Conference on their outputs in same course. For example, violations Intelligent Tutoring Systems - Volume 9684, ITS detected by ASPA are aimed to be importable directly 2016, Springer-Verlag, Zagreb, Croatia, 2016, pp. to GradeTool, which would even more fasten the assess- 122–132. doi:10.1007/978- 3- 319- 39583- 8_12. ment process when grader would not need to even mark [9] E. S. Wiese, M. Yen, A. Chen, L. A. Santos, A. Fox, already found violations. On the other hand, the future Teaching Students to Recognize and Implement development of Mímir could be automatic generation of Good Coding Style, in: Conference on Learning @ autograder test cases, which could then be imported to Scale - L@S ’17, 2017, pp. 41–50. autograder, which would then save time on assignment [10] r2c, Semgrep: Code scanning at ludicrous speed., setup phase. 2023. URL: https://semgrep.dev/docs/.
Next iterations for these development actions are [11] E. Kaila, M.-J. Laakso, T. Rajala, E. Kurvinen, planned to be done for the fall 2023 CS1 course and ex- A model for gamifying programming education: pand range to other programming courses as well. In University-level programming course quantified, in: the future, we also could expand our analysis to cover 2018 41st International Convention on Information topics related to the support services of code generation, and Communication Technology, Electronics and such as use and role of artificial intelligence, or taking an Microelectronics (MIPRO), IEEE, Opatija, 2018, pp. indepth look into the discussions and common problems 0689–0694. doi:10.23919/MIPRO.2018.8400129. the teaching assistants encounter on the online support [12] V. Karavirta, P. Ihantola, T. Koskinen, Serviceplatforms. Oriented Approach to Improve Interoperability of ELearning Systems, in: 2013 IEEE 13th International Conference on Advanced Learning Technologies, References 2013, pp. 341–345. doi:10.1109/ICALT.2013.105, iSSN: 2161-377X. [1] U. Nikula, O. Gotel, J. Kasurinen, A Motivation [13] R. Luukkainen, J. Kasurinen, U. Nikula, V. LenarGuided Holistic Rehabilitation of the First Program- duzzi, ASPA: A static analyser to support learning ming Course, ACM Transactions on Computing and continuous feedback on programming courses. Education 11 (2011) 1–38. doi:10.1145/2048931. an empirical validation, in: Proceedings of the 2048935. ACM/IEEE 44th International Conference on Soft[2] U. Nikula, J. Järvinen, Ohjelmoinnin perusteet ware Engineering: Software Engineering Education (6 op) / Introduction to Programming (6 cr), and Training, ICSE-SEET ’22, Association for Com2023. URL: https://sisu.lut.fi/student/courseunit/ puting Machinery, New York, NY, USA, 2022, p. 29–39. doi:10.1145/3510456.3514149. [25] L. Nichols, K. Dewey, M. Emre, S. Chen, B. Hard[14] JASP Team, JASP (Version 0.14.1)[Computer soft- ekopf, Syntax-based improvements to plagiarism ware], 2020. URL: https://jasp-stats.org/. detectors and their evaluations, in: Proceedings of [15] J. Kasurinen, U. Nikula, Estimating programming the 2019 ACM Conference on Innovation and Techknowledge with Bayesian knowledge tracing, ACM nology in Computer Science Education, ITiCSE ’19, SIGCSE Bulletin 41 (2009) 313–317. doi:10.1145/ Association for Computing Machinery, New York, 1595496.1562972. NY, USA, 2019, p. 555–561. doi:10.1145/3304221. [16] Labtool-2019 group, Welcome to the 3319789.
labtool wiki!, 2022. URL: https://github.com/ [26] T. Hynninen, A. Knutas, J. Kasurinen, Plagiarism UniversityOfHelsinkiCS/labtool/wiki. networks: finding instances of copied answers in [17] T. Ahoniemi, T. Reinikainen, Aloha - a grading tool an online introductory programming environment, for semi-automatic assessment of mass program- in: Proceedings of the 17th Koli Calling Internaming courses, in: Proceedings of the 6th Baltic tional Conference on Computing Education ReSea Conference on Computing Education Research: search, Koli Calling ’17, Association for ComputKoli Calling 2006, Baltic Sea ’06, Association for ing Machinery, 2017, pp. 187–188. doi:10.1145/ Computing Machinery, New York, NY, USA, 2006, 3141880.3141906.
p. 139–140. doi:10.1145/1315803.1315830. [27] L. Prechelt, G. Malpohl, M. Philippsen, Finding [18] T. Ahoniemi, E. Lahtinen, T. Reinikainen, Improv- plagiarisms among a set of programs with JPlag, ing pedagogical feedback and objective grading, in: Journal of Universal Computer Science 8 (2002) Proceedings of the 39th SIGCSE Technical Sympo- 1016–1038. sium on Computer Science Education, SIGCSE ’08, Association for Computing Machinery, New York, NY, USA, 2008, p. 72–76. doi:10.1145/1352135.
1352162. [19] S. Parihar, Z. Dadachanji, P. K. Singh, R. Das,
A. Karkare, A. Bhattacharya, Automatic grading and feedback using program repair for introductory programming courses, in: Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 92–97. doi:10.1145/3059009.3059026. [20] E. Hegarty-Kelly, D. A. Mooney, Analysis of an automatic grading system within first year computer science programming modules, in: Proceedings of 5th Conference on Computing Education Practice, CEP ’21, Association for Computing Machinery, New York, NY, USA, 2021, p. 17–20.
doi:10.1145/3437914.3437973. [21] J.-Y. Kuo, H.-C. Lin, P.-F. Wang, Z.-G. Nie, A feedback system supporting students approaching a high-level programming course, Applied Sciences 12 (2022). doi:10.3390/app12147064. [22] J. Hofstadt, P. Cothren, A modern, fast and powerful gui framework for python., 2023. URL: https: //github.com/hoffstadt/DearPyGui/wiki. [23] H. T. Jankowitz, Detecting plagiarism in student pascale programs, The Computer Journal 31 (1988) 1–8. doi:10.1093/comjnl/31.1.1. [24] S. Mann, Z. Frew, Similarity and originality in code:
Plagiarism and normal variation in student assignments, in: Proceedings of the 8th Australasian Conference on Computing Education-Volume 52, volume 52, 2006, pp. 143–150. doi:10.5555/1151869. 1151888.