Distributed Pair Programming (DPP) is widely known to promote collaboration and knowledge sharing among novice programmers, while it engages them in carrying out programming assignments. Moreover, DPP is a means of experiencing agile software development techniques that are considered important in the software market. In this paper, we share some experiences on using the DPP system of SCEPPSys for carrying out assignments in an undergraduate Object Oriented Programming (OOP) course. Specifically, we focus on the information recorded during problem solving and the statistics reported by the system and how this information can be utilized for monitoring both the course regarding the fulfillment of its goals and the programming habits and progress of students. Some proposals are made towards extending the possibilities of SCEPPSys for generating automatically more sophisticated reports that would support instructors in more successfully monitoring a course and students. The ultimate goal of such an enhanced monitoring is to improve students' software quality.
Distributed Pair Programming (DPP) systems allow two programmers to collaborate remotely in order to apply the Pair Programming (PP) [Cockburn and Williams 2000] technique from separate locations. The model of PP originated from the software industry as a part of Extreme Programming (XP) [Williams et al. 2000]. It involves two programmers working on the same workstation and sharing one computer in order to develop software. In the context of pair programming, the team members adopt two specific roles: one programmer acts as the "driver" and the other one as the "navigator" (also called "observer"). The driver has possession of keyboard and mouse and types the programming code, while the navigator reviews the inserted code and gives guidelines to the driver. Driver and navigator are in constant collaboration in order to design and develop the program code in common, while they should frequently alternate roles. PP is based on close collaboration, continuous knowledge transfer, negotiation and sharing of programming skills. Programmers not only receive the advantages of group work, they also produce high quality software in a shorter time [Sanjay andVanshi 2010, Zacharis 2009]. The only difference between DPP and PP is DPP's allowing geographically-distributed teams to collaborate and share program code. However, such collaboration is only feasible if an underlying infrastructure enables for all necessary interactions.
A considerable number of solutions used to implement DPP were built as plugins for the Eclipse IDE. These plugins support DPP for the Java programming language, within a very popular development environment. A detailed review of all available DPP solutions reveals that most tools lack effective instructional or student support [Tsompanoudi and Satratzemi 2011]. Although they cover the basic requirements of DPP, they cannot address common problems of PP, such as unequal contributions from each member of the student pair.
12:98 • S. Xinogalos, M. Satratzemi, D. Tsompanoudi and A. Chatzigeorgiou For this purpose SCEPPSys (Scripted Collaboration in an Educational Pair Programming System) was developed using an existing Eclipse plugin for DPP. Compared to other plugins like Sangam [Ho et al. 2004] and XPairtise [Schümmer and Lukosch 2009], SCEPPSys saves and analyzes users' interactions, and helps educators in organizing and monitoring DPP classes. To the best of our knowledge only RIPPLE [Boyer et al. 2008] provides logging capability of users' interactions, allowing researchers to reconstruct user sessions for further study. However, the logged information is provided only as raw data and further processing is required in order to extract useful information.
In this paper, we share some experiences on using the DPP system of SCEPPSys for carrying out assignments in an undergraduate Object Oriented Programming (OOP) course based on Java. Specifically, we focus on the information recorded during problem solving and the statistics reported by the system and how this information can be utilized for monitoring both the course regarding the fulfillment of its goals and the programming habits and progress of students. Some proposals are made towards extending the possibilities of SCEPPSys for reporting automatically more sophisticated reports that would support instructors in more successfully monitoring a course and students.
SCEPPSys is based on a typical client-server architecture and consists of: a server for dispatching messages between the clients; a database for storing user's accounts, information about courses, groups and assignments, shared projects and statistics; a web-based authoring tool used by instructors for scripting DPP [Tsompanoudi and Satrtatzemi 2014]; and an Eclipse plugin installed by students. In the next sections the process that an instructor applies for setting up a course, as well as a typical DPP session carried out by students are briefly described. Interested readers can find further information in a detailed description of SCEPPSys [Tsompanoudi et al. 2015].
The instructor can easily set up a course using the web-based authoring tool of SCEPPSys. First of all the list of learning goals (e.g. constructor definition, object construction, inheritance) that will be used for characterizing the various tasks assigned to students must be defined. Then the assignments can be defined and scheduled in the form of collaboration scripts. A collaboration script involves the definition of: (1) participants -students enrolled to the course; (2) groups or pairs that can be formed randomly, with comparable skill or contribution levels, or freely; (3) assignments that comprise of problem solving tasks or steps characterized by one of the learning goals (not visible to students) and having an accompanying hint that can be optionally consulted by students; (4) task distribution policies -rotating role switching of driver/observer, balanced knowledge switching aiming at achieving symmetry in skill acquisition (learning goals) or free switching. The system monitors the problem solving approach adopted by students and reports various statistics described in the next section.
Students use an Eclipse plugin for applying DPP. A DPP session starts when group members meet online and request a pair programming session. Then a shared project is automatically generated inside the workspace of both students and the programming tasks are displayed in a separate area. Students solve the tasks by adopting the roles of the driver and navigator and switch roles according to the task distribution policy. During the session a text-based chat provides a means of communication and coordination between the team members. To motivate students, metrics like driving time and individual participation rates are displayed in the workspace and students may retrieve useful hints for each step during the problem solving process. Students may submit the assignment on session close or continue the DPP session at another time.
Monitoring an OOP Course Through Assignments in a DPP System • 12:99
SCEPPSys records a variety of information during the problem solving process and calculates statistics for each student per project. The statistics reported are presented in Table I. Moreover, the mean value of the aforementioned elements is calculated along with the number of projects submitted per assignment for each one of the utilized distribution policies (none/free, rotating roles, balanced). The statistics calculated and reported by SCEPPSys in combination with the projects' grades can be used as metrics for: monitoring the fulfillment of the courses' goals in general detecting difficulties with specific OOP concepts/constructs detecting students' progress in programming detecting undesirable behaviors (e.g. plagiarism)
In the following sections the indications provided by the most important of the aforementioned statistics are briefly analyzed.
Both the total time for developing a project and the time for actually writing code (driving time) provide indications for the level of difficulty of an assignment, as well as the difficulties of students with the underlying OOP concepts. Moreover, the total and driving time of an assignment help realize students' workload that must be in accordance with the ECTS of the course.
The difference of the total and driving time, or else the driving/total time ratio, helps instructors detect odd or even extreme behaviors. For example, a very small total time or a total time that is approximately equal to the driving time are strong indications of either "copying a solution" or "working offline" (most probably in isolation) and logging in the system just for copying and pasting either the plagiarized or offline solution correspondingly. Both behaviors are problematic and denote non-academic behavior, non-collaborative problem solving or cooperation problems. Instructors can draw safer conclusions if the minimum time necessary to collaboratively solve a specific assignment through the system is recorded. This can be ideally accomplished by having themselves or knowledgeable assistants solve the assignment, or else by estimating the necessary time based on their experience from lab sessions.
Every step of an assignment can have a corresponding hint, providing support to students. Depending on the difficulty of a specific step and whether it deals with a new OOP concept/construct the help provided can be more or less verbose. Moreover, it can contain small pieces of source code. The number of hints retrieved by students is another indication of the degree of difficulty or students' 12:100 • S. Xinogalos, M. Satratzemi, D. Tsompanoudi and A. Chatzigeorgiou confidence on their solution for an assignment. This means that the more difficult an assignment is, or alternatively the less confident students are for their solution, a bigger number of hints is retrieved.
One of the main advantages of DPP is the collaboration and exchange of knowledge between the members of a pair. The communication between the collaborators can be accomplished using the incorporated chat tool. The number of messages sent during code development provides indications for the degree of cooperation and communication between the members of a pair, as well as for the difficulty of an assignment. Of course, students can and do use other means for communication during problem solving, such as Skype and Facebook.
The number of synchronized program executions (sync runs) provides information regarding the problem solving strategy adopted from students. Instructors always emphasize the advantages of incremental development and testing, but unfortunately students tend to write large portions of a program before they actually test it. When implementing programs in OOP languages this delayed testing is even more prominent due to the fact that usually an important portion of a class or classes has to be implemented along with the main method in order to test the program. Students tend to complete the definition of a class, or even the classes needed, before they write the main method and test their solution. SCEPSSyS can contribute to changing students' problem solving strategies through scripted collaboration. Specifically, students can be asked to incrementally implement and test a class by appropriate steps in the collaboration script and moreover they can be asked to check the results until then by running their project. However, it is clear that students can overlook the script's suggestion and not test their project. So, examining the number of synchronized program executions can help monitor students' problem solving strategies.
Moreover, the number of sync runs in combination with other statistics, such as the total and driving time, number of hints retrieved and messages sent, can indicate potential difficulties in achieving the goals of an assignment. For example, a large number of sync runs combined with an average (or less than average) total and driving time can indicate the application of the desirable incremental development and testing strategy. On the other hand, a large number of sync runs combined with a large amount of total and driving time (above average) can indicate difficulties in achieving the goals of the assignment.
SCEPPSys was used during the academic year 2015-16 in a 3 rd semester compulsory course on OOP at the Department of Applied Informatics, University of Macedonia, Greece. Students were taught for 3 hours per week in the lab and had the chance to carry out programming assignments during the 13 weeks of the course. The assignments were optional and their grade was calculated only in the case that students passed the final exams. The previous years the assignments were carried out individually by the participating students as homework. This year students had the chance to select whether they would like to solve the assignments individually, or in pairs using the DPP system of SCEPPSys. Initially 48 pairs of students registered for the DPP assignments, while 47 pairs submitted at least one out of the six projects assigned. At the end of the course a questionnaire was delivered to the participating students in order to obtain their feedback on the DPP assignments.
The six projects covered the main learning units of the course and utilized small and manageable steps with corresponding hints. The only exception was the sixth project that included just five steps, one for each one of the necessary classes. In Table II the learning unit covered by each project is presented, along with the mean grade (in the scale of 0 to 10) and the mean values of the metrics briefly analyzed in the previous section.
Students' mean grades in all the projects were very good (at least 8.72). Even at the last project the mean grade was 8.76, with the difference that it was submitted by fewer pairs. As already mentioned, the last project was a special case, since it included one step for each class and no hints. Moreover, it included most of the OOP concepts/constructs taught in the course.
Students tend to give up the effort as the course reaches the end. As can be seen in Table II the number of projects submitted decreases importantly in the last two assignments (approximately the last three weeks of the course). It is quite certain that instructors teaching programming courses, and especially OOP, would agree that such courses are cognitively demanding and the cognitive overload of students leads a certain percentage of them in dropping out of their responsibilities at some point, such as attending lectures and carrying out assignments. Of course, we must remind that in the specific case, assignments were not obligatory. Moreover, we must mention that 41% of the students that participated in the DPP assignments had not passed the first semester "Procedural Programming" course based on C. It is clear that understanding the reasons for dropping out needs a much deeper analysis of the available data in order to draw solid conclusions.
Implementing programs is widely known that is a time consuming activity, especially for novices.
Based on the data collected by the system, it comes out that students working in pairs spent approximately two to four hours for writing the code for an assignment. However, nearly one fourth to one third of this time was spent on actually writing code. The only exception was the last and most demanding -in terms of the concepts covered, LOC, lack of detailed steps (each step corresponded to a whole class) and hints -project, where just 21% of the total time was spent on writing code. It is reasonable to assume that most of the time was spent on studying the corresponding material, reading hints, communicating with the collaborator and of course thinking about the solution.
12:102 • S. Xinogalos, M. Satratzemi, D. Tsompanoudi and A. Chatzigeorgiou
Incremental development and testing, as already mentioned, is an important aspect that should be reinforced among students. However, especially in OOP, this is not always easy. Students tend to implement the classes needed and then write the main method for testing their code. In scripted collaboration the instructor has the chance to reinforce the idea of incremental development and testing by asking students partially implement a class and create objects for testing each method from the very first steps. Moreover, students can be reminded in the textual description of the steps to run their project. Of course, students can just ignore such advice. From the data recorded by the system it turns out that students do run the programs, but it cannot be told in certainty that they do it during problem solving for testing their program, or because they have to make continuous corrections due to raised exceptions and/or logical errors that lead to wrong output. However, at least 8 sync runs were recorded in average for each project, which is an indication of incremental development and testing. If the time of each sync run and its (un)successful completion was recorded we could draw safer conclusions regarding students' problem solving strategies.
Although several students use alternative means for communication, and especially Skype and Facebook, the average number of messages sent by the members of the pairs was unexpectedly high. Specifically, 86 to 153 messages were sent in average for each project. This is definitely a strong indication of collaborative work and exchange of perceptions and knowledge.
An important feature of SCEPPSys that helps the instructor to monitor the coverage of the intended OOP concepts/constructs through the assignments, is reporting the frequency that each learning goal has been addressed in the context of the assignments so far. This feature gives the chance to adjust the assignments and the learning goals they aim at. Ideally, this must be done in combination with the recorded difficulty for each learning goal as the course progresses. Since the projects are graded on-line in a step by step manner and each step is characterized by its main learning goal, the average grade per learning goal can be calculated and used for detecting students' difficulties and taking the appropriate actions both in lectures and assignments.
In the on-line questionnaire that was answered voluntarily by 58 out of the 94 students that participated in the DPP assignments, the majority agreed or completely agreed that the quality of the assignments was good (86.3%) and that the assignments covered in a high degree the content of the course (84.5%).
Based on the assumptions made in the previous section about the various statistics recorded and using the data reported in Table II, some conclusions regarding the difficulty of the assignments can be drawn.
Considering the number of projects submitted in combination with the grades of the assignments it is obvious that the last assignment (#6) was the most difficult one. This was anticipated since it combines most of the OOP concepts taught: inheritance, ArrayList, event handling and for the first time concepts from the Collection Framework (Comparator) and binary files. This makes it clear that students should be given more time (usually it is ten days) for this last assignment.
Based on the reduced number of projects submitted (28) the next most difficult assignment was the one referring to GUI and event handling (#5), which also utilized inheritance. The number of submitted projects decreased gradually from 45/46 to 39, 35 and finally less than 30 in the last two projects. It seems that this is more or less the time point during the course at which the amount of required knowledge overwhelms a number of students. As a consequence, dropping out appears and instructors should pay special attention in finding ways to support and even more motivate students to keep on the effort. However, the performance of the pairs that submitted the last two projects was still very good.
Although students' performance in the assignment regarding object collections and more specifically the ArrayList collection was very good (average grade 9.15), there are strong indications that it was a demanding assignment (#3) for students, maybe difficult for some of them. Specifically, this assignment demanded more than 4 hours of work in average for each pair, and a big number (153) of messages sent through the incorporated chat tool in order to collaboratively reach a solution. Moreover, 65% of the available hints were retrieved by the students and 25 sync runs of their projects were made. The number of sync runs indicates that various problems came up during problem solving that required correcting and re-running the project. Of course, we must stress that students finally managed to collaboratively reach a good solution, as well as the fact that difficulties with ArrayList collections have been recorded in previous studies (Xinogalos et al. 2008).
The conclusions drawn from the statistics calculated by the system are partially confirmed by students' replies on the questionnaire presented in Table III Out of the 58 students that filled in the questionnaire: The last assignment (#6) that demanded the usage of inheritance, ArrayList collections, event handling, the Collection Framework (Comparator) and binary files was the most difficult for students. Specifically, 63% considered this assignment as difficult (32%) or of high difficulty (31%), while another 17.2% did not submit this assignment. The assignment #5 was the second more difficult assignment. It referred to GUI, event handling and also utilized inheritance. 58.6% of the students considered this assignment to be difficult (43.1%), or even of high difficulty (15.5%). The next more difficult assignment according to students was the one referring to inheritance and polymorphism (#4). Specifically, 50% of the students considered the assignment of medium difficulty and 15.5% as difficult or of high difficulty. The statistics recorded by the system show that this assignment demanded the less time, while the smallest percentage (56%) of hints was retrieved in comparison with the rest of the assignments. Probably, the assignment demanded off-line studying and preparation. Finally, the assignment on object collections and ArrayList (#3) was considered by 37.9% of the students to be of medium difficulty, while the statistics recorded indicated a higher degree of difficulty. Despite the insights provided by students' replies, we must mention that the questionnaire was filled in after the end of the course. If students were asked to rate the difficulty of each assignment just after finishing it (for example, by entering the specific information as a comment at the end of their project) and not recalling after several weeks their difficulty with assignments, the results of the specific question would be more trustworthy.
12:104 • S. Xinogalos, M. Satratzemi, D. Tsompanoudi and A. Chatzigeorgiou
The benefits of DPP are numerous and have been recorded in the literature. The system of SCEPPSys utilized in this study has the advantage of recording various data and reporting statistics that could be used for monitoring an OOP course based on Java regarding the fulfillment of its goals and taking the appropriate actions in time. Such statistics can be used for detecting difficulties with specific concepts, the quality of collaboration between pairs and potential undesirable behaviors, as well as students' progress in programming. An important extension of SCEPPSys would be the ability to provide more advanced reports towards this direction that would automate the process of detecting:
potential difficulties with specific OOP concepts/constructs: by calculating the average grade achieved by students for each learning goal in the context of each assignment and all the assignments collectively. detecting students' progress in programming: by calculating and reporting important changes in the grades of his/her projects, as well as the contribution of a student in the projects. detecting undesirable behaviors (e.g. copying a solution, or working offline): by comparing the total and driving time for a project (if there is a small difference between them then probably a solution was just entered in the system and not collaboratively developed), and also with a minimum required time defined by the instructor. It is obvious that in order to achieve this goal and provide really comprehensive reports that will save a great deal of time to instructors and give them even more valuable information both for monitoring the course and each student, the system has to be used in real situations. This study had exactly this goal, to start exploring the indications provided by the reported statistics, validating and using them for automatically creating reports that will be a useful tool in the hands of instructors. A step further would be to examine the possibility of incorporating to the system abilities for calculating simple educational software metrics with the aim of enhancing the quality of students' programs [Xinogalos and Ivanović 2013].