-Application of educational and interactive simulation games to teach important concepts is an area that has attracted several Software Engineering researchers and educators attention. Previous research and studies on usage of simulation games in classroom to train students have demonstrated positive learning outcomes. Peer code review is a recommended best practice during software development which consists of systematically examining the source code of peers before releasing the software to Quality Assurance (QA). Practitioners and Researchers have proposed several best practices on various aspects of peer code review such as the size of code to be reviewed (in terms of lines of code), inspection rate and usage of checklists. We describe a single player educational simulation game to train students on best practices of peer code review. We define learning objectives, create a scenario in which the player plays the role of a project manager and a scoring system (as a factor of time, budget, quality and technical debt). We design a result screen showing the trace of events and reasoning (learning through success and failure as well as discovery) behind the points awarded to the player. We conduct a survey of the players by conducting a quiz before and after the game play and demonstrate the effectiveness of our approach.
Keywords—Peer Code Review, Simulation Game, Software Engineering Education and Training, Teaching Critical Decision Making
I.
Software Engineering (SE) being a practice-oriented and
applied field is taught primarily (at University Level) using
instructor-driven classroom lectures as well as team-based
projects requiring hands-on skills. In comparison to classroom
lectures, team-based hands-on projects require more active
participation and experiential learning. SE Educators have
proposed and shown positive student learning outcomes by
teaching certain concepts using simulation games. Some of the
advantages of teaching using simulation games are
incorporation of real-world dynamics such as critical decision making
under multiple and conflicting goals, encountering unexpected
and unanticipated events, allowing exploration of alternatives
(discovery learning) and allowing incorporation of learning
through doing and failure [
To develop a web-based interactive educational SE simulation game or environment for teaching benefits and best-practices of peer-code review process.
To investigate a learning framework and model based on discovery learning, learning from failure, evidence and reasoning for teaching concepts on the practice of peer code review.
To evaluate the proposed learning framework and tool by conducting experiments and collecting feedback from users.
II.
CONTRIBUTIONS
In this Section, we discuss closely related work and state our novel research contributions in context to the related work. We conduct a literature survey of papers published on the topic of teaching Software Engineering concepts using simulation games. Table I shows the result of our literature review. Table I lists 9 papers in reverse chronological order and reveals that teaching Software Engineering concepts using simulation games is an area that has attracted several researchers attention from the year 2000 until 2013. We characterize 9 papers based on the tool name, year, simulation topic, University and interface. We infer that teaching software engineering processes and project management are the two most popular target areas for simulation games. The game interfaces various from simple command line and menu driven model to animated and 3D interfaces. Researchers have also experimented with board games in addition to computer-based games. In context to closely related work, the study presented in this paper makes the following novel contributions: 1)
While there has been work done in the area of teaching Software Engineering processes and project management skills, our work is the first in the area of building and investigating simulation games for teaching best practices for peer code review.
We propose a simulation game to teach peer code review practices based on learning by failure, success and discovery. We demonstrate the effectiveness of our approach, discuss the strengths and limitations of our tool based on conducting user experiments and collecting their feedback.
III.
In this game we define 12 learning objectives covering multiple aspects of peer code review. These learning objectives are captured in our pre game questions1 and post game questions2. Table II shows 6 of the 12 learning objectives (due to space constraint), corresponding decisions and the questions which we asked to the player in game. Table II shows the structure that we follow to design the game questions. Each question is designed keeping in mind the learning objectives of the game. Based on the learning objectives we come up with a situation or a decision which best questions that objective. An evaluation question is then formed around this decision which challenges the player’s knowledge to its best. We therefore present different situations to the player where they have to make decisions like whom to assign the task of review process (Figure 1), what inspection rate to choose (Figure 2), when to start with review, steps required to foster good code review culture in team etc. A decision can map to one or more learning objectives. Similarly two or more evaluation questions may map to one decision.
We introduce unexpected events and unforeseen circumstances (such as internal conflicts between the team members, attrition and change in deadline or demand from the customer) in the game to make it more realistic. Unexpected events are unobservable and our goal is to examine the response and the decision making ability of the player to unexpected circumstances. Figure 3 shows a screenshot for the simulation game in which the player is presented with an unexpected
2http://bit.ly/1GEdBBX situation. As shown in Figure 3, the project manager is encountered with a situation wherein a developer quits the team or organization just one month before the release date. C. Scoring System
1) Technical Debt: We apply Technical Debt3 concept or metaphor as one of the elements of our scoring system. We calibrate our scoring system such that incorrect decisions (quick and dirty solutions) lead to accumulation of technical debt. Figure 4 shows technical debt score of various players (data collected during our experimental evaluation of the simulation game) as the game progresses from start to finish. Figure 4 reveals various behaviors: we observe cases in which technical debt gets build-up due to lack of knowledge and incorrect decisions and on the other hand we notice cases in which prudent decisions lead to controlled technical debt. Each decision a player makes has certain weight or point associated to it varying from 1 to 5. We take the maximum range value i.e. 5 as standard TD (Technical Debt) point against which all calculations are made. For every decision taken by the player, we calculate the deviation of player’s current decision point from the standard TD point and find the average TD point value obtained so far. We then calculate the equivalent percentage value of this TD point relative to the standard TD point which is the overall TD incurred by player so far in the game (refer to Equations 1, 2, 3 and 4).
deviationV alue = (standardT D decisionW eight) (1) sumT D = sumT D + deviationV alue T Davg = sumT D=decisionCounter T D = (T Davg=standardT D) 100 (2) (3) (4) 2) Time: Time is the most valuable resource in any project and time management is another key aspect of managing a project. It is therefore required by the player to properly plan the project time and associated decisions to meet project deadline. Every decision that a player takes in game has a positive or negative impact. The magnitude of impact varies depending on the choice or decision made by the player (as shown in Figure 5). Series of decisions taken by a player either
Learning Objective The earlier a defect is found, the better. The longer a defect remains in an artifact, the more embedded it will become and the more it will cost to fix Authors should annotate source code before the review begins Use of checklists substantially improve results for both authors and reviewers prevent them from meeting the deadline or they are able to complete the project successfully. All of the decisions have a path time associated with them. This path time gets deducted from the remaining time as the player proceeds further in game (Equation 5 and 6).
remainingT ime = remainingT ime pathT ime timeScore (days) = remainingT ime (5) (6) 3) Budget: At the beginning of game, player is allotted a budget of Rs 2 million to complete the project. The scoring system that we have built, tracks player’s utilisation of this budget. If at any point of time player has consumed the entire budget value, they can go no further in game. Figure 6 shows different player behaviours in terms of budget utilisation. Budget remaining at the end of game is the combined effect of the cost consumed to perform review, developer’s recruitment, buying tools, team incentives etc. Refer to Equation 7 and 8 to see how path cost for player’s decision is used to obtain the cost score.
remainingC ost = remainingC ost
pathC ost
costScore (Rs) = remainingC ost
(7)
(8)
4) Quality: We incorporate the concept of software quality
in our scoring system to keep a check of code review
consistency maintained by the player. The scoring system captures
the quality standards from the beginning of the project. We use
defect density (defects/kLOC) to measure the software quality
maintained during the game. Code bases with a defect density
of 1:0 (or 1 defect for every 1; 000 lines of code) are considered
good quality software[
Each game consists of a problem space, initial state and
single (or a set of) goal states. A problem space is a
mathematical abstraction in form of a tree (refer to Figure 8) where
the root represents starting game state, nodes represent states
of the game (decisions in our case), edges represent moves and
leaves represent final states (marked as red circles) [
At the end of game, each player gets a score reflecting their overall performance. This score produced, takes into account all the factors like path taken, remaining time, budget consumed, project quality index and technical debt accumulated at the end of game. Following are the steps depicting the procedure to compute the performance of each player during the execution of game and based on the decisions made by the player.
1) 2) 3) 4)
Each decision a player takes has a weight (Wd) associated with it, which varies from +1 to +5 (represented in Figure 8). As the player proceeds in game, the weight associated with each decision keeps on accumulating and is stored in Psum (refer to Equation 12). Final value of Psum is then used to determine whether a player followed a poor, optimal, good or an excellent path during the game.
Psum =
X Wd (out of 50)
(12)
Next we scale and obtain the equivalent values for
cost (Cf), time (Tf) and quality (Qf) remaining at the
end of game. Project attributes sum (Pattr sum) holds
the average of equivalent values of these three project
attributes (Equation 13) [
Pattr sum = (Cf + Tf + Qf )=3 (out of 50) (13) Score sum (Sf’) is obtained by adding the values calculated in step 1 and 2 (Equation 14).
Sf0 (out of 100) = Psum + Pattr sum (14) We then make use of TDavg (refer to Equation 3) to calculate the final score (Sf) in equation 15.
To meet the game’s objective, it is necessary to ensure that player is learning throughout the course of game. The player is given a proper feedback at each step of the game so they can observe the effect of decisions taken by them on the project. Every decision of player has a direct impact on the four parameters of scoring system i.e. time, cost, quality and technical debt, which is made visible by the presence of four dynamic meters in game screens (demonstrated in Figure 1, 2 and 3). The value of score meters change after each decision and makes player aware of the consequences of his decision on the available resources. In the end, final values in score meters obtained for the player is compared with the ideal values that should be there (refer to Figure 9). It helps student compare the consumption of resources done during their project course and the quality standards that they managed to maintain. Along with step wise feedback, we also provide a detailed analysis of player’s performance at the end of the game. We reflect to player’s the decisions taken by them during the game and provide feedback in form of remarks. The remark that a player gets, help them discover about the correctness of their decisions. For example, if a player chooses to recruit a full time reviewer (refer to screenshot in Figure 1) they are reminded that there is a more experienced and expert co developer present in the team who can carry out this task. In case they select a very fast inspection rate like 900 or above LOC/hour (Figure 2) they are told that with this high inspection rate they can conclude that the reviewer is not looking at the code at all. This in depth analysis helps player trace the events and reasoning (learning through success and failure as well as discovery) behind the points awarded to them (Figure 10 shows remarks provided to player).
IV. EXPERIMENTAL ANALYSIS AND PERFORMANCE
EVALUATION
We conduct experiments to evaluate our proposed approach by asking 17 students (10 Undergraduate or Bachelors in Computer Science and 7 Graduate or Masters in Computer Science students) to play the game. Table III, IV presents the list of questions and their responses in the pre-game questionnaire4. Sf = S0 f (1
T Davg=standardT D) (15)
TABLE III.
QUESTIONNAIRE RESULTS BEFORE THE GAME PLAY
[
Survey score values (in pre game and post game survey) of students who took part in the game evaluation The pre-game questionnaire consists of questions having one correct answer (2, 3, 5, 7, 8, 9, 10 and 11) and also questions having more than one correct answer (1, 4 and 6). Table III displays the questions, answer choices, correct answer(s) and the responses of 17 students. Table III reveals that 30% of the respondents selected the correct choice for the most optimal inspection rate for peer code review and only 18% selected the correct answer for optimal code review size. We asked such questions to test the knowledge of the students and investigate improvement after playing our game. As shown in Figure III, our pre-game questionnaire consists of questions on various aspects of peer code review.
Table IV displays a set of inter related questions, answer choices (common for all questions) and the responses of 17 students for the same. We asked students these questions to test their knowledge of existing review processes. The questions test the efficiency of review processes when compared in different categories like cost of using that process, time taken to carry out review process, review effectiveness and resource consumption. It can be observed that 47:05% of students think that tool assisted review process is more efficient in terms of cost and only 29:42% are aware that manual review processes actually have low investment cost. The general belief that tool assisted reviews are better that manual review processes in every aspect is evident from Table IV, despite the fact that both of the review process find just as many bugs.
As observed from Table III and IV, there are total 11 questions in the survey questionnaire, which we made mandatory for each student to answer. To obtain the score of pre and post game survey questions we assign a weight age of 1 mark to each question. For every wrong response +0 marks were given and for each right response selected +1/n marks were given (n is the number of correct answers for that question).
As mentioned above, questions in survey have both single and multiple correct answers, therefore value of n varies from 1 to 3 (n=1 for Q2, 3, 5, 7, 8, 9, 10, 11; n=2 for Q4; n=3 for 1,6). Using this scoring criteria, we made a bar graph to see the improvement in score before and after playing the game (refer to Figure 11). It can be seen that mean score of students in pre game survey is 4:44 which increases to 7:75 in post game survey. Thus, there is an average improvement of 74:54% i.e. 3:31 in score. In terms of absolute score a maximum improvement of 5:67 (Student ID 10) and minimum improvement of 1:5 (Student ID 2) can be seen.
Figure 12 is a bar graph representing how survey scores vary for each question, before and after game play. The values in Figure 12 are the values for each question that we obtained from the responses of 17 students who took part in evaluation.
We follow the same scoring criteria that we use to draw Figure 11. It explores the improvement trend that exists for each and every question, before and after the game play. As visible from the graph, there are some questions which could be categorized easy as more than 50% of students could answer them correctly in pre game survey (Q1, 3, 5 and 9). Similarly there is another set of questions which students found hard and only 30% or less could answer correctly like Q2, Q4, Q6, Q7 and Q8. A mean pre game score of 6:86 is observed for these questions, which then raises to 11:98 in post game survey.
Thus an average improvement of 74:63% (5:12 in absolute) is observed, which is similar to that observed in Figure 11.
In absolute terms, a maximum improvement of 12 (Q7) and minimum improvement of 2 (Q9 and Q11) is observed.
We perform post game survey once the students are done playing the game. It is divided into two sections. First section contains all the questions specified in Table III and IV, to test student’s learning after the game play. There is another section which requires the player to rate the game on different aspects like learning done throughout the game, score and analysis provided at the end of game, motivation to try again on strategy or decision failure and introduction of new concepts or practices which student was not aware of. Figure 13 represents the box plot demonstrating the distribution of ratings provided by the player to game in these categories. It can be seen that learning done throughout the game has a median of 4 with most of the rating distribution spread from 3:75 to 4 with 7 outliers (3, 5) as visible from Figure 13. Similar information can be obtained about other aspects too from Figure 13.
V.
CONCLUSION
We describe an education and interactive software engineering simulation game to teach important peer code review concepts and best practises. We define 12 learning objectives covering various aspects of peer code review and design our game as well as scoring system to teach the pre-defined learning goals. We evaluate the effectiveness of our approach by conducting experiments involving 17 undergraduate and graduate students. We observe a variety of responses and behaviour across various players. We found that students lack basic knowledge of peer code review and its standard industrial practices. Our experiments reveal that there is a significant improvement in the knowledge of the participants after playing the game. We observe that students find simulation game more engaging and interesting platform for learning.