What Matters to Students – A Rationale Management Case Study in Agile Software Development Mathias Schubanz Claus Lewerentz Software and Systems Research Group Software and Systems Research Group Brandenburg University of Technology Brandenburg University of Technology Cottbus, Germany Cottbus, Germany M.Schubanz@b-tu.de Claus.Lewerentz@b-tu.de Abstract—Documenting design decisions and their rationale so-called design rationale (DR) allow for traceability and (Design Rationale, DR) in software development projects is vital comprehension of typical “why”-questions on the products for supporting the comprehension of the product, product quality, and processes. Furthermore, decision management creates an and future maintenance. Although an increasing number of re- search publications address this topic, systematic approaches and essential opportunity for reflection and learning. supporting DR tools are found very rarely in practice. In software The relevance of DR for software developers was examined engineering education, DR is usually not well covered in teaching. and confirmed several times, for example, by Tang et al. [46]. The lack of suitable decision documentation is mainly an issue in In their study, software engineers confirmed that they capture agile software development. In agile approaches, documentation DR and consider this to be relevant in their work processes. is regarded as less important than working products. To explore possibilities for integrating decision documentation into Scrum Despite the many supporting arguments and the advocates for processes for educational software development projects, we the documentation and use of DR, sustaining and managing conducted a series of eight case studies. These were part of decisions and their rationale is said to be applied seldom in software lab courses in three universities, i.e., BTU Cottbus, PUT practice [2], [6]. Its intangible nature is just one of many Poznan, University of Stuttgart, with about 400 participants in driving factors behind this contradiction. Another is the lack 82 project teams. We introduced additional process elements in Scrum and developed a lightweight capture technique to support of adequate process integration and tool support for handling the decision capture. rationale information [30]. Furthermore, the structured and This paper describes the case study setup and corresponding systematic handling of decisions is only dealt with very implementation and, thus, an example approach of managing selectively in teaching, as described in Kleebaum et al. [23]. rationale in Scrum. Additionally, it presents a data analysis of The explicit management and documentation of decisions the students’ most relevant decisions documented throughout the case studies. We conclude the paper with a discussion on the seem particularly crucial in agile software development (ASD). observations we made during the case study executions and the The very concept of agility promotes frequent independent applicability of the approach in educational software projects. decision making by the single developer or the development Index Terms—rationale management, agile software develop- team. At the same time, ASD deprioritizes documentation in ment, scrum, teaching, case study, decision types, design decision favor of working software (cf. Agile Manifesto [3]). Moreover, ASD promotes efficiency in knowledge transfer I. M OTIVATION through direct informal face-to-face communication [5], [10], During a software development project, the members of the [15], [35]. In addition to the fact that developers, in general, are project team make many decisions. These decisions concern reluctant to document decisions, especially when it is unclear technical issues on all levels of detail as the software archi- which ones are to be documented [1], they understand ASD tecture or the selection of particular implementation platform in particular as the liberation from recording any information technology as well as organizational aspects as the prioritiza- at all [44]. Sometimes developers also tend to refer to source tion of requirements or the use of specific development tools. code as the only real documentation artifact [38], [39], [44]. The overall set of decisions profoundly influences the However, even well-structured and readable code only covers quality properties of the developed product as well as the the ”What?“, not the ”Why?“ and especially not the ”Why efficiency and effectiveness of the development process itself. not?“. Explicit and conscious handling of decisions together with As part of an ongoing research project [42] with a focus their rationale (i.e., decision alternatives, selection criteria, that is now tailored to ASD, we try to empirically analyze the and reasoning) and observed consequences is vital for the described area of conflict with the help of grounded theory [9] comprehension and sustainable maintenance processes for and propose corresponding solutions. In order to do so, we software products [7] [14]. The management of decisions and have observed the work of students and conducted eight case S. Krusche, S. Wagner (Hrsg.): SEUH 2020 17 Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). studies, where they were required to document their most collaboration models and organizational models with the help important decisions actively. These case studies were carried of a tool called Sysiphus, or the Architecture Tradeoff Analysis out within the scope of software engineering lab projects in the Method (ATAM) by Kazman et al. [22] which evaluates an computer science courses of three universities, i.e., Branden- architecture against a set of defined quality goals to derive burg University of Technology Cottbus - Senftenberg (BTU), analyses and rationale. Poznan University of Technology (PUT), and the University Later work focused primarily on overcoming the obstacle of Stuttgart (US). We aimed to use the experience and data of capturing decisions. According to Burge and Brown [7], from the case studies to answer the following three research the problem to be solved is that many possible applications questions (RQ) and at the same time, achieve the following of rationale do not come into effect because the methods learning objective (LO): and tools are not in place to support these opportunities. RQ1 What are suitable ways to introduce decision captur- Approaches that try to mitigate this obstacle include Ishino and ing and reflection in an agile development process? Jin [21] or Myers et al. [36] that integrated decision capture directly into CAD tools and thus into the working environment RQ2 Which types of decisions are crucial in agile educa- of the engineers. The same idea with a closer relationship tional software development projects? to software development was implemented in DecDec tool RQ3 To what extent do students capture decision alterna- by Hesse et al. [17], which is directly integrated into the tives and their rationale? Eclipse IDE [13] and thus into the toolchain of a devel- ................................................................................................ oper. As a documentation tool for decision knowledge, it is LO1 Raise the awareness of students for decision making geared towards collaborative and incremental decision making and rationale management in software engineering. processes. Similarly, Helaba [33] offers a joint workspace The remainder of the paper is structured as follows: Sec- to support communication around design artifacts and activ- tion II discusses related work in the field of rationale man- ities in multidisciplinary teams. There are many other tools agement in general and in an educational context. Section III available to assist in capturing decisions. An overview of the presents the case study setup and details on its execution. area of software architecture documentation in this context is Section IV elaborates on our results with respect to our re- provided, for example, by Tang et al. [45]. search questions and the learning objective. Subsequently, we discuss these findings and some conclusions in Section V. We B. Rationale Management in Agile Software Development conclude by describing potential future work in Section VI. Based on previous work, research presented many con- II. R ELATED W ORK tributions on the topic of documenting decisions and their In this section, we discuss related work. Section II-A rationale in the context of the increasingly popular ASD. elaborates on the topic of rationale management in the context Among other aspects, they present new decision modeling of software engineering. Section II-B focusses on rationale approaches for ASD in general and Scrum in particular. For management in ASD. Finally, Section II-C examines related instance, Waagenaar et al. [50] propose a concrete Scrum work on teaching rationale management. artifact model or Wang et al. [51] elaborate on safety-related documentation by introducing safety-epics and safety-stories. A. Rationale Management in Software Engineering Other papers also focus on identifying best practices for Early research on rationale management dates back to the documentation in ASD. For instance, Hoda et al. [19] present a 1970s and aimed at the use in politics (cf. Kunz and Rit- structured approach to experience-based patterns in agile docu- tel [28]). The ideas were quickly translated into first practical mentation. With five concrete patterns, they strive to enable de- projects. Conklin and Yakemovic [11] reported on the system- velopers to create exactly the right amount of documentation. atic capture and use in various engineering disciplines which Rüping presents another much more comprehensive pattern already took place in the 1980s. A few years later, research approach in his book on agile documentation [40]. Based on in the field of Human-Computer-Interaction (HCI) recognized experience and concrete cases from previous projects, Rüping the relevance of managing rationale, and various contribu- derives documentation patterns tailored to several problems in tions appeared (cf. Fischer et al. [16], Lee and Lai [32], ASD. MacLean et al. [34]). These then gradually served as a basis Researchers also developed ASD-specific tools to capture for research on rationale management in software engineering. rationale. Echo [31], for instance, facilitates agile requirements Much of this research identified the decision making process gathering under consideration of the relationship to decisions as a central point for improvement. Accordingly, many ap- and their rationale. Voigt et al. [49] also proposed sprintDoc, proaches propose models, tools, and methods for documenting which introduces developing documentation artifacts (wiki decisions. To name just a few, these include the Architecture pages). Due to the existing history, changes to issues are Rationalization Method (ARM) by Tang and Han [47] which directly linked to changes in the documentation and thus attaches rationale to software architecture with the help of fine- traceable. Other approaches even employ machine-learning grained model elements, an extendable meta-model facilitating techniques to extract undocumented design decisions (cf. the RUSE model by Wolf [52] which integrates system models, Bhat et al. [4]). S. Krusche, S. Wagner (Hrsg.): SEUH 2020 18 C. Rationale Management in Teaching taken over by graduate students (MA program). Occasionally Despite the many contributions from research addressing the not enough graduate students were available. In these cases, trade-off between agile principles and rationale management, the SM was taken over by the undergraduate students or by there are so far only a few contributions in the area of the study organizers. In the latter case, the study organizers, integrating decision documentation into agile process models. however, did not get involved in the selection of important In practice, developers seem not to have a high awareness of decisions. the importance of DR and DR is yet very rarely included in After an initial setup and exploration phase, the teams software engineering education. The literature describes only worked in two-week (BTU) or four-week (PUT, US) devel- a few approaches. For instance, Kleebaum et al. [23] integrate opment sprints. In total, the case studies included 82 devel- rationale management in the context of a software develop- opment teams with about 350 undergraduate developers and ment project both in the lecture and in practical development 50 graduate students (cf. Table I). In this context, it must be tasks. Lago et al. [29] go a similar way here. In the context of a noted that the graduate students often worked with several software architecture lecture, the topic of software architecture teams simultaneously, for example, as PO for one team and decision making is dealt with and then deepened by a specially as SM for another team. designed card game, called DecidArch. Other approaches TABLE I OVERVIEW OF CONDUCTED CASTE STUDIES . include a card game developed by Schriek et al. [41] who intend to prompt the students to consider design elements University Semester Teams more intensively or De Boer et al. [12] who teach students BTU 2015 Winter 2 how to elicit, communicate, and document architecture design PUT 2016 Summer 27 decisions. BTU 2016 Winter 5 PUT 2017 Summer 31 III. C ASE S TUDY D ESIGN US 2017 Summer 1 In this section, we present the case study design. Initially, BTU 2017 Winter 4 Section III-A gives a general overview of how the case BTU 2018 Winter 6 study was carried out. We then describe the pilot study that BTU 2019 Winter 61 was conducted in advance and its findings (Section III-B). 82 Section III-C explains how we have incorporated these find- ings into the case study setup. Subsequently, Section III-D B. Pilot Case Study describes the data collection procedure. A description of the According to research question RQ1, the main objective was analysis procedure is provided in Section III-E, followed by to integrate the capture and documentation of decisions into elaborations on the validation procedure in Section III-F. a Scrum process. It was one of our main goals to change the textbook version of the Scrum process only as much as A. Case Study Overview necessary because we had to assume that most students in that Between 2015 and 2019, we conducted eight case studies early phase of their studies used the Scrum process for the with computer science students from the Brandenburg Uni- first time. Furthermore, too many additional process elements versity of Technology Cottbus - Senftenberg (BTU), Technical contradict the first agile principle (cf. agile manifesto [3]). University Poznan (PUT), and the University of Stuttgart To meet these requirements, we conducted a pilot case study (US). We integrated the case studies into mandatory software over one semester with two teams at the BTU in advance of development projects as part of the Bachelor’s or Master’s the case studies listed in Table I. Thus, in the first two sprints, study programs. we conducted workshops together with the students. During The tasks of small student teams were to develop com- these workshops, we discussed issues being significant to the prehensive software products during the lecturing period of students, elicited alternatives, made decisions, and documented one semester, i.e., about 15 weeks. The products were mostly them. interactive web or desktop applications, e.g., online versions Based on the experiences from the interaction among the of popular board games. The expected work effort for a project students, the interaction between the students and the teaching was between 500 h (PUT) and 1.000 h (BTU, US), thus 100- staff as well as the dynamics of the decision making, we 200 h per person. identified the need for the following extensions to the Scrum In addition to the development task, the students attended process, called Extension Requirements (ER) in the following: complimentary lectures that discussed software engineering ER1 The team has to have a clear vision of who will ex- techniques. The lectures did neither specifically address deci- ecute tasks related to the case study. Thus, one team sion making techniques nor the topic of rationale management. member is responsible for recording the decisions. All projects used the Scrum process, which served as the ER2 The case study setup has to embed this responsibility common agile organizational structure for the case study. The in clearly defined process elements that provide a typical project team consisted of four to five undergraduate context and time frame within the sprint. students (BA program) working as developers. The roles of Product Owner (PO) and Scrum Master (SM) were typically 1 At the time of publication, the case study was still ongoing. S. Krusche, S. Wagner (Hrsg.): SEUH 2020 19 ER3 The quality of recorded decisions needs to be assured Backlog Daily Scrum through corresponding measures to enable rationale Refinement reuse, as discussed by Thurimella et al. [48]. ER4 Capturing decisions must be regulated in a way that Review certain documentation templates and tools are to be Scrum Master used. Additionally, the case that a team might not agree on a set of decisions to capture has to be Retrospective Sprint Planning addressed as well. Decision Scrum Master Decision Review Capture C. Case Study Setup To address the ERs listed above and enable a systematic ap- Fig. 1. Altered Sprint Procedure. proach to capture decisions, we decided to extend the textbook Scrum by two process elements, as shown in Figure 1. were converted into images using screenshots and uploaded Decision Capture At the end of each sprint planning to the students’ project repositories. (ER2) the Scrum Master (ER1) is responsible for This approach showed insufficient usability and transferred identifying and documenting the three most relevant too much responsibility to the case study organizers. Addition- decisions for the current sprint together with the ally, the generated mindmaps did not show optimal readability. team. Once recorded, these have to be uploaded to Subsequent changes due to, for instance, changed requirements the project repository. or feedback could not be incorporated. We needed to either recreate the documentation entirely or save the raw data along Decision Review At the end of the sprint, during the retro- with the results to regenerate the mindmap. We also found spective (ER1), the Scrum Master (ER2) is responsi- that students rarely documented their decisions when none of ble for reviewing the documented decisions together the case study organizers were attending the sprint planning. with the team (ER3). In case of changes, incon- All of these issues were reasons for the authors to refine the sistencies, or recently emerged important decisions, capture technique for the planned case studies. the Scrum Master has to revise the documentation. (ER3). Autonomy in Decision Capture: With the goal of more active engagement of the students, In addition to these extensions, a case study guideline was the authors decided to give the students more responsibility provided to students. It contained additional instructions on and thus to increase the autonomy in the documentation of how to carry out the identification and selection of the most decisions. It was necessary to choose another handling of the relevant decisions if there is no immediate consent (ER4). templates. For the first three case studies in 2015 (BTU) and The guidelines also contained the requirements for the use of 2016 (BTU, PUT), Excel templates were compiled based on documentation templates and the corresponding tools (ER4). the previous QOC-based [34] modeling and distributed to the For more information, please refer to Section III-D. students. This way, the responsibility lay solely with the team, Furthermore, a git [8] repository was made available to the respectively, the Scrum Master, as envisioned in the case study students, which provided the documentation templates (ER4), setup in Section III-C. Additionally, sprint planning meetings as well as an extensive collection of sample documentation no longer had to be accompanied by the case study organizers (ER4). However, there were no instructions on what types of for decision documentation. This change required the students decisions should be documented in order not to restrict them to manage decisions consciously and independently and not in their thinking and decision making. to let this happen as a passive process triggered by a third D. Data Collection – From Mindmaps to Markdown Records party. According to the case study setup (cf. Section III-C) the Scrum Master was responsible for uploading the completed The data collection process varied throughout the different documentation to the project repository at the end of the sprint. parts of the case studies. Apart from the pilot study, it was carried out by the students over the entire period. The MADR – Markdown Architecture Decision Records: capture technique, as well as the associated tools and templates Based on the feedback and experience of the 2016 case studies for recording decisions, were repeatedly examined for their as well as a simultaneously conducted industry case study, we usability and improved throughout the pilot case study and refined the handling of templates in the following case studies. the case studies. A crucial point of criticism was the manageability. Working Initially, during the pilot case study, we have chosen with Excel files proved to be impractical and cumbersome. lightweight tool support, i.e., Text2Mindmap [20], to record Thus, it was perceived as too time-consuming to perform decisions. For this purpose, we used the Questions Options reviews of the captured decisions by the team or revise the Criteria (QOC) model [34] model and mapped it using Text- documented decisions. Accordingly, the case study organizers ToMindmap. By this means, the case study organizers captured sought alternatives in dialogue with the students and the the decisions made during the architecture workshops of the industry partner. The choice fell on the use of Markdown (MD) pilot study. After the workshops, the generated mind maps files. MD is a lightweight markup syntax, which makes it easy S. Krusche, S. Wagner (Hrsg.): SEUH 2020 20 to build documents [37]. Since both students and industry answers to be inappropriate. Furthermore, we tried to reduce partners version their development projects with git [8] and the individual influence of the authors during data analysis by its management frontend GitLab [18], the use of MD also coding the data separately by each author. Only afterward, offered direct integration into the tool environment, since the results were compared and standardized to a uniform GitLab renders MD files in the web interface nicely. categorization. Besides, the management of MD files in git, combined with Concerning threats to external validity, one needs to be a corresponding branching strategy, enables collaborative work aware that the software development labs and their associated with short feedback cycles. Since we already used git in the lectures had an impact (as discussed in Section V). It was case study, it seemed reasonable to make the MD templates hardly possible to mitigate this influence. The limited project available in a public GitHub repository (cf. [43]). duration of only a few weeks and the steep learning curve also The revised methodology, including the new templates, affected the external validity. Nevertheless, the free choice of was supposed to be implemented in the first case study of development topics coupled with the high number of partici- 2017 (PUT), but could not be implemented for organizational pating groups and the resulting large amount of documented reasons. The recording using Excel templates was applied decisions should be favorable to external validity. once again. During the second case study in 2017 with the University of Stuttgart (US), the students used the new IV. R ESULT E VALUATION methodology. Recording vital decisions and their rationale In this section, we try to give answers to the research with the help of MD files attracted such interest that our questions and the learning objective outlined in the Motivation partners from the University of Stuttgart have forked and section based on the results of the case studies. As a basis for further refined the original GitHub project. The result is a new this discussion, we surveyed the students after completing the open-source (OS) project that was not only used in the second case study. The survey comprised three predefined questions case study in 2017, which is an OS project itself (cf. [26]) (survey questions SQ1 to SQ3) and one open question for but has reached into the OS community. Currently, more than individual feedback. Participation was voluntary. The survey ten completely independent OSS projects employ Markdown resulted in a response of 56 answers. Considering all of the Architecture Decision Records (MADR) [25] to capture their students who already completed the case study by the time of decisions. Besides, other developers from the community have publication, the response rate is about 18%. worked on the MADR GitHub project (cf. [24]). Based on the results of this follow-up survey, Section IV-A Subsequently, we used MADR for the following case studies elaborates on the achievement of learning objective LO1. in 2017, 2018, and 2019 (BTU). Thanks, among other things, Subsequently, Section IV-B answers research question RQ1 to the comprehensive instructions, there was hardly any neg- as far as possible and reports on the students’ perspective ative feedback from the students. in which Scrum phase decisions have to be documented. In Section IV-C, we then report on our findings on the types E. Data Analysis – Coding and Classification of decisions captured by the students and thus try to answer To answer research question RQ2, we searched the literature RQ2. Section IV-D concludes with a discussion on the scope for a comprehensive classification of decision types, e.g., and nature of the information collected for a decision, i.e., Kruchten [27]. However, in our opinion, the classifications research question RQ3. found did not provide structures clear enough to be intuitive and intelligible for a data analysis like this. Accordingly, we A. LO1 – Raise the Awareness for Rationale Management used an inductive approach and coded the data. Each of the To check the achievement of our learning goal, we asked the authors did this individually. Subsequently, we tried to derive students to rate the usefulness of the conducted case study. The a consistent categorization from the codes, which resulted in question had a dedicated focus on the communication within several discussions and involved several iterations. Discrepan- the team, respectively, with stakeholders (survey question SQ1, cies in the two result sets had to be identified and also resolved cf. Figure 2): As can be seen, it turned out that roughly equal on a case-by-case basis. After further iterations, we refined parts of the students strongly agree or agree on the usefulness the codes into a simple and straightforward classification. The of decision documentation (∼ 37%), respectively disagree or results can be found in Section IV-C. disagree strongly (∼ 43%). F. Validitation Procedures We have taken various measures to address potential threats to validity. Concerning internal validity, for example, we did not get involved in the selection of decisions to be documented. The authors explicitly pointed out to the students that the case Fig. 2. Results for SQ1: usefulness of documented decisions, in percent. study will not affect the evaluation of the lecture. Another aspect of internal validity was the integration of free-text Besides, we asked students whether they reused documented answers into the follow-up survey. This way, students could decisions (survey question SQ2, cf. Figure 3) and if so, for provide individual answers if they considered the predefined what purpose. Here about 38% denied, and a further 14% S. Krusche, S. Wagner (Hrsg.): SEUH 2020 21 replied that they used the decisions merely as a template for recording further decisions. The remaining 48% of the students opposed this by stating an explicit use case. These included Fig. 3. Results for SQ2: reuse of captured decisions, in percent. the following scenarios: • A review was conducted. Fig. 4. Results for SQ3: phases to capture decision in, in percent. • A wrong decision was made and had to be reconsidered. • A bug had to be fixed. • A similar problem had to be decided on. points throughout the sprint. Reasons given by the students • One team member needed advice. include the fact that it was just forgotten and was made up • The requirements had changed. for at the end of the sprint. Sometimes it also happened that the vital decisions only emerged in the course of the sprint. The above answers indicate that the participating students Correspondingly, the integration of the decision capture into have become aware of the topic and its importance. Several sprint planning is a possible and preferred option, although it students also confirmed this impression in an open question is not the only feasible option for process integration. on feedback on the case study. For example, one student Still, the authors, like the students, consider the integration wrote: “captured decisions help you to understand the problem of the decision capture into the sprint planning to be the thoroughly and [. . . ] have a better picture of the project and preferred way, since working with the recorded and thus its structure so that you can develop better software.“ communicated decisions during the sprint is thereby made B. RQ1 – Suitability of the Chosen Approach possible. As described earlier, building on initial ideas and the C. RQ2 – What Matters to Students? results of the pilot case study, we decided on an approach to Based on the classification derived from the coding, Table II integrate rationale management into Scrum (cf. Section III). lists 702 currently recorded decisions grouped by the corre- Accordingly, for research question RQ1, we can only discuss sponding categories. Decisions that we could not attribute to the applicability of our specific approach. As the answers to any of the available classification types are listed under N/A. questions SQ1 and SQ2 show, there is no uniform opinion With about 2% and 12 decisions respectively, this includes among the participating students. However, a considerable decisions that were not clear enough as well as decisions in share of the students considers the decision capture approach which the students deliberately wrote nonsense. Accordingly, to be helpful and applicable. In this context, it was vital a successful mapping of more than 98% of available decisions for us to find out how the participants in the case study to the categorization appears sufficiently accurate for us and assessed the timing of the decision capture. Accordingly, we confirms the appropriateness of the chosen classification. have asked students to indicate phases in which they consider TABLE II documentation of decisions to be most appropriate (survey AGGREGATED NUMBER OF CAPTURED DECISIONS PER TYPE . question SQ3, cf. Figure 4). Several answers could be given. Again, there is no uniform opinion on the part of the Decision type Count Percent students. However, a clear majority (59%) voted for capturing Definition / refinement of features / requirements 133 19 decisions during the sprint planning. The second most votes Development libraries / frameworks 100 14.3 came with 43% and 39% for review and retrospective, the Software architecture 97 13.8 phases at the end of a sprint. Still, just over a quarter of Development process 89 12.7 the respondents (27%) were in favor of capturing decisions Software quality measures (testing, etc.) 67 9.5 Prioritization of tasks / features 66 9.4 while working on the backlog. The least preferred approach is Development tools 62 8.8 decision capture during planning poker (21%). We explicitly Development platform 49 7 neglect that 23% of the students who argue for capturing de- Deployment (Other in Fig. 5) 10 1.4 cisions during the Daily Scrum because it would conceptually Communication within the team (Other in Fig. 5) 10 1.4 contradict the Daily Scrum. To-Do decisions (Other in Fig. 5) 7 1 The three most frequently mentioned responses also reflect N/A (Other in Fig. 5) 12 1.7 our observations during the execution of the case studies. 702 Although the case study setup clearly states that the most important decisions should be captured during sprint planning, The resulting distribution of decision types within all cap- the students occasionally documented them at various other tured decisions shows that there is not a single relevant S. Krusche, S. Wagner (Hrsg.): SEUH 2020 22 These two together account for a little less than 20% in the first sprint, while it is about 40% and more from the fourth sprint onwards. Fig. 6. Aggregated number of captured decisions per sprint, absolute. Fig. 5. Distribution of decision types over all case studies.2 For reading the graph, it is necessary to know that the projects have lasted a maximum of seven sprints. The de- decision type, but rather a group of important types. As can velopment of the open-source project, to which the students be seen in Figure 5, the five most frequently documented of the University of Stuttgart contributed (cf. Section III-D), decision types account for more than two-thirds of all captured was continued afterward and resulted in further documented decisions (∼ 69%). Combining the seven most documented decisions. These are listed in the ”Subsequent Dev.” column. decision types even accounts for nearly 88% of all decisions. Furthermore, it is necessary to read the graph in the context On closer inspection of these seven, one can see that the most of the total number of captured decisions per sprint. As one frequently documented decision type ”Definition / refinement can see from Figure 6, this has changed permanently. Although of features / requirements“ is closely related to the ”Prioritiza- the students were required to write down the three most tion of tasks / features.“ Topics related to requirements analysis important decisions, they sometimes recorded less or none at and its prioritization together represent a considerable part all. In individual cases, they even recorded more. In the first of all documented decisions with roughly 29%. A second sprint, the number of documented decisions was still very high thematic group also stands out on closer inspection, solving (avg = 2.7 decisions), but decreased afterward. During the implementation problems relating to software architecture and third sprint, the students recorded significantly less (avg = 1.7 those relating to the use of frameworks and libraries. Fre- decisions). As the development projects at the PUT lasted only quently, they go hand in hand anyway. These two together three sprints (cf. Section III), from the fourth sprint on only the account for about 28% of all decisions and thus have a teams of the US and the BTU documented their decisions (24 considerable influence on software development processes. teams in total). In the following, the amount of documented Even more striking is the evolution of documented de- decisions is somewhat more stable again. cisions over time. In Figure 7, we have broken down the It should be noted that for sprint five and six, there is no different types of decisions by the sprints from which they data from the 2019 case study of the BTU available at the originate. Upon mere observation, considerable differences time of publication. Accordingly, only 18 teams documented in the distribution of the types of documented decisions are in sprints five and six. In particular, this means that in the noticeable. In the first sprints, for example, one can find fourth and fifth sprint, an average of about 2.3 decisions per an above-average number of decisions on the development team was documented, and in the sixth sprint, an average of tools, the libraries / frameworks used, and especially the de- 2.1 decisions per team was documented. For the seventh sprint velopment platform. These become considerably less over (4 teams) as well as the subsequent development (1 team), the time. The decisions addressing the development platform, number of documented decisions increased again. However, which in the beginning make up almost 20%, then disappear each of these two, with 12 decisions each, accounted for just almost completely. Equally noteworthy is the progress of under two percent of the decisions recorded and thus have decisions on software quality. They play a subordinate role little significance. at the beginning and then gain considerably in importance In summary, the trend analysis also provides interesting throughout the sprints. Especially in the third sprint, the topic insights for answering the research question RQ2. However, of software quality plays a vital role, with a share of 25% in there is no simple answer to this question. Besides, the all documented decisions. Subsequently, relevance decreases character of the projects, in which the case studies took place, again. Just as important is the development of the decisions on features / requirements as well as on the software architecture. 2 All decision types with less than 2% share are subsumed under Other. S. Krusche, S. Wagner (Hrsg.): SEUH 2020 23 Fig. 7. Relative distribution of decision types over all case studies per sprint. influences the results too. All projects started from scratch and did have a rather brief lifespan. D. RQ3 – What Information has been Documented? Another aspect that has been of interest to us while exercis- ing the case studies is the quantity of information students cap- ture for a decision. Specifically, we are interested in whether students only document the issue itself and the decision to solve or whether they also record additional information such as the alternatives considered. Also, this information, for example, can be accompanied by further rationales. TABLE III D ECISIONS CLASSIFIED BY THE EXTENT OF DOCUMENTED INFORMATION . Fig. 8. Distribution of decisions classified by the amount of avail. information. Given answer type Count Percent Alternatives & Rationale 273 38.9 all cases (∼ 76%), students recorded two or more solution Multiple alternatives 262 37.3 alternatives for the documented decision. Additional rationales One alternative 123 17.5 on the selected solution alternative and the reasoning behind No alternative (Other in Fig. 8) 20 2.9 were recorded in about 39%. This number includes cases with Rationale only (Other in Fig. 8) 9 1.3 simple explanations formulated in one sentence as well as Explicit exclusion (Other in Fig. 8) 1 0.1 N/A (Other in Fig. 8) 14 2 statements in which the rationale behind the decision was 702 broken down into, e.g., positive and negative decision criteria. To answer research question RQ3, we went through all V. D ISCUSSION documented decisions again and categorized them according The results presented in this paper suggest that the capture to the degree to which the students completed the templates. of decisions and their rationale is an issue that should have Content-related considerations did not play a role. its place in software engineering education. Although the In analogy to the left column from Table III, we dis- opinions from the follow-up surveys are not uniformly positive tinguished between the simple naming of the problem, the with the students, the early stage of their studies is affecting documentation of the selected solution alternative, the naming their judgment. Some students lack the experience that would of none, one, or more alternatives. We also classified the change the assessment of such an approach. We observed that provision of further rationale according to the template. the more advanced the students were in their studies, the more The result was that about 94% of all cases, at least one positive was the feedback on the setup of our case study. considered solution alternative was specified for the docu- Furthermore, according to the opinion of the authors, the mented decisions (cf. Figure 8). In nearly three-quarters of concrete context in which the respective case study took place S. Krusche, S. Wagner (Hrsg.): SEUH 2020 24 influenced on the quantity of the documented decisions, the making and rationale management in software engineering. amount of information recorded per decision as well as the Moreover, the management of decisions and their rationale, as type of documented decisions. For the authors, this impression already mentioned in related works, can certainly be seen as is based on, for example, the complementary lectures that a controversial issue. implicitly and unconsciously emphasize certain software en- gineering aspects. A repeatedly increased number of captured VI. F UTURE W ORK decisions on the topic of software quality aspects in the same Upon completion of the current case study, it is necessary sprints support this observation. to enter the new data and update the evaluation. Building on In the following, we briefly discuss the results of the this, we see the necessity to integrate the lessons learned from research questions and the achievement of the learning goal: the case studies into teaching in a more structured way. To RQ1 – Suitable Ways to Introduce Rationale Management: this end, it is also necessary to increase the response rate The follow-up surveys with the participating students showed to the follow-up surveys carried out subsequently with the that the case study setup provides a viable way to integrate participating students. In this way, we can incorporate more the capture and management of decisions and their rationale and more detailed feedback into further rationale management into an agile process. The selected implementation also offers applications. Another aspect is addressing the topic of software sufficient possibilities for adaptation to one’s own needs. architecture decision making in the context of the supplemen- Feedback and experience have also shown, however, that it tary lectures to the software development project, as done by, is not the only viable way to capture decisions. Adjustments e.g., Lago et al. [29]. are easy to implement, especially in the phases to which the The logical next step is to generalize the approach used here decision recording is coupled. for the Scrum process in teaching. Thereby the experiences of the case studies, as well as the feedback of the students, play RQ2 – Crucial Types of Decisions: a crucial role. The goal is to make the approach applicable to For research question RQ2, it has to be stressed that there is other scenarios, if not to other agile process models. no simple answer. A simple analysis of the distribution of all Another strategy for future work is to conduct a similar documented decisions would be very short-sighted. A closer case study setup in an industrial context. It has been done in a look at the available data reveals that the phase of development single case so far, however, for comparability with industrial has a significant influence on which decisions are essential projects, considerably more data needs to be collected. An- for students. In the specific case of the case studies, the other approach is to compare the findings with applications organizational structure of the software development projects in the open-source community. To this end, it would be also has a notable influence here. Almost all projects started necessary to find projects that systematically record decisions from scratch and only lasted for a limited time. Accordingly, and make them accessible. The decision data could then be the students needed to make technical decisions that would not compared with the data from the industry and from the case occur at such frequency in projects running for a prolonged studies with the students. However, comparability from the time. The credit points awarded in the respective courses are perspective of the applied process model is questionable here. a limiting factor here. We hypothesize that only the type of essential decisions will All in all, it can be emphasized that two major topics be comparable. account for a large proportion of the documented decisions. One relates to requirements analysis, features, their refinement ACKNOWLEDGMENT as well as the sequence of realization. Decisions on how The authors gratefully acknowledge the fruitful cooperation to implement these features, how are they mapped to the with our colleagues from PUT Poznan and the University software architecture, which libraries and frameworks are to of Stuttgart, namely Miroslaw Ochodek, Bartosz Walter, and be employed, define the other one. Oliver Kopp. Furthermore, we thank all students for participat- RQ3 – What Information do Students Capture?: With ing in the case studies and taking some extra time and effort research question RQ3, the authors wanted to find out to to support us. Without all of them, these case studies would what extent students document a decision. It turned out that not have been possible. they document more than expected. In the pilot case study, R EFERENCES students sometimes only did what the guidelines specified. Throughout the case studies, however, the students provided [1] A LEXEEVA , Z., P EREZ -PALACIN , D., AND M IRANDOLA , R. Design several considered alternatives for a decision in the majority of Decision Documentation: A Literature Overview. In European Confer- ence on Software Architecture (2016), Springer, pp. 84–101. the cases. More than that, the students added further rationale [2] BASS , L., C LEMENTS , P., AND K AZMAN , R. Software Architecture in for the decisions made in more than a third of the cases. Practice, 2 ed. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 2003. LO1 – Raise the Awareness for Rationale Management: [3] B ECK , K., B EEDLE , M., VAN B ENNEKUM , A., C OCKBURN , A., C UN - In the follow-up surveys conducted with the participating NIGHAM , W., F OWLER , M., H IGHSMITH , J., H UNT, A., J EFFRIES , R., K ERN , J., M ARICK , B., M ARTIN , R. C., S CHWABER , K., S UTHER - students, it has become apparent that a considerable number LAND , J., AND T HOMAS , D. Manifesto for Agile Software Develop- of students have become aware of the topic of decision ment. http://agilemanifesto.org/, February 2001. S. Krusche, S. Wagner (Hrsg.): SEUH 2020 25 [4] B HAT, M., S HUMAIEV, K., B IESDORF, A., H OHENSTEIN , U., AND [29] L AGO , P., C AI , J. F., DE B OER , R. C., K RUCHTEN , P., AND V ERDEC - M ATTHES , F. Automatic Extraction of Design Decisions From Issue CHIA , R. Decidarch: Playing Cards as Software Architects. In Management Systems: A Machine Learning Based Approach. In Eu- Proceedings of the 52nd Hawaii International Conference on System ropean Conference on Software Architecture (2017), Springer, pp. 138– Sciences (2019). 154. [30] L AT OZA , T. D., AND M YERS , B. A. Hard-To-Answer Questions About [5] B RIAND , L. C. Software Documentation: How Much is Enough? In Code. In Evaluation and Usability of Programming Languages and Proceedings of the 7th European Conference on Software Maintenance Tools (2010), ACM, p. 8. and Reengineering (2003), IEEE, pp. 13–15. [31] L EE , C., G UADAGNO , L., AND J IA , X. An Agile Approach to Capturing [6] B URGE , J. E. Design Rationale: Researching Under Uncertainty. Arti- Requirements and Traceability. In Proceedings of the 2nd International ficial Intelligence for Engineering Design, Analysis and Manufacturing Workshop on Traceability in Emerging Forms of Software Engineering 22, 4 (2008), pp. 311–324. (TEFSE) (2003), vol. 20. [7] B URGE , J. E., AND B ROWN , D. C. Software Engineering Using [32] L EE , J., AND L AI , K.-Y. What’s in Design Rationale? Hum.-Comput. RATionale. Journal of Systems and Software 81, 3 (2008), 395–413. Interact. 6 (September 1991), 251–280. [8] C HACON , S., AND S TRAUB , B. Pro Git. Apress, 2014. [33] L OPEZ , M. G., H AESEN , M., L UYTEN , K., AND C ONINX , K. Helaba: [9] C HARMAZ , K., AND B ELGRAVE , L. L. Grounded Theory. In The A System to Highlight Design Rationale in Collaborative Design Pro- Blackwell Encyclopedia of Sociology. American Cancer Society, 2015. cesses. In Cooperative Design, Visualization, and Engineering. Springer, [10] C LEAR , T. Documentation and Agile Methods: Striking a Balance. 2015, pp. 175–184. SIGCSE Bull. 35, 2 (June 2003), 12–13. [34] M AC L EAN , A., YOUNG , R. M., B ELLOTTI , V. M. E., AND M ORAN , [11] C ONKLIN , E. J., AND YAKEMOVIC , K. C. B. A Process-Oriented T. P. Questions, Options, and Criteria: Elements of Design Space Approach to Design Rationale. Human–Computer Interaction 6 (Sept. Analysis. Hum.-Comput. Interact. 6 (September 1991), 201–250. 1991), 357–391. [35] M ELNIK , G., AND M AURER , F. Direct Verbal Communication as a [12] D E B OER , R. C., FARENHORST, R., AND VAN V LIET, H. A Commu- Catalyst of Agile Knowledge Sharing. In Agile Development Conference nity of Learners Approach to Software Architecture Education. In 22nd (2004), IEEE, pp. 21–31. Conference on Software Engineering Education and Training (2009), [36] M YERS , K. L., Z UMEL , N. B., AND G ARCIA , P. Automated Capture IEEE, pp. 190–197. of Rationale for the Detailed Design Process. In AAAI/IAAI (1999), [13] D ES R IVI ÈRES , J., AND W IEGAND , J. Eclipse: A Platform for In- pp. 876–883. tegrating Development Tools. IBM Systems Journal 43 (April 2004), [37] OVADIA , S. Markdown for Librarians and Academics. Behavioral & 371–383. Social Sciences Librarian 33, 2 (2014), 120–124. [14] D UTOIT, A. H., M C C ALL , R., M ISTR ÍK , I., AND PAECH , B. Rationale [38] R EEVES , J. W. What is Software Design. C++ Journal 2, 2 (1992), Management in Software Engineering: Concepts and Techniques. In 14–12. Rationale Management in Software Engineering. Springer, 2006, pp. 1– [39] RUBIN , E., AND RUBIN , H. Supporting Agile Software Development 48. Through Active Documentation. Requirements Engineering 16, 2 (2011), [15] DYB Å , T., AND D INGSØYR , T. Empirical Studies of Agile Software De- 117–132. velopment: A Systematic Review. Information and software technology [40] R ÜPING , A. Agile Documentation: a Pattern Guide to Producing 50, 9-10 (2008), 833–859. Lightweight Documents for Software Projects. John Wiley & Sons, 2005. [16] F ISCHER , G., L EMKE , A. C., M C C ALL , R., AND M ORCH , A. I. Mak- [41] S CHRIEK , C., VAN DER W ERF, J.-M. E., TANG , A., AND B EX , F. ing Argumentation Serve Design. Hum.-Comput. Interact. 6 (September Software Architecture Design Reasoning: A Card Game to Help Novice 1991), 393–419. Designers. In European Conference on Software Architecture (2016), [17] H ESSE , T.-M., K UEHLWEIN , A., AND ROEHM , T. DecDoc: A Tool for Springer, pp. 22–38. Documenting Design Decisions Collaboratively and Incrementally. In [42] S CHUBANZ , M. Design Rationale Capture in Software Architecture: 1st International Workshop on Decision Making in Software ARCHitec- What has to be Captured? In Proceedings of the 19th International ture (MARCH) (2016), IEEE, pp. 30–37. Doctoral Symposium on Components and Architecture (2014), ACM, [18] H ETHEY, J. M. GitLab Repository Management. Packt Publishing Ltd, pp. 31–36. 2013. [43] S CHUBANZ , M. Making Decisions Sustainable in [19] H ODA , R., N OBLE , J., AND M ARSHALL , S. How Much is Just Enough? Agile Software Development. GitHub, March 2017. Some Documentation Patterns on Agile Projects. EuroPLoP2010; 15th https://github.com/schubmat/DecisionCapture/. European Pattern Languages of Programs (2010). [44] S ELIC , B. Agile Documentation, Anyone? IEEE Software 26, 6 (2009), [20] I ONA , J. Text 2 Mind Map. The School Librarian 65, 3 (2017), 150. 11–12. [21] I SHINO , Y., AND J IN , Y. An Information Value Based Approach to [45] TANG , A., AVGERIOU , P., JANSEN , A., C APILLA , R., AND BABAR , Design Procedure Capture. Advanced Engineering Informatics 20, 1 M. A. A Comparative Study of Architecture Knowledge Management (2006), 89–107. Tools. Journal of Systems and Software 83, 3 (2010), 352–370. [22] K AZMAN , R., K LEIN , M., AND C LEMENTS , P. ATAM: Method [46] TANG , A., BABAR , M. A., G ORTON , I., AND H AN , J. A Survey of for Architecture Evaluation. Tech. rep., CMU / Software Engineering Architecture Design Rationale. Journal of Systems and Software 79, 12 Institute, 2000. (2006), 1792–1804. [23] K LEEBAUM , A., J OHANSSEN , J. O., PAECH , B., AND B RUEGGE , [47] TANG , A., AND H AN , J. Architecture Rationalization: A Methodology B. Teaching Rationale Management in Agile Project Courses. In for Architecture Verifiability, Traceability and Completeness. In ECBS Tagungsband des 16. Workshops ”Software Engineering im Unterricht (2005), pp. 135–144. der Hochschulen” (2019). [48] T HURIMELLA , A., S CHUBANZ , M., P LEUSS , A., AND B OTTERWECK , [24] KOPP, O. Markdown Architectural Decision Records. GitHub, Mar. G. Guidelines for Managing Requirements Rationales. Software, IEEE 2017. https://adr.github.io/madr/. 34, 1 (2017), 82 – 90. [25] KOPP, O., A RMBRUSTER , A., AND Z IMMERMANN , O. Markdown [49] VOIGT, S., H ÜTTEMANN , D., G OHR , A., AND G ROSSE , M. Agile Architectural Decision Records: Format and Tool Support. In ZEUS Documentation Tool Concept. In Developments and Advances in (2018), pp. 55–62. Intelligent Systems and Applications (Cham, 2018), Á. Rocha and L. P. Reis, Eds., Springer International Publishing, pp. 67–79. [26] KOPP, O., B INZ , T., B REITENB ÜCHER , U., AND L EYMANN , F. [50] WAGENAAR , G., H ELMS , R., DAMIAN , D., AND B RINKKEMPER , S. Winery– A Modeling Tool for TOSCA-Based Cloud Applications. Artefacts in Agile Software Development. In International Conference In International Conference on Service-Oriented Computing (2013), on Product-Focused Software Process Improvement (2015), Springer, Springer, pp. 700–704. pp. 133–148. [27] K RUCHTEN , P. An Ontology of Architectural Design Decisions in [51] WANG , Y., B OGICEVIC , I., AND WAGNER , S. A Study of Safety Software Intensive Systems. In 2nd Groningen Workshop on Software Documentation in a Scrum Development Process. In Proceedings of Variability (2004), pp. 54–61. the XP2017 Scientific Workshops (2017), ACM, pp. 22:1–22:5. [28] K UNZ , W., AND R ITTEL , H. W. J. Issues as Elements of Information [52] W OLF, T. Rationale-Based Unified Software Engineering Model. VDM Systems. Tech. rep., Systemforschung, Heidelberg, Germany Science Verlag, Saarbrücken, Germany, 2008. Design, University of California, Berkeley, 1970. S. Krusche, S. Wagner (Hrsg.): SEUH 2020 26