=Paper=
{{Paper
|id=Vol-2531/paper04
|storemode=property
|title=What Matters to Students - A Rationale Management Case Study in Agile Software Development
|pdfUrl=https://ceur-ws.org/Vol-2531/paper04.pdf
|volume=Vol-2531
|authors=Mathias Schubanz,Claus Lewerentz
|dblpUrl=https://dblp.org/rec/conf/seuh/SchubanzL20
}}
==What Matters to Students - A Rationale Management Case Study in Agile Software Development==
https://ceur-ws.org/Vol-2531/paper04.pdf
What Matters to Students –
A Rationale Management Case Study
in Agile Software Development
Mathias Schubanz Claus Lewerentz
Software and Systems Research Group Software and Systems Research Group
Brandenburg University of Technology Brandenburg University of Technology
Cottbus, Germany Cottbus, Germany
M.Schubanz@b-tu.de Claus.Lewerentz@b-tu.de
Abstract—Documenting design decisions and their rationale so-called design rationale (DR) allow for traceability and
(Design Rationale, DR) in software development projects is vital comprehension of typical “why”-questions on the products
for supporting the comprehension of the product, product quality, and processes. Furthermore, decision management creates an
and future maintenance. Although an increasing number of re-
search publications address this topic, systematic approaches and essential opportunity for reflection and learning.
supporting DR tools are found very rarely in practice. In software The relevance of DR for software developers was examined
engineering education, DR is usually not well covered in teaching. and confirmed several times, for example, by Tang et al. [46].
The lack of suitable decision documentation is mainly an issue in In their study, software engineers confirmed that they capture
agile software development. In agile approaches, documentation DR and consider this to be relevant in their work processes.
is regarded as less important than working products. To explore
possibilities for integrating decision documentation into Scrum Despite the many supporting arguments and the advocates for
processes for educational software development projects, we the documentation and use of DR, sustaining and managing
conducted a series of eight case studies. These were part of decisions and their rationale is said to be applied seldom in
software lab courses in three universities, i.e., BTU Cottbus, PUT practice [2], [6]. Its intangible nature is just one of many
Poznan, University of Stuttgart, with about 400 participants in driving factors behind this contradiction. Another is the lack
82 project teams. We introduced additional process elements in
Scrum and developed a lightweight capture technique to support of adequate process integration and tool support for handling
the decision capture. rationale information [30]. Furthermore, the structured and
This paper describes the case study setup and corresponding systematic handling of decisions is only dealt with very
implementation and, thus, an example approach of managing selectively in teaching, as described in Kleebaum et al. [23].
rationale in Scrum. Additionally, it presents a data analysis of The explicit management and documentation of decisions
the students’ most relevant decisions documented throughout the
case studies. We conclude the paper with a discussion on the seem particularly crucial in agile software development (ASD).
observations we made during the case study executions and the The very concept of agility promotes frequent independent
applicability of the approach in educational software projects. decision making by the single developer or the development
Index Terms—rationale management, agile software develop- team. At the same time, ASD deprioritizes documentation in
ment, scrum, teaching, case study, decision types, design decision favor of working software (cf. Agile Manifesto [3]).
Moreover, ASD promotes efficiency in knowledge transfer
I. M OTIVATION through direct informal face-to-face communication [5], [10],
During a software development project, the members of the [15], [35]. In addition to the fact that developers, in general, are
project team make many decisions. These decisions concern reluctant to document decisions, especially when it is unclear
technical issues on all levels of detail as the software archi- which ones are to be documented [1], they understand ASD
tecture or the selection of particular implementation platform in particular as the liberation from recording any information
technology as well as organizational aspects as the prioritiza- at all [44]. Sometimes developers also tend to refer to source
tion of requirements or the use of specific development tools. code as the only real documentation artifact [38], [39], [44].
The overall set of decisions profoundly influences the However, even well-structured and readable code only covers
quality properties of the developed product as well as the the ”What?“, not the ”Why?“ and especially not the ”Why
efficiency and effectiveness of the development process itself. not?“.
Explicit and conscious handling of decisions together with As part of an ongoing research project [42] with a focus
their rationale (i.e., decision alternatives, selection criteria, that is now tailored to ASD, we try to empirically analyze the
and reasoning) and observed consequences is vital for the described area of conflict with the help of grounded theory [9]
comprehension and sustainable maintenance processes for and propose corresponding solutions. In order to do so, we
software products [7] [14]. The management of decisions and have observed the work of students and conducted eight case
S. Krusche, S. Wagner (Hrsg.): SEUH 2020 17
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
studies, where they were required to document their most collaboration models and organizational models with the help
important decisions actively. These case studies were carried of a tool called Sysiphus, or the Architecture Tradeoff Analysis
out within the scope of software engineering lab projects in the Method (ATAM) by Kazman et al. [22] which evaluates an
computer science courses of three universities, i.e., Branden- architecture against a set of defined quality goals to derive
burg University of Technology Cottbus - Senftenberg (BTU), analyses and rationale.
Poznan University of Technology (PUT), and the University Later work focused primarily on overcoming the obstacle
of Stuttgart (US). We aimed to use the experience and data of capturing decisions. According to Burge and Brown [7],
from the case studies to answer the following three research the problem to be solved is that many possible applications
questions (RQ) and at the same time, achieve the following of rationale do not come into effect because the methods
learning objective (LO): and tools are not in place to support these opportunities.
RQ1 What are suitable ways to introduce decision captur- Approaches that try to mitigate this obstacle include Ishino and
ing and reflection in an agile development process? Jin [21] or Myers et al. [36] that integrated decision capture
directly into CAD tools and thus into the working environment
RQ2 Which types of decisions are crucial in agile educa-
of the engineers. The same idea with a closer relationship
tional software development projects?
to software development was implemented in DecDec tool
RQ3 To what extent do students capture decision alterna- by Hesse et al. [17], which is directly integrated into the
tives and their rationale? Eclipse IDE [13] and thus into the toolchain of a devel-
................................................................................................
oper. As a documentation tool for decision knowledge, it is
LO1 Raise the awareness of students for decision making
geared towards collaborative and incremental decision making
and rationale management in software engineering.
processes. Similarly, Helaba [33] offers a joint workspace
The remainder of the paper is structured as follows: Sec- to support communication around design artifacts and activ-
tion II discusses related work in the field of rationale man- ities in multidisciplinary teams. There are many other tools
agement in general and in an educational context. Section III available to assist in capturing decisions. An overview of the
presents the case study setup and details on its execution. area of software architecture documentation in this context is
Section IV elaborates on our results with respect to our re- provided, for example, by Tang et al. [45].
search questions and the learning objective. Subsequently, we
discuss these findings and some conclusions in Section V. We B. Rationale Management in Agile Software Development
conclude by describing potential future work in Section VI.
Based on previous work, research presented many con-
II. R ELATED W ORK tributions on the topic of documenting decisions and their
In this section, we discuss related work. Section II-A rationale in the context of the increasingly popular ASD.
elaborates on the topic of rationale management in the context Among other aspects, they present new decision modeling
of software engineering. Section II-B focusses on rationale approaches for ASD in general and Scrum in particular. For
management in ASD. Finally, Section II-C examines related instance, Waagenaar et al. [50] propose a concrete Scrum
work on teaching rationale management. artifact model or Wang et al. [51] elaborate on safety-related
documentation by introducing safety-epics and safety-stories.
A. Rationale Management in Software Engineering Other papers also focus on identifying best practices for
Early research on rationale management dates back to the documentation in ASD. For instance, Hoda et al. [19] present a
1970s and aimed at the use in politics (cf. Kunz and Rit- structured approach to experience-based patterns in agile docu-
tel [28]). The ideas were quickly translated into first practical mentation. With five concrete patterns, they strive to enable de-
projects. Conklin and Yakemovic [11] reported on the system- velopers to create exactly the right amount of documentation.
atic capture and use in various engineering disciplines which Rüping presents another much more comprehensive pattern
already took place in the 1980s. A few years later, research approach in his book on agile documentation [40]. Based on
in the field of Human-Computer-Interaction (HCI) recognized experience and concrete cases from previous projects, Rüping
the relevance of managing rationale, and various contribu- derives documentation patterns tailored to several problems in
tions appeared (cf. Fischer et al. [16], Lee and Lai [32], ASD.
MacLean et al. [34]). These then gradually served as a basis Researchers also developed ASD-specific tools to capture
for research on rationale management in software engineering. rationale. Echo [31], for instance, facilitates agile requirements
Much of this research identified the decision making process gathering under consideration of the relationship to decisions
as a central point for improvement. Accordingly, many ap- and their rationale. Voigt et al. [49] also proposed sprintDoc,
proaches propose models, tools, and methods for documenting which introduces developing documentation artifacts (wiki
decisions. To name just a few, these include the Architecture pages). Due to the existing history, changes to issues are
Rationalization Method (ARM) by Tang and Han [47] which directly linked to changes in the documentation and thus
attaches rationale to software architecture with the help of fine- traceable. Other approaches even employ machine-learning
grained model elements, an extendable meta-model facilitating techniques to extract undocumented design decisions (cf.
the RUSE model by Wolf [52] which integrates system models, Bhat et al. [4]).
S. Krusche, S. Wagner (Hrsg.): SEUH 2020 18
C. Rationale Management in Teaching taken over by graduate students (MA program). Occasionally
Despite the many contributions from research addressing the not enough graduate students were available. In these cases,
trade-off between agile principles and rationale management, the SM was taken over by the undergraduate students or by
there are so far only a few contributions in the area of the study organizers. In the latter case, the study organizers,
integrating decision documentation into agile process models. however, did not get involved in the selection of important
In practice, developers seem not to have a high awareness of decisions.
the importance of DR and DR is yet very rarely included in After an initial setup and exploration phase, the teams
software engineering education. The literature describes only worked in two-week (BTU) or four-week (PUT, US) devel-
a few approaches. For instance, Kleebaum et al. [23] integrate opment sprints. In total, the case studies included 82 devel-
rationale management in the context of a software develop- opment teams with about 350 undergraduate developers and
ment project both in the lecture and in practical development 50 graduate students (cf. Table I). In this context, it must be
tasks. Lago et al. [29] go a similar way here. In the context of a noted that the graduate students often worked with several
software architecture lecture, the topic of software architecture teams simultaneously, for example, as PO for one team and
decision making is dealt with and then deepened by a specially as SM for another team.
designed card game, called DecidArch. Other approaches TABLE I
OVERVIEW OF CONDUCTED CASTE STUDIES .
include a card game developed by Schriek et al. [41] who
intend to prompt the students to consider design elements University Semester Teams
more intensively or De Boer et al. [12] who teach students BTU 2015 Winter 2
how to elicit, communicate, and document architecture design PUT 2016 Summer 27
decisions. BTU 2016 Winter 5
PUT 2017 Summer 31
III. C ASE S TUDY D ESIGN
US 2017 Summer 1
In this section, we present the case study design. Initially, BTU 2017 Winter 4
Section III-A gives a general overview of how the case BTU 2018 Winter 6
study was carried out. We then describe the pilot study that BTU 2019 Winter 61
was conducted in advance and its findings (Section III-B). 82
Section III-C explains how we have incorporated these find-
ings into the case study setup. Subsequently, Section III-D B. Pilot Case Study
describes the data collection procedure. A description of the According to research question RQ1, the main objective was
analysis procedure is provided in Section III-E, followed by to integrate the capture and documentation of decisions into
elaborations on the validation procedure in Section III-F. a Scrum process. It was one of our main goals to change
the textbook version of the Scrum process only as much as
A. Case Study Overview necessary because we had to assume that most students in that
Between 2015 and 2019, we conducted eight case studies early phase of their studies used the Scrum process for the
with computer science students from the Brandenburg Uni- first time. Furthermore, too many additional process elements
versity of Technology Cottbus - Senftenberg (BTU), Technical contradict the first agile principle (cf. agile manifesto [3]).
University Poznan (PUT), and the University of Stuttgart To meet these requirements, we conducted a pilot case study
(US). We integrated the case studies into mandatory software over one semester with two teams at the BTU in advance of
development projects as part of the Bachelor’s or Master’s the case studies listed in Table I. Thus, in the first two sprints,
study programs. we conducted workshops together with the students. During
The tasks of small student teams were to develop com- these workshops, we discussed issues being significant to the
prehensive software products during the lecturing period of students, elicited alternatives, made decisions, and documented
one semester, i.e., about 15 weeks. The products were mostly them.
interactive web or desktop applications, e.g., online versions Based on the experiences from the interaction among the
of popular board games. The expected work effort for a project students, the interaction between the students and the teaching
was between 500 h (PUT) and 1.000 h (BTU, US), thus 100- staff as well as the dynamics of the decision making, we
200 h per person. identified the need for the following extensions to the Scrum
In addition to the development task, the students attended process, called Extension Requirements (ER) in the following:
complimentary lectures that discussed software engineering ER1 The team has to have a clear vision of who will ex-
techniques. The lectures did neither specifically address deci- ecute tasks related to the case study. Thus, one team
sion making techniques nor the topic of rationale management. member is responsible for recording the decisions.
All projects used the Scrum process, which served as the
ER2 The case study setup has to embed this responsibility
common agile organizational structure for the case study. The
in clearly defined process elements that provide a
typical project team consisted of four to five undergraduate
context and time frame within the sprint.
students (BA program) working as developers. The roles of
Product Owner (PO) and Scrum Master (SM) were typically 1 At the time of publication, the case study was still ongoing.
S. Krusche, S. Wagner (Hrsg.): SEUH 2020 19
ER3 The quality of recorded decisions needs to be assured Backlog Daily Scrum
through corresponding measures to enable rationale Refinement
reuse, as discussed by Thurimella et al. [48].
ER4 Capturing decisions must be regulated in a way that Review
certain documentation templates and tools are to be Scrum Master
used. Additionally, the case that a team might not
agree on a set of decisions to capture has to be Retrospective Sprint Planning
addressed as well. Decision Scrum Master Decision
Review Capture
C. Case Study Setup
To address the ERs listed above and enable a systematic ap-
Fig. 1. Altered Sprint Procedure.
proach to capture decisions, we decided to extend the textbook
Scrum by two process elements, as shown in Figure 1. were converted into images using screenshots and uploaded
Decision Capture At the end of each sprint planning to the students’ project repositories.
(ER2) the Scrum Master (ER1) is responsible for This approach showed insufficient usability and transferred
identifying and documenting the three most relevant too much responsibility to the case study organizers. Addition-
decisions for the current sprint together with the ally, the generated mindmaps did not show optimal readability.
team. Once recorded, these have to be uploaded to Subsequent changes due to, for instance, changed requirements
the project repository. or feedback could not be incorporated. We needed to either
recreate the documentation entirely or save the raw data along
Decision Review At the end of the sprint, during the retro- with the results to regenerate the mindmap. We also found
spective (ER1), the Scrum Master (ER2) is responsi- that students rarely documented their decisions when none of
ble for reviewing the documented decisions together the case study organizers were attending the sprint planning.
with the team (ER3). In case of changes, incon- All of these issues were reasons for the authors to refine the
sistencies, or recently emerged important decisions, capture technique for the planned case studies.
the Scrum Master has to revise the documentation.
(ER3). Autonomy in Decision Capture:
With the goal of more active engagement of the students,
In addition to these extensions, a case study guideline was
the authors decided to give the students more responsibility
provided to students. It contained additional instructions on
and thus to increase the autonomy in the documentation of
how to carry out the identification and selection of the most
decisions. It was necessary to choose another handling of the
relevant decisions if there is no immediate consent (ER4).
templates. For the first three case studies in 2015 (BTU) and
The guidelines also contained the requirements for the use of
2016 (BTU, PUT), Excel templates were compiled based on
documentation templates and the corresponding tools (ER4).
the previous QOC-based [34] modeling and distributed to the
For more information, please refer to Section III-D.
students. This way, the responsibility lay solely with the team,
Furthermore, a git [8] repository was made available to the
respectively, the Scrum Master, as envisioned in the case study
students, which provided the documentation templates (ER4),
setup in Section III-C. Additionally, sprint planning meetings
as well as an extensive collection of sample documentation
no longer had to be accompanied by the case study organizers
(ER4). However, there were no instructions on what types of
for decision documentation. This change required the students
decisions should be documented in order not to restrict them
to manage decisions consciously and independently and not
in their thinking and decision making.
to let this happen as a passive process triggered by a third
D. Data Collection – From Mindmaps to Markdown Records party. According to the case study setup (cf. Section III-C)
the Scrum Master was responsible for uploading the completed
The data collection process varied throughout the different
documentation to the project repository at the end of the sprint.
parts of the case studies. Apart from the pilot study, it
was carried out by the students over the entire period. The MADR – Markdown Architecture Decision Records:
capture technique, as well as the associated tools and templates Based on the feedback and experience of the 2016 case studies
for recording decisions, were repeatedly examined for their as well as a simultaneously conducted industry case study, we
usability and improved throughout the pilot case study and refined the handling of templates in the following case studies.
the case studies. A crucial point of criticism was the manageability. Working
Initially, during the pilot case study, we have chosen with Excel files proved to be impractical and cumbersome.
lightweight tool support, i.e., Text2Mindmap [20], to record Thus, it was perceived as too time-consuming to perform
decisions. For this purpose, we used the Questions Options reviews of the captured decisions by the team or revise the
Criteria (QOC) model [34] model and mapped it using Text- documented decisions. Accordingly, the case study organizers
ToMindmap. By this means, the case study organizers captured sought alternatives in dialogue with the students and the
the decisions made during the architecture workshops of the industry partner. The choice fell on the use of Markdown (MD)
pilot study. After the workshops, the generated mind maps files. MD is a lightweight markup syntax, which makes it easy
S. Krusche, S. Wagner (Hrsg.): SEUH 2020 20
to build documents [37]. Since both students and industry answers to be inappropriate. Furthermore, we tried to reduce
partners version their development projects with git [8] and the individual influence of the authors during data analysis by
its management frontend GitLab [18], the use of MD also coding the data separately by each author. Only afterward,
offered direct integration into the tool environment, since the results were compared and standardized to a uniform
GitLab renders MD files in the web interface nicely. categorization.
Besides, the management of MD files in git, combined with Concerning threats to external validity, one needs to be
a corresponding branching strategy, enables collaborative work aware that the software development labs and their associated
with short feedback cycles. Since we already used git in the lectures had an impact (as discussed in Section V). It was
case study, it seemed reasonable to make the MD templates hardly possible to mitigate this influence. The limited project
available in a public GitHub repository (cf. [43]). duration of only a few weeks and the steep learning curve also
The revised methodology, including the new templates, affected the external validity. Nevertheless, the free choice of
was supposed to be implemented in the first case study of development topics coupled with the high number of partici-
2017 (PUT), but could not be implemented for organizational pating groups and the resulting large amount of documented
reasons. The recording using Excel templates was applied decisions should be favorable to external validity.
once again. During the second case study in 2017 with
the University of Stuttgart (US), the students used the new IV. R ESULT E VALUATION
methodology. Recording vital decisions and their rationale In this section, we try to give answers to the research
with the help of MD files attracted such interest that our questions and the learning objective outlined in the Motivation
partners from the University of Stuttgart have forked and section based on the results of the case studies. As a basis for
further refined the original GitHub project. The result is a new this discussion, we surveyed the students after completing the
open-source (OS) project that was not only used in the second case study. The survey comprised three predefined questions
case study in 2017, which is an OS project itself (cf. [26]) (survey questions SQ1 to SQ3) and one open question for
but has reached into the OS community. Currently, more than individual feedback. Participation was voluntary. The survey
ten completely independent OSS projects employ Markdown resulted in a response of 56 answers. Considering all of the
Architecture Decision Records (MADR) [25] to capture their students who already completed the case study by the time of
decisions. Besides, other developers from the community have publication, the response rate is about 18%.
worked on the MADR GitHub project (cf. [24]). Based on the results of this follow-up survey, Section IV-A
Subsequently, we used MADR for the following case studies elaborates on the achievement of learning objective LO1.
in 2017, 2018, and 2019 (BTU). Thanks, among other things, Subsequently, Section IV-B answers research question RQ1
to the comprehensive instructions, there was hardly any neg- as far as possible and reports on the students’ perspective
ative feedback from the students. in which Scrum phase decisions have to be documented. In
Section IV-C, we then report on our findings on the types
E. Data Analysis – Coding and Classification of decisions captured by the students and thus try to answer
To answer research question RQ2, we searched the literature RQ2. Section IV-D concludes with a discussion on the scope
for a comprehensive classification of decision types, e.g., and nature of the information collected for a decision, i.e.,
Kruchten [27]. However, in our opinion, the classifications research question RQ3.
found did not provide structures clear enough to be intuitive
and intelligible for a data analysis like this. Accordingly, we A. LO1 – Raise the Awareness for Rationale Management
used an inductive approach and coded the data. Each of the To check the achievement of our learning goal, we asked the
authors did this individually. Subsequently, we tried to derive students to rate the usefulness of the conducted case study. The
a consistent categorization from the codes, which resulted in question had a dedicated focus on the communication within
several discussions and involved several iterations. Discrepan- the team, respectively, with stakeholders (survey question SQ1,
cies in the two result sets had to be identified and also resolved cf. Figure 2): As can be seen, it turned out that roughly equal
on a case-by-case basis. After further iterations, we refined parts of the students strongly agree or agree on the usefulness
the codes into a simple and straightforward classification. The of decision documentation (∼ 37%), respectively disagree or
results can be found in Section IV-C. disagree strongly (∼ 43%).
F. Validitation Procedures
We have taken various measures to address potential threats
to validity. Concerning internal validity, for example, we did
not get involved in the selection of decisions to be documented.
The authors explicitly pointed out to the students that the case
Fig. 2. Results for SQ1: usefulness of documented decisions, in percent.
study will not affect the evaluation of the lecture. Another
aspect of internal validity was the integration of free-text Besides, we asked students whether they reused documented
answers into the follow-up survey. This way, students could decisions (survey question SQ2, cf. Figure 3) and if so, for
provide individual answers if they considered the predefined what purpose. Here about 38% denied, and a further 14%
S. Krusche, S. Wagner (Hrsg.): SEUH 2020 21
replied that they used the decisions merely as a template for
recording further decisions. The remaining 48% of the students
opposed this by stating an explicit use case. These included
Fig. 3. Results for SQ2: reuse of captured decisions, in percent.
the following scenarios:
• A review was conducted.
Fig. 4. Results for SQ3: phases to capture decision in, in percent.
• A wrong decision was made and had to be reconsidered.
• A bug had to be fixed.
• A similar problem had to be decided on.
points throughout the sprint. Reasons given by the students
• One team member needed advice.
include the fact that it was just forgotten and was made up
• The requirements had changed.
for at the end of the sprint. Sometimes it also happened that
the vital decisions only emerged in the course of the sprint.
The above answers indicate that the participating students
Correspondingly, the integration of the decision capture into
have become aware of the topic and its importance. Several
sprint planning is a possible and preferred option, although it
students also confirmed this impression in an open question
is not the only feasible option for process integration.
on feedback on the case study. For example, one student
Still, the authors, like the students, consider the integration
wrote: “captured decisions help you to understand the problem
of the decision capture into the sprint planning to be the
thoroughly and [. . . ] have a better picture of the project and
preferred way, since working with the recorded and thus
its structure so that you can develop better software.“
communicated decisions during the sprint is thereby made
B. RQ1 – Suitability of the Chosen Approach possible.
As described earlier, building on initial ideas and the C. RQ2 – What Matters to Students?
results of the pilot case study, we decided on an approach to Based on the classification derived from the coding, Table II
integrate rationale management into Scrum (cf. Section III). lists 702 currently recorded decisions grouped by the corre-
Accordingly, for research question RQ1, we can only discuss sponding categories. Decisions that we could not attribute to
the applicability of our specific approach. As the answers to any of the available classification types are listed under N/A.
questions SQ1 and SQ2 show, there is no uniform opinion With about 2% and 12 decisions respectively, this includes
among the participating students. However, a considerable decisions that were not clear enough as well as decisions in
share of the students considers the decision capture approach which the students deliberately wrote nonsense. Accordingly,
to be helpful and applicable. In this context, it was vital a successful mapping of more than 98% of available decisions
for us to find out how the participants in the case study to the categorization appears sufficiently accurate for us and
assessed the timing of the decision capture. Accordingly, we confirms the appropriateness of the chosen classification.
have asked students to indicate phases in which they consider
TABLE II
documentation of decisions to be most appropriate (survey AGGREGATED NUMBER OF CAPTURED DECISIONS PER TYPE .
question SQ3, cf. Figure 4). Several answers could be given.
Again, there is no uniform opinion on the part of the Decision type Count Percent
students. However, a clear majority (59%) voted for capturing Definition / refinement of features / requirements 133 19
decisions during the sprint planning. The second most votes Development libraries / frameworks 100 14.3
came with 43% and 39% for review and retrospective, the Software architecture 97 13.8
phases at the end of a sprint. Still, just over a quarter of Development process 89 12.7
the respondents (27%) were in favor of capturing decisions Software quality measures (testing, etc.) 67 9.5
Prioritization of tasks / features 66 9.4
while working on the backlog. The least preferred approach is
Development tools 62 8.8
decision capture during planning poker (21%). We explicitly
Development platform 49 7
neglect that 23% of the students who argue for capturing de-
Deployment (Other in Fig. 5) 10 1.4
cisions during the Daily Scrum because it would conceptually Communication within the team (Other in Fig. 5) 10 1.4
contradict the Daily Scrum. To-Do decisions (Other in Fig. 5) 7 1
The three most frequently mentioned responses also reflect N/A (Other in Fig. 5) 12 1.7
our observations during the execution of the case studies. 702
Although the case study setup clearly states that the most
important decisions should be captured during sprint planning, The resulting distribution of decision types within all cap-
the students occasionally documented them at various other tured decisions shows that there is not a single relevant
S. Krusche, S. Wagner (Hrsg.): SEUH 2020 22
These two together account for a little less than 20% in the
first sprint, while it is about 40% and more from the fourth
sprint onwards.
Fig. 6. Aggregated number of captured decisions per sprint, absolute.
Fig. 5. Distribution of decision types over all case studies.2 For reading the graph, it is necessary to know that the
projects have lasted a maximum of seven sprints. The de-
decision type, but rather a group of important types. As can velopment of the open-source project, to which the students
be seen in Figure 5, the five most frequently documented of the University of Stuttgart contributed (cf. Section III-D),
decision types account for more than two-thirds of all captured was continued afterward and resulted in further documented
decisions (∼ 69%). Combining the seven most documented decisions. These are listed in the ”Subsequent Dev.” column.
decision types even accounts for nearly 88% of all decisions. Furthermore, it is necessary to read the graph in the context
On closer inspection of these seven, one can see that the most of the total number of captured decisions per sprint. As one
frequently documented decision type ”Definition / refinement can see from Figure 6, this has changed permanently. Although
of features / requirements“ is closely related to the ”Prioritiza- the students were required to write down the three most
tion of tasks / features.“ Topics related to requirements analysis important decisions, they sometimes recorded less or none at
and its prioritization together represent a considerable part all. In individual cases, they even recorded more. In the first
of all documented decisions with roughly 29%. A second sprint, the number of documented decisions was still very high
thematic group also stands out on closer inspection, solving (avg = 2.7 decisions), but decreased afterward. During the
implementation problems relating to software architecture and third sprint, the students recorded significantly less (avg = 1.7
those relating to the use of frameworks and libraries. Fre- decisions). As the development projects at the PUT lasted only
quently, they go hand in hand anyway. These two together three sprints (cf. Section III), from the fourth sprint on only the
account for about 28% of all decisions and thus have a teams of the US and the BTU documented their decisions (24
considerable influence on software development processes. teams in total). In the following, the amount of documented
Even more striking is the evolution of documented de- decisions is somewhat more stable again.
cisions over time. In Figure 7, we have broken down the It should be noted that for sprint five and six, there is no
different types of decisions by the sprints from which they data from the 2019 case study of the BTU available at the
originate. Upon mere observation, considerable differences time of publication. Accordingly, only 18 teams documented
in the distribution of the types of documented decisions are in sprints five and six. In particular, this means that in the
noticeable. In the first sprints, for example, one can find fourth and fifth sprint, an average of about 2.3 decisions per
an above-average number of decisions on the development team was documented, and in the sixth sprint, an average of
tools, the libraries / frameworks used, and especially the de- 2.1 decisions per team was documented. For the seventh sprint
velopment platform. These become considerably less over (4 teams) as well as the subsequent development (1 team), the
time. The decisions addressing the development platform, number of documented decisions increased again. However,
which in the beginning make up almost 20%, then disappear each of these two, with 12 decisions each, accounted for just
almost completely. Equally noteworthy is the progress of under two percent of the decisions recorded and thus have
decisions on software quality. They play a subordinate role little significance.
at the beginning and then gain considerably in importance In summary, the trend analysis also provides interesting
throughout the sprints. Especially in the third sprint, the topic insights for answering the research question RQ2. However,
of software quality plays a vital role, with a share of 25% in there is no simple answer to this question. Besides, the
all documented decisions. Subsequently, relevance decreases character of the projects, in which the case studies took place,
again. Just as important is the development of the decisions on
features / requirements as well as on the software architecture. 2 All decision types with less than 2% share are subsumed under Other.
S. Krusche, S. Wagner (Hrsg.): SEUH 2020 23
Fig. 7. Relative distribution of decision types over all case studies per sprint.
influences the results too. All projects started from scratch and
did have a rather brief lifespan.
D. RQ3 – What Information has been Documented?
Another aspect that has been of interest to us while exercis-
ing the case studies is the quantity of information students cap-
ture for a decision. Specifically, we are interested in whether
students only document the issue itself and the decision to
solve or whether they also record additional information such
as the alternatives considered. Also, this information, for
example, can be accompanied by further rationales.
TABLE III
D ECISIONS CLASSIFIED BY THE EXTENT OF DOCUMENTED INFORMATION .
Fig. 8. Distribution of decisions classified by the amount of avail. information.
Given answer type Count Percent
Alternatives & Rationale 273 38.9 all cases (∼ 76%), students recorded two or more solution
Multiple alternatives 262 37.3 alternatives for the documented decision. Additional rationales
One alternative 123 17.5
on the selected solution alternative and the reasoning behind
No alternative (Other in Fig. 8) 20 2.9
were recorded in about 39%. This number includes cases with
Rationale only (Other in Fig. 8) 9 1.3
simple explanations formulated in one sentence as well as
Explicit exclusion (Other in Fig. 8) 1 0.1
N/A (Other in Fig. 8) 14 2
statements in which the rationale behind the decision was
702 broken down into, e.g., positive and negative decision criteria.
To answer research question RQ3, we went through all V. D ISCUSSION
documented decisions again and categorized them according The results presented in this paper suggest that the capture
to the degree to which the students completed the templates. of decisions and their rationale is an issue that should have
Content-related considerations did not play a role. its place in software engineering education. Although the
In analogy to the left column from Table III, we dis- opinions from the follow-up surveys are not uniformly positive
tinguished between the simple naming of the problem, the with the students, the early stage of their studies is affecting
documentation of the selected solution alternative, the naming their judgment. Some students lack the experience that would
of none, one, or more alternatives. We also classified the change the assessment of such an approach. We observed that
provision of further rationale according to the template. the more advanced the students were in their studies, the more
The result was that about 94% of all cases, at least one positive was the feedback on the setup of our case study.
considered solution alternative was specified for the docu- Furthermore, according to the opinion of the authors, the
mented decisions (cf. Figure 8). In nearly three-quarters of concrete context in which the respective case study took place
S. Krusche, S. Wagner (Hrsg.): SEUH 2020 24
influenced on the quantity of the documented decisions, the making and rationale management in software engineering.
amount of information recorded per decision as well as the Moreover, the management of decisions and their rationale, as
type of documented decisions. For the authors, this impression already mentioned in related works, can certainly be seen as
is based on, for example, the complementary lectures that a controversial issue.
implicitly and unconsciously emphasize certain software en-
gineering aspects. A repeatedly increased number of captured VI. F UTURE W ORK
decisions on the topic of software quality aspects in the same Upon completion of the current case study, it is necessary
sprints support this observation. to enter the new data and update the evaluation. Building on
In the following, we briefly discuss the results of the this, we see the necessity to integrate the lessons learned from
research questions and the achievement of the learning goal: the case studies into teaching in a more structured way. To
RQ1 – Suitable Ways to Introduce Rationale Management: this end, it is also necessary to increase the response rate
The follow-up surveys with the participating students showed to the follow-up surveys carried out subsequently with the
that the case study setup provides a viable way to integrate participating students. In this way, we can incorporate more
the capture and management of decisions and their rationale and more detailed feedback into further rationale management
into an agile process. The selected implementation also offers applications. Another aspect is addressing the topic of software
sufficient possibilities for adaptation to one’s own needs. architecture decision making in the context of the supplemen-
Feedback and experience have also shown, however, that it tary lectures to the software development project, as done by,
is not the only viable way to capture decisions. Adjustments e.g., Lago et al. [29].
are easy to implement, especially in the phases to which the The logical next step is to generalize the approach used here
decision recording is coupled. for the Scrum process in teaching. Thereby the experiences of
the case studies, as well as the feedback of the students, play
RQ2 – Crucial Types of Decisions: a crucial role. The goal is to make the approach applicable to
For research question RQ2, it has to be stressed that there is other scenarios, if not to other agile process models.
no simple answer. A simple analysis of the distribution of all Another strategy for future work is to conduct a similar
documented decisions would be very short-sighted. A closer case study setup in an industrial context. It has been done in a
look at the available data reveals that the phase of development single case so far, however, for comparability with industrial
has a significant influence on which decisions are essential projects, considerably more data needs to be collected. An-
for students. In the specific case of the case studies, the other approach is to compare the findings with applications
organizational structure of the software development projects in the open-source community. To this end, it would be
also has a notable influence here. Almost all projects started necessary to find projects that systematically record decisions
from scratch and only lasted for a limited time. Accordingly, and make them accessible. The decision data could then be
the students needed to make technical decisions that would not compared with the data from the industry and from the case
occur at such frequency in projects running for a prolonged studies with the students. However, comparability from the
time. The credit points awarded in the respective courses are perspective of the applied process model is questionable here.
a limiting factor here. We hypothesize that only the type of essential decisions will
All in all, it can be emphasized that two major topics be comparable.
account for a large proportion of the documented decisions.
One relates to requirements analysis, features, their refinement ACKNOWLEDGMENT
as well as the sequence of realization. Decisions on how The authors gratefully acknowledge the fruitful cooperation
to implement these features, how are they mapped to the with our colleagues from PUT Poznan and the University
software architecture, which libraries and frameworks are to of Stuttgart, namely Miroslaw Ochodek, Bartosz Walter, and
be employed, define the other one. Oliver Kopp. Furthermore, we thank all students for participat-
RQ3 – What Information do Students Capture?: With ing in the case studies and taking some extra time and effort
research question RQ3, the authors wanted to find out to to support us. Without all of them, these case studies would
what extent students document a decision. It turned out that not have been possible.
they document more than expected. In the pilot case study,
R EFERENCES
students sometimes only did what the guidelines specified.
Throughout the case studies, however, the students provided [1] A LEXEEVA , Z., P EREZ -PALACIN , D., AND M IRANDOLA , R. Design
several considered alternatives for a decision in the majority of Decision Documentation: A Literature Overview. In European Confer-
ence on Software Architecture (2016), Springer, pp. 84–101.
the cases. More than that, the students added further rationale [2] BASS , L., C LEMENTS , P., AND K AZMAN , R. Software Architecture in
for the decisions made in more than a third of the cases. Practice, 2 ed. Addison-Wesley Longman Publishing Co., Inc., Boston,
MA, USA, 2003.
LO1 – Raise the Awareness for Rationale Management: [3] B ECK , K., B EEDLE , M., VAN B ENNEKUM , A., C OCKBURN , A., C UN -
In the follow-up surveys conducted with the participating NIGHAM , W., F OWLER , M., H IGHSMITH , J., H UNT, A., J EFFRIES , R.,
K ERN , J., M ARICK , B., M ARTIN , R. C., S CHWABER , K., S UTHER -
students, it has become apparent that a considerable number LAND , J., AND T HOMAS , D. Manifesto for Agile Software Develop-
of students have become aware of the topic of decision ment. http://agilemanifesto.org/, February 2001.
S. Krusche, S. Wagner (Hrsg.): SEUH 2020 25
[4] B HAT, M., S HUMAIEV, K., B IESDORF, A., H OHENSTEIN , U., AND [29] L AGO , P., C AI , J. F., DE B OER , R. C., K RUCHTEN , P., AND V ERDEC -
M ATTHES , F. Automatic Extraction of Design Decisions From Issue CHIA , R. Decidarch: Playing Cards as Software Architects. In
Management Systems: A Machine Learning Based Approach. In Eu- Proceedings of the 52nd Hawaii International Conference on System
ropean Conference on Software Architecture (2017), Springer, pp. 138– Sciences (2019).
154. [30] L AT OZA , T. D., AND M YERS , B. A. Hard-To-Answer Questions About
[5] B RIAND , L. C. Software Documentation: How Much is Enough? In Code. In Evaluation and Usability of Programming Languages and
Proceedings of the 7th European Conference on Software Maintenance Tools (2010), ACM, p. 8.
and Reengineering (2003), IEEE, pp. 13–15. [31] L EE , C., G UADAGNO , L., AND J IA , X. An Agile Approach to Capturing
[6] B URGE , J. E. Design Rationale: Researching Under Uncertainty. Arti- Requirements and Traceability. In Proceedings of the 2nd International
ficial Intelligence for Engineering Design, Analysis and Manufacturing Workshop on Traceability in Emerging Forms of Software Engineering
22, 4 (2008), pp. 311–324. (TEFSE) (2003), vol. 20.
[7] B URGE , J. E., AND B ROWN , D. C. Software Engineering Using [32] L EE , J., AND L AI , K.-Y. What’s in Design Rationale? Hum.-Comput.
RATionale. Journal of Systems and Software 81, 3 (2008), 395–413. Interact. 6 (September 1991), 251–280.
[8] C HACON , S., AND S TRAUB , B. Pro Git. Apress, 2014. [33] L OPEZ , M. G., H AESEN , M., L UYTEN , K., AND C ONINX , K. Helaba:
[9] C HARMAZ , K., AND B ELGRAVE , L. L. Grounded Theory. In The A System to Highlight Design Rationale in Collaborative Design Pro-
Blackwell Encyclopedia of Sociology. American Cancer Society, 2015. cesses. In Cooperative Design, Visualization, and Engineering. Springer,
[10] C LEAR , T. Documentation and Agile Methods: Striking a Balance. 2015, pp. 175–184.
SIGCSE Bull. 35, 2 (June 2003), 12–13. [34] M AC L EAN , A., YOUNG , R. M., B ELLOTTI , V. M. E., AND M ORAN ,
[11] C ONKLIN , E. J., AND YAKEMOVIC , K. C. B. A Process-Oriented T. P. Questions, Options, and Criteria: Elements of Design Space
Approach to Design Rationale. Human–Computer Interaction 6 (Sept. Analysis. Hum.-Comput. Interact. 6 (September 1991), 201–250.
1991), 357–391. [35] M ELNIK , G., AND M AURER , F. Direct Verbal Communication as a
[12] D E B OER , R. C., FARENHORST, R., AND VAN V LIET, H. A Commu- Catalyst of Agile Knowledge Sharing. In Agile Development Conference
nity of Learners Approach to Software Architecture Education. In 22nd (2004), IEEE, pp. 21–31.
Conference on Software Engineering Education and Training (2009), [36] M YERS , K. L., Z UMEL , N. B., AND G ARCIA , P. Automated Capture
IEEE, pp. 190–197. of Rationale for the Detailed Design Process. In AAAI/IAAI (1999),
[13] D ES R IVI ÈRES , J., AND W IEGAND , J. Eclipse: A Platform for In- pp. 876–883.
tegrating Development Tools. IBM Systems Journal 43 (April 2004), [37] OVADIA , S. Markdown for Librarians and Academics. Behavioral &
371–383. Social Sciences Librarian 33, 2 (2014), 120–124.
[14] D UTOIT, A. H., M C C ALL , R., M ISTR ÍK , I., AND PAECH , B. Rationale [38] R EEVES , J. W. What is Software Design. C++ Journal 2, 2 (1992),
Management in Software Engineering: Concepts and Techniques. In 14–12.
Rationale Management in Software Engineering. Springer, 2006, pp. 1– [39] RUBIN , E., AND RUBIN , H. Supporting Agile Software Development
48. Through Active Documentation. Requirements Engineering 16, 2 (2011),
[15] DYB Å , T., AND D INGSØYR , T. Empirical Studies of Agile Software De- 117–132.
velopment: A Systematic Review. Information and software technology [40] R ÜPING , A. Agile Documentation: a Pattern Guide to Producing
50, 9-10 (2008), 833–859. Lightweight Documents for Software Projects. John Wiley & Sons, 2005.
[16] F ISCHER , G., L EMKE , A. C., M C C ALL , R., AND M ORCH , A. I. Mak- [41] S CHRIEK , C., VAN DER W ERF, J.-M. E., TANG , A., AND B EX , F.
ing Argumentation Serve Design. Hum.-Comput. Interact. 6 (September Software Architecture Design Reasoning: A Card Game to Help Novice
1991), 393–419. Designers. In European Conference on Software Architecture (2016),
[17] H ESSE , T.-M., K UEHLWEIN , A., AND ROEHM , T. DecDoc: A Tool for Springer, pp. 22–38.
Documenting Design Decisions Collaboratively and Incrementally. In [42] S CHUBANZ , M. Design Rationale Capture in Software Architecture:
1st International Workshop on Decision Making in Software ARCHitec- What has to be Captured? In Proceedings of the 19th International
ture (MARCH) (2016), IEEE, pp. 30–37. Doctoral Symposium on Components and Architecture (2014), ACM,
[18] H ETHEY, J. M. GitLab Repository Management. Packt Publishing Ltd, pp. 31–36.
2013. [43] S CHUBANZ , M. Making Decisions Sustainable in
[19] H ODA , R., N OBLE , J., AND M ARSHALL , S. How Much is Just Enough? Agile Software Development. GitHub, March 2017.
Some Documentation Patterns on Agile Projects. EuroPLoP2010; 15th https://github.com/schubmat/DecisionCapture/.
European Pattern Languages of Programs (2010). [44] S ELIC , B. Agile Documentation, Anyone? IEEE Software 26, 6 (2009),
[20] I ONA , J. Text 2 Mind Map. The School Librarian 65, 3 (2017), 150. 11–12.
[21] I SHINO , Y., AND J IN , Y. An Information Value Based Approach to [45] TANG , A., AVGERIOU , P., JANSEN , A., C APILLA , R., AND BABAR ,
Design Procedure Capture. Advanced Engineering Informatics 20, 1 M. A. A Comparative Study of Architecture Knowledge Management
(2006), 89–107. Tools. Journal of Systems and Software 83, 3 (2010), 352–370.
[22] K AZMAN , R., K LEIN , M., AND C LEMENTS , P. ATAM: Method [46] TANG , A., BABAR , M. A., G ORTON , I., AND H AN , J. A Survey of
for Architecture Evaluation. Tech. rep., CMU / Software Engineering Architecture Design Rationale. Journal of Systems and Software 79, 12
Institute, 2000. (2006), 1792–1804.
[23] K LEEBAUM , A., J OHANSSEN , J. O., PAECH , B., AND B RUEGGE , [47] TANG , A., AND H AN , J. Architecture Rationalization: A Methodology
B. Teaching Rationale Management in Agile Project Courses. In for Architecture Verifiability, Traceability and Completeness. In ECBS
Tagungsband des 16. Workshops ”Software Engineering im Unterricht (2005), pp. 135–144.
der Hochschulen” (2019). [48] T HURIMELLA , A., S CHUBANZ , M., P LEUSS , A., AND B OTTERWECK ,
[24] KOPP, O. Markdown Architectural Decision Records. GitHub, Mar. G. Guidelines for Managing Requirements Rationales. Software, IEEE
2017. https://adr.github.io/madr/. 34, 1 (2017), 82 – 90.
[25] KOPP, O., A RMBRUSTER , A., AND Z IMMERMANN , O. Markdown [49] VOIGT, S., H ÜTTEMANN , D., G OHR , A., AND G ROSSE , M. Agile
Architectural Decision Records: Format and Tool Support. In ZEUS Documentation Tool Concept. In Developments and Advances in
(2018), pp. 55–62. Intelligent Systems and Applications (Cham, 2018), Á. Rocha and L. P.
Reis, Eds., Springer International Publishing, pp. 67–79.
[26] KOPP, O., B INZ , T., B REITENB ÜCHER , U., AND L EYMANN , F.
[50] WAGENAAR , G., H ELMS , R., DAMIAN , D., AND B RINKKEMPER , S.
Winery– A Modeling Tool for TOSCA-Based Cloud Applications.
Artefacts in Agile Software Development. In International Conference
In International Conference on Service-Oriented Computing (2013),
on Product-Focused Software Process Improvement (2015), Springer,
Springer, pp. 700–704.
pp. 133–148.
[27] K RUCHTEN , P. An Ontology of Architectural Design Decisions in
[51] WANG , Y., B OGICEVIC , I., AND WAGNER , S. A Study of Safety
Software Intensive Systems. In 2nd Groningen Workshop on Software
Documentation in a Scrum Development Process. In Proceedings of
Variability (2004), pp. 54–61.
the XP2017 Scientific Workshops (2017), ACM, pp. 22:1–22:5.
[28] K UNZ , W., AND R ITTEL , H. W. J. Issues as Elements of Information
[52] W OLF, T. Rationale-Based Unified Software Engineering Model. VDM
Systems. Tech. rep., Systemforschung, Heidelberg, Germany Science
Verlag, Saarbrücken, Germany, 2008.
Design, University of California, Berkeley, 1970.
S. Krusche, S. Wagner (Hrsg.): SEUH 2020 26