Goal models have long been used in the literature to model and reason about stakeholders' intentions. Prior work proposed several studies aimed at investigating what utility stakeholders derive from constructing and analyzing goal models. We designed and conducted an initial empirical study that explores the construction stage of goal modeling, asking whether stakeholders bene t from manually drawing their own model. We recruited eight quali ed participants and asked each to create a goal model for a decision they were considering while talking out loud. Half of the participants in this study used BloomingLeaf, while the remaining participants drew goal models by hand. Using information gathered through an online pre-study questionnaire, we constructed a goal model for each participant. Participants' compared our generated models with their manually created ones. We used open coding to nd themes and categories for qualitative responses. Analysis was mixed on whether participants preferred their own model to the researcher generated one. The results of which have implications for future studies, as well as goal model adoption and automation.
In early project planning, goal-oriented requirements engineering (GORE)
approaches have been advocated to help stakeholders make trade-o decisions.
While there have been many GORE languages [
Our aim is to investigate the construction stage of goal modeling. This study
is a rst attempt at exploring the questions proposed by Grubb [
The remainder of this paper is organized as follows. Section 2 introduces the methodology of our study. Section 3 describes and interprets our study observations. We discuss implications and future work in Sect. 4. 2
This study was reviewed by the Smith College Institutional Review Board (Protocol: 18-110), see supplemental information online1 for the study protocol. Study Design. Participants were asked to complete a pre-session questionnaire, a single in-person modeling session, and a brief post-study questionnaire. Throughout the study, participants were asked to explore one of three motivating scenarios that was currently relevant to their life: choosing between majors, planing for after graduation, and deciding between study abroad options.
In the pre-study survey, participants were asked who, what, and why questions to elicit scenario information, which allowed us to create goal models of their trade-o decisions, known as the AUTO model. Four participants each were randomly assigned to either the Paper or Tool group.
In the in-person modeling session, participants were trained on the syntax
and semantics of Tropos goal models (either on paper or in BloomingLeaf).
To demonstrate Tropos language constructs, we used a modernized version of
the Trusted Computing example [
In the optional post-study questionnaire, participants were asked if their decisions changed after the in-person session or if they had any further thoughts about the scenario they investigated.
Recruitment and Remuneration. We recruited eight participants in the summer of 2019. Participants were required to be pro cient in English, without any formal training in goal modeling, and registered undergraduate students at Smith College in the 2019 school year. Each participant was paid $15.00 upon completion of the in-person session. The participants' information was kept anonymous and demographic information was not collected.
Data Analysis. With participants' consent, in-person modeling sessions were
recorded and transcribed. Two of the authors independently used open-coding on
the transcripts to explore themes [
Understanding Tropos (and BloomingLeaf ). We begin by answering RQ1: Were novice participants able to understand and use goal model constructs? If so, what di culties were experienced by novices when being introduced to goal modeling and Tropos? After watching the tutorial videos, participants were able to identify actors, decomposition links, and contribution links. Some participants were confused about the di erence between the speci c types of contribution links. For example, one participant asked, \What's the di erence between ++ and ++S?", while another asked \Why do some [links] have two negatives and others have one negative?". We were able to clarify participants' questions about model semantics prior to continuing; thus, we believe we can compare observations between participants.
Participants were encouraged to continue asking questions throughout the session for additional clari cation. We observed two additional patterns over the remainder of the modeling. First, participants asked clarifying questions about model elements, including intentions. For example, a participant asked if \soft goals are things that don't have cut-o s?". These questions were most likely to occur at the beginning of the modeling process when participants had not yet fully transferred concepts from the Trusted Computing example. Second, participants asked how factors in their scenario could be added to the model. This style of question was most common once participants identi ed the main goals of their scenario. For example, participants asked, \Is my interest a resource or a task?" and \Should this link be and?". Overall, we found that novice participants were able to understand and use goal modeling constructs with assistance. Model Understanding. Next, we consider RQ2: To what extent were participants able to understand the AUTO model? After each participant constructed the USER model, the researcher showed them the AUTO model and asked questions to understand how well it t the participant's own scenario. None of the participants removed any nodes or connections from the AUTO model. Four participants proposed new connections between existing intentions and new intentions to be added. Two participants proposed how an intention could be further decomposed. After reviewing the AUTO model, all of the participants were able to present a clear cut answer about the major trade-o s and decisions in their own scenario, which were not clearly indicated in their pre-study questionnaire; thus, all participants were able to understand the AUTO model. This provides additional evidence that participants understood the goal model constructs. Comparing Auto Model with Self-created Model. Following from the previous question, we look at RQ3: Did participants choose to extend the USER or AUTO model? Five participants chose the AUTO model (3 Paper & 2 Tool), while three chose the USER model (1 Paper & 2 Tool). For participants who chose the AUTO model, their reasoning was that the AUTO model was more organized or had more information, sample responses included, \I think auto is more clear and the arrows make more sense" and \...there are more direct connections between all of the components that go into my goals...". For participants who chose the USER model, they found it contains more detailed information, for example, \I'll extend the one I just made, it has a lot of links I don't want to recreate on auto". In reviewing the participants' rationale, we became concerned that participants may have chosen the AUTO model because it was more visually appealing than their own model, which is addressed in our future work. Utility Evaluation. Finally and most importantly, we looked at RQ4: What utility (if any) was described by participants or observed by researchers? To evaluate what utility participants gain from constructing the model, we coded the responses to the post-study questionnaire and the transcripts of the in-person sessions, including the results to the question, \what did you learn from each model?". We observed four aspects of utility in the model construction activities in this study.
(1) Elicit Underlying Assumptions & Motivations. When talking out loud, participants discussed their motivation behind achieving a certain goal while putting that goal on the canvas. Thinking in the context of goal modeling helped participants to explore tacit information, though most of this information does not appear in the nal model. Two comments included in this category were, \probably it is also a goal [to become pre-dental], 'cause after coming to Smith, I nd that I really want to take a lot of chemistry courses and want to eventually make it a major", and \one of the goals of neuroscience is to achieve a deep understanding in one of the hard sciences, a task would be taking a bunch of upper-level classes".
(2) Explicitly Consider Structural Relationships. Study participants responded that they learned how to break down a bigger goal into smaller tasks and considered how to weigh di erent factors in the process. This lead to new insights about how they could structurally think about the ultimate goal of their scenario. One participant said that she learned \...a di erent way to weigh things, compare things that are required vs possibilities. A new way to put thoughts down". Another participant said that this made her \...look at things as smaller steps, which is really valuable. It very much so set into stark terms the decisions. I'll have to make in the future, which I've been putting o ".
(3) Learned Something New. Participants responded that after modeling their goals, some of the factors or goals were presented and linked within the model which they had not seriously considered before. This forced them to reconsider a perspective or factor that they had previously ignored. Two sample responses were, \I wasn't thinking a lot about how di erent perspectives would contribute to my decision-making, especially [my] pre-health advisor...but in this model it represents that.", and \I think one of the biggest conclusions is that industry job would satisfy less of my personal goals than teaching or research".
(4) Update Conclusions. Finally, participants responded that their personal preference was further reinforced or changed after the modeling experience. Some said that the idea of \soft goals" helped them to see the broader picture of the scenario, which in uenced their decisions. For example, one participant said, \I think my preference is towards the french major now. It shifted more in that direction.", while another reinforced their concerns, saying, \I guess I was thinking a lot about money issues, but when you draw [out the model] it feels real and pressing." 4
Threats to Validity. The observations made in this study may be erroneous
due to threats to validity, speci cally construct validity. Subjects may have
gained utility from discussing their scenarios. We did not separate the utility
in talking through a problem from the utility of modeling the problem. Future
studies will isolate this factor and compare them with the results of Kwan and
Yu [
Discussion of Utility. In Sect. 3 we found evidence of utility; thus, we reject the null hypothesis as stated, but this is insu cient to answer our study question: To what extent do participants gain value in manually drawing goal models (in a tool or on paper)? Since this was an exploratory study, none of the observations were conclusive but instead inform our future work. Future Work. We expect to re ne our protocol and make the changes described above to address our threats to validity. We will repeat this study as a series of controlled experiments with larger numbers of participants over longer periods of observation, isolating each variable in this study. In future work, we intend to explore participants rationale for choosing the AUTO or USER model.
To investigate the e ects of `talking out loud', we will compare the use of goal modeling to other methods of choosing between trade-o s (e.g., pro and con lists). For each of these studies we should consider novice users (representing stakeholders), trained modelers, and stakeholders not involved in the modeling process. Finally, we hope to partner with software organizations to study goal model construction in an industrial context.
Summary In this paper, we presented an exploratory study of the utility of goal model construction. We found that participants understood and were able to use model constructs and found utility in both the USER and AUTO models. Participants gained insights into their assumptions, goals, and the structural nature of their scenarios with some participants changing their decision. Future work will repeat our study with trained modelers and address the issues noted in our discussion above.