=Paper=
{{Paper
|id=Vol-1370/paper_3
|storemode=property
|title=Observational Studies of new i* Users: Challenges and Recommendations
|pdfUrl=https://ceur-ws.org/Vol-1370/paper_3.pdf
|volume=Vol-1370
|dblpUrl=https://dblp.org/rec/conf/caise/Horkoff15
}}
==Observational Studies of new i* Users: Challenges and Recommendations==
https://ceur-ws.org/Vol-1370/paper_3.pdf
1st International iStar Teaching Workshop (iStarT 2015)
Observational Studies of new i* Users: Challenges and
Recommendations
Jennifer Horkoff
Department of Human Computer Interaction, City University London
horkoff@city.ac.uk
Abstract. As users and teachers of i* and related frameworks, we work with
certain assumptions about which concepts and conventions are easy or more
difficult for new i* users to learn and use. As part of a recent research initiative
examining the possible synergies between goal modeling and creativity tech-
niques for requirements engineering, I have conducted an exploratory study ex-
amining groups of graduate and undergraduate students working together to
draw i* models. Several observations were made about the way students
worked together to draw models, including the areas and concepts in which they
had particular difficulty. I summarize and report on these findings, in the hope
that it can have a positive impact on the way i* and other goal modeling frame-
works are taught or introduced to new users.
Keywords: iStar, Pedagogy, Observational Studies, Modeling, Education
1 Introduction
When teaching goal models (such as i* [1]) to new users, we gradually gain an im-
pression of the concepts and relations with which users struggle. Typically we gain
these insights through questions asked in classes, tutorials, office hours, or by grading
models as part of student assignments (see [2], for an example of observations made
from student assignments). Occasionally we have the opportunity to observe students
creating their models, perhaps as part of an in-class work session, but typically, as
goal model instructors, we make our observations either before or after student mod-
eling, not as the modeling occurs.
As part of an exploratory study aimed at understanding potential synergies between
creativity techniques and conceptual modelling for RE, the author had the opportunity
to observe nine groups of one to four students (23 participants in total) work together
to create i* models. A short description of the exploratory study, including results
from the creativity perspective appears in [2]. Although the study was performed as
part of a larger project focusing on creativity and goal modeling, many observations
made during the study were relevant specifically for goal modeling. In this paper I
describe my observations from the goal modeling perspective, including general ob-
servations on student difficulties and experiences. A first attempt is made to provide
ideas and recommendations to help mitigate some of the student difficulties. It is my
13
hope that these observations and recommendations can help to improve the way we
teach goal modeling in practice.
The setup of the study is described in more detail in Section 2. Study observations
are summarized in Section 3, while Section 4 provides some initial advice and rec-
ommendations. Section 5 concludes the paper.
2 Study Setup
In order to explore the relationship between creativity and goal modeling, the author
performed nine one-hour sessions with groups of one to four students, all of whom
had some coursework experience with goal (i*) models. Students were recruited from
the Requirements Engineering (RE) course taught at City University (City U), open to
both Undergraduate and Masters students. Two groups, used as pilot studies for the
two treatments, were comprised of current graduate students and researchers at the
center who had taken the RE course in the past. Sessions involved a total of 23 partic-
ipants, (two undergrad and 21 Masters Students) studying a range of Information
System-related topics, including Business Analysis and Design, Software Engineer-
ing, Information and Technology, Business Systems, and Business Computing.
Although our study was exploratory in nature, we were guided by research ques-
tions, including: a) should goal modeling be performed before or after creativity tech-
niques? b) can goal models effectively capture creative ideas? c) does the structure of
the models impede creative thought?
In the sessions, student were given an i* refresher, a small example Strategic De-
pendency (SD) and Strategic Rationale (SR) model. They were then given a toy sce-
nario involving a parking garage, and were asked to sketch a goal model and come up
with creative ideas guided by selected creativity triggers (participation, connectivity,
convenience, boundaries). Goal models were drawn on large sheets of paper. Five
groups performed goal modeling then creative thinking, while the other four groups
did the reverse. Approximately 20-30 minutes of the session were devoted to goal
modeling. Participants reflected on the process via a short survey, including the order-
ing of activities and potential synergies between modeling and creative thought.
The author facilitated the session, explaining tasks, making observations and taking
notes. When progress was stalled, the author would try to provide guidance, but oth-
erwise did not interfere. Audio and video were not taken; the facilitator took notes and
kept the resulting models. From the perspective of the students, there were no explicit
rewards, but the sessions were advertised as a chance to practice their i* skills with an
i* “expert”. I was available to answer any i*-related questions and to help them with
their models when requested. Observations on the submitted assignments for this
session of the course (“module” in UK terms) are provided in a companion paper [3].
3 Observations
Some observations made during the sessions were in line with previous observations
made from assignments and teaching experience. For example, students had difficul-
14
ty distinguishing between goals and tasks and they had to be urged to make their SD
models consistent with their SR models. Other observations were more surprising.
Progress. First, although I realized that timing in the activities would be tight, it
was surprising how little progress most groups made with their i* models. Many
groups (4/9) made progress on an SD model, but did not get a chance to start SR
modeling. Even when working on their SD models, groups made fairly slow progress.
Often much time was spent in discussing ideas, dependencies and actors, with very
little writing or drawing of the content of their discussions. It took groups some time
to even begin modeling. Those groups who did get to some SR modeling struggled a
lot with internal SR relationships (decomposition, means-ends, contribution). Statis-
tics on the number of i* elements and relations drawn by each group can be found in
Table 1. We can see that the first group of pilot participants, P1, who had more expe-
rience in i* modeling, performed better than the rest of the groups (P2 and S1-7).
SD First. While observing the students starting with SD models, I noticed they of-
ten came up with many ideas for elements that did not fit well in the constraints of
SDs. For example, reading the domain description, students would find a softgoal
vehicle safety. This could be drawn as a dependency, but is most likely a softgoal of
the Manager, represented inside an SR actor. In these cases, the students either had to
figure out how to draw the element as part of a dependum, or temporarily drop the
concept. As the process of drawing an SD is consuming, students didn’t think to rec-
ord these elements for later SR use. In this way, students were either overloading their
SD models or losing ideas for elements that better belonged in the SR models.
System Actors. Students also had difficultly knowing that they should explicitly
model the as-is system (in this case Parking Garage System) as an explicit actor. Other
actors like Attendant and Driver were identified easily, but more abstract actors were
less obvious. In retrospect, the example i* model they had been given as a refresher
was too simple, not IT-specific, and therefore did not have an example system actor.
Table 1: Resultant Goal Model Counts from Study (blank cell = 0 count)
P1 P2 S1 S2 S3 S4 S5 S6 S7*
SD Actors 4 4 4 4 4 3 5 4 2 4 4
Goal 2 2 4 1 1 1 1 1 1 1 1
Softgoal 4 4 7 6 2 2 3 3 5
Task 1 1 1 1 2 2 1 1
Resource 1 5 1 1
Sum Dependencies 6 8 17 9 4 3 5 5 2 5 6
SR Goal 4 1 1 1 1
Task 11 3 2 2 2 4 4
Softgoal 5 1 2
Resource 1 1 2 2 3 1
Sum SR Elements 21 5 4 4 5 7 8
Decomposition 14 4 1 2 1 5 5
Means-end 2 1 1 2
Contribution 1 2
* drew 3 separate models
15
Actor vs. Resource. Similarly, 7/9 groups debated whether or not the parking lot
sensor, measuring occupancy at each level, was an actor or a resource. In several cas-
es I stepped in to help the groups, explaining that it would depend on whether the
sensor had intentionality, its own set of goals, tasks, and dependencies. The idea that a
non-human entity could be an actor was not intuitive.
Dependency Rule. As this study was conducted using students at City U, I had the
opportunity to evaluate some of the i* conventions specific to this university, devel-
oped based on their modeling and teaching experiences. As part of their course mate-
rial, students were taught a particular rule helping them to move from SD to SR mod-
els, specifically, the dependum in a dependency relationship in an SD model is always
modelled inside the boundary of the depender in the SR diagram [5]. Similarly, ac-
cording to City U convention, goal, task and resource dependencies correspond to
decreasing levels of delegation. The rationale for this convention is that in a goal de-
pendency the dependee has great freedom in how to satisfy the goal, thus the level of
delegation is stronger, while there is less freedom and weaker delegation for a task,
and even less so for a resource.
Observing the students, the first dependency rule caused some confusion and had
to be reiterated and reviewed several times. There were a few cases where dependen-
cies seemed sensible, but violated the rule. For example, in the left side of Figure 1,
the Driver depends on the Attendee to Give Directions, which seems to be a sensible de-
pendency. The rule would state that Give Directions must be inside the Driver, when
really it’s the Attendee Giving Directions, as in the associate SR, which is wrong accord-
ing to the rule (note that a further City U convention is that SR models have no de-
pendums). As tasks have a greater degree of delegation than resources, perhaps a
more correct way to model this would be to model the dependency as a resource, as
shown on the right of the Figure. However, these types of rewriting solutions were not
easy for the students to come up with on their own; usually they could only do so with
some help from the author/facilitator.
Tables. I observed that many of the students started their SD models by drawing ta-
bles, recommended in their course slides. This appeared to be a helpful intermediate
way for them to start the course slides, less intimidating than drawing shapes.
Student Example Before Potential Solution
Driver Give Attendee Driver Directions Attendee
directions
Driver Attendee Driver Attendee
Car
parked
Car
Give Give
parked
directions Directions directions
Wrong according to rule Rule followed
Figure 1: Example of City U Dependency Rule Violated and Followed
16
Other observations. Although students were given short questions guiding how to
start i* modeling, almost all participants ignored the instructions, perhaps because
there were already too many study-related sheets and forms to read. Also, there didn’t
seem to be an obvious difference in modelling effectiveness between pre-formed and
randomly formed teams of modelers. Finally, the effects of physical modeling should
be considered. It’s difficult to know if using tools would help the students, making it
easier for them to start the process of modeling, or easier to maintain consistency
between SD and SR. Several groups complained about having to model on paper, i.e.,
it was hard to move shapes and fix mistakes. However, using a tool with the modeling
view, for example, projected on a screen, may have inhibited collaboration and team-
work. Future studies should look more closely at paper vs. electronic modeling.
Creativity. In terms of the relationships between goal models and creativity, al-
most all participants indicated that if they had to do the activity again, they would
perform both creativity and goal modeling: there was a general agreement that these
activities work well together. Participants were often able to express their creative
ideas in terms of the model, e.g., adding new actors or softgoals. Goal modeling did
not appear to over-constrain creative thought, but gave group members a common
understanding by which to ground their ideas. More details can be found in [6].
4 Recommendations
Based on the observations of this exploratory study, I can provide recommendations
for future i* instruction. First, it’s important not to underestimate the amount of effort
and time it takes new learners to draw i* models, particularly in the first steps of get-
ting the modeling started. Although we want to encourage independent thought, it
may be useful in some contexts, e.g., tutorial exercises, to provide skeleton models
with some completed actors and elements as a starting point for modeling.
Regarding the common practice of starting with SD models, the rationale behind
this practice is clear – SD models consist of a simpler subset of goal modeling con-
cepts. However, observations have led me to believe that dependencies are not the
most intuitive or easily found concepts by new learners. In many cases it may be easi-
er to say that an actor has a goal or performs a task than to identify specific types of
dependencies between actors. I suggest that students start with a hybrid SD/SR model,
including actors, boundaries, elements inside boundaries, and some dependencies, but
without filling in the details of the internal SR relationships. An example of such a
model is shown in Figure 2, capturing the concepts of an SD, but allowing modelers
to record SR ideas. In fact, the process of SD and SR modeling is ideally iterative, but
it was not clear during the observations if the students were willing or able to change
their SD models based on their SR progress. Starting the modeling process in this way
could help to emphasize the tight linkage between SD and SR.
Observations indicate that we, as instructors, must place more emphasis on less
“concrete” actors such as various systems; our examples should include actors of this
type. Similarly, we should provide examples in which we decide whether an entity is
a resource or an actor, i.e., should a system entity be ascribed intentionality or not?
17
Figure 2: Example Hybrid SD/SR Model Possibly Helpful for New Learners
Dependency rules, such as those taught at City U, seem to cause confusion more than
provide guidance. As the intention of teaching new i* users is to get them thinking
systematically from a broader perspective, one could argue for less rules altogether.
The use of tables as an intermediate artifact appeared to be helpful for students. We
could also think about using tables to help create SR models, e.g., list all the
goals/tasks for an actor without worrying about internal links or dependencies. It
would be helpful if the tools we used in teaching had tabular views of actors and de-
pendencies, even if these views only showed a subset of the information in the model.
5 Conclusions
The author has used her experience observing new i* users in an exploratory experi-
ment to understand modeling challenges, providing a list of recommendations based
on these observations. Although the observations and recommendations are specific to
i*, many are applicable to other goal modeling frameworks using actors and depend-
encies. These recommendations are meant to act as a starting point for discussion,
with the hope that follow-up work will test their effectiveness.
References
1. E. Yu et al., eds. Social modeling for requirements engineering. MIT Press, 2011.
2. J. Horkoff et al. Reflective Analysis of the Syntax and Semantics of the i* Frame-
work. RIGIM'08. Springer, 2008. 249-260.
3. J. Horkoff, N. Maiden. Creativity and Conceptual Modeling for Requirements Engineer-
ing. CreaRE: 5th Int. Workshop on Creativity in Requirements Engineering. 2015.
4. A. Bennaceur, J. Lockerbie, J. Horkoff. On the Learnability of i*: Experiences from a New
Teacher. Submitted to iStarT’15.
5. A. Bennaceur, N. Maiden. Lecture 5: Requirements Modeling II, i* Strategic Rationale
Modelling. City University. 2014
18