=Paper=
{{Paper
|id=Vol-1829/iStar17_paper_16
|storemode=property
|title=Goal Modeling without Stress: An Empirical Study of User Engagement
|pdfUrl=https://ceur-ws.org/Vol-1829/iStar17_paper_16.pdf
|volume=Vol-1829
|authors=Amy Kwan,Eric Yu
|dblpUrl=https://dblp.org/rec/conf/istar/KwanY17
}}
==Goal Modeling without Stress: An Empirical Study of User Engagement==
https://ceur-ws.org/Vol-1829/iStar17_paper_16.pdf
Goal modeling without stress:
An empirical study of user engagement
Amy Kwan1,3 and Eric Yu1,2
1
Faculty of Information, University of Toronto, Toronto, Canada
2
Department of Computer Science, University of Toronto, Toronto, Canada
3
Rotman School of Management, University of Toronto, Toronto, Canada
Abstract.
Our research explores the approach of having the researcher/facilitator create
the model for discussion rather than expecting the subject matter experts who
may know the process well, but are not proficient with the model’s notation.
This paper presents the design, approach, and preliminary findings from a case
study (using two different processes) where a group of subject matter experts
were introduced to goal-oriented modeling by way of having the facilitator
model the process upon reading the detailed textual process documentation and
gathering input in the form of a survey prior to showing the model to the group.
This study aims to illustrate the benefits of having a facilitator who is
comfortable with the models to enable the discussion with subject matter
experts to focus on the benefits and value of the information expressed through
the models by removing any obstacle or stresses due to a participant’s
discomfort or unfamiliarity with modeling.
Keywords: i* framework, goal modeling, operational processes
1 Introduction
A business process is a standardized set of activities that accomplish a specific task,
such as processing a customer’s order [1]. Being able to accurately construct business
process models requires the knowledge of the process itself, i.e., how the data flows
(data flow diagrams), the sequential actions (process flow diagrams) or the goal and
dependencies (goal-oriented modeling). However, being able to accurately construct
any of the mentioned models would also require knowledge of the model along with
its respective notation. It can already be overwhelming to anyone unfamiliar with the
models to read them, let alone attempt to construct one by him or herself. Thus, the
ability to efficiently and effectively capture all of the elements and information of a
business process to analyze it when working directly with subject matter experts who
execute the process can be quite limiting if the expectation of modeling and analyzing
the processes is left on the shoulders of subject matter experts who are unfamiliar
with the concept, methodologies, and notations of modeling.
In our work we try to address this challenge by taking the approach of having
someone who understands the model first attempt modeling the processes using
existing documentation (i.e., from detailed textual operational process manuals) and
then setting up workshops with the subject matter experts of the process to validate
the accuracy of the process-focused elements. This approach allows the facilitator and
subject matter experts to focus on the benefits of discussion of the models in terms of
what the model expresses, without becoming too concerned about the technical
correctness of the models. This approach effectively removes the pressures or
discomfort of creating a model that the subject matter expert is unfamiliar with.
2 Context of Empirical Case Study
This industrial case study was conducted by the first author in a public sector
organization’s finance operations back-office with subject matter experts for the two
respective processes chosen for this study. The processes chosen for this study were
1) payment failures and 2) reversing a transaction. The intent was to have a modeler-
facilitator who also had direct familiarity with the business and the department where
the internal processes are executed (i.e., as opposed to a modeler-facilitator who was
an external consultant or internal business analyst outside of the operational
department). With regard to the participants, subject matter experts at various levels
of experience and seniority were invited to participate. The department where this
study was conducted had multiple layers of controls and oversight for each of its
operational processes, i.e., most processes had three layers of oversight: Processor,
Reviewer and Supervisor. In order to obtain in-depth feedback from the subject matter
experts for this study, we chose included participants at all three levels with varying
lengths and depths of experience and responsibilities for the process, thus, each group
included subjects who had either a processor, reviewer or supervisor role for that
process to get in-depth viewpoints from every actor/stakeholder involved in executing
the process. The two groups of participants were distinct. We present the observations
from the workshops that assessed two operational business processes within an
industry organizational setting. This contribution aims to provide a different approach
to capturing the benefits of modeling by focusing on the expressiveness of modeling
rather than the technical notations of modeling.
3 Methodology and Study Design
The workshops set up to observe and gather information for each process was
extensive and conducted over three in-person sessions/meetings with the participants
for each process. This section of the paper outlines the details of preparing for the
workshops as well as the general/common steps and approach at each of the
facilitated in-person sessions with the participants.
The following steps were taken in preparation for the workshops: 1) A copy of the
operational guide for the process was obtained. This guide is a textual documentation
that outlines detail including the tools required to perform the process (i.e.,
applications and access to files), the pre-requisites for each process (i.e., any
dependencies of other input processes) and the steps, written out step-by-step,
required to execute the process. 2) The modeler-facilitator created As-Is Data Flow
Diagram (DFD) and As-Is Process Flow Diagram (PFD) based on the operational
guides. 3) A survey for each process was created. The questions were designed to: A)
Gather a high-level profile of the subject matter expert and his/her role in the context
of executing the process, B) Get a baseline of the participants familiarity and
understanding of the type of model to get a sense of whether the subject matter expert
may have familiar with the notations of the models prior to having it presented to
him/her, C) Have the subject matter expert articulate in his/her own words his/her
understanding of his/her role in the process as well as his/her understanding of the
goal/outcome that is to be achieved upon completion of the process, D) Validate the
general accuracy of the model and how the steps were documented and represented by
the modeler-facilitator, and, E) Ask if the subject matter expert could identify directly
from the model any process gaps or opportunities for improvement that were not
previously identified prior to reviewing the visual representation of the process.
For measure (C) the intention was also to gather more in-depth information and
perspective about the goals of the processes, which would be used to model the i* at a
later stage for another step of the workshop. This is because the information provided
in the operational guides was not substantial enough to use to create the i*, given that
the nature of the operational guide was process-focused and not goal-focused.
This next section of the paper outlines summarizes the work facilitated at each of
the workshops as well as the general/common steps and approach at each of the
facilitated in-person sessions with the subject matter experts.
Working Session 1 for Payment Failures process: 1) Participants were given a
survey that asked: their role and level of experience with the process, their familiarity
and comfort with data flow diagrams, their goals and what constitutes success or a job
well-done, and who they depend on and who they think depends on them for the
process. 2) Participants were given a physical hard-copy of a pre-drafted version of
the DFD, a walkthrough of what the model displayed, and were asked to: confirm the
accuracy of the information provided in the model (marking corrections directly on
the hard copy using the “red” pen provided), comment what was working well and/or
not working well (i.e., gaps, inefficiencies, etc.) upon review of the DFD provided,
and provide suggestions on how to improve the process (marking suggestions directly
on the hard copy using the “blue” pen provided). 3) Post Working Session 1:
Researcher/facilitator analyzed the survey results and created: an updated As-Is DFD
model, called As-Is(2), a To-Be DFD model incorporating suggested improvements,
and an updated As-Is i* SD model incorporating information from the survey.
Working Session 1 for process of Reversing a Transaction: The steps followed
in the Reversing Transactions process were identical to that of the Payment Failures
process outlined above with the exception of one difference in step 3 (walkthrough of
prepared model and completion of the survey). An As-Is PFD (instead of a DFD))
was provided. The rationale behind providing two different process/detail-oriented
models is we wanted to observe possibility of whether the type of diagram/model
initially presented would influence the type of feedback and comments provided. We
also wanted to get a broader sense and range of what types of diagrams/models and
notations the participants were familiar and/or comfortable with.
Working Session 2 for Payment Failures process: 1) Met with participants and
showed them the DFD models and asked for confirmation that the models accurately
captured their updates and suggestions. 2) Introduced participants to goal-modeling
through presentation of the As-Is i* SD model and asked to: confirm the accuracy of
the information provided in the model (marking corrections directly on the hard copy
using the “red” pen provided), and make note of any updates or suggestions of goals
(marking suggestions directly on the hard copy using the “blue” pen provided). 3)
Post Working Session 2: Researcher/facilitator analyzed the notes made on the
models by the participants and created: an updated As-Is i* SD model incorporating
suggested improvements, a To-Be i* SD model, and an As-Is i* SR model.
Working Session 2 for process of Reversing a Transaction: The steps followed
in the Reversing Transactions process were identical to that of the Payment Failures
process outlined above with the exception of one difference in step 3 (review of the
prepared models). A PFD (instead of a DFD)) was reviewed.
Working Session 3 for Payment Failures process: 1) Met with participants and
showed them the i* SD models and asked for confirmation that the models accurately
captured their updates and suggestions. 2) Introduced participants to the As-Is i* SR
model and had a discussion. 3) Post Working Session 3: Researcher/facilitator
analyzed the survey results and updated/finalized all i* models (As-Is and To-Be i*
SD and SR models) 4) Next step (to be scheduled at the time of this report) will be to
have the working group of participants provide overall feedback about the
effectiveness and learnings from these workshops and how they were helpful (or not)
in facilitating the exposure to them of goal-modeling.
Working Session 3 for process of Reversing a Transaction: The steps followed
in the Reversing Transactions process were identical to that of the Payment Failures
process outlined above.
4 Results, Observations and Findings
Working Session 1: Payment Failures: Information gathered from the surveys
indicated that none of the participants knew what a DFD was prior to the start of the
workshop. Once the participants were walked through the DFD by the workshop
facilitator, they were quickly able to pick-up the concept of what they were reading
and comfortable with making corrections to the existing draft. However from a
practical application perspective, at first the participants were unsure of what to make
of the DFD and whether knowing or referring to the DFD would provide any value.
However, after some discussions among the team, they were able to identify
opportunities where data was captured but not currently utilized. These unused data
could potentially generate value through analytics.
Reversing a Transaction: Information gathered from the surveys indicated that
most of the participants in this group knew what a PFD was prior to the start of the
workshop (contrary to the experience with the DFD in Process 1). Because of the
group’s general comfort with PFD models, it was observed that they needed little
direction and were comfortable diving right into providing feedback and suggestions
on the copies provided.
General Observations and Findings: When comparing survey results and
feedback between the session where the DFD was used and the session where the
PFD was used, it was observed that these levels of participants, who were all
operational staff, were much more familiar and comfortable with the PFD. This was
evident through both the time spent in relation to the level/depth of comments
provided. Those in the group working with the PFD spent less time on the individual
component of the workshop exercise (completing the survey and providing feedback
on the models) compared to those in the group working the DFD.
Working Session 2: Payment Failures: Similar to the DFD, prior to the
workshop, none of the participants had heard of or been exposed to the concept of
goal-modeling. Introducing the i* model was more foreign and challenging for the
participants. However, once the concept of the i* model was walked through with the
group, they appeared highly curious and engaged when seeing the bigger picture of
how their work affects (or is affected by) other actors in order to achieve the goals
that would lead to successfully executing the process. Feedback from the discussions
led us to believe that the employees were much more motivated and appreciated the
impact of their work after seeing the i*. When asked for feedback on the approach,
participants acknowledged that having the researcher/facilitator provide the models
(instead of having the participants learn and create the models) removed the stress of
learning the model and allowed them to focus on discussing the process itself.
Reversing a Transaction: Observations made, related to goal-modeling, were the
same as compared to that of the Payment Failure process.
General Observations and Findings: Given that both groups had no prior
exposure or knowledge to goal-modeling (or i*) we were able to start on equal footing
with both groups. The initial reaction upon introducing the i* to both groups was quite
similar; the participants seemed quite intrigued and curious about the i* models. Upon
a short introduction and walkthrough of the i* using only basic/limited notations, we
asked participants if they would find value in modeling the processes from a goal-
oriented view (in addition to the process-oriented view). While participants were
interested and gave feedback that they had a higher sense of appreciation of how their
work fit into the bigger picture and created a sense of pride/value for their work, they
also indicated through discussion that they did not see learning how to create the
models themselves would be an effective use of their time. The analogy was along the
lines of “I find value in seeing and reading the map, but I don’t find value in taking
the time to create the map myself”.
The discussion and feedback from this working session suggests that introducing
and sharing goal-oriented models could correlate with increased employee job
satisfaction and motivation.
Working Session 3: General Observations and Findings: Both groups were
provided with a hard copy of the To-Be i* SD model. All of the participants were
genuinely enthusiastic and interested in seeing the updated i* models. Soon after
receiving a copy, we observed the participants self-initiated in marking up the
diagrams to provide further feedback. We did not dig further or make anything of this
observation. However, we did observe that the participants seemed quite comfortable
with the i* models despite it being only the second time they had seen it. The As-Is i*
SR models were shown to the participant, but given the time limitations of booking
time with participants, we did not have the opportunity to take a deep-dive with the
SR models. There is an opportunity to expand this section of the study at a later time.
5 Concluding Remarks and Future Research
We described the design, approach and observations of two case studies where the
respective subject matter expert groups were provided with a pre-drafted/prepared
copy of a process-oriented model (DFD or PFD) and a goal-oriented model (i*). The
case study was set up to investigate the approach of having participants with subject
matter expertise comment on a diagram that was prepared for discussion rather than
putting the onus on them to create a model from scratch. Thus, they can focus on
discussing the expressiveness of the models and the information that the models
provide rather than the technical notation which the subject matter experts may not be
familiar with. We found that the group was more comfortable with participating once
it was established at the start of each session that there would be no expectations of
them to create their own models from scratch. There is an opportunity for future
research to continue/extend this study to have the subject matter experts/staff learn the
language and produce the models on their own so that we can compare the models
between models produced by the modeler-facilitator and staff for dimensions such as
accuracy, number/types of improvements identified, and/or the efficiency or cost-
benefit to the department of having inter-departmental staff produce the models.
One surprising finding was how much more engaged the group was with goal-
oriented modeling compared to that of process or data flow modeling. This suggests
that there could be a correlation with increased employee motivation when they are
able to see how their individual contributions fit into the bigger picture of the overall
process and its goals. We plan to do further research in this area in the future.
Another important hypothesis to investigate in future research would be to explore
the idea that having a modeler-facilitator who does not work in the department or
understand the operational processes in depth would lead to poor/inaccurate models,
i.e., modeling language and notations would be correct, but the modeled/represented
process itself could be inaccurate.
Our research on this project continues as we continue to work with the group and
gather feedback on the working sessions.
Acknowledgments. This research was supported by an organization in Toronto, ON.
6 References
1. Baltzan P, Philips A (2010) Business Driven Technology, Fourth Edition, McGraw Hill, pp.
571