<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>A Case Study on itsVALUE with an Exemplary Case</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Jim R. Wilken</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Lasse Guckenberger</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Nico Löser</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Jack D. Rittelmeyer</string-name>
          <email>jack.rittelmeyer@uni-rostock.de</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Henning D. Richter</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Kurt Sandkuhl</string-name>
          <email>kurt.sandkuhl@uni-rostock.de</email>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Institute of Computer Science, University of Rostock</institution>
          ,
          <addr-line>Albert-Einstein-Str. 22, 18057 Rostock</addr-line>
          ,
          <country country="DE">Germany</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>School of Engineering, Jönköping University</institution>
          ,
          <addr-line>Jönköping</addr-line>
          ,
          <country country="SE">Sweden</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>This paper presents a case study of the itsVALUE method, a value-oriented approach used to model and analyze IT services. The case study investigates the application of the itsVALUE method in the exemplary case of the Business Informatics Research Conference 2022 organisation. Finally, this paper also intends to discuss the method's efectiveness, which include pointing out some weaknesses and providing suggestions for improvement. A semi-structured interview with the conference organizers was conducted to collect the value streams and administrative processes for the event. The phases "Understand and Define Stakeholder Value" and "Understand and Define the Current Value Stream" of the itsVALUE method were applied, modelled using the ADOxx Modeller and afterwards assessed. The results indicate the strengths of the method regarding structured value stream analysis but also pinpoint challenges, especially with respect to the usability of the ADOxx Modeller. The framework for structuring the evaluation was done as a balanced scorecard: method documentation, process eficiency, and practical utility of results were some key evaluated aspects. The paper concludes by giving recommendations for refinement of the itsVALUE method, improvements regarding the ADOxx-based modelling tool, along with some propositions for future research by expanding and generalizing the evaluation results.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;itsVALUE</kwd>
        <kwd>value stream modeling</kwd>
        <kwd>value streams</kwd>
        <kwd>ADOxx</kwd>
        <kwd>ITIL</kwd>
        <kwd>case study</kwd>
        <kwd>semi-structured interview</kwd>
        <kwd>balanced scorecard</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        Under modern, competitive, and rapidly changing market conditions, each organization should regularly
reassess and enhance its processes in order to stay eficient and provide added value for all stakeholders.
Therefore, one of the aspects involved in optimization includes value modeling, reflecting a systematic
way of understanding how value flows within an organization. Value modeling is indispensable
in practice since it enables organizations to find and eradicate ineficiencies, align processes to the
stakeholder needs, and ensures every component of the workflow adds to value creation in general
[
        <xref ref-type="bibr" rid="ref1">1</xref>
        ] [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]. Traditional value modelling methods usually focus on economic transactions alone and do not
consider intangible benefits like stakeholder satisfaction, ease of use, or sustainability like e ³value for
example. [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]
      </p>
      <p>
        Due to this lack, the method itsVALUE was developed as an integrated framework for the modeling
and analysis of IT services from a value-oriented viewpoint. itsVALUE takes into consideration not
only explicit, measurable forms of value but also integrates implicit, perceived values relevant to a
variety of stakeholders. Such a holistic approach is required by modern IT services to satisfy complex
demands coming from a diverse set of stakeholders. Further, the itsVALUE approach consists of a
method framework, a notation, and a modeling tool called ADOxx Modeller, which allows putting the
method into practice to easily create and manage models. [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]
      </p>
      <p>
        A first case study and evaluation have already been carried out for the itsVALUE approach using
a real-life example with a company [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ]. Based on the results of this initial evaluation, itsVALUE did
undergo significant revisions. The main focus of this paper is the evaluation of the revised version of
the itsVALUE method by conducting a thorough case study on the application of the approach in the
organization of the Business Informatics Research (BIR) Conference 2022. An interview is conducted
with the conference organizers and the information from this interview is used to apply the itsVALUE
method. The evaluation of the method will be conducted and presented by using a Balanced Scorecard
approach. Ulf Seigerroth and Kurt Sandkuhl demonstrate in their article ’Method Engineering in
Information Systems Analysis and Design: A Balanced Scorecard Approach for Method Improvement’
how this approach can be used for method improvement [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]. Building on that, we can show those
aspects that need further refinement and give suggestions for improvements in respect to the practical
utility and usability of the method.
      </p>
    </sec>
    <sec id="sec-2">
      <title>2. Theoretical Background</title>
      <p>This chapter summarizes the most important theoretical background relevant for this research project.
First, the theory behind value streams and value stream modeling will be explained, followed by the
development of the itsVALUE method so far.</p>
      <sec id="sec-2-1">
        <title>2.1. Value Streams &amp; Value Stream Modeling</title>
        <p>
          Value streams encompass all activities necessary to develop and deliver a product or service to the
customer. They include both value-adding and non-value-adding business processes, providing a
comprehensive view of a company’s entire operations. By organizing a company’s activities into
value streams, a clear overview is created, showing what the company truly delivers. Services can
be analyzed for elements that obstruct the workflow and for activities that do not add value. Such
activities, often referred to as ’waste’, should be eliminated. [
          <xref ref-type="bibr" rid="ref7">7</xref>
          ] Value Stream Modeling has its origins
in Lean Production. Value Stream Mapping is a general approach used to visualize all necessary steps
within a value stream, aiming to optimize processes. This optimization can target areas such as lead
times, waiting times, or transportation times. [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ] There are various approaches to value stream analysis,
each with diferent focal points. Traditional approaches focus on the value being created, outlining the
necessary steps and processes to generate that value, with particular emphasis on the timing of each
process. [
          <xref ref-type="bibr" rid="ref8">8</xref>
          ] [
          <xref ref-type="bibr" rid="ref9">9</xref>
          ] [
          <xref ref-type="bibr" rid="ref10">10</xref>
          ] Modern approaches go beyond this perspective. They analyze, for example, the
lfow of information within the value stream and explore optimization opportunities. [
          <xref ref-type="bibr" rid="ref11">11</xref>
          ] Additionally,
aspects like the IT infrastructure used are given greater consideration in modern approaches. [
          <xref ref-type="bibr" rid="ref12">12</xref>
          ]
In summary, value streams consist of all activities necessary to create a service, and Value Stream
Modeling aims to represent these activities in order to optimize existing value streams.
        </p>
      </sec>
      <sec id="sec-2-2">
        <title>2.2. itsVALUE</title>
        <p>
          The itsVALUE method is an innovative approach to modeling and analyzing value creation in IT services.
It was developed to address existing gaps in value stream analysis and value modeling approaches,
which traditionally focus on optimizing process times and exchanging economic value. The method
builds on established concepts from service, value, and enterprise modeling and specifically integrates
the requirements and recommendations of ITIL 4, chosen for its practical relevance. The method
consists of several components that describe the steps for modeling, analyzing, and (re)designing
value streams in IT services. These components are summarized in a methodological framework that
considers both existing and new value streams. A key feature of the method is its iterative approach:
Four iterations are recommended to develop the necessary insights into the analyzed value stream,
especially when modeling a completely new service. This recommendation is based on the positive
results achieved by applying similar iterative approaches in other contexts like in [
          <xref ref-type="bibr" rid="ref13">13</xref>
          ] [
          <xref ref-type="bibr" rid="ref14">14</xref>
          ] [
          <xref ref-type="bibr" rid="ref15">15</xref>
          ] [
          <xref ref-type="bibr" rid="ref4">4</xref>
          ] [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ].
        </p>
        <p>
          Another central aspect of the itsVALUE method is the inclusion of stakeholder perspectives and
consideration of digitalization. The method aims to analyze and model the perceived benefits,
importance, and advantages of an IT service from the stakeholder’s viewpoint. This goes beyond traditional
approaches that mainly focus on economic transactions and process optimizations. Concepts such as
digitization rate, data availability and data usage are incorporated to enable a comprehensive evaluation
of value streams [
          <xref ref-type="bibr" rid="ref13">13</xref>
          ] [
          <xref ref-type="bibr" rid="ref14">14</xref>
          ] [
          <xref ref-type="bibr" rid="ref15">15</xref>
          ].
        </p>
        <p>
          The itsVALUE methodology integrates various established modeling approaches. These include the
Value Stream Method (VSM) and Value Stream Method and Notation (VSMN), which extend traditional
value stream modeling by considering both information processing and stakeholder perspectives.
Additionally, the IT-Self-Service Blueprint (IT S-SB) concept is used to integrate IT technologies into
traditional service blueprinting approaches. Another key element is the 4EM, a participatory enterprise
modeling procedure that supports stakeholder integration and provides concepts for modeling the
impact of context on value creation [
          <xref ref-type="bibr" rid="ref13">13</xref>
          ] [
          <xref ref-type="bibr" rid="ref14">14</xref>
          ], [
          <xref ref-type="bibr" rid="ref15">15</xref>
          ].
        </p>
        <p>
          The practical application of the itsVALUE method has been evaluated in one case study. This study
demonstrated that the method and its associated models are highly relevant for value-oriented IT
service modeling. Participants in these studies showed significant interest in the future application of
the method and the itsVALUE modeler. Experienced modelers found the itsVALUE notation supportive,
while less experienced users occasionally faced dificulties with its complexity. However, it was also
recognized that further iterations and adjustments of the method are needed to improve its applicability
and user-friendliness [
          <xref ref-type="bibr" rid="ref13">13</xref>
          ] [
          <xref ref-type="bibr" rid="ref14">14</xref>
          ] [
          <xref ref-type="bibr" rid="ref15">15</xref>
          ].
        </p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>3. Case Study</title>
      <p>The case study is divided into two main parts: the case study itself and its evaluation (section 4). The
aim of this approach is to identify potential weaknesses in the itsVALUE method and to evaluate the
entire method based on evaluation criteria. The conduction of the method as well as the evaluation are
carried out by three of the authors that have not worked with itsVALUE in any way before. Because of
that, they were able to use and experience the method like a normal real first time user.</p>
      <p>As an example case to apply the method, we chose the organization of a scientific conference (BIR
2022). We precisely chose this case to see if and how itsVALUE is applicable for non-profit oriented
processes as well as because we have the organizers of the conference in house, enabling us to collect
the necessary data for modeling in high detail. A semi-structured interview was conducted with the
organizers of this conference to collect the data. The interview is described in detail in section 3.1.
The diferent itsVALUE models were modeled based on the data collected from this interview using
the ADOxx Modeller. The results are described in section 3.3. However, during the application of the
itsVALUE method, only the phases ”Understand and Define Stakeholder Value” and ”Understand and
Define the Current Value Stream” regarding existing value streams were considered. The phase ”Create
a New Value Stream Supporting Stakeholder Value” was not conducted, because only existing value
streams should be considered and it is not necessary to create no processes for this case.</p>
      <sec id="sec-3-1">
        <title>3.1. Interview Planning &amp; Guide Development</title>
        <p>
          The central method of data collection in this study are semi-standardized interviews. This choice is
based on the methodological guidelines of Robert K. Yin, as described in his book ’Case Study Research:
Design and Methods’ [
          <xref ref-type="bibr" rid="ref16">16</xref>
          ], as well as on the concepts of Kallio et al. [
          <xref ref-type="bibr" rid="ref17">17</xref>
          ]. Semi-standardized interviews
make it possible to collect detailed information about the experiences, opinions and attitudes of the
experts interviewed. The interview guide, additionally based on the recommendations of Gläser et al.
[
          <xref ref-type="bibr" rid="ref18">18</xref>
          ], allows for both open-ended and targeted questions tailored to the specific objectives of the study.
This type of interview ofers the flexibility to adapt to the dynamics of the conversation while ensuring
that all relevant topics are taken into account.
        </p>
        <sec id="sec-3-1-1">
          <title>3.1.1. Development of the interview guide</title>
          <p>
            Following the recommendations of Gläser et al., the interview guide was designed to allow for open
discussion, but also to include targeted questions geared towards the specific research objectives. This
methodological approach promotes efective communication, takes into account the cultural context of
the interviewees and avoids the uncritical adoption of subjective theories as facts. The development of
the guide followed a structured five-phase approach that includes the design, development, piloting,
implementation and evaluation of it [
            <xref ref-type="bibr" rid="ref18">18</xref>
            ] [
            <xref ref-type="bibr" rid="ref17">17</xref>
            ]:
          </p>
          <p>Phase 1: Establishing the basic premises and objectives
In the first phase, a "deep understanding of BIR Conference 2022 organizers’ perspectives on their daily
interactions with the administrative and organizational processes of the conference" was identified as
the main objective of the guide. Another focus was on analyzing cost structures and value streams of
the conference. The aim of the interview was to document the experiences, challenges and needs of the
organizers and to identify the diferent value streams of the conference.</p>
          <p>Phase 2: Integration and analysis of existing knowledge
The second phase focused on the integration of existing knowledge and an in-depth literature review to
refine the content and methodological direction of the guide. The aim was to create a sound basis for
the development of the guide, including specific questions that encourage open answers. This approach
should provide deep insights into the experiences of the organizers and the diferent aspects of the
value streams.</p>
          <p>Phase 3: Development of the structured interview guide
In the third phase, the preliminary interview guide was created, which is divided into several key areas:
Introduction, project identification and definition, data collection and analysis, evaluation of benefits,
detailed process steps, registration and venue, cost analysis, evaluation of the conference, general
questions, final questions and a debriefing.</p>
          <p>Phase 4: Piloting and ethical considerations
During the piloting phase of the interview guide, particular attention was paid to ethical aspects in
order to comply with research standards. This included maintaining the confidentiality of the experts
and obtaining consent for the use of data. The pilot test was conducted with two individuals who had
relevant expertise but were not part of the main study sample. The first participant was a retired expert
in event planning with over 30 years of experience. Their extensive background in managing complex
projects provided valuable insights into the practical aspects of the interview. The second participant
was an academic with experience in qualitative research methodologies, allowing us to evaluate the
guide from both practical and theoretical perspectives. Several specific adjustments were made to the
interview guide based on the feedback from the pilot test:
• Flow of the Interview: The sequence of questions within each section was adjusted to improve
the logical flow and the rapport-building process with interviewees. In particular, we moved
questions regarding the project scope earlier in the interview (e.g., shifting the question on
external service providers) to provide more context before discussing the stakeholders’ roles.
• Duration of the Interview: It became apparent that the original guide could extend the interview
beyond the planned 80 minutes. To address this, several questions were condensed or merged.
For instance, in the section on cost analysis, the questions regarding distribution of the total
budget and cost distribution over the lifetime were combined into a single, streamlined query,
which helped reduce redundancy without sacrificing the depth of the responses.</p>
          <p>
            These refinements, guided by the pilot test as recommended by Yin [
            <xref ref-type="bibr" rid="ref16">16</xref>
            ], ensured that the interview
guide was both efective in capturing the required data and manageable for the participants.
          </p>
          <p>Phase 5: Finalizing and conducting the interviews
In the final phase, the final interview guide was carefully prepared for conducting the planned interview.
The interview, which lasted around 80 minutes, made a significant contribution to supporting the
objectives of the study and creating a sound data basis for the subsequent analysis.</p>
        </sec>
        <sec id="sec-3-1-2">
          <title>3.1.2. Selection of experts</title>
          <p>
            The selection of experts is crucial for the quality and type of information to be collected in an interview.
Suitable interview partners were identified on the basis of several criteria, but most importantly of the
direct involvement of the individuals in these administrative and organizational processes of the BIR
conference 2022. This means that the selection of experts is mostly based on convenience sampling,
because they were easy to access and fulfilled the criteria. The availability and willingness of potential
interviewees and their ability to provide accurate and relevant information were also taken into account
to ensure the depth and quality of the data collected [
            <xref ref-type="bibr" rid="ref18">18</xref>
            ].
          </p>
        </sec>
      </sec>
      <sec id="sec-3-2">
        <title>3.2. The Semi-structured Interview Guideline</title>
        <p>The interview guideline is systematically structured into nine key sections, each designed to guide
the interviewer through distinct areas relevant to the research objectives, focusing on BIR 2022. This
approach ensures both comprehensive data collection and suficient flexibility to explore participants’
individual experiences and perspectives. Each section is timed and organized as follows:
• Introduction (3 min): The interview begins with a brief introduction, outlining the purpose of
the research and explaining the significance of the interview. Inter-viewees are informed about
how their data will be used, and consent for recording is obtained. A transparent and open
atmosphere is created by allowing interviewees to ask questions or request breaks at any time.
• Project Identification and Definition (22 min): This is the longest section, where interviewees
discuss the key objectives and scope of the BIR Conference. It begins with questions about the
conference’s main goals and how they align with the university’s broader strategic goals. The
scope of the project is explored, including roles, stakeholders, external service providers, and
the timeline. Specific focus is placed on stakeholders’ involvement and expectations, and the
organizers’ roles and values are analyzed in depth.
• Data Collection and Analysis (10 min): This section examines the financial aspects of the BIR
Conference. It includes questions about initial, ongoing, and hidden costs, as well as the expected
benefits of the event. The goal is to understand both the tangible and intangible outcomes of the
conference and how these benefits will be quantified.
• Evaluation of the Benefits (10 min): The interview shifts towards assessing how the benefits
of the BIR Conference are measured and evaluated. This section includes questions about the
methods used to quantify the benefits, historical benchmarks, and how strategic advantages are
evaluated, particularly in relation to the university’s long-term goals.
• Detailed Process Steps (10 min): The focus here is on the detailed steps of organizing
the conference, from planning and registration to peer review processes and session
planning. Interviewees are asked to provide a thorough description of the processes involved,
including the management of paper submissions and reviews, to capture the operational workflow.
• Registration and Venue (4 min): This shorter section looks at the logistical aspects of the
conference, such as registration fees and venue arrangements, including any special agreements
with local accommodation providers.
• Cost Analysis (5 min): Cost-related questions are explored in greater depth, focusing on the
distribution of the conference budget, cost reserves, and the tech-nical infrastructure required to
support the event. Specific attention is paid to contingency planning for technical failures.
• Evaluation of the Conference (8 min): The interviewees are asked to provide insights into the
post-event evaluation processes, including the documentation of outcomes, the structuring of
results, and recommendations for future actions. The aim is to capture how the success of the
conference will be assessed and what improvements can be made.
• General Questions (8 min): This final section wraps up with broader reflections, including
lessons learned from previous IT projects and the identification of best practices. Success criteria
for the BIR Conference are also discussed, along with continuous improvement mechanisms for
future events.</p>
      </sec>
      <sec id="sec-3-3">
        <title>3.3. itsVALUE Modeling</title>
        <p>For modeling, we focused on one key value stream of the case study: the conference paper submission
process. This process was selected as it provided the richest information from the interviews and
therefore allowed for accurate modeling. The paper submission process is central to the case study, as
it involves a variety of stakeholders and interactions. The detailed information gathered during the
interviews provides a solid foundation for modeling and analyzing this process. Furthermore, various
stakeholders were identified during the paper submission process:
• Scientists: The authors who submit their work to the conference.
• Host (University of Rostock): The institution that may act as host or co-organizer.
• Participants: The conference attendees who will benefit from the work presented.
• Review committee: The reviewers who evaluate the submitted papers.
• Conference organizers: The individuals or organizations who plan and run the conference.
• External service providers: These include, for example, hotels and catering companies that
provide services for the conference.</p>
        <p>After identifying the stakeholders, the diferent values for these groups were determined. These
values were then modeled in VPM for each stakeholder and a VSB. In addition to modelling the value
streams, an SVM was created that visualizes the relationships and values for all the stakeholders. This
map provides an overview of the interests and added value that each stakeholder derives from the paper
submission process. Because of page restrictions, no models are shown here, but are available upon
request.</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Evaluation</title>
      <p>After finishing the application of the itsVALUE method by creating the recommended models, the
method should be evaluated. In the first step of the evaluation, evaluation criteria were defined (section
4.1) and then applied. The results of the evaluation will be described in section 4.2.</p>
      <p>
        A Balanced Scorecard approach was chosen for the evaluation. The article [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ] describes in detail the
process of developing a Balanced Scorecard and its use as a tool for optimizing specific methods. Four
perspectives relevant to the corresponding method were identified for the Balanced Scorecard. Criteria
were defined for each perspective, related to the previously identified goals. Various indicators were
then defined and described for each criterion, explaining how they can be measured. The Balanced
Scorecard was developed for the "Method for Information Demand Analysis" and demonstrates its
application for various user groups [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ]. Furthermore, the transferability of the approach to other
methods was investigated, using the 4EM method as an example [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ]. However, only two of the
formerly four perspectives were considered, as the other two perspectives were not applicable. Based
on the experiences gained, the authors provided recommendations for indicators that can be used
with other methods [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]. These two perspectives, namely the quality of the method documentation
and process eficiency, were also considered for the Balanced Scorecard in this work. Additional
perspectives stem from the already conducted case study on the evaluation of itsVALUE by Henning D.
Richter and Birger Lantow. The evaluation was published in two separate articles. The first article
focuses on the method and the associated results [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ], while the second article addresses the notation,
the tool, and the conducted modeling session [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ].
      </p>
      <p>Based on the already conducted evaluation, the method results, the ADOxx modeler, and the notation
were also included as respective perspectives in the Balanced Scorecard. Thus, the Balanced Scorecard
developed for this case study comprises the following perspectives:
• Quality of the method documentation
• Eficiency of the process
• Practicality of the method results
• Perceived benefit of the modeling tool
• Understandability of the notation</p>
      <p>For each of these perspectives, various sub-goals and additional research questions were formulated.
To make these sub-goals measurable, corresponding indicators were defined for each sub-goal. In the
evaluation, each indicator is assigned a value justified by the detailed answering of the associated
research question. This helps the reader understand how the writers came up with the indicators. The
evaluation questions, sub-goals, and indicators for the individual focal points are presented in the
following section 4.1.</p>
      <p>It should be emphasized that the values were selected based on the subjective perception of the
authors of this paper. They were established during a joint discussion and thoroughly debated among the
authors. It is important to emphasize that the values are heavily influenced by our personal assessments
as well as the challenges we encountered.</p>
      <sec id="sec-4-1">
        <title>4.1. Evaluation Criteria</title>
        <p>As explained previously, we identified five focal points for the evaluation and determined appropriate
evaluation criteria for each. These evaluation criteria will be presented in detail in this section. Each
table contains an abbreviation for the sub-goal (e.g., D.1, D.2.1), the corresponding sub-goal, the
associated research question, and the respective indicator.</p>
        <p>The first table presents the selected evaluation criteria for the quality of method documentation. In
total, five sub-goals have been identified:</p>
        <p>Table 2 shows the evaluation criteria for the focus on process eficiency. We considered 6 sub-goals
and formulated the corresponding evaluation questions. The sub-goal P.2 is further divided into the
phases of the method to clarify the time required for each respective phase.</p>
        <p>For the focus on methodological results, five sub-goals were considered. The sub-goal M.1 was
divided into three subordinate goals corresponding to the three models. This is shown in Table 3.</p>
        <p>For the focus area ’Perceived Benefits of the ADOXX Modeler’, four sub-goals have been identified.
These sub-goals are listed in Table 4.</p>
        <p>For the focus ’Clarity of the notation’, four sub-goals have been identified (see Table 5).</p>
      </sec>
      <sec id="sec-4-2">
        <title>4.2. Evaluation Results</title>
        <p>In this section, the results of the evaluation are presented, categorized by the respective focus areas
from the quality of method documentation (Section 4.2.1), the process eficiency (Section 4.2.2), the
practicality of method results (Section 4.2.3), over the perceived usability of the ADOxx modeller
(Section 4.2.4) to the notation (Section 4.2.5).</p>
        <sec id="sec-4-2-1">
          <title>4.2.1. Quality of Method Documentation</title>
          <p>The review of the method documentation, including the application manual and the Installation Guide,
took three hours. The application manual comprises 23 pages, while the Installation Guide is relatively
brief. The documentation was generally perceived as complete. The application manual provides a
compact overview of the notation and the corresponding metamodels, and in its second part, it details
the itsVALUE method, examining each phase individually. Installation procedures are described in the
Installation Guide available on the OMiLAB Community platform.</p>
          <p>The content was deemed accurate, with no errors found in the method section. The Application
Manual efectively and clearly explains the notation, metamodels, and models, without going into
excessive detail. The table of Visual Representations ofers an excellent overview of the available class
relations and their associated visualizations. The ’Important Notes on the ADOxx Modeller’ may not
be particularly clear for newcomers to the method. To enhance understanding, these notes could be
improved with the inclusion of relevant screenshots or more detailed descriptions.</p>
          <p>The second part of the Application Manual is very clear and comprehensible. The exemplary
ifctional case is well-chosen, and it is particularly helpful that the method is described step-by-step
through this case. This approach allows newcomers to quickly engage with the method and understand
it progressively through a concrete example. Additionally, the language used in the explanations
is suficiently simple to introduce even non-experts to the method efectively. The selected graphics
efectively guide the user through the process of the method. However, the explanation of the calculation
of the S and H scores could be improved for greater clarity. The Application Manual would be more
useful with detailed information and screenshots from the ADOxx modeller.</p>
          <p>The installation guide requires significant improvement to enhance its clarity and comprehensibility.
Currently, it is disorganized and presents numerous ambiguities to the reader. Firstly, there is an
error in the initial paragraph, rendering the sentence incomprehensible. The reader is left to infer the
intended message.The guide proceeds with a step-by-step explanation for downloading and installing
the environment. However, there are critical issues with these instructions. It would be beneficial
to provide a direct link to the specified modeling toolkit, as the current description does not clearly
identify which toolkit is being referred to. Additionally, the guide references a tutorial on the adoxx.org
website. Unfortunately, the specified tutorial cannot be found on the website. To address this, including
a direct link to the intended tutorial or ofering a more precise description would be advantageous.A
screenshot of the installation guide is attached.What the authors of this paper additionally wish for is
the integration of a modeled case study into the ADOxx Modeller as an example. This would allow
users to familiarize themselves with the corresponding functions in the Modeller. Furthermore, it
would be desirable to include a more complex case study in the documentation to demonstrate how the
various models would appear in in a larger and more detailed context. Due to many issues with the
tool, it would also be beneficial to maintain a public database of known errors and their corresponding
solutions. Thus, the following two deficiencies have been identified: "Better description of the use of
the ADOxx Modeller" and "unclear and dificult-to-understand installation guide". The scorecard with
the corresponding evaluations is shown in Table 7.</p>
        </sec>
        <sec id="sec-4-2-2">
          <title>4.2.2. Process Eficiency</title>
          <p>The entire process took 28 hours. Table 8 shows how the efort is distributed across the diferent sections
of the method. The ’Understand and Define Stakeholder Value’ section took 6.5 hours. The second part,
’Understand and Define the Current Value Stream,’ required 16 hours. It should be noted that most of
this time was spent calculating the KPIs, which had to be done manually due to issues with the ADOxx
Modeller. These problems are discussed in more detail in Section 5. Without this additional efort, i.e.,
with automatic calculation by the tool, the time required would have been significantly less. The efort
for the third step, ’Enhance Stakeholder Value: Define the Future Value Stream,’ was about 5 hours. With
the successful calculation of the KPIs, the efort required for the process was considered satisfactory.
The process was considered efective because the various phases and resulting models created a clear
and understandable structure. This structure allows the modeler to gain a comprehensive overview
of the entire process. Each phase contributes to capturing and optimizing the value streams in the
chosen case study. By the end of the process, all value streams deemed essential by the modeler are fully
identified and optimized. The process was perceived as correct.The scorecard with the corresponding
evaluations is shown in Table 9.</p>
        </sec>
        <sec id="sec-4-2-3">
          <title>4.2.3. Practicality of the Method Results</title>
          <p>The VPM was considered extremely helpful in identifying the baseline and the values of the stakeholders.
However, it is dificult to estimate the level of detail at which the corresponding values should be
identified and incorporated into the model without making the model overly complex and losing its
significance. Additionally, it was challenging to determine which stakeholders are relevant enough to
include in the analysis.</p>
          <p>After the second phase, the VPM can quickly become confusing, as all identified components of the
value stream must be mapped with the value perceptions. The SVM was also very helpful as it clearly
represents all stakeholders, their values, and the relationships and dependencies among them. The
relation class “Afects” does not really provide valuable information as it is theoretically applicable to
all objects. Distinguishing between “Contradicts” and “Hinders” was also often not easy.</p>
          <p>The VSB provides a precise overview of the individual steps within a value stream, clearly representing
the sequence of process steps. This helps in understanding process flows and identifying fail points.
However, it was challenging to choose an appropriate granularity for the process. The “Environments”
can be efectively used to group activities and diferent actors, helping to keep the model organized.</p>
          <p>The PSM clearly illustrates how the supporting components support the activities and processes.
However, the definition of a supporting component could be clearer. It should be more precisely defined
when a component qualifies as a supporting component. Additionally, the distinction between the PSM
and the VSB could be more clearly highlighted.</p>
          <p>The support and hindrance factors S and H are theoretically very meaningful as they help analyze
the efectiveness and eficiency of the value stream and identify targeted improvement measures. In
practical application, however, it has been shown that dificulties in determining and calculating the
metrics impair their significance. One challenge is determining the probabilities of occurrence, as these
can only be based on estimates without precise data. It is also essential that the automatic calculation
of the KPIs works, as manual calculation would significantly reduce the usefulness of the KPIs in the
practical modeling process.</p>
          <p>The method itself was considered complete, understandable, and correct. However, the models
can become very confusing in complex case studies. Additionally, the information required for the
individual models is substantial. Many pieces of information have to be estimated because it would be
too time-consuming to survey all stakeholders, or the stakeholders would also have to estimate this
information. This leads to the information being less authentic and potentially incorrectly estimated.</p>
          <p>The scorecard with the corresponding evaluations is shown in Table 10.</p>
        </sec>
        <sec id="sec-4-2-4">
          <title>4.2.4. Perceived usability of the ADOxx Modeller</title>
          <p>The ADOxx Modeller encountered numerous dificulties during the modeling process. The first problems
arose during the installation. Only one out of three people was able to successfully install the software.
For the other two individuals, installation was not possible due to an ”SQL database” error. Despite
several attempts, the error could not be resolved. Nevertheless, the ADOxx Modeller was installed on
one system. However, further dificulties arose while working with the tool. One problem was that the
names of the components could not be changed, only the descriptions could be adjusted. This could lead
to problems when viewing the models, as the components retain their default names and are therefore
dificult to identify. Additionally, the automatic calculation of KPIs was not possible, requiring manual
calculations. Manual calculation is very time-consuming and could only be partially completed in our
case. The links between the models, which should have been set automatically, also did not work in
this case. Overall, the user interface could be made more intuitive to facilitate ease of use.</p>
          <p>All these small problems led to significant limitations in the application of the method. In this specific
case, it is unclear whether the problems arose due to user errors or if they are general issues with the
ADOxx Modeller. However, it can be generally concluded that the user-friendliness of the modeling
tool has significant room for improvement.</p>
          <p>In conclusion, it should also be mentioned that when all functions work properly, the ADOxx Modeller
is a very useful tool for the method. Particularly, the automatic calculation of KPIs would be a great
help in applying the method. The tool is theoretically complete, as it covers all phases of the itsVALUE
method and can also calculate the corresponding KPIs. All relations and components of the method are
also present. The accuracy could only be assessed with dificulty, as the corresponding functions could
not be fully utilized in this case. The comprehensibility is improvable, as many errors and questions
arose during use, and, as described above, the operation could be made more intuitive. The results in
form of the scorecard are shown in Table 11.</p>
        </sec>
        <sec id="sec-4-2-5">
          <title>4.2.5. Notation</title>
          <p>The notation could be made more understandable, especially regarding the visual representations.
The "Constraint" class is shown as a white box on a white background, making it hard to distinguish.
Many of the classes are diferentiated only by color, which might be confusing for viewers with little
experience with the method. To make the classes clearer, it would be helpful to add symbols to the
rectangles. For example, a computer symbol could be used for IT systems.</p>
          <p>Another critical point is the color scale used for the ”Relevant Factor” and”Current Performance
Level” Green stands for ”High” and Blue for ”Outstanding” which is not intuitive. Additionally, the
symbols for the ”Relevant Factor” and ”Current Performance Level” are too small, they would be better
if they were larger. The symbols for events (Start, Common, End) and the labels on the arrows of the
relation classes could also be clearer.</p>
          <p>Overall, the notation is understandable after some practice. It is correct, with no obvious errors. The
notation is complete, and all models and components of the method are well represented. Because of
that, the scores rank from middle to high for this are (see Table 12).</p>
        </sec>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>5. Limitations</title>
      <p>This study has several limitations that should be considered for a comprehensive evaluation. These
limitations pertain to both the execution of the case study and the evaluation process.</p>
      <p>The case example was not optimally chosen, as it involved a process that was too unstandardized and
extensive. As a result, modeling was only possible at a high level of abstraction. Furthermore, it should
be noted that we lack expertise in organizing scientific conferences such as BIR. Our knowledge about
the conference and its procedures is based solely on publicly available information and an interview we
conducted. The interview lasted only 1.5 hours and was our main source of information. Consequently,
data collection was relatively challenging and required many assumptions, which may not always
accurately reflect actual conditions. Therefore, we decided to consider only the submission of the
conference paper as a value stream in order to limit the scope of the case study. A validation of these
assumptions with experts and stakeholders did not take place. Another limiting factor was that the
modeling tool, due to previously mentioned issues, could only be used by one person, which afected the
duration and quality of the modeling. Several functions of the tool were not usable, so the KPIs could
only be calculated manually or not at all. Finally, the section Create a New Value Stream Supporting
Stakeholder Value was not conducted, as our case study focused on existing value streams.</p>
      <p>
        The evaluation had several limitations. Firstly, we are not method experts and are new to the applied
method. A method expert would likely evaluate the method diferently, as they would prioritize diferent
aspects. Our time for the evaluation and the conduct of the case study was limited due to various
factors. Additionally, the evaluation was influenced by issues with the ADOxx Modeller. These technical
dificulties could have impacted our impressions of the method, even though we tried to disregard
them. The chosen case example also influenced the evaluation. With a more intuitive case example, the
evaluation might have turned out diferently. The evaluation is solely based on the experiences of the
three authors, who jointly modeled a case example. The applied balanced scorecard method is explicitly
recommended for methods that are used by many practitioners and applied in numerous use cases.[
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]
A separation between the persons who create and evaluate the assessment and those who evaluate the
method could not be maintained for various reasons. The rating using a score of 1-5 based on only
one evaluation has very limited significance. To reduce subjectivity, it would have been important to
have the evaluation conducted by multiple individuals. We tried to enhance our rating with a textual
description to better explain the scores. Due to these limitations, the evaluation can only provide a
limited implication for the evaluation of the method and does not possess strong evidence. It could
be argued that another limitation would be that end-users were not involved in the evaluation of the
modeling technique, but we argue that the three researchers who applied the technique can be seen
like end-users, because they had not got any previous experience with itsVALUE before like a new
end-user would have. However, it is a limitation that the balanced scorecard approach does not address
the distinction between modeling experts and end-users.
      </p>
    </sec>
    <sec id="sec-6">
      <title>6. Summary &amp; Outlook</title>
      <p>The aim of this case study was to conduct an evaluation of the itsVALUE method. The purpose of this
evaluation was to assess the method from the user’s perspective, identify weaknesses, and suggest
potential improvements. These insights are intended to contribute to the further development and
optimization of the method. For this purpose, a case example was selected and the method was applied
to it. The chosen case example was the organization of the BIR, a scientific conference. The information
for the case example was gathered through an interview with the conference organizers. The evaluation
shows that various aspects have potential for improvement. The methodology documentation was
generally rated positively, but a more precise description of the installation process would be desirable
to enhance the installation guide. Despite this limitation, the comprehensibility of the installation guide
must be emphasized, as it clearly and vividly explains the method. The process of the method was
assessed as eficient and understandable, with the diferent phases building logically and coherently
upon one another. The various models provide a good overview and serve their purpose in many cases.
However, there is a risk that these models may become confusing in complex scenarios. The indicators
could only be evaluated theoretically, as automatic calculation was not possible. The biggest weakness
of the itsVALUE approach is the ADOxx modeller. Many dificulties were encountered while using the
modeller, indicating that user-friendliness can be improved. This is particularly unfortunate, as the
functions ofered by the ADOxx modeller are very practical and useful. The notation is generally good
but could be made more understandable, especially in its visual representation. It is important to consider
the limitations associated with both the case study and the evaluation. For our case study, the primary
constraints were the complexity of the chosen scenario and the issues encountered with the ADOxx
Modeller. These factors should be taken into account in the assessment. It would have maybe been
more straightforward to select a more clearly defined case with monetary value streams. The evaluation
has several limitations that need to be mentioned: The results were obtained solely by us, representing
a very small user base. Furthermore, the outcomes are heavily influenced by our subjective perceptions,
specific issues, and the chosen case study. Overall, the results provide only limited implications for the
assessment of the method and do not constitute strong evidence. For future research, several fields of
action arise from this. Firstly, the evaluation should be conducted with a larger number of users to
obtain a broader data base and to verify whether the results of our evaluation can be generalized to
other users. In this context, it would also be sensible to select various case studies to exclude potential
influences of specific case studies on the evaluation. The results from this extended evaluation could
then be compared with the results of our initial evaluation. Furthermore, it is advisable to integrate the
weaknesses and improvement suggestions we identified into the method and subsequently re-evaluate
using our evaluation criteria to assess the impact of the measures. Additionally, it would also be
conceivable to conduct an evaluation focusing on specific aspects, such as documentation, to examine
particular elements in more detail.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          <article-title>[1] AXELOS, ITIL 4 Managing Professional Create, Deliver and Support, The Stationery Ofice</article-title>
          , London,
          <year>2019</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          <article-title>[2] AXELOS, ITIL Foundation, ITIL 4 edn, The Stationery Ofice</article-title>
          , London,
          <year>2019</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>H.</given-names>
            <surname>Richter</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Lantow</surname>
          </string-name>
          ,
          <article-title>It-service value modeling: a systematic literature analysis</article-title>
          ,
          <source>in: International Conference on Business Information Systems</source>
          , Springer,
          <year>2021</year>
          , pp.
          <fpage>267</fpage>
          -
          <lpage>278</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>H.</given-names>
            <surname>Richter</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Lantow</surname>
          </string-name>
          , itsvalue
          <article-title>- a method supporting value-oriented itsm</article-title>
          , in: R. A.
          <string-name>
            <surname>Buchmann</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          <string-name>
            <surname>Polini</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          <string-name>
            <surname>Johansson</surname>
          </string-name>
          , D. Karagiannis (Eds.),
          <source>Perspectives in Business Informatics Research</source>
          , Springer International Publishing, Cham,
          <year>2021</year>
          , pp.
          <fpage>133</fpage>
          -
          <lpage>149</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>H. D.</given-names>
            <surname>Richter</surname>
          </string-name>
          ,
          <article-title>A case study on itsvalue to evaluate its method, notation and adoxx modeller</article-title>
          , in: E¯. Nazaruka,
          <string-name>
            <given-names>K.</given-names>
            <surname>Sandkuhl</surname>
          </string-name>
          , U. Seigerroth (Eds.),
          <source>Perspectives in Business Informatics Research</source>
          , Springer International Publishing, Cham,
          <year>2022</year>
          , pp.
          <fpage>69</fpage>
          -
          <lpage>85</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>K.</given-names>
            <surname>Sandkuhl</surname>
          </string-name>
          , U. Seigerroth,
          <article-title>Method engineering in information systems analysis and design: a balanced scorecard approach for method improvement</article-title>
          ,
          <source>Software and Systems Modeling</source>
          <volume>18</volume>
          (
          <year>2019</year>
          )
          <fpage>1833</fpage>
          -
          <lpage>1857</lpage>
          . doi:
          <volume>10</volume>
          .1007/s10270-018-0692-3.
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>A.-A.</given-names>
            <surname>Dadashnejad</surname>
          </string-name>
          ,
          <string-name>
            <surname>C.</surname>
          </string-name>
          <article-title>Valmohammadi, Investigating the efect of value stream mapping on operational losses: a case study</article-title>
          ,
          <source>Journal of engineering, design and technology 16</source>
          (
          <year>2018</year>
          )
          <fpage>478</fpage>
          -
          <lpage>500</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>A. M.</given-names>
            <surname>Faisal</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Ravi</surname>
          </string-name>
          ,
          <article-title>Simulation modeling and analysis for the implementation of total quality management using value stream mapping in labour-intensive small and medium-sized enterprises</article-title>
          ,
          <source>in: 2018 2nd International Conference on Trends in Electronics and Informatics (ICOEI)</source>
          , IEEE,
          <year>2018</year>
          , pp.
          <fpage>1442</fpage>
          -
          <lpage>1444</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>A. M.</given-names>
            <surname>Faisal</surname>
          </string-name>
          ,
          <article-title>Predictive simulation modeling and analytics of value stream mapping for the implementation of lean manufacturing: a case study of small and medium-sized enterprises (smes)</article-title>
          ,
          <source>in: 2018 Second International Conference on Intelligent Computing and Control Systems (ICICCS)</source>
          , IEEE,
          <year>2018</year>
          , pp.
          <fpage>582</fpage>
          -
          <lpage>585</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>A.</given-names>
            <surname>Azizi</surname>
          </string-name>
          , et al.,
          <article-title>Designing a future value stream mapping to reduce lead time using smed-a case study</article-title>
          ,
          <source>Procedia Manufacturing</source>
          <volume>2</volume>
          (
          <year>2015</year>
          )
          <fpage>153</fpage>
          -
          <lpage>158</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>L.</given-names>
            <surname>Hartmann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Meudt</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Seifermann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Metternich</surname>
          </string-name>
          ,
          <article-title>Value stream method 4.0: holistic method to analyse and design value streams in the digital age</article-title>
          ,
          <source>Procedia cirp 78</source>
          (
          <year>2018</year>
          )
          <fpage>249</fpage>
          -
          <lpage>254</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>M.</given-names>
            <surname>Lewin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Voigtländer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Fay</surname>
          </string-name>
          ,
          <article-title>Method for process modelling and analysis with regard to the requirements of industry 4.0: An extension of the value stream method</article-title>
          ,
          <source>in: IECON 2017-43rd Annual conference of the IEEE industrial Electronics Society</source>
          , IEEE,
          <year>2017</year>
          , pp.
          <fpage>3957</fpage>
          -
          <lpage>3962</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <surname>B. J. D. K. Robert Andrei</surname>
            <given-names>Buchmann</given-names>
          </string-name>
          , Andrea Polini,
          <source>Perspectives in Business Informatics Research</source>
          , 1 ed., Springer Cham, Cham,
          <year>2021</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <surname>U. S. E¯rika</surname>
            <given-names>Nazaruka</given-names>
          </string-name>
          , Kurt Sandkuhl,
          <source>Perspectives in Business Informatics Research</source>
          , 1 ed., Springer Cham, Cham,
          <year>2022</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <surname>K. H. W. U. Dimitris Karagiannis</surname>
            ,
            <given-names>Moonkun</given-names>
          </string-name>
          <string-name>
            <surname>Lee</surname>
          </string-name>
          ,
          <source>Domain-Specific Conceptual Modeling</source>
          , 1 ed., Springer Cham, Cham,
          <year>2022</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>R. K.</given-names>
            <surname>Yin</surname>
          </string-name>
          ,
          <source>Case Study Research: Design and Methods</source>
          , 5th ed.,
          <source>SAGE Publications</source>
          ,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <given-names>H.</given-names>
            <surname>Kallio</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Åstedt Kurki</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Pietilä</surname>
          </string-name>
          ,
          <article-title>Systematic methodological review: developing a framework for a qualitative semi-structured interview guide</article-title>
          ,
          <source>Journal of Advanced Nursing</source>
          <volume>72</volume>
          (
          <year>2016</year>
          )
          <fpage>2954</fpage>
          -
          <lpage>2965</lpage>
          . doi:
          <volume>10</volume>
          .1111/jan.13031.
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>J.</given-names>
            <surname>Gläser</surname>
          </string-name>
          , G. Laudel,
          <article-title>Experteninterviews und qualitative Inhaltsanalyse als Instrumente rekonstruierender Untersuchungen, 3 ed</article-title>
          .,
          <source>VS Verlag für Sozialwissenschaften, Wiesbaden</source>
          ,
          <year>2009</year>
          . Lektorat: Frank Engelhardt.
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <given-names>M.</given-names>
            <surname>Lundqvist</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Sandkuhl</surname>
          </string-name>
          , E. Holmqvist, U. Seigerroth,
          <article-title>Method for information demand analysis: Ida user guide and handbook</article-title>
          , Abgerufen von Jönköping University website,
          <year>2011</year>
          . URL: https: //urn.kb.se/resolve?urn=urn:nbn:se:hj:
          <fpage>diva</fpage>
          -
          <lpage>36787</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>K.</given-names>
            <surname>Sandkuhl</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Wißotzki</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Stirna</surname>
          </string-name>
          ,
          <source>Die 4EM-Methode im Überblick</source>
          , Springer Berlin Heidelberg, Berlin, Heidelberg,
          <year>2013</year>
          , pp.
          <fpage>75</fpage>
          -
          <lpage>87</lpage>
          . URL: https://doi.org/10.1007/978-3-
          <fpage>642</fpage>
          -31093-
          <issue>5</issue>
          _6. doi:
          <volume>10</volume>
          . 1007/978-3-
          <fpage>642</fpage>
          -31093-
          <issue>5</issue>
          _
          <fpage>6</fpage>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>