=Paper=
{{Paper
|id=Vol-3154/short8
|storemode=property
|title=Investigating Hybrid Site Visits as a Methodology for Multi-Stakeholder Engagement in Contextual Requirements Analysis
|pdfUrl=https://ceur-ws.org/Vol-3154/short8.pdf
|volume=Vol-3154
|authors=Peter Fröhlich,Jelena Rosic,Matthias Neubauer,Josué Manuel Rivera Velázquez,Matthias Baldauf,Lisa Diamond
|dblpUrl=https://dblp.org/rec/conf/chi/0003RNVBD22
}}
==Investigating Hybrid Site Visits as a Methodology for Multi-Stakeholder Engagement in Contextual Requirements Analysis==
https://ceur-ws.org/Vol-3154/short8.pdf
Investigating Hybrid Site Visits as a Methodology for
Multi-Stakeholder Engagement in Contextual
Requirements Analysis
Peter Fröhlich1 , Jelena Rosic1 , Matthias Neubauer2 , Josué Manuel Rivera Velázquez3 ,
Matthias Baldauf4 and Lisa Diamond1
1
AIT Austrian Institute of Technology, Center for Technology Experience, Giefinggasse 4, 1120 Vienna, Austria
2
University of Applied Sciences, Logistikum – Department of Logistics, Wehrgrabengasse 1-3, 4400 Steyr, Austria
3
Cerema, Project team STI, 1 avenue du Colonel Roche, Toulouse 31400, France
4
Eastern Switzerland University of Applied Sciences, IPM Institute for Information and Process Management,
Rosenbergstrasse 59, 9001 St.Gallen, Switzerland
Abstract
Engaging stakeholders in capturing contextual requirements for automated transport systems can
be a challenging task, especially when they are locally distributed and limitations such as traveling
restriction hinder personal presence. This paper presents experiences from hybrid site visits applied in
contextual inquiry studies related to automated ground transport systems. The results rely on in-depth
interviews and a follow-up survey with organizers and participants of the hybrid site visits. Initial
findings reveal that physical participation is the preferred mode of attendance for contextual inquiry.
Online participation may complement physical observations. However, the interaction between remote
and on-site participants needs to be well facilitated, e.g. by own remote facilitator taking care of questions
in the online chat.
Keywords
Requirements analysis, computer-supported collaborative work, contextual inquiry
1. Introduction
Capturing user requirements and understanding the context of use represents a main challenge
for ubiquitous computing systems [1]. The ever-increasing complexity of sociotechnical systems
makes this task even more challenging, especially when it comes to envisioning new paradigms
of operation and work in industry [2]. Some of the most significant challenges in this respect
need to be faced when integrating automated road transport in logistics work processes, due to
the sheer complexity and demanding timescales of logistics workflows, the large number of
involved actors, and the unclear perspectives of automated vehicles as an emerging technology.
A large number of methodologies for capturing requirements have been proposed, reaching
from traditional requirements engineering [3, 4, 5, 6], sociotechnical theory, technology accep-
tance modelling [5], and contextual requirements research in Human-Computer Interaction
[7, 8]. A key principle common to all this research is to engage a wide spectrum of stakeholders
who will be affected by the system or who should be using it in the future. Also, in line with
AutomationXP22: Engaging with Automation, CHI'22, April 30, 2022, New Orleans, LA
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
the contextual requirements and design paradigm [8], situating future stakeholders and users
in the actual and expected usage context is always preferable.
When dealing with automated transport logistics, such contextual investigations can be
realized by so-called site visits. Site visits allow stakeholders to experience the logistics site,
to inspect current workflows as well as working conditions and propose future opportunities
for automation and assistance technologies. However, limitations such as (1) geographical
distribution, (2) time and budget constraints, (3) restrictions due to climate emissions, or (4)
special health-related restrictions (during the COVID-19 pandemics), may hinder the feasibility
of face-to-face site visits. This demands for adapted site visits formats, which allow for locally
distributed, collaborative requirements elicitation.
In this paper, we report experiences from a recent requirements gathering activity related to
future automated road transport logistics systems in the European research project AWARD [9],
and we share experiences gathered with “hybrid site visits”, as a means to gather contextual
requirements. In the next section, we briefly describe this hybrid site visit approach, which
consists of two different participation formats: (1) “physical” - participants are physically on-site
and (2) “virtual” – participants attended remotely via a teleconference setting. We then provide
the gathered results and methodological suggestions for future research in the area of locally
distributed, collaborative contextual requirements research.
2. METHOD
Requirements capturing was realized through a mixed-methods approach, which aimed at
gathering data through several channels, to gain deep insights into the workplace requirements
of future automated road transport logistics systems. Apart from the site visits discussed in this
paper, the data gathering methods included an online survey, remote in-depth interviews, and
an online international workshop (see [6] for further details).
At each of the four use case sites, visits were organized that maximized attendance in times
of COVID-caused travel restrictions, but that still provided information on the opportunities
and challenges. This was realized by hybrid site visits - a mixture of a workshop format and a
guided tour through the respective site, which could be attended physically and virtually.
For each use case, site visits in person as well as virtual tours provided an opportunity to get
familiar with the site and to discuss specific user requirements. In a virtual visit, participants
were able to live stream the use case site and operations via a Microsoft Teams teleconference.
Depending on COVID-19 restrictions at the time of the visit, sites were also able to provide
in-person attendance. Overall, four site visits, each dedicated to one use case (cf. Table 1), were
conducted. Apart from the port use case, all other use cases allowed on-site participation.
The representatives for each use case usually first presented the specifics of the project in
regard to their on-site operations. Afterwards, a guided tour was conducted. In the tour, aspects
such as the transport infrastructure, transport vehicles, loading/unloading points and workplaces
of relevant process participants were able to observe real live operations. Furthermore, site
visit participants could ask questions. This was done using a smartphone for live streaming and
when regulations allowed, a video material was recorded, anonymized if required and made
available for internal use among the project partners.
Table 1
Site Visits performed during the requirements gathering phase.
Participants Forklift Port Hub-to-Hub Airport
Physical 6 1 17 4
Virtual 21 40 22 27
3. RESULTS
After the site visits, dedicated interviews with the site visit organizers from each use case were
conducted. In the interviews, their experience of organizing, guiding the tour, interacting with
participants, and conclusions on the positive aspects as well as drawbacks were discussed. For
the participants in the site visits, both physical and virtual, a post attendance survey with
open-ended questions was implemented to assess their experience of both modes of the visits.
3.1. Perceived benefits of hybrid site visits
Reports from the site visit participants as well as from the organizers express expectations about
understanding of the on-site processes and operations in each mode of attendance (physical
and virtual) and sufficient learning about the complexity of the automated logistics tasks ahead.
In general, the opportunity offered in hybrid site visits to inspect the usage context in a flexible
setup, either on-site or remote, to get familiar with the local conditions and to ask questions,
was perceived as a much-appreciated format.
As expected, there were many positive comments about this format due to the opportunity
to make appointments for larger groups of people and to fit the site visit to individual working
schedules and companies’ policies, without being forced to travel. The main benefit experienced
within the physical mode was the possibility to grasp the actual size of surroundings, nature of
movement on the site, timings of operations, as well as vehicles and participants’ interactions.
However, outdoor operations are generally noisy and the hybrid form of the site visit allowed
physically present participants to use the online communication channel to hear what the guide
was talking about. With the hybrid possibility, no information was lost due to unexpected
or noisy conditions. This way, it was possible to observe the complete workflow without
interrupting it. Provided that technical broadcasting was good, virtual participants could feel
well-oriented about the important topics, get the impression of being emerged in the test site,
and explore some details of the procedures with expert explanations.
More particular comparisons and suggestions for how to improve future hybrid site visits are
further presented from the point of view of organizers and participants.
3.2. The organizers perspective
Among the organizers, the general agreement was expressed in favor of the physical mode of
the visit which offers the opportunity for more fruitful interaction in their opinion. In the light
of COVID restrictions, this attitude also relates to the more general working conditions where
meeting people in person for collaboration was missing since the start of the project.
From the practical perspective of the guide of the visit, one way to improve online interaction
in future situations would be for participants to instruct the guide on the site depending on
what they would like to see, and possibly a separate moderator for chat and questions. Learning
about the site of operations and the processes, as well as having good interactions was not
dependent solely on technical and practical issues. In fact, social gatherings in pre-COVID
times such as having a coffee together, meeting in the evening for a drink in the bar or in the
hotel are considered to deepen the talks and general understanding of the topics.
Animating people to be more talkative in online communication is needed generally, and
this is reflected in the small number of questions. Providing more structured possibilities for
discussion between physical and virtual participants in hybrid mode would be the way to tackle
these limitations. Also, involving online participants through the perspective of a specific
scenario beforehand and discussing it during the visit in smaller groups could help interaction.
On the part of the virtual mode of attendance, the video stream was the main source of
getting familiar with the surroundings and the site of operations. In the on-site events, the
participants were able to extend their visit, explore more, look around, and interact which
resulted in more questions asked and more engaging participation. This was compared to
previous online participation where it was somewhat expected that people would have a limited
range of questions because of restricted insight into surroundings and operations.
Technical aspects of video presentation where stream and recording would offer more detailed
visual perspectives and protocols with more opportunities for interaction and discussion would
significantly improve future virtual events as agreed by participants and organizers. Organizers
also suggest a pre-recording of the routes and trajectories on the site that would be available to
the participants at any time and allow further discussion and reflection.
The biggest benefit of the virtual site visits was the possibility to introduce the project and
operations to as many people as possible compared to physical participation which is limited
by the number of allowed participants, organizational matters and safety on the site. Yet,
organizers’ belief is that for the core partnership and people involved in the actual development,
physical attendance can offer a better understanding because these actors also have domain
knowledge that directs discussion about future operations on the site.
Regardless of the mode of presentation, organizers recognized additional challenges and
limitations when presenting the use case during the site visit. For example, in public areas apply
different regulations for traffic and participants than on closed sites. Presentation challenges
range from having noisy environments to complicated regulations preventing actual access to
an area or not being allowed to live stream or record the video. For example, not being able
to have a tour inside the factory prevented getting insight into interactions with the logistics
system, how participants in the process see themselves working with the system, etc.
3.3. The participants’ perspective
As respondents in the post attendance survey reported on both modes of attendance almost in
equal numbers. Since the majority had a chance to experience both modes, a direct comparison
could be drawn. The responses pointed to aspects such as possibilities for interaction, evaluation
of participant’s own role during the visit, how others’ presence affected the experience, technical
aspects, tools and protocols used, and guidance of the site visit tour.
Concerning the virtual mode of presentation, there were expectedly fewer possibilities
for interactions. The big number of participants in online meetings may have caused some
insecurity for the participants to pose a question and some comments in the chat were missed.
Physical presence benefited from having the opportunity to ask questions and reflect during
the demonstration. It was also easier to interact with other participants but more difficult to
communicate with the guide of the tour. The reports from participants typically considered
on-site tours better for accessing a larger portion of the site and easier for asking questions.
Issues that come with the virtual mode of attendance were mostly technical such as the camera
angle or connection problems. On the part of the tour guides and the information about the
on-site operations, participants appreciated good and instructive explanations and where the
situation allowed, detailed visual inspections and close-ups of the operations that helped them
to learn about the complete workflow and the infrastructure. In the future, for hybrid modes, a
written program or agenda was suggested to help participants understand better what they see
and in a proper context.
In addition to the limitations and benefits of the modes of attendance that depend on external
factors, the participants were also asked about their role as participants, whether they took
a more active or passive role and the reasons that contributed to it. For virtual participation,
participants reported being more passive and believed that on-site visits and physical attendance
would have provided a more active role. In online communication, being fully aware of other
participants and their interests can be lacking while physical interaction can more easily lead to
background discussions that are not on the agenda. Virtual visits could benefit from separating
participants into smaller dedicated groups instead of general big groups. Those who reported
an active role in the physical site visit, participated with questions and reflections when they
could not fully understand the presentation. However, the possibility for discussions in physical
interactions was affected by on-site noise that made it difficult to follow.
4. CONCLUSIONS
Engaging stakeholders in capturing contextual requirements can be a challenging task, especially
when the targeted application contexts and use case sites are difficult to reach. In this paper, we
systematically compare different setups of site visits. We could generally confirm that site visits
are perceived as a suitable way for gathering a detailed understanding and learning about the
complexity of investigated operations.
Providing hybrid formats for site visits was recognized both by organizers and participants
as an efficient means to get every project participant a common level of knowledge. Concluding
from our comparative analysis of two modes of hybrid site visits – physical vs virtual – we con-
firmed our expectation that there are many reasons for certain aspects of physical participation
to be the preferred mode of attendance. This is reflected in the need for spontaneous face to
face interactions and discussions as a way of learning about the project. The challenge of how
to serve both modes of attendance simultaneously in a hybrid setting is related to moderation
of discussions, technical obstacles and noise. Additionally, besides restrictions imposed by the
COVID-19 pandemic, physical attendance was recognized as challenging due to distance and
time spent on travel. Although more limited in terms of the number of participants, some
participants would still choose physical mode while for others the virtual visit was the only
possible attendance. In the future, recommendations for fostering both modes of attendance
envision a stepwise program that would provide common grounds and share information about
the context, and guidance for the attendees during the whole session.
Acknowledgments
This work is part of the project AWARD, which has received funding from the European
Union’s Horizon 2020 research and innovation programme under grant agreement No 101006817.
The content of this paper reflects only the author’s view. Neither the European Commission
nor CINEA is responsible for any use that may be made of the information it contains.
References
[1] M. Augstein, T. Neumayr, S. Pimminger, C. Ebner, J. Altmann, W. Kurschl, Contextual design
in industrial settings: Experiences and recommendations, in: ICEIS, 2018, pp. 429–440.
doi:1 0 . 5 2 2 0 / 0 0 0 6 6 7 4 9 0 4 2 9 0 4 4 0 .
[2] G. Schuh, A. Gützlaff, F. Sauermann, M. Schmidhuber, Socio-technical requirements for
production planning and control systems., in: Conference on Production Systems and
Logistics, CPSL 2020, Institutionelles Repositorium der Leibniz, Universität Hannover, 2022.
doi:1 0 . 1 5 4 8 8 / 9 6 5 8 .
[3] K. Czarnecki, Requirements engineering in the age of societal-scale cyber-physical systems:
The case of automated driving, in: 2018 IEEE 26th International Requirements Engineering
Conference (RE), IEEE, 2018, pp. 3–4.
[4] C. Sippl, F. Bock, C. Lauer, A. Heinz, T. Neumayer, R. German, Scenario-based systems
engineering: An approach towards automated driving function development, in: 2019 IEEE
International Systems Conference (SysCon), IEEE, 2019, pp. 1–8.
[5] M. Neubauer, O. Schauer, Human factors in the design of automated transport logistics, in:
N. A. Stanton (Ed.), Advances in Human Aspects of Transportation, Springer International
Publishing, Cham, 2018, pp. 1145–1156.
[6] P. Fröhlich, M. Gafert, L. Diamond, M. Reinthaler, M. Neubauer, H. Florian, S. Koskinen,
Towards a comprehensive understanding of stakeholder requirements for automated road
transport logistics, CEUR Workshop Proceedings 2905 (2021).
[7] B. V. Meldert, L. D. Boeck, Introducing autonomous vehicles in logistics: a review from
a broad perspective, Working Papers of Department of Decision Sciences and Infor-
mation Management, Leuven 543558, KU Leuven, Faculty of Economics and Business
(FEB), Department of Decision Sciences and Information Management, Leuven, 2016. URL:
https://ideas.repec.org/p/ete/kbiper/543558.html.
[8] K. Holtzblatt, H. Beyer, Contextual Design: Evolved, Synthesis Lectures on Human-Centered
Informatics, Morgan & Claypool Publishers, 2014.
[9] All Weather Autonomous Real logistics operations and Demonstrations (AWARD), 2022.
URL: https://award-h2020.eu, accessed: April, 2022.