=Paper=
{{Paper
|id=Vol-3328/workshop2
|storemode=property
|title=Re-envisioning Interaction in the (General) Aviation Cockpit through Tangibles
|pdfUrl=https://ceur-ws.org/Vol-3328/workshop2.pdf
|volume=Vol-3328
|authors=Sebastian S. Feger,Christopher Katins,Philippe Palanque,Thomas Kosch
|dblpUrl=https://dblp.org/rec/conf/etis/FegerKPK22
}}
==Re-envisioning Interaction in the (General) Aviation Cockpit through Tangibles==
https://ceur-ws.org/Vol-3328/workshop2.pdf
Re-envisioning Interaction in the (General) Aviation
Cockpit through Tangibles
Sebastian S. Feger1 , Christopher Katins2 , Philippe Palanque3 and Thomas Kosch2
1
LMU Munich, Geschwister-Scholl-Platz 1, 80539 Munich, Germany
2
Humboldt University of Berlin, Rudower Chaussee 25, 12489 Berlin-Adlershof, Germany
3
ICS-IRIT, Université Paul Sabatier - Toulouse III, 118, route de Narbonne, 31062 Toulouse, France
Abstract
Lowering pilot workload is a crucial challenge in general aviation (GA). GA refers to the branch of aviation
characterized by non-professional and non-commercial pilot operations. Today’s digital technology
support in these flight operations is mostly limited to consumer tablets and navigation apps. Researchers
and practitioners envision increased pilot support in the near future through mixed-reality applications
and additional machine-supported systems. Controlling, filtering, and manipulating this wealth of new
digital data introduced in the GA cockpit will require new tangible input and output techniques and
devices. While this is especially true in the mostly tight and shaky cockpits of small GA aircraft, we see
opportunities for the design of tangibles used in other branches of aviation and even in autonomous or
drone operations. To this end, we want to bring together researchers and practitioners concerned with
the design of future aviation cockpits and work on a strategy to develop dedicated scientific outlets for
HCI and tangible interaction research in (general) aviation.
Keywords
Pilot Workload, Tangible Cockpit Inputs and Outputs, Information Selection and Manipulation
1. Background
Airplanes are among the most common modes of everyday transportation in many countries
and rural areas, sometimes even preceding cars or trains [1]. Aviation characterized by non-
professional and non-commercial civilian pilots who fly to conduct routine tasks is referred
to as General Aviation (GA) [2]. In the US alone, around half a million GA pilots fly close to
200.000 planes and generate an annual economic activity surpassing 150 billion US dollars [3].
At the same time, and in stark contrast to commercial aviation, GA contributes a by far larger
number of accidents per flight than any other aviation branch [4]. This workshop is in line with
recent calls for Human-Computer Interaction (HCI) researchers and practitioners to study and
improve cockpit interaction in GA [5].
Public repositories like the US’ NTSB Aviation Accident Database [6] point to pilot error due
to inexperience, missing routine, misinterpretation, and heavy workload as some of the most
common causes for accidents [7]. This is not surprising, given the limited number of hours
Proceedings of ETIS 2022, November 7–10, 2022, Toulouse, France
$ sebastian.feger@ifi.lmu.de (S. S. Feger); christopher.katins@hu-berlin.de (C. Katins); palanque@irit.fr
(P. Palanque); thomas.kosch@hu-berlin.de (T. Kosch)
0000-0002-0287-0945 (S. S. Feger)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
Figure 1: A consumer tablet with a navigation app is usually the most advanced technology support
that general aviation pilots have.
pilots require to log for maintaining their flight permissions. In the US for example, GA pilots
are only required to take a biennial flight review to maintain their privileges [8]. They are
not obliged to spend more time in the air. In addition, GA planes, often single-piston aircraft
with two to six seats and an average age of 38 years [3] in the US, are still a major part of the
worldwide GA fleet. In many cases, a moving map on a tablet, as shown in Figure 1, strapped
to the pilot’s leg or attached inside the cockpit is the only digital flight support that GA pilots
get in their single-pilot operations [9]. Yet, the interaction with such devices in the GA aircraft
is heavily limited for several reasons: (1) the noisy environment renders voice interaction
ineffective; (2) shaky and turbulent operations in small aircraft impact the efficiency of touch
interactions; (3) consumer tablet screens are difficult to read in bright sunlight, especially in the
air; and (4) two-dimensional displays fail to render the full spectrum of information present in
the three-dimensional maneuvering space.
The first two barriers already represent motivations for designing and introducing tangible
artifacts in the cockpit to improve the interaction with information displays. This call for
tangibility is in line with previous research that envisioned tangible (airline) cockpits [10, 11, 12].
This interaction should allow the pilot to keep their attention focused on the outside environment
and must therefore present an intuitive way of data selection and manipulation. Further, we
envision the need for more sophisticated and intuitive tangible input devices to filter and
manipulate data in the three-dimensional space. In particular, mixed reality holds the potential
to effectively visualize information in the 3D environment. Figure 2 shows one of our pilot
studies with a mixed reality headset in a sophisticated GA aircraft simulator [13]. The pilot
received a number of new information regarding his environment that he would normally not
have direct access to. Those include the highlighting of airports and runways and information
related to surrounding traffic. To control the information and visualization load, our study
participants, who are all pilots, asked for easy-to-use filter mechanisms. Here, input options are
Figure 2: Picture of a pilot who tests the interaction with augmented en-route flight information
through a mixed reality headset in a high-fidelity Diamond DA 40 simulator.
even more limited: not only is voice interaction not an option, mixed reality headsets do not
offer a touchscreen, and gaze and gesture control have shown to be inefficient.
These are strong examples that motivate the importance of research in the direction of
tangible artifacts for control in the general aviation cockpit. Further, we recognize that research
around tangibility in the airplane cockpit go beyond this general aviation scope. Therefore, we
further welcome discussions around additional topics, including but not limited to: commercial
and military aviation, drone operations, flight training, air traffic control, and autonomous
aircraft operations.
This paper is structured as follows. While Sections 1-5 report mainly on the original workshop
proposal and setting [14], Section 6 presents concrete activities and findings from the workshop.
2. Organizers
Sebastian Feger, LMU Munich
# sebastian.feger@ifi.lmu.de
Sebastian Feger is a computer scientist and postdoctoral UX researcher at LMU Munich. He
holds a US private pilot license and pushes research on interaction design in and outside of the
cockpit to contribute to greater safety in general aviation.
Christopher Katins, HU Berlin
# christopher.katins@hu-berlin.de
Christopher Katins is a PhD student and HCI researcher at the Humboldt University of Berlin.
His research focuses on mixed reality and its applications in the cockpit of civil aviation aircraft.
He currently is a student pilot.
Philippe Palanque, Interactive Critical Systems Group at IRIT
# palanque@irit.fr
Philippe Palanque is a professor of computer science and head of the Reliability System and
Software research department at IRIT. Philippe researched multimodal interaction and pilot
support in civil and military aviation.
Thomas Kosch, HU Berlin
# thomas.kosch@hu-berlin.de
Thomas Kosch is a professor at the Department of Computer Science at the Humboldt University
of Berlin. He is researching AI-driven mixed-reality interfaces that extend and augment human
cognition.
3. Workshop Structure
We hosted a half-day (4h) workshop at ETIS’22:
• Welcome (10 minutes)
• Introduction round with a focus on individual backgrounds and expectations (40 minutes)
• Presentation and discussion of at least two demos (30 minutes). Participants can volunteer
to present a demo. The organizers will prepare at least two demos: (1) an assistive system
based on an e-ink display and (2) a control unit for augmented reality experiences.
• Short break (10 minutes).
• Discussion and presentation of several key questions: What are current research chal-
lenges for the design of tangible devices in (general) aviation? How can we address those
challenges? (30 minutes)
• Ideation and presentation of ideas for tangible devices in (general) aviation. (30 minutes)
• Prototyping of the ideas (1h). The participants will work in small groups.
• Prototype presentation (15 minutes)
• Discussing next steps and wrap-up (15 minutes)
4. Post-Workshop Plans
We want to establish a network of researchers and practitioners designing systems for (general)
aviation. As part of this, we discuss next steps in developing a stronger scientific agenda with
the participants. In particular, we envision the following post-workshop activities:
• Host curated conversations in a hybrid/remote setting that will introduce experts to
the wider HCI community.
• Create an international scientific outlet based on the workshop discussions and
most importantly the curated conversations. That scientific outlet (i.e., an international
symposium or a conference) will likely feature a broad spectrum of topics, ranging from
traditional to future aviation spaces and the role that HCI can play in transforming and
shaping them.
• Finally, we propose a special issue on HCI and (general) aviation in general, and tangible
devices in the cockpit more specifically, as a more immediate outlet for research.
Figure 3: A depiction of the complexity of airspace interpretation in 2D.
5. Workshop Setting
We hosted a half-day (4h) workshop. We used simple prototyping materials, including cardboard
and basic tools that allow to shape those materials.
6. Workshop Activities and Findings
In the first part of the workshop, we introduced the participants to challenges in GA and
presented taxonomies around fault classification in aviation. In the context of concrete examples,
we demonstrated the inadequacy of current GA tools in supporting pilots. For example, Figure 3
shows a common airspace map in the US. In this context, we discussed that a 2D representation
of 3D airspace requires heavy pilot workload while assessing flight constraints in complex
airspace. We identified Augmented Reality (AR) as a future key technology to provide (GA)
pilots with live information in their surrounding 3D airspace [15]. However, we also note
that controlling the AR experience will be difficult, as common interaction modalities (i.e.,
voice, gaze, and pointing) are largely inapplicable in the often small and noisy GA cockpits. In
response, we experimented with tangible control elements like the sliders depicted in Figure 4.
They make use of metaphors that are well-known to most GA pilots and foresee the control of
Figure 4: These control elements make use of metaphors that are well-known to GA pilots and allow to
control the information density in GA AR experiences.
Figure 5: A prototype of a tactile map that provides pilots with an overview of points and objects of
interest in their vicinity. This prototype foresees an interaction between these tactile items and the
information displayed in future AR tools.
the AR information density.
Finally, we envisioned and prototyped tangible control elements that could be used for even
more fine-grained control of AR experiences. Figure 5 shows a prototype of a tactile map that is
placed on top of a yoke. The tactile elements relate to objects (e.g., airplanes) and points (e.g.,
airports) of interest. The workshop participants envisioned that the pilots could not only get a
tangible overview of the surrounding through this map, but even select those items for further
highlighting and control in a connected AR experience.
References
[1] M. W. Babcock, The economic significance of general aviation airports in rural areas, in:
Journal of the Transportation Research Forum, volume 39, 2000, pp. 141–156.
[2] International Civil Aviation Organization, Operation of Aircraft. Part I, 2010.
[3] General Aviation Manufacturers Association, GAMA Databook 2019, 2019.
[4] Annual Summary of US Civil Aviation Accidents, 2019. URL: https://www.ntsb.gov/safety/
data/Pages/AviationDataStats2019.aspx.
[5] S. S. Feger, F. Ehrentraut, C. Katins, P. Palanque, T. Kosch, Hci for general aviation: Current
state and research challenges, Interactions 29 (2022) 60–65. URL: https://doi.org/10.1145/
3564040. doi:10.1145/3564040.
[6] NTSB Aviation Accident Database & Synopses, 2022. URL: https://www.ntsb.gov/Pages/
AviationQuery.aspx.
[7] D. Wiegmann, T. Faaborg, A. Boquet, C. Detwiler, K. Holcomb, S. Shappell, Human Error
and General Aviation Accidents: A Comprehensive, Fine-Grained Analysis Using HFACS
(2005).
[8] 14 CFR 61.56 – Flight review., 2022. URL: https://www.ecfr.gov/current/title-14/chapter-I/
subchapter-D/part-61/subpart-A/section-61.56.
[9] R. E. Joslin, Human Factors Hazards of IPADS In General Aviation Cockpits, in: Proceedings
of the Human Factors and Ergonomics Society Annual Meeting, volume 57, 2013, pp. 56–60.
doi:10.1177/1541931213571015.
[10] C. Letondal, J.-L. Vinot, S. Pauchet, C. Boussiron, S. Rey, V. Becquet, C. Lavenir, Being in the
sky: Framing tangible and embodied interaction for future airliner cockpits, in: Proceedings
of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction,
2018, pp. 656–666.
[11] J.-L. Vinot, C. Letondal, S. Pauchet, S. Chatty, Could tangibility improve the safety of
touch-based interaction? exploring a new physical design space for pilot-system interfaces,
in: Proceedings of the International Conference on Human-Computer Interaction in
Aerospace, 2016, pp. 1–8.
[12] J. A. L. d. Castillo, N. Couture, The aircraft of the future: towards the tangible cockpit,
in: Proceedings of the International Conference on Human-Computer Interaction in
Aerospace, 2016, pp. 1–8.
[13] C. Katins, Mixed Reality in General Aviation for Pilot Workload Support, 2022.
[14] S. S. Feger, C. Katins, P. Palanque, T. Kosch, Tangible interaction in the (general) aviation
cockpit (2022).
[15] C. Katins, S. S. Feger, T. Kosch, No margin for errors: Using extended reality to augment
users in safety-critical environments (2022).