Re-envisioning Interaction in the (General) Aviation Cockpit through Tangibles

Abstract

Lowering pilot workload is a crucial challenge in general aviation (GA). GA refers to the branch of aviation characterized by non-professional and non-commercial pilot operations. Today's digital technology support in these flight operations is mostly limited to consumer tablets and navigation apps. Researchers and practitioners envision increased pilot support in the near future through mixed-reality applications and additional machine-supported systems. Controlling, filtering, and manipulating this wealth of new digital data introduced in the GA cockpit will require new tangible input and output techniques and devices. While this is especially true in the mostly tight and shaky cockpits of small GA aircraft, we see opportunities for the design of tangibles used in other branches of aviation and even in autonomous or drone operations. To this end, we want to bring together researchers and practitioners concerned with the design of future aviation cockpits and work on a strategy to develop dedicated scientific outlets for HCI and tangible interaction research in (general) aviation.

Keywords

• eol>Pilot Workload
• Tangible Cockpit Inputs and Outputs
• Information Selection and Manipulation

1. Background

Airplanes are among the most common modes of everyday transportation in many countries and rural areas, sometimes even preceding cars or trains [ 1 ]. Aviation characterized by nonprofessional and non-commercial civilian pilots who fly to conduct routine tasks is referred to as General Aviation (GA) [ 2 ]. In the US alone, around half a million GA pilots fly close to 200.000 planes and generate an annual economic activity surpassing 150 billion US dollars [ 3 ]. At the same time, and in stark contrast to commercial aviation, GA contributes a by far larger number of accidents per flight than any other aviation branch [ 4 ]. This workshop is in line with recent calls for Human-Computer Interaction (HCI) researchers and practitioners to study and improve cockpit interaction in GA [ 5 ].

Public repositories like the US’ NTSB Aviation Accident Database [ 6 ] point to pilot error due to inexperience, missing routine, misinterpretation, and heavy workload as some of the most common causes for accidents [ 7 ]. This is not surprising, given the limited number of hours pilots require to log for maintaining their flight permissions. In the US for example, GA pilots are only required to take a biennial flight review to maintain their privileges [ 8 ]. They are not obliged to spend more time in the air. In addition, GA planes, often single-piston aircraft with two to six seats and an average age of 38 years [ 3 ] in the US, are still a major part of the worldwide GA fleet. In many cases, a moving map on a tablet, as shown in Figure 1, strapped to the pilot’s leg or attached inside the cockpit is the only digital flight support that GA pilots get in their single-pilot operations [ 9 ]. Yet, the interaction with such devices in the GA aircraft is heavily limited for several reasons: (1) the noisy environment renders voice interaction inefective; (2) shaky and turbulent operations in small aircraft impact the eficiency of touch interactions; (3) consumer tablet screens are dificult to read in bright sunlight, especially in the air; and (4) two-dimensional displays fail to render the full spectrum of information present in the three-dimensional maneuvering space.

The first two barriers already represent motivations for designing and introducing tangible artifacts in the cockpit to improve the interaction with information displays. This call for tangibility is in line with previous research that envisioned tangible (airline) cockpits [ 10, 11, 12 ]. This interaction should allow the pilot to keep their attention focused on the outside environment and must therefore present an intuitive way of data selection and manipulation. Further, we envision the need for more sophisticated and intuitive tangible input devices to filter and manipulate data in the three-dimensional space. In particular, mixed reality holds the potential to efectively visualize information in the 3D environment. Figure 2 shows one of our pilot studies with a mixed reality headset in a sophisticated GA aircraft simulator [ 13 ]. The pilot received a number of new information regarding his environment that he would normally not have direct access to. Those include the highlighting of airports and runways and information related to surrounding trafic. To control the information and visualization load, our study participants, who are all pilots, asked for easy-to-use filter mechanisms. Here, input options are even more limited: not only is voice interaction not an option, mixed reality headsets do not ofer a touchscreen, and gaze and gesture control have shown to be ineficient.

These are strong examples that motivate the importance of research in the direction of tangible artifacts for control in the general aviation cockpit. Further, we recognize that research around tangibility in the airplane cockpit go beyond this general aviation scope. Therefore, we further welcome discussions around additional topics, including but not limited to: commercial and military aviation, drone operations, flight training, air trafic control, and autonomous aircraft operations.

This paper is structured as follows. While Sections 1-5 report mainly on the original workshop proposal and setting [ 14 ], Section 6 presents concrete activities and findings from the workshop.

2. Organizers

Sebastian Feger, LMU Munich

# [e-mail] Sebastian Feger is a computer scientist and postdoctoral UX researcher at LMU Munich. He holds a US private pilot license and pushes research on interaction design in and outside of the cockpit to contribute to greater safety in general aviation.

Christopher Katins, HU Berlin

# [e-mail] Christopher Katins is a PhD student and HCI researcher at the Humboldt University of Berlin. His research focuses on mixed reality and its applications in the cockpit of civil aviation aircraft. He currently is a student pilot.

Philippe Palanque, Interactive Critical Systems Group at IRIT

# [e-mail] Philippe Palanque is a professor of computer science and head of the Reliability System and Software research department at IRIT. Philippe researched multimodal interaction and pilot support in civil and military aviation.

Thomas Kosch, HU Berlin

# [e-mail] Thomas Kosch is a professor at the Department of Computer Science at the Humboldt University of Berlin. He is researching AI-driven mixed-reality interfaces that extend and augment human cognition.

3. Workshop Structure

We hosted a half-day (4h) workshop at ETIS’22: • Welcome (10 minutes) • Introduction round with a focus on individual backgrounds and expectations (40 minutes) • Presentation and discussion of at least two demos (30 minutes). Participants can volunteer to present a demo. The organizers will prepare at least two demos: (1) an assistive system based on an e-ink display and (2) a control unit for augmented reality experiences. • Short break (10 minutes). • Discussion and presentation of several key questions: What are current research challenges for the design of tangible devices in (general) aviation? How can we address those challenges? (30 minutes) • Ideation and presentation of ideas for tangible devices in (general) aviation. (30 minutes) • Prototyping of the ideas (1h). The participants will work in small groups. • Prototype presentation (15 minutes) • Discussing next steps and wrap-up (15 minutes)

4. Post-Workshop Plans

We want to establish a network of researchers and practitioners designing systems for (general) aviation. As part of this, we discuss next steps in developing a stronger scientific agenda with the participants. In particular, we envision the following post-workshop activities: • Host curated conversations in a hybrid/remote setting that will introduce experts to the wider HCI community. • Create an international scientific outlet based on the workshop discussions and most importantly the curated conversations. That scientific outlet (i.e., an international symposium or a conference) will likely feature a broad spectrum of topics, ranging from traditional to future aviation spaces and the role that HCI can play in transforming and shaping them. • Finally, we propose a special issue on HCI and (general) aviation in general, and tangible devices in the cockpit more specifically, as a more immediate outlet for research.

5. Workshop Setting

We hosted a half-day (4h) workshop. We used simple prototyping materials, including cardboard and basic tools that allow to shape those materials.

6. Workshop Activities and Findings

In the first part of the workshop, we introduced the participants to challenges in GA and presented taxonomies around fault classification in aviation. In the context of concrete examples, we demonstrated the inadequacy of current GA tools in supporting pilots. For example, Figure 3 shows a common airspace map in the US. In this context, we discussed that a 2D representation of 3D airspace requires heavy pilot workload while assessing flight constraints in complex airspace. We identified Augmented Reality (AR) as a future key technology to provide (GA) pilots with live information in their surrounding 3D airspace [ 15 ]. However, we also note that controlling the AR experience will be dificult, as common interaction modalities (i.e., voice, gaze, and pointing) are largely inapplicable in the often small and noisy GA cockpits. In response, we experimented with tangible control elements like the sliders depicted in Figure 4. They make use of metaphors that are well-known to most GA pilots and foresee the control of the AR information density.

Finally, we envisioned and prototyped tangible control elements that could be used for even more fine-grained control of AR experiences. Figure 5 shows a prototype of a tactile map that is placed on top of a yoke. The tactile elements relate to objects (e.g., airplanes) and points (e.g., airports) of interest. The workshop participants envisioned that the pilots could not only get a tangible overview of the surrounding through this map, but even select those items for further highlighting and control in a connected AR experience.

References

[1] M. W. Babcock , The economic significance of general aviation airports in rural areas , in: Journal of the Transportation Research Forum , volume 39 , 2000 , pp. 141 - 156 .

[2] International Civil Aviation Organization , Operation of Aircraft. Part I , 2010 .

[3] General Aviation Manufacturers Association , GAMA Databook 2019 , 2019 .

[4] Annual Summary of US Civil Aviation Accidents , 2019 . URL: https://www.ntsb.gov/safety/ data/Pages/AviationDataStats2019.aspx.

[5] S. S. Feger , F. Ehrentraut , C. Katins , P. Palanque , T. Kosch, Hci for general aviation: Current state and research challenges , Interactions 29 ( 2022 ) 60 - 65 . URL: https://doi.org/10.1145/ 3564040. doi: 10 .1145/3564040.

[6] NTSB Aviation Accident Database & Synopses, 2022 . URL: https://www.ntsb.gov/Pages/ AviationQuery.aspx.

[7] D. Wiegmann , T. Faaborg , A. Boquet , C. Detwiler , K. Holcomb , S. Shappell , Human Error and General Aviation Accidents: A Comprehensive, Fine-Grained Analysis Using HFACS ( 2005 ).

[8] 14 CFR 61 . 56 - Flight review., 2022 . URL: https://www.ecfr.gov/current/title-14/chapter-I/ subchapter-D/part-61/subpart-A/section-61. 56 .

[9] R. E. Joslin , Human Factors Hazards of IPADS In General Aviation Cockpits , in: Proceedings of the Human Factors and Ergonomics Society Annual Meeting , volume 57 , 2013 , pp. 56 - 60 . doi: 10 .1177/1541931213571015.

[10] C. Letondal , J.-L. Vinot , S. Pauchet , C. Boussiron , S. Rey , V. Becquet , C. Lavenir , Being in the sky: Framing tangible and embodied interaction for future airliner cockpits , in: Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction , 2018 , pp. 656 - 666 .

[11] J.-L. Vinot , C. Letondal , S. Pauchet , S. Chatty , Could tangibility improve the safety of touch-based interaction? exploring a new physical design space for pilot-system interfaces , in: Proceedings of the International Conference on Human-Computer Interaction in Aerospace , 2016 , pp. 1 - 8 .

[12] J. A. L. d. Castillo , N. Couture , The aircraft of the future: towards the tangible cockpit , in: Proceedings of the International Conference on Human-Computer Interaction in Aerospace , 2016 , pp. 1 - 8 .

[13] C. Katins , Mixed Reality in General Aviation for Pilot Workload Support, 2022 .

[14] S. S. Feger , C. Katins , P. Palanque , T. Kosch, Tangible interaction in the (general) aviation cockpit ( 2022 ).

[15] C. Katins , S. S. Feger , T. Kosch, No margin for errors: Using extended reality to augment users in safety-critical environments ( 2022 ).