What Can Be Learnt from Engineering Safety Critical Partly-Autonomous Systems when Engineering Recommender Systems Camille Fayollas Abstract Célia Martinie Human-Automation Design main target is to design sys- Philippe Palanque tems in such a way that the couple system operator per- Eric Barboni forms as efficiently as possible. Means for such designs in- ICS-IRIT, University of Toulouse, clude identifying functions (on the system side) and tasks 118, route de Narbonne, 31042 Toulouse, France (on the operator’s side) and balancing the allocation of {lastname}@irit.fr tasks and functions between operators and the systems being operated. Allocating functions to the most suitable ac- tor has been the early driver of function allocation [18]. The philosophy of recommender systems is that the system will Yannick Deleris provide a set of options for the users to select from. Such AIRBUS Operations, behavior can be easily connected to previous work on lev- 316 Route de Bayonne,3 1060 els of automation as defined by Sheridan [34] and lessons Toulouse, France can be drawn from putting together these two views. When yannick.deleris@airbus.com these automations (including the one of recommender sys- tems) are not adequately designed (or correctly understood by the operator), they may result in so called automation surprises [25, 32] that degrade, instead of enhance, the overall performance of the operations. This position paper identifies issues related to bringing recommender systems in the domain of safety critical interactive systems. While their advantages are clearly pointed out by their advocates, Copyright is held by the author/owner(s). limitations are usually hidden or overlooked. We present EICS’16, June 21-24, 2016, Bruxelles, Belgium. this argumentation in the case of the ECAM (Electronic Centralised Aircraft Monitor) of which some behavior could be considered as similar to the one of a recommender sys- 14 tem. We also highlight some engineering aspects of deploy- instead of enhance as expected, the overall performance ing recommender systems in the safety critical domain. of the operations and might lead to incidents or even ac- cidents [25]. The issue of usability of recommenders sys- Author Keywords tems has also been identified in the early days [35] even Automation, recommender systems, transparent automa- though only perceived as an interactive application being tion, operator tasks, performance used by users and not an interaction taking place with a partly-autonomous system. Work in that domain focuses ACM Classification Keywords on usefulness of the recommender systems and on their D.2.2 [Design Tools and Techniques]: Computer-aided soft- usability or user experience [20]. ware engineering (CASE); H.5.3 [Group and Organization Interfaces] This paper argues that having an automation-centered view on recommender systems helps to identify design issues Introduction related to their user interfaces and could inform design de- Human-Automation Design main target is to design sys- cisions and evaluation of these systems. Such a perspec- tems in such a way that the couple system operator per- tive could also help understanding issues that have to be forms as efficiently as possible. Allocating functions to the addressed prior to the deployment of such systems in the most suitable actor has been the early driver of function al- context of safety critical interactive systems. location as advocated by Fitts [18]. Such work is known as MABA-MABA (Men Are Better At âĂŞ Machine Are Bet- The next section proposes a short overview of the main ter At) where the underlying philosophy is that automate concepts related to automations and focuses on the human as many functions as possible was perceived as adequate aspects of automation. Then the paper positions recom- (see for instance [11]). This technology-centered view has mender systems within that context and highlights similari- led to unsafe and unusable systems [32] and it became ties with other systems as well as their specificities. The pa- clear that the design of usable partly-autonomous systems per then presents the case study of the ECAM (Electronic is a difficult task. Centralised Aircraft Monitor) and how this system relates to recommender systems. Last section lists design and engi- Recommender systems belong to this trend of work on au- neering issues related to the deployment of recommender tomation, even though that characteristic is not put forward systems in the area of safety critical interactive systems. or even ignored by their designers and promoters. When the word automation is connected to recommender systems Even though those levels can support the understanding it is usually for explaining that the recommender system is of automation they cannot be used as a mean for assess- evolving autonomously as in “automated collaborative filter- ing the automation of a system which has to be done at a ing” used for instance in GroupLens [21]. much finer grain i.e., “function” by “function”. However, if a detailed description of the “functions” is provided they make When automation is not adequately designed, or correctly it possible to support both the decision and the design pro- perceived and then understood by the operator, they may cess of migrating a function from the operator’s activity to result in so called automation surprises [33] that degrade, the system or vice versa. 15 As stated in [27], automated systems can operate at spe- cific levels within this continuum and automation can be applied not only to the output functions but also to input functions. Figure 2 presents the four-stage model of human information processing as introduced in [27]. HIGH 10. The computer decides every- thing, acts autonomously, ignoring the human 9. Informs the human only if it, the computer, decides to 8. Informs the human only if Figure 2: Simple four-stages model of human information asked, or processing. 7. Executes automatically, then necessarily informs the human, The first stage refers to the acquisition and recording of and multiple forms of information. The second one involves con- 6. Allows the human a restricted scious perception, and manipulation of processed and re- time to veto before automatic trieved information in working memory. The third stage is execution, or where decisions are accomplished by cognitive processes. 5. Executes that suggestion if the The last one contains the implementation of a response or human approves, or action consistent with decision made in the previous stage. 4. Suggests one alternative The first three stages in that model represent how the op- 3. Narrows the selection down to a erator processes the information that is rendered by the few, or interactive system. The last stage identifies the response 2. The computer offers a complete from the user that may correspond to providing input to the set of decision/action alternatives, controlled system by means of the interactive system (flow or of events from the user towards the controlled system. LOW 1. The computer offers no as- sistance: human must take all decisions and actions Figure 1: Levels of automation of decision and action selection from [34] and [27] Figure 3: Four classes of system functions (that can be automated) The model in Figure 2 (about human information process- ing) has a similar counterpart in system’s functions as shown in Figure 3. Each of these functions can be automated 16 to different degrees. For instance, the sensory process- to analyze the effect of the level of automation and empha- ing stage (in Figure 2) could be migrated to the informa- size the importance of a simulation framework to have a tion acquisition stage (in Figure 3) by developing hardware feedback on design choices before deploying the system. sensors. The second stage in the human model could be Finally, several techniques have been coined to provide automated by developing inferential algorithms (as for in- support for formal verification of human automation inter- stance in recommender systems). The third stage involves action [9], which aim at providing tools for checking confor- selection from several alternatives which can be easily im- mance between what the system has to perform, and what plemented with algorithms. The final stage called action the user is responsible for. For each of these techniques implementation refers to the execution of the choice. Au- and methods, human automation interaction is dealt with as tomation of this stage may involve different levels of ma- a whole and thus focusing on goal-related tasks. chine execution and could even replace physical effectors (e.g., hand or voice) of the operator [28]. The stages of hu- Recommender systems as man information processing as well as their corresponding partly-autonomous system classes of system functions are used to analyze and de- Recommender systems may be based on different ap- sign which tasks are performed by the human operator and proaches. They may implement content-based filtering, which functions are performed by the system (also called knowledge-based filtering, collaborative filtering or hybrid function allocation as defined in [15]). filtering [8]. We don’t describe here the various types of recommender systems in details and encourage the inter- Based on this theoretical framework, several techniques ested reader to see [8] for a very detailed and pedagogic and methods have been proposed to analyze, design and survey. We here focus on content-based approaches for evaluate human automation interaction. Proud et al. pro- recommender systems as it is a relevant filtering approach posed the LOA (Level Of Autonomy) Assessment Tool [30] for interactive cockpits. Figure 4 presents a typical architec- (based on a LOA Assessment Scale) which produces an- ture of a content-based recommender system. The system alytical summaries of the appropriate LOA for particular stores a set of items and each item is described accord- functions and has been applied to an Autonomous Flight ing to a set of attributes. In such systems the user is de- Management System. Cummings et al. [13] identified a re- scribed according to these attributes too, this description finement mechanism for the decision making step, to help being named user profile. According to the user profile and in deciding which one of the human or of the system should the attributes of the set of items, the systems proposes to perform a given decision task. More generally, techniques the user a set of recommendations. based on cognitive task analysis, such as the one proposed in [3], help in understanding precisely the different tasks Recommender Systems and Levels of Automation: that are actually performed by the human operator. Model According to the levels of automation presented in Fig- based approaches take advantage of task analysis and ure 1 recommender systems typically fall within levels 2 propose to systematically ensure consistency and coher- to 4 depending on the number of alternatives presented to ence between task models and system behavioral descrip- the user. Rules for designing autonomous systems would tion [22]. Johansson et al. [19] developed a simulation tool thus apply to recommender systems to avoid know issues 17 • Action implementation: this stage corresponds to the presentation on the user interface of the selected items to the operator. That presentation of informa- tion can be sometimes enriched with argumentation about the rationale for selecting items [12]. It is also important to note that, following that presentation of information, user interaction is usually available allow- ing users to browse the list of recommended items, to access more information about them and to select the desired one. Such operator behavior is taken ex- Figure 4: Content-based recommendation principle (from [8]) plicitly into account by the operator behavior model of Figure 2 and detailed below. such as automation surprises [32] and [25]. Issue of trans- Recommender Systems and operators behavior: As parency of automation has been also identified for recom- far as user activity is concerned the operator behavior de- mender systems [14] while a process and a notation to sys- scribed in Figure 2 can be refined for describing interaction tematically engineer transparency for partly-autonomous with a recommender system. systems was proposed in [7]. According to the four stages of human information process- Recommender Systems and systems functions: The ing proposed in Figure 2 it is easy to relate to the recom- recommender system behavior in Figure 4 and the classes mender system behavior: of system functions in Figure 3 can be related as follows: • Information acquisition: this stage corresponds to • Sensory processing: while interacting with the rec- the process in the recommender systems browsing ommender system this activity would consist in all the internal source of items being candidates for rec- operators’ information sensing both from the recom- ommendation. Information can be entirely stored at mender and from the system under use. Localiza- initialization or gathered during execution. tion of the information from the recommender system might deeply impact that sensing. • Information analysis: this stage corresponds to the recommender system process of correlating user • Perception/working memory: at this stage informa- profile with items description. tion from the recommender system will be integrated with the information presented by the system. It is • Decision and action selection: this stage corre- important to note that human errors such as interfer- sponds to the filtering process of the recommender ence, overshooting a stop rule... [31] and thus should system selecting amongst the list of candidate items be avoided (and is not possible detected, recovered and ranking them before presentation to the operator. or mitigated). 18 • Decision making: this corresponds to the decision (using the System Display (SD)); ii) the display of warnings of the operator for selecting one of the recommenda- about system failures and procedures that have to be com- tions presented by the recommender system (if more pleted by the pilot to process the detected warning (using than one is offered). the Warning Display (WD)) and iii) the production of aural and visual alerts (using several lights and loudspeakers in • Response selection: this is the actual selection of the cockpit). one of the recommender system recommendation. As stated above, that stage might involve additional The SD and WD are displayed, in the cockpit of the A380, cycles within this 4 stages model (while operators on two separated Display Units (DU). These two DUs are interact with the recommender systems e.g. browsing highlighted in Figure 5 and are part of the eight of DUs the recommendation or accessing more information composing the Control and Display System (CDS). The about a given recommendation). CDS is the interactive system in aircraft cockpits (flight decks) that offers various operational services of major im- portance for flight crew. It displays aircraft parameters, and The mapping between recommender system processes enables the flying crew to graphically interact with these and Parasuraman’s system functions as well as the map- parameters using a keyboard and a mouse (KCCU for Key- ping between activities done with recommender systems board and Cursor Control Unit) to control aircraft systems. and Parasuraman’s stages of human information process- ing, provides support to analyze the impact of the level of automation of the recommender system functions on the operators’ task. Illustrative example To exemplify the concept presented above in the context of a safety critical system, we present a case study from the avionics domain: the ECAM (Electronic Centralized Air- craft Monitor) in the Airbus family. The ECAM is a system that monitors aircraft systems (e.g., the engines) and re- lays to the pilots data about their state (e.g., if their use is limited due to a failure) and the procedures that have to be Figure 5: WD and SD in the cockpit of the A380 achieved by the pilots to recover from the failure. The ECAM system is composed of several systems. More As presented in Figure 6, if the ECAM has created, simulta- particularly, the Flight Warning System (FWS) is in charge neously, several warning messages, it sorts them, in order of the processing of data from the monitoring of the aircraft to obtain a display order, according to three inhibition mech- systems. This processing enables: i) the display of infor- anisms: mation about the status of the aircraft systems parameters 19 • Their priority: a priority is associated to each warning Figure 7 presents an example of the display of warning message; messages (one red warning and three amber warnings) and their associated recovery procedures on the ECAM. • FWS internal behavior: some warning messages In this example, the red warning (L1 in Figure 7) informs may be inhibited in case of presence of others warn- the pilot that the autopilot is not working anymore. There- ing messages (for instance, the “APU fault” warning fore, the first amber warning (L2 in Figure 7) informs the message is not displayed if the “APU fire” is already pilot that the auto-thrust is not working anymore. The cor- detected); responding recovery procedure (L3 in Figure 7) indicates • The current flight phase: some warning messages to the pilot that s/he has to take responsibility for the thrust are only displayed when the aircraft is in a given flight by moving the thrust levers. The second amber warning (L4 phase (for instance, flight management systems fail- and L5 in Figure 7) informs the pilot that the flight control ures are not displayed after landing). laws are not working anymore. The corresponding recovery procedures (L6 in Figure 7) indicates to the pilot that s/he has to take responsibility for the aircraft speed that must be under 0.82 MACH. Figure 6: Principle for the display of the warnings Therefore, the warning messages are displayed (in the pro- Figure 7: Example of the display of warning messages on the ECAM (from [10]) cessed display order) to the pilots, within the WD, with three different colors, representing their priority level: These warnings messages notification are similar to rec- ommendations in recommender systems (see, for instance, • Red warnings that require immediate actions from the the one presented in Figure 4) in the sense that the sys- pilots (e.g., the loss of an engine); tem sorts the warning messages and their associated re- • Amber warnings that require non-immediate actions covery procedures and proposes, to the pilots, an order from the pilots (e.g., a fault within the APU); for their treatment. In this example, the system indicates to the pilot that the auto-thrust management function is off • Green warnings that only require monitoring from the and indicates to the pilot that s/he shall manually move the pilots but do not present any danger. 20 thrust levers (lines 2 and 3 in Figure 4). Using Parasura- or safety critical domain. [14] presents the evaluation of man’s models, we can analyze that the system fall within a recommender system for single pilot operations but no level 4 of automation (Figure 1). In other cases, the sys- information is given about the design and development of tem may propose a list of prioritized alarms and recovery the underlying system and nor about its user interface and procedures, which make the system fall within level 3 of au- interaction techniques. tomation (Figure 1). As inferred in the Parasuraman’s level of automation, the more alternatives the system proposes, We have considered several engineering approaches to the less automated. examine these issues. First, the ICO user interface design techniques enables to develop usable and reliable interac- Design and engineering issues of recommender tion techniques [24]. Complemented with task modelling systems in safety critical domain (for describing operators goals and tasks to be performed This section tries to identify the potential of recommender to reach these goals), it can be used to analyze user’s task systems as well as the design issues related to their engi- w.r.t. system’s behavior [22]. At last, task models can also neering. be used to assess whether the user or the system should handle a particular task in a particular context [22]. All of The classification framework of recommender systems pro- these techniques aim at finding the optimal collaboration posed in [29] identifies multiple application domains where solution between the user and the system but were not ap- recommender systems have been deployed (as an excerpt plied with a recommender systems, even with the AMAN is presented in Figure 8). (Arrival Manager) advisory tool for air traffic control which could be also considered as a simple recommender sys- tem [23]. However these approaches do not deal with the possible dynamic change of behavior of the system, especially if it has machine learning capabilities (reinjecting operators’ se- lections in the items information). Additionally, considering that the safety-critical user interfaces require additional de- sign and development paths, we identified the following set of issues that must be considered if the system is (partly) autonomous: • What is usability of a recommender system in a criti- Figure 8: Classification framework of recommender systems cal context and how to evaluate it (as operators follow (from [29]) extensive training and have deep knowledge of the behavior of the supervised systems), It is interesting to note that none of them target at critical • How to guarantee the safety and dependability of the 21 possible interactions when browsing recommended systems and that existing classifications in that domain are items, applicable to recommender systems. • How to guarantee the safety and dependability of the We have shown on a simple example from the aviation do- underlying recommender system behavior, main that current systems exhibits some of the characteris- tics of recommender systems. We have also highlighted de- • How to analyze and prevent operators’ errors, sign and development issues that currently prevent recom- • How to assess and design responsibility, authority mender from being deployed in the context of safety critical and liability between the recommender system and systems but we have also highlighted some of the problems the operators (for instance in aircraft the entire au- to be addressed. thority belong to the captain and not to the first offi- Future work deals with the definition of engineering ap- cer), proaches for building reliable and fault-tolerant recom- • How to design and specify interaction techniques mender systems following what has been done in the past where autonomous behavior from the system inter- for interactive cockpit applications as presented in [16] fere with operator input (including the question on and [36]. It is important to note that trade-off between prop- how to model that formally [7]), erties (such as usability and dependability as presented in [17]) will also be present in the case of recommender • How to design interaction so that the operators can systems in safety critical applications. foresee the systems’ future steps and states and the impact of selecting one recommendation instead of References another one, [1] Johnny Accot, Stéphane Chatty, Sébastien Maury, and Philippe Palanque. 1997. Formal transducers: • How to design interactions when the automation (rec- models of devices and building bricks for the design ommendation) can fail and how to notify the operators of highly interactive systems. In Design, Specification about degradation of the recommender system (for and Verification of Interactive Systems’ 97. Springer, each of the stages in Figure 3), 143–159. • How to enhance and evaluate aspects of user experi- [2] Johnny Accot, Stéphane Chatty, and Philippe ence, while fulfilling the constraints of a safety-critical Palanque. 1996. A formal description of low level in- system which has to be secure, safe, reliable and us- teraction and its application to multimodal interactive able. systems. In Design, Specification and Verification of Interactive Systems’ 96. Springer, 92–104. [3] Julie A Adams, Curtis M Humphrey, Michael A Summary and Conclusion Goodrich, Joseph L Cooper, Bryan S Morse, Cameron This position paper proposed to consider recommender Engh, and Nathan Rasmussen. 2009. Cognitive systems as partly-autonomous systems. We have demon- task analysis for developing unmanned aerial vehi- strated that their behavior is similar to the ones of autonomous 22 cle wilderness search support. Journal of cognitive rep., République Française, Ministère de l’Ecologie, du engineering and decision making 3, 1 (2009), 1–26. Développement durable et de l’Énergie. [4] ARINC. 2002. ARINC 661 Cockpit Display System [11] Alphonse Chapanis. 1996. Human factors in systems Interfaces to User Systems. ARINC Specification 661. engineering. John Wiley & Sons, Inc. (2002). [12] Carlos Iván Chesnevar and Ana G Maguitman. 2004. [5] Rémi Bastide, Philippe Palanque, Duc-Hoa Le, Jaime Arguenet: An argument-based recommender system Muñoz, and others. 1998. Integrating rendering spec- for solving web search queries. In Intelligent Systems, ifications into a formalism for the design of interactive 2004. Proceedings. 2004 2nd International IEEE Con- systems. In Design, Specification and Verification of ference, Vol. 1. IEEE, 282–287. Interactive Systems’ 98. Springer, 171–190. [13] Mary L Cummings and Sylvain Bruni. 2009. Collabora- [6] Olivier Bau and Wendy E Mackay. 2008. OctoPocus: tive Human–Automation Decision Making. In Springer a dynamic guide for learning gesture-based command handbook of automation. Springer, 437–447. sets. In Proceedings of the 21st annual ACM sympo- [14] Arik-Quang V Dao, Kolina Koltai, Samantha D Cals, sium on User interface software and technology. ACM, Summer L Brandt, Joel Lachter, Michael Matessa, 37–46. David E Smith, Vernol Battiste, and Walter W Johnson. [7] R Bernhaupt, M Cronel, F Manciet, C Martinie, and P 2015. Evaluation of a recommender system for single Palanque. 2015. Transparent Automation for Assess- pilot operations. Procedia Manufacturing 3 (2015), ing and Designing better Interactions between Oper- 3070–3077. ators and Partly-Autonomous Interactive Systems. In [15] Andy Dearden, Michael Harrison, and Peter Wright. Proceedings of the 5th International Conference on 2000. Allocation of function: scenarios, context and Application and Theory of Automation in Command the economics of effort. International Journal of and Control Systems. ACM, 129–139. Human-Computer Studies 52, 2 (2000), 289–318. [8] Jesús Bobadilla, Fernando Ortega, Antonio Hernando, [16] Camille Fayollas, Jean-Charles Fabre, Philippe and Abraham Gutiérrez. 2013. Recommender systems Palanque, Eric Barboni, David Navarre, and Yannick survey. Knowledge-Based Systems 46 (2013), 109– Deleris. 2013. Interactive cockpits as critical applica- 132. tions: a model-based and a fault-tolerant approach. [9] Matthew L Bolton, Ellen J Bass, and Radu I Si- International Journal of Critical Computer-Based Sys- miniceanu. 2013. Using formal verification to evaluate tems 17 4, 3 (2013), 202–226. human-automation interaction: A review. IEEE Trans- [17] Camille Fayollas, Célia Martinie, P Palanque, Yannick actions on Systems, Man, and Cybernetics: Systems Deleris, J-C Fabre, and David Navarre. 2014. An Ap- 43, 3 (2013), 488–503. proach for Assessing the Impact of Dependability on [10] Bureau d’Enquêtes et d’Analyse. 2012. Rapport final- Usability: Application to Interactive Cockpits. In De- Accident survenu le 1er juin 2009 à l’Airbus A330-203 pendable Computing Conference (EDCC), 2014 Tenth immatriculé F-GZCP exploité par Air France vol AF European. IEEE, 198–209. 447 Rio de Janeiro-Paris. Technical Report. Tech. [18] Paul M Fitts. 1951. Human engineering for an effective air-navigation and traffic-control system. (1951). 23 [19] Björn Johansson, Åsa Fasth, Johan Stahre, Juhani Interaction (TOCHI) 16, 4 (2009), 18. Heilala, Swee Leong, Y Tina Lee, and Frank Riddick. [25] Everett Palmer. 1995. Oops, it didn’t arm’- A case 2009. Enabling flexible manufacturing systems by us- study of two automation surprises. In International ing level of automation as design parameter. In Winter Symposium on Aviation Psychology, 8 th, Columbus, Simulation Conference. Winter Simulation Conference, OH. 227–232. 2176–2184. [26] Raja Parasuraman and Victor Riley. 1997. Humans [20] Bart P Knijnenburg, Martijn C Willemsen, Zeno Gant- and automation: Use, misuse, disuse, abuse. Human ner, Hakan Soncu, and Chris Newell. 2012. Explaining Factors: The Journal of the Human Factors and Er- the user experience of recommender systems. User gonomics Society 39, 2 (1997), 230–253. Modeling and User-Adapted Interaction 22, 4-5 (2012), [27] Raja Parasuraman, Thomas B Sheridan, and Christo- 441–504. pher D Wickens. 2000. A model for types and levels of [21] Joseph A Konstan, Bradley N Miller, David Maltz, human interaction with automation. IEEE Transactions Jonathan L Herlocker, Lee R Gordon, and John Riedl. on systems, man, and cybernetics-Part A: Systems 1997. GroupLens: applying collaborative filtering to and Humans 30, 3 (2000), 286–297. Usenet news. Commun. ACM 40, 3 (1997), 77–87. [28] Raja Parasuraman and Christopher D Wickens. 2008. [22] Célia Martinie, Philippe Palanque, Eric Barboni, Marco Humans: Still vital after all these years of automation. Winckler, Martina Ragosta, Alberto Pasquini, and Human Factors: The Journal of the Human Factors Paola Lanzi. 2011. Formal tasks and systems mod- and Ergonomics Society 50, 3 (2008), 511–520. els as a tool for specifying and assessing automation [29] Deuk Hee Park, Hyea Kyeong Kim, Il Young Choi, and designs. In Proceedings of the 1st International Con- Jae Kyeong Kim. 2012. A literature review and classifi- ference on Application and Theory of Automation in cation of recommender systems research. Expert Sys- Command and Control Systems. IRIT Press, 50–59. tems with Applications 39, 11 (2012), 10059–10072. [23] Celia Martinie, Philippe Palanque, Alberto Pasquini, [30] Ryan W Proud, Jeremy J Hart, and Richard B Mrozin- Martina Ragosta, Eric Rigaud, and Sara Silvagni. ski. 2003. Methods for determining the level of au- 2012. Using complementary models-based ap- tonomy to design into a human spaceflight vehicle: a proaches for representing and analysing ATM systems’ function specific approach. Technical Report. DTIC variability. In Proceedings of the 2nd International Con- Document. ference on Application and Theory of Automation in [31] James Reason. 1990. Human error. Cambridge uni- Command and Control Systems. IRIT Press, 146– versity press. 157. [32] Nadine B Sarter, David D Woods, and Charles E [24] David Navarre, Philippe Palanque, Jean-Francois Billings. 1997. Automation surprises. Handbook of Ladry, and Eric Barboni. 2009. ICOs: A model-based human factors and ergonomics 2 (1997), 1926–1943. user interface description technique dedicated to in- [33] Thomas B Sheridan and Raja Parasuraman. 2005. teractive systems addressing usability, reliability and Human-automation interaction. Reviews of human scalability. ACM Transactions on Computer-Human factors and ergonomics 1, 1 (2005), 89–129. 24 [34] Thomas B Sheridan and William L Verplank. 1978. Hu- [36] A Tankeu-Choitat, David Navarre, P Palanque, Yannick man and computer control of undersea teleoperators. Deleris, Jean-Charles Fabre, and Camille Fayollas. Technical Report. DTIC Document. 2011. Self-checking components for dependable in- [35] Kirsten Swearingen and Rashmi Sinha. 2001. Be- teractive cockpits using formal description techniques. yond algorithms: An HCI perspective on recommender In Dependable Computing (PRDC), 2011 IEEE 17th systems. In ACM SIGIR 2001 Workshop on Recom- Pacific Rim International Symposium on. IEEE, 164– mender Systems, Vol. 13. Citeseer, 1–11. 173. 25