Automation Space: Towards a Design Space for Everyday Automation Jessica Bongard Abstract ZHAW Zurich University of No longer only experts are confronted with (semi-)auto- Applied Sciences mated systems, yet automation has founds its way into our Zurich, Switzerland everyday lifes in various forms and applications. In this pa- jes.bongard@gmail.com per, we introduce our ongoing work towards a design space for “Everyday Automation” to uncover the dimensions of Matthias Baldauf respective approaches, identify research gaps and promis- FHS St.Gallen University of ing future applications as well as to allow for transferring Applied Sciences experiences and knowledge between different types of au- St.Gallen, Switzerland tomated systems. Based on a literature review, we derived matthias.baldauf@fhsg.ch first dimensions for such a dedicated design space, such as the domain, the task type, the type of user interaction, Peter Fröhlich and the automation level. For a visual presentation of this AIT Austrian Institute of “Automation Space”, we propose a so-called morphological Technology box which might provide a suitable tool for overviewing the Vienna, Austria diverse manifestations of automation in everyday life and for peter.froehlich@ait.ac.at supporting ideation of novel approaches. Author Keywords Automation; design space; automation domain; automation level. CCS Concepts •Human-centered computing → Human computer inter- action (HCI); ________________________________________________________ Workshop proceedings Automation Experience across Domains In conjunction with CHI'20, April 26th, 2020, Honolulu, HI, USA Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Website: http://everyday-automation.tech-experience.at Introduction dimensions identified so far as well as a promising visual- Whether a fully automatic vacuum cleaner in the living room ization approach. or a self-sufficient service for municipal information and ap- plications: automation appears in numerous forms in our Method everyday life and is constantly evolving. Everyday Automa- In order to determine this design space and the dimensions tion [5] is a very broad and complex topic, which is particu- of Everyday Automation, we started to conduct a literature larly driven by recent advances in Artificial Intelligence (AI) research. Used sources and search engines for scientific and “smart” devices at affordable prices. works include the ACM Digital Library, the IEEE Xplore Dig- ital Library, the AIS eLibrary, as well as Research Gate and Everyday automation can be understood as a union of the Google Scholar. At the center of this review are keywords definitions of automation and everyday life. It is a process and keyword combinations which were derived from contri- in which individual functions or entire activities are trans- butions to last year’s CHI workshop on Everyday Automa- ferred from humans to machines [15], and which focuses tion Experience [5] and included “everyday automation”, on a person in their immediate, everyday environment. The “smart technology”, “smart devices”, “everyday interaction”, immediate everyday environment of a person is determined “digital assistants”, “home automation” and “smart city”. The in particular by routines and habits, but also by mobility and search terms are expanded during search with newly ac- social interactions, for example. quired knowledge. The main inclusion criterion for a study was that the source must contain recent approaches and From a scientific perspective, a categorization of the nu- examples for everyday automation. merous appearances of automation in our everyday life is relevant in order to provide a comprehensive overview The analysis of the documents was done according to the and structure and to uncover possible gaps in research and Quantitative Content Analysis (cf. [16]). This method is con- promising future applications. By analyzing existing au- sidered to be particularly appropriate as the method aims at tomation approaches for everyday tasks and by identifying structuring certain themes and contents and filters out and potential variants, we strive to unfold the so-called “design summarizes aspects of the material. space” of everyday automation. Design spaces have a long history in HCI research. Examples include the work by Bux- Preliminary Dimensions ton who introduced a taxonomy of input devices [2] and the From this literature research, we identified and selected five work by Ballagas et al. [1] who presented a design space preliminary core dimensions for a design space of Every- for using smartphones for ubiquitous input. Other examples day Automation. In the following, we briefly introduce these include a design space for driver-based automotive user in- dimensions and present corresponding examples from liter- terfaces by Kern and Schmidt [12] and a design space for ature. interactive public displays by Müller et al. [19]. Presence of the System In the following, we introduce our ongoing work on creat- Everyday Automation applications can be differentiated ac- ing the Automation Space, a design space for Everyday cording to the presence of a physical system. Based on Automation. We report on the method, preliminary core the analysis of the examples, a division of the applications ple, an autonomous delivery droid (e.g., [10]) takes over into virtual and physical presence could be determined. the complete delivery of orders, while a sports wearable For example, virtual systems are smart assistance systems (e.g., [8]) only signals flow state feedback and recommen- in cars [11], digital representatives [17] or virtual reality dations for further activity to the wearer. indoor navigation systems [9]. Autonomous drones [18], fully-automated coffee makers [7] and automatic vacuum User Interaction cleaner robots [14] are examples of physical systems, i.e. Six different user interactions for Everyday Automation ap- physical representations of the automated object. plications were identified: stationary or mobile external device, hardware buttons, touch interface, hand gestures, Domain voice interface and eye gestures. Stationary or mobile ex- The following application domains of Everyday Automation ternal devices include in particular computers, tablets or were identified from the literature examples: Education, smartphones, as well as hardware controllers, cameras, Health and Sports, Shopping and Restaurant, Transporta- and wearables. Examples of the various interaction modal- tion, Home, Security and Government. Frequently, several ities include automated passport control in a stationary different areas of application are mentioned for the same sluice (e.g., [4]), autonomous drone control by hand ges- example. For instance, food recognition of smart refrig- tures (e.g., [18]), and automatic language translation via a erators (for automating ordering processes, e.g.) can be voice interface (e.g., [20]). used at home, but also in restaurants (e.g., [6]). Further- more, interaction with displays based on eye movement can Automation Level serve as a public information display in the museum, but According to Parasuraman et al. [21], the degree of au- can also be used as a game for waiting areas in the hospi- tomation is divided into three areas: fully manual, semi- tal (e.g., [24]). automated and fully automated. Where manual means that a task is carried out exclusively by humans and is Automated Task therefore only listed for the sake of completeness. Semi- Based on the analysis of the Everyday Automation exam- automated means that a task is carried out by combining ples, it was found that key words identified for the auto- the advantages of human skills with the advantages of the mated task are covered by the dimensions suggested by machine [13]. In an AR-based system helping patients to Parasuraman et al. [21]: Information acquisition, information test their blood at home, a combination of human action analysis, decision and action selection and action imple- and machine support takes place (e.g., [3]). Fully automatic mentation. While information acquisition describes purely means that a task is completed completely and exclusively sensory functions for capturing data from the environment, by the machine. At a sans-checkout grocery store such as information analysis deals with processing the captured Amazon Go, the scanning of the items and the payment data. The decision or action selection deals with the deriva- process are carried out completely automatically (e.g., [22]). tion of further action steps and the action implementation includes the actual execution of an action selection and usually replaces the hand or voice of a person. For exam- Figure 1: Preliminary “Automation Space” as a morphological box: The lines indicate different appearances of Everyday Automation. Visualization The red line symbolizes an automatic vacuum cleaner Everyday Automation covers a diverse and complex range robot [14], the green line an AR-based system helping pa- of applications. Therefore it is not trivial to find an appro- tients to test their blood at home [3] and the blue line a vir- priate and suitable form for visualizing the corresponding tual reality indoor navigation system [9]. Each additional design space. We propose a representation of the design path through the dimensions might inspire a novel Everyday space based on the concept of the morphological box, Automation application. which has its origin in the creativity techniques. This form of visualization is based on the division of a subject into its Conclusion and Outlook elementary components, whereby the dimensions for each In this paper, we presented our ongoing work on creating component are determined and a combination of the ele- a design space for “Everyday Automation”. From a litera- ments is ultimately displayed [23]. The aspects mentioned ture review, we identified first core dimensions: presence above reflect parallels and central elements of a design of the system, domain, automated task, user interaction, space. This form of visualization is therefore considered to and automation level. For visualizing these dimensions and be particularly suitable for compactly visualizing a design the various existing specifications, we proposed a morpho- space with many dimensions and manifestations. logical box. This approach provides a compact overview of manifestations and particularly supports the ideation of Figure 1 presents a morphological box for the preliminary novel applications. version of the design space with aforementioned dimen- sions. Each vertical path from top to bottom through all In future work, we will complete this first version of the Au- the dimensions represents a potential appearance of an tomation Space by further dimensions. Additionally, we automated system in an everyday setting. In Figure 1, plan to evaluate complementary alternative visualization three above-mentioned examples from literature are drawn: approaches beyond the currently used morphological box. REFERENCES Algorithm. In Proceedings of the 2019 International [1] R. Ballagas, J. Borchers, M. Rohs, and J. G. Sheridan. Conference on Artificial Intelligence and Computer 2006. The smart phone: a ubiquitous input device. Science. 303–308. IEEE Pervasive Computing 5, 1 (2006), 70–77. [7] Marc Hassenzahl and Holger Klapperich. 2014. [2] William Buxton. 1983. Lexical and Pragmatic Convenient, clean, and efficient?: the experiential Considerations of Input Structures. SIGGRAPH costs of everyday automation. In Proceedings of the Comput. Graph. 17, 1 (Jan. 1983), 31–37. DOI: 8th Nordic Conference on Human-Computer http://dx.doi.org/10.1145/988584.988586 Interaction: Fun, Fast, Foundational. ACM, 21–30. [3] Tom Djajadiningrat, Pei-Yin Chao, SeYoung Kim, [8] Hayati Havlucu, Terry Eskenazi, Bariş Akgün, Marleen Van Leengoed, and Jeroen Raijmakers. 2016. Mehmet Cengiz Onbaşlı, Aykut Coşkun, and Oğuzhan Mime: An AR-based System Helping Patients to Test Özcan. 2018. Flow state feedback through sports their Blood at Home. In Proceedings of the 2016 ACM wearables: A case study on tennis. In Proceedings of Conference on Designing Interactive Systems. the 2018 Designing Interactive Systems Conference. 347–359. 1025–1039. [4] Abdulsalam Dukyil, Ahmed Mohammed, and [9] Darlis Herumurti, Anny Yuniarti, Imam Kuswardayan, Mohamed Darwish. 2016. An optimization approach Wijayanti Nurul Khotimah, and Wahyu Widyananda. for a RFID-enabled passport tracking system. In 2017. Virtual reality navigation system in virtual mall Proceedings of the 4th International Conference on environment. In Proceedings of the 3rd International Control, Mechatronics and Automation. 189–194. Conference on Communication and Information Processing. 209–213. [5] Peter Fröhlich, Matthias Baldauf, Thomas Meneweger, Ingrid Erickson, Manfred Tscheligi, Thomas Gable, [10] Elin Janebäck and Matilda Kristiansson. 2019. Friendly Boris de Ruyter, and Fabio Paternò. 2019. Everyday robot delivery: Designing an autonomous delivery Automation Experience: Non-Expert Users droid for collaborative consumption. Master’s thesis. Encountering Ubiquitous Automated Systems. In [11] Katharina Keller, Kim Valerie Carl, Hendrik Jöntgen, Extended Abstracts of the 2019 CHI Conference on Benjamin M Abdel-Karim, Max Mühlhäuser, and Oliver Human Factors in Computing Systems (CHI EA ’19). Hinz. 2019. " KITT, where are you?" why smart Association for Computing Machinery, New York, NY, assistance systems in cars enrich people’s lives. In USA, Article Paper W25, 8 pages. DOI: Adjunct Proceedings of the 2019 ACM International http://dx.doi.org/10.1145/3290607.3299013 Joint Conference on Pervasive and Ubiquitous [6] Xiaoyan Gao, Xiangqian Ding, Ruichun Hou, and Ye Computing and Proceedings of the 2019 ACM Tao. 2019. Research on Food Recognition of Smart International Symposium on Wearable Computers. Refrigerator Based on SSD Target Detection 1120–1132. [12] Dagmar Kern and Albrecht Schmidt. 2009. Design Computing, Systems, Networks, and Applications. Space for Driver-Based Automotive User Interfaces. In 43–48. Proceedings of the 1st International Conference on [19] Jörg Müller, Florian Alt, Daniel Michelis, and Albrecht Automotive User Interfaces and Interactive Vehicular Schmidt. 2010. Requirements and Design Space for Applications (AutomotiveUI ’09). Association for Interactive Public Displays. In Proceedings of the 18th Computing Machinery, New York, NY, USA, 3–10. ACM International Conference on Multimedia (MM DOI:http://dx.doi.org/10.1145/1620509.1620511 ’10). Association for Computing Machinery, New York, [13] Marcel Langer and Dirk Söffker. 2011. Human NY, USA, 1285–1294. DOI: guidance and supervision of a manufacturing system http://dx.doi.org/10.1145/1873951.1874203 for semi-automated production. In 2011 IEEE Jordan [20] Hermann Ney. 2001. Stochastic modelling: from Conference on Applied Electrical Engineering and pattern classification to language translation. In Computing Technologies (AEECT). IEEE, 1–6. Proceedings of the workshop on Data-driven methods [14] Hyunsoo Lee and Amarnath Banerjee. 2015. in machine translation-Volume 14. Association for Intelligent scheduling and motion control for household Computational Linguistics, 1–5. vacuum cleaning robot system using simulation based [21] Raja Parasuraman, Thomas B Sheridan, and optimization. In 2015 Winter Simulation Conference Christopher D Wickens. 2000. A model for types and (WSC). IEEE, 1163–1171. levels of human interaction with automation. IEEE [15] Dietrich Manzey. 2012. Systemgestaltung und Transactions on systems, man, and cybernetics-Part Automatisierung. Springer Berlin Heidelberg, Berlin, A: Systems and Humans 30, 3 (2000), 286–297. Heidelberg, 333–352. DOI: [22] Alex Polacco and Kayla Backes. 2018. The amazon go http://dx.doi.org/10.1007/978-3-642-19886-1_19 concept: Implications, applications, and sustainability. [16] Philipp Mayring and Thomas Fenzl. 2014. Qualitative Journal of Business and Management 24, 1 (2018), Inhaltsanalyse. Springer Fachmedien Wiesbaden, 79–92. Wiesbaden, 543–556. DOI: [23] Christian Schawel and Fabian Billing. 2018. http://dx.doi.org/10.1007/978-3-531-18939-0_38 Morphologischer Kasten. Springer Fachmedien [17] Hila Mehr, H Ash, and D Fellow. 2017. Artificial Wiesbaden, Wiesbaden, 219–221. DOI: intelligence for citizen services and government. Ash http://dx.doi.org/10.1007/978-3-658-18917-4_57 Cent. Democr. Gov. Innov. Harvard Kennedy Sch., no. [24] Mélodie Vidal, Andreas Bulling, and Hans Gellersen. August (2017), 1–12. 2013. Pursuits: spontaneous interaction with displays [18] Silvia Mirri, Catia Prandi, and Paola Salomoni. 2019. based on smooth pursuit eye movement and moving Human-Drone Interaction: state of the art, open issues targets. In Proceedings of the 2013 ACM international and challenges. In Proceedings of the ACM joint conference on Pervasive and ubiquitous SIGCOMM 2019 Workshop on Mobile AirGround Edge computing. 439–448.