If when is better than if (and while might help): on the importance of influencing mental models in EUD (a pilot study) Giuseppe Gallittoa, Barbara Treccania and Massimo Zancanaroa,b a Department of Psychology and Cognitive Science, University of Trento, Rovereto (Trento), Italy b Fondazione Bruno Kessler – FBK,, Povo (Trento), Italy Abstract In this paper, we present a preliminary study aimed at improving the users’ mental model of an automatic smart home system based on trigger-action rules. We hypothesized that a computational model of how the rules are evaluated and activated, coupled with a linguistic form of the rules that clarifies the difference between events and states, may improve accuracy in identifying buggy rules, which may eventually increase end-users’ acceptance of this type of systems. This pilot study was conducted with non-programmers and provided some evidence in support of this hypothesis. Keywords 1 End-User Development (EUD), trigger-action rules, mental models 1. Introduction End-User Development (EUD) is defined as the possibility for people without programming experience to create or modify their applications [12]. EUD approaches focus on empowering users beyond their involvement in the design phases, as advocated by user-centered design, and propose a vision in which design, learning, and development are an inherent part of technology in use [6, 16]. Empowering users is particularly important today, as the Internet of Things (IoT) is pushing for digitalizing everyday objects: in this respect, EUD may prove crucial to facilitate the adoption of this technology [1, 17]. Programming is difficult for non-experts because it often requires expressing solutions in ways that are not familiar to them [15]. The concept of rule may provide an intelligible metaphor for programming digital technologies since it embeds the idea that specific actions need to be taken in specific situations [5]. In the context of IoT-based smart devices, the programming approach based on contextual rules has evolved in the so-called trigger-action programming for which a rule takes the specific form of an action that is performed upon the occurrence of condition. Such a metaphor is supposed to be easily graspable by users since IoT devices are usually either sensors that detect events in the world or actuators that operate changes in the world (of both). The simplicity of this approach is demonstrated by the success of web-based services, like IFTTT (https://ifttt.com/) [20]. Nevertheless, for an effective programming of IoT devices, it is important to allow more expressive triggering conditions and more elaborate actions [1, 3, 8, 19]. When the triggering conditions become more complicated (for example allowing multiple conditions), the rule metaphor becomes less simple and users are more prone to errors, for example by inaccurately composing events or mistaking events with states [10, 2]. Another source of confusion are users’ wrong or inaccurate mental models of the actual functioning of the system: for example, because users assume default states when there is none [21]. Indeed, mental models are important cognitive constructs that can explain how people interact with the world and specifically with complex artifacts [7]. Roughly, a mental model is a simplified EMPATHY: Empowering People in Dealing with Internet of Things Ecosystems. Workshop co-located with AVI 2020, Island of Ischia, Italy EMAIL: giuseppe.gallitto@unitn.it (G. Gallitto); barbara.treccani@unitn.it (B. Treccani); massimo.zancanaro@unitn.it (M.Zancanaro) ORCID: 0000-0001-5185-0206 (G. Gallitto); 0000-0001-8028-0708 (B. Treccani); 0000-0002-1554-5703 (M. Zancanaro) © 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) representation of the mechanism and working of an artifact that a user develops in order to make sense of the artifact itself and to effectively use it [13, 18]. Users’ mental models are not necessarily correct and complete representations of an artifact, but they may have both predictive and explanatory power for understanding the interaction between the user and the artifact. Mental models can and should be communicated to the user by proper design [14] but it might also be effective to communicate them verbally, by providing the user with an adequate description of how the system works. Regardless of their correctness, different mental models of the same object may lead to dramatically different user-object interactions: Halasz and Moran’s study [9] is a compelling example of how two different, albeit both correct, descriptions of the functioning of a reverse-polish calculator can lead to different performances. 2. Research hypothesis The present study aims at investigating how end-users conceptualize EUD in smart homes and at providing simple but effective design hints to improve understanding of trigger-action rules. Specifically, we would like to encourage accurate mental models by providing a better representation of how rule-based systems works and by fostering a recognition of different important concepts by imposing linguistic constraints to the rules. Regarding the first aspect, there is some of evidence that proper and short instructions may contribute to develop accurate mental models [11] and that mental models arising from an explicit description of the working mechanism (hereafter called “computational models”) are more effective than models arising from a simple description of the procedure/rules of a given system (hereafter called “descriptive models”) [9]. We therefore posit Hypothesis 1 as follows: • Hypothesis 1: a short description of the cyclical mechanism of evaluation and activation of the rules may improve the understanding of the effects of trigger-action rules. Regarding the second aspect, one key point is the difference between events and states [10]. In some EUD implementations there is the possibility to associate a condition involving events and/or states with the trigger [4]. Yet the distinction between events and states is difficult to understand because they are often closely related [8] (for example, the state “the door is open” is related to the event “the door opens”). Several types of error may be attributed to the confusion between these two concepts [2]. We therefore posit Hypothesis 2 as follows: • Hypothesis 2: expressing the rule through a linguistic form that clarifies the event/state distinction may improve the understanding of the effects of the rule; we proposed the form WHEN WHILE DO 3. The study The study is based on the one presented by Brackenbury and colleagues [2] with some modifications. It consists of eight scenarios in a smart-home setting, each one accompanied by two rules that may or may not realize the purposes stated in the scenario. In some cases the rules are correct (i.e., they realize the described scenario) while in other cases they are buggy (i.e., conditions described in the scenario do not activate the rules or their activation have outcomes other than those intended – i.e., different from those described in the scenario). Before the study, the users are offered a short tutorial with some examples about the distinction between events and states (see Figure 1) and they have been told that rules have the form: WHEN WHILE DO . However, they can be shortened as IF THEN . Following this, the scenarios were presented in random order. In the rules of the scenarios, the use of the WHEN/WHILE/DO form is alternated (in a randomized way) with the form IF/THEN. Half of the subjects receive a computational description of how the system works that specifies the cyclical nature of the rule evaluation-and-activation system (i.e., a “computational model”) while the other half receive a simpler description of the possible rule structures (i.e., a “descriptive model”). Therefore, the study has two between-subject conditions (i.e., computational vs. descriptive models; cf., Hypothesis 1) and two within-subject conditions (i.e., the two possible format/structure of the rules; cf., Hypothesis 2). Figure 1: Picture from the tutorial illustrating the difference between events and states 3.1. Preliminary results The study has been piloted with 12 bachelor students from the Department of Psychology and Cognitive Science of the University of Trento; none of them had knowledge of, or prior experience with, computer programming. Although the results are preliminary, they are promising and clearly show a trend that supports both hypotheses. 0.86 0.84 0.82 0.80 Mean accuracy (range: 0-1) 0.78 0.76 0.74 0.72 0.70 0.68 0.66 0.64 descriptive computational WHEN/WHILE IF Model Figure 2. Mean accuracy in the recognition of rules’ behavior for the two types of model (descriptive vs. computational) and the two rule formats (When/While vs. If) As suggested by the plot in Figure 2, accuracy in the recognition of rules’ behavior is higher when the rules are presented in the WHEN/WHILE format than when presented in the IF format (Hypothesis 2) and for participants who have been exposed to the computational model rather than the descriptive model (Hypothesis 1). An analysis of variance (ANOVA) partially confirmed the trends by showing a significant main effect of the format of the rules. The main effect of the model was not significant (see Table 1). Table 1. Repeated measures ANOVA with one between-subjects factor (mental model) and one within-subjectsfactor (format) Repeated Measures Analysis of Variance Sigma-restricted parameterization Effective hypothesis decomposition SS Degr. of - Freedom MS F p Intercept 12.45522 1 12.45522 160.9903 0.000000 mental model 0.01077 1 0.01077 0.1393 0.717651 Error 0.69630 9 0.07737 FORMAT 0.08148 1 0.08148 5.2105 0.048352* FORMAT*mental model 0.00067 1 0.00067 0.0431 0.840228 Error 0.14074 9 0.01564 4. Conclusion and further works The results of this pilot study are initial evidence supporting our hypotheses: Our manipulations seem to be effective in increasing the participants’ understanding of the rules’ behavior and of their effects. Consistent with our hypotheses, the use of a WHEN/WHILE format for the rules and (possibly) the computational description of the system seems to improve users’ mental models of how the smart- home automatic system works. The full study is ongoing. It will include a measure of comprehension of the training and more detailed analyses of the results. 5. Acknowledgements This work has been supported by the Italian Ministry of Education, University and Research (MIUR) under grant PRIN 2017 "EMPATHY: EMpowering People in deAling with internet of THings ecosYstems" (Progetti di Rilevante Interesse Nazionale – Bando 2017, Grant 2017MX9T7H). 6. References [1] Ardito, C. et al. 2018. From smart objects to smart experiences: An end-user development approach. International Journal of Human-Computer Studies. 114, (Jun. 2018), 51–68. DOI:https://doi.org/10.1016/j.ijhcs.2017.12.002. [2] Brackenbury, W. et al. 2019. How Users Interpret Bugs in Trigger-Action Programming. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19 (Glasgow, Scotland Uk) [3] Davidoff, S., et al. 2006. Principles of smart home control. In Proceedings of ACM Ubicomp [4] Desolda, G. et al. 2017. Empowering End Users to Customize their Smart Environments: Model, Composition Paradigms, and Domain-Specific Tools. ACM Transactions on Computer-Human Interaction. 24, 2 (Apr. 2017), 1–52. [5] Dey, A.K. et al. 2006. iCAP: Interactive Prototyping of Context-Aware Applications. In Proceedings of Pervasive Computing 2006 (\, 2006). [6] Fischer, G. et al. 2004. Meta-design: a manifesto for end-user development. Communications of the ACM. 47, 9:33-37 [7] Gentner, D. and Stevens, A.L. eds. 1983. Mental models. Erlbaum. [8] Ghiani, G. et al. 2017. Personalization of Context-Dependent Applications Through Trigger- Action Rules. ACM Transactions on Computer-Human Interaction. 24, 2 (Apr. 2017), 1–33. DOI:https://doi.org/10.1145/3057861. [9] Halasz, F.G. and Moran, T.P. 1983. Mental models and problem solving in using a calculator. Proceedings of the SIGCHI conference on Human Factors in Computing Systems - CHI ’83 (Boston, Massachusetts, United States, 1983), 212–216. [10] Huang, J. and Cakmak, M. 2015. Supporting mental model accuracy in trigger-action programming. Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing - UbiComp ’15 (Osaka, Japan, 2015), 215–225. [11] Kulesza T., et al., 2012. Tell Me More?: The Effects of Mental Model Soundness on Personalizing an Intelligent Agent. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12). [12] Lieberman, H., Paternò, F., Klann, M., Wulf, V., 2006. End-user development: an emerging paradigm. In: Lieberman, Henry, Paternò, Fabio, Wulf, Volker (Eds.), End User Development. Springer, The Netherlands, pp. 1–8. [13] Norman, D.A. 1983. Some Observations on Mental Models. In Gentner, D. and Stevens, A.L. eds. Mental models. Erlbaum. [14] Norman, D.A. 2013. The design of everyday things. Basic Books. [15] Pane, J.F. et al. 2001. Studying the language and structure in non-programmers’ solutions to programming problems. International Journal of Human-Computer Studies. 54, 2 (Feb. 2001), 237–264. [16] Paternò, F. 2013. End User Development: Survey of an Emerging Field for Empowering People. ISRN Software Engineering. 2013, (2013), 1–11. [17] Paternò, F. and Santoro, C. 2019. End-user development for personalizing applications, things, and robots. International Journal of Human-Computer Studies. 131:120–130. [18] Payne, S.J. 2009, Mental Models in Human-Computer Interaction. In Sears, A. and Jacko, J.A. eds. Human-computer interaction. Fundamentals. CRC Press. [19] Ur, B. et al. 2014. Practical trigger-action programming in the smart home. Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI ’14 (Toronto, Ontario, Canada, 2014), 803–812. [20] Ur, B. et al. 2016. Trigger-Action Programming in the Wild: An Analysis of 200,000 IFTTT Recipes. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ’16 (Santa Clara, California, USA, 2016), 3227–3231. [21] Yarosh, S. and Zave, P. 2017. Locked or Not?: Mental Models of IoT Feature Interaction. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ’17 (Denver, Colorado, USA, 2017), 2993–2997.