=Paper=
{{Paper
|id=Vol-2702/sample-2col
|storemode=property
|title=If when is better than if (and while might help): on the importance of influencing mental models in EUD (a pilot study)
|pdfUrl=https://ceur-ws.org/Vol-2702/EMPATHY_2020_paper_2.pdf
|volume=Vol-2702
|authors=Giuseppe Gallitto,Barbara Treccani,Massimo Zancanaro
|dblpUrl=https://dblp.org/rec/conf/avi/GallittoTZ20
}}
==If when is better than if (and while might help): on the importance of influencing mental models in EUD (a pilot study)==
https://ceur-ws.org/Vol-2702/EMPATHY_2020_paper_2.pdf
If when is better than if (and while might help): on the
importance of influencing mental models in EUD (a pilot study)
Giuseppe Gallittoa, Barbara Treccania and Massimo Zancanaroa,b
a
Department of Psychology and Cognitive Science, University of Trento, Rovereto (Trento), Italy
b
Fondazione Bruno Kessler – FBK,, Povo (Trento), Italy
Abstract
In this paper, we present a preliminary study aimed at improving the users’ mental model of
an automatic smart home system based on trigger-action rules. We hypothesized that a
computational model of how the rules are evaluated and activated, coupled with a linguistic
form of the rules that clarifies the difference between events and states, may improve accuracy
in identifying buggy rules, which may eventually increase end-users’ acceptance of this type
of systems. This pilot study was conducted with non-programmers and provided some evidence
in support of this hypothesis.
Keywords 1
End-User Development (EUD), trigger-action rules, mental models
1. Introduction
End-User Development (EUD) is defined as the possibility for people without programming
experience to create or modify their applications [12]. EUD approaches focus on empowering users
beyond their involvement in the design phases, as advocated by user-centered design, and propose a
vision in which design, learning, and development are an inherent part of technology in use [6, 16].
Empowering users is particularly important today, as the Internet of Things (IoT) is pushing for
digitalizing everyday objects: in this respect, EUD may prove crucial to facilitate the adoption of this
technology [1, 17].
Programming is difficult for non-experts because it often requires expressing solutions in ways that
are not familiar to them [15]. The concept of rule may provide an intelligible metaphor for programming
digital technologies since it embeds the idea that specific actions need to be taken in specific situations
[5].
In the context of IoT-based smart devices, the programming approach based on contextual rules has
evolved in the so-called trigger-action programming for which a rule takes the specific form of an
action that is performed upon the occurrence of condition. Such a metaphor is supposed to be easily
graspable by users since IoT devices are usually either sensors that detect events in the world or
actuators that operate changes in the world (of both).
The simplicity of this approach is demonstrated by the success of web-based services, like IFTTT
(https://ifttt.com/) [20]. Nevertheless, for an effective programming of IoT devices, it is important to
allow more expressive triggering conditions and more elaborate actions [1, 3, 8, 19]. When the
triggering conditions become more complicated (for example allowing multiple conditions), the rule
metaphor becomes less simple and users are more prone to errors, for example by inaccurately
composing events or mistaking events with states [10, 2]. Another source of confusion are users’ wrong
or inaccurate mental models of the actual functioning of the system: for example, because users assume
default states when there is none [21].
Indeed, mental models are important cognitive constructs that can explain how people interact with
the world and specifically with complex artifacts [7]. Roughly, a mental model is a simplified
EMPATHY: Empowering People in Dealing with Internet of Things Ecosystems. Workshop co-located with AVI 2020, Island of Ischia,
Italy EMAIL: giuseppe.gallitto@unitn.it (G. Gallitto); barbara.treccani@unitn.it (B. Treccani); massimo.zancanaro@unitn.it (M.Zancanaro)
ORCID: 0000-0001-5185-0206 (G. Gallitto); 0000-0001-8028-0708 (B. Treccani); 0000-0002-1554-5703 (M. Zancanaro)
© 2020 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
representation of the mechanism and working of an artifact that a user develops in order to make sense
of the artifact itself and to effectively use it [13, 18]. Users’ mental models are not necessarily correct
and complete representations of an artifact, but they may have both predictive and explanatory power
for understanding the interaction between the user and the artifact.
Mental models can and should be communicated to the user by proper design [14] but it might also
be effective to communicate them verbally, by providing the user with an adequate description of how
the system works. Regardless of their correctness, different mental models of the same object may lead
to dramatically different user-object interactions: Halasz and Moran’s study [9] is a compelling example
of how two different, albeit both correct, descriptions of the functioning of a reverse-polish calculator
can lead to different performances.
2. Research hypothesis
The present study aims at investigating how end-users conceptualize EUD in smart homes and at
providing simple but effective design hints to improve understanding of trigger-action rules.
Specifically, we would like to encourage accurate mental models by providing a better
representation of how rule-based systems works and by fostering a recognition of different important
concepts by imposing linguistic constraints to the rules.
Regarding the first aspect, there is some of evidence that proper and short instructions may
contribute to develop accurate mental models [11] and that mental models arising from an explicit
description of the working mechanism (hereafter called “computational models”) are more effective
than models arising from a simple description of the procedure/rules of a given system (hereafter called
“descriptive models”) [9]. We therefore posit Hypothesis 1 as follows:
• Hypothesis 1: a short description of the cyclical mechanism of evaluation and activation of the
rules may improve the understanding of the effects of trigger-action rules.
Regarding the second aspect, one key point is the difference between events and states [10]. In some
EUD implementations there is the possibility to associate a condition involving events and/or states
with the trigger [4]. Yet the distinction between events and states is difficult to understand because they
are often closely related [8] (for example, the state “the door is open” is related to the event “the door
opens”). Several types of error may be attributed to the confusion between these two concepts [2]. We
therefore posit Hypothesis 2 as follows:
• Hypothesis 2: expressing the rule through a linguistic form that clarifies the event/state
distinction may improve the understanding of the effects of the rule; we proposed the form WHEN
WHILE DO
3. The study
The study is based on the one presented by Brackenbury and colleagues [2] with some modifications.
It consists of eight scenarios in a smart-home setting, each one accompanied by two rules that may or
may not realize the purposes stated in the scenario. In some cases the rules are correct (i.e., they realize
the described scenario) while in other cases they are buggy (i.e., conditions described in the scenario
do not activate the rules or their activation have outcomes other than those intended – i.e., different
from those described in the scenario).
Before the study, the users are offered a short tutorial with some examples about the distinction
between events and states (see Figure 1) and they have been told that rules have the form: WHEN
WHILE DO . However, they can be shortened as IF
THEN .
Following this, the scenarios were presented in random order. In the rules of the scenarios, the use
of the WHEN/WHILE/DO form is alternated (in a randomized way) with the form IF/THEN.
Half of the subjects receive a computational description of how the system works that specifies the
cyclical nature of the rule evaluation-and-activation system (i.e., a “computational model”) while the
other half receive a simpler description of the possible rule structures (i.e., a “descriptive model”).
Therefore, the study has two between-subject conditions (i.e., computational vs. descriptive models;
cf., Hypothesis 1) and two within-subject conditions (i.e., the two possible format/structure of the rules;
cf., Hypothesis 2).
Figure 1: Picture from the tutorial illustrating the difference between events and states
3.1. Preliminary results
The study has been piloted with 12 bachelor students from the Department of Psychology and
Cognitive Science of the University of Trento; none of them had knowledge of, or prior experience
with, computer programming.
Although the results are preliminary, they are promising and clearly show a trend that supports both
hypotheses.
0.86
0.84
0.82
0.80
Mean accuracy (range: 0-1)
0.78
0.76
0.74
0.72
0.70
0.68
0.66
0.64
descriptive computational WHEN/WHILE
IF
Model
Figure 2. Mean accuracy in the recognition of rules’ behavior for the two types of model (descriptive
vs. computational) and the two rule formats (When/While vs. If)
As suggested by the plot in Figure 2, accuracy in the recognition of rules’ behavior is higher when
the rules are presented in the WHEN/WHILE format than when presented in the IF format (Hypothesis
2) and for participants who have been exposed to the computational model rather than the descriptive
model (Hypothesis 1).
An analysis of variance (ANOVA) partially confirmed the trends by showing a significant main
effect of the format of the rules. The main effect of the model was not significant (see Table 1).
Table 1. Repeated measures ANOVA with one between-subjects factor (mental model) and one
within-subjectsfactor (format)
Repeated Measures Analysis of Variance Sigma-restricted parameterization Effective hypothesis
decomposition
SS Degr. of - Freedom MS F p
Intercept 12.45522 1 12.45522 160.9903 0.000000
mental model 0.01077 1 0.01077 0.1393 0.717651
Error 0.69630 9 0.07737
FORMAT 0.08148 1 0.08148 5.2105 0.048352*
FORMAT*mental model 0.00067 1 0.00067 0.0431 0.840228
Error 0.14074 9 0.01564
4. Conclusion and further works
The results of this pilot study are initial evidence supporting our hypotheses: Our manipulations
seem to be effective in increasing the participants’ understanding of the rules’ behavior and of their
effects. Consistent with our hypotheses, the use of a WHEN/WHILE format for the rules and (possibly)
the computational description of the system seems to improve users’ mental models of how the smart-
home automatic system works.
The full study is ongoing. It will include a measure of comprehension of the training and more
detailed analyses of the results.
5. Acknowledgements
This work has been supported by the Italian Ministry of Education, University and Research (MIUR)
under grant PRIN 2017 "EMPATHY: EMpowering People in deAling with internet of THings
ecosYstems" (Progetti di Rilevante Interesse Nazionale – Bando 2017, Grant 2017MX9T7H).
6. References
[1] Ardito, C. et al. 2018. From smart objects to smart experiences: An end-user development
approach. International Journal of Human-Computer Studies. 114, (Jun. 2018), 51–68.
DOI:https://doi.org/10.1016/j.ijhcs.2017.12.002.
[2] Brackenbury, W. et al. 2019. How Users Interpret Bugs in Trigger-Action Programming.
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19
(Glasgow, Scotland Uk)
[3] Davidoff, S., et al. 2006. Principles of smart home control. In Proceedings of ACM Ubicomp
[4] Desolda, G. et al. 2017. Empowering End Users to Customize their Smart Environments: Model,
Composition Paradigms, and Domain-Specific Tools. ACM Transactions on Computer-Human
Interaction. 24, 2 (Apr. 2017), 1–52.
[5] Dey, A.K. et al. 2006. iCAP: Interactive Prototyping of Context-Aware Applications. In
Proceedings of Pervasive Computing 2006 (\, 2006).
[6] Fischer, G. et al. 2004. Meta-design: a manifesto for end-user development. Communications of
the ACM. 47, 9:33-37
[7] Gentner, D. and Stevens, A.L. eds. 1983. Mental models. Erlbaum.
[8] Ghiani, G. et al. 2017. Personalization of Context-Dependent Applications Through Trigger-
Action Rules. ACM Transactions on Computer-Human Interaction. 24, 2 (Apr. 2017), 1–33.
DOI:https://doi.org/10.1145/3057861.
[9] Halasz, F.G. and Moran, T.P. 1983. Mental models and problem solving in using a calculator.
Proceedings of the SIGCHI conference on Human Factors in Computing Systems - CHI ’83
(Boston, Massachusetts, United States, 1983), 212–216.
[10] Huang, J. and Cakmak, M. 2015. Supporting mental model accuracy in trigger-action
programming. Proceedings of the 2015 ACM International Joint Conference on Pervasive and
Ubiquitous Computing - UbiComp ’15 (Osaka, Japan, 2015), 215–225.
[11] Kulesza T., et al., 2012. Tell Me More?: The Effects of Mental Model Soundness on Personalizing
an Intelligent Agent. In Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems (CHI ’12).
[12] Lieberman, H., Paternò, F., Klann, M., Wulf, V., 2006. End-user development: an emerging
paradigm. In: Lieberman, Henry, Paternò, Fabio, Wulf, Volker (Eds.), End User Development.
Springer, The Netherlands, pp. 1–8.
[13] Norman, D.A. 1983. Some Observations on Mental Models. In Gentner, D. and Stevens, A.L. eds.
Mental models. Erlbaum.
[14] Norman, D.A. 2013. The design of everyday things. Basic Books.
[15] Pane, J.F. et al. 2001. Studying the language and structure in non-programmers’ solutions to
programming problems. International Journal of Human-Computer Studies. 54, 2 (Feb. 2001),
237–264.
[16] Paternò, F. 2013. End User Development: Survey of an Emerging Field for Empowering People.
ISRN Software Engineering. 2013, (2013), 1–11.
[17] Paternò, F. and Santoro, C. 2019. End-user development for personalizing applications, things, and
robots. International Journal of Human-Computer Studies. 131:120–130.
[18] Payne, S.J. 2009, Mental Models in Human-Computer Interaction. In Sears, A. and Jacko, J.A.
eds. Human-computer interaction. Fundamentals. CRC Press.
[19] Ur, B. et al. 2014. Practical trigger-action programming in the smart home. Proceedings of the
32nd annual ACM conference on Human factors in computing systems - CHI ’14 (Toronto,
Ontario, Canada, 2014), 803–812.
[20] Ur, B. et al. 2016. Trigger-Action Programming in the Wild: An Analysis of 200,000 IFTTT
Recipes. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI
’16 (Santa Clara, California, USA, 2016), 3227–3231.
[21] Yarosh, S. and Zave, P. 2017. Locked or Not?: Mental Models of IoT Feature Interaction.
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ’17
(Denver, Colorado, USA, 2017), 2993–2997.