In this paper, we describe a pilot study aimed to explore strategies used by non-programmer users to test trigger-action rules for customizing an IoT device. The main goal of our research was to examine strategies used by participants to detect and solve errors. In the pilot study, we asked non-programmers to imagine testing a set of rules, some of which were bugged. The pilot study was meant to understand the feasibility of this approach to investigate users' mental models while performing this kind of task.
In this paper, we present a pilot study to explore the strategies of non-programmer users in debugging trigger-action (TA) rules to customize an IoT educational tool. In particular, we wanted to investigate how naive users approach debugging and the characteristics of the strategies they adopt.
In recent years, the rapid diffusion of IoT has brought end-users to the center of a complex ecosystem
made of interconnected objects and web services, changing the way they live [
In this context, the End-User Development (EUD) paradigm has allowed non-technical users
without programming skills to customize the behavior of their devices and applications [
The relative simplicity and applicability of TAP to IoT have attracted a lot of interest [
Only very recently, a few studies have been carried out that focused on the problem of rule errors in
EUD and explored debugging approaches to support end-users in customizing their IoT devices [
However, several aspects remain to be clarified. In particular, we still do not know much about how
end-users approach debugging and what strategies they adopt [
The present study aimed to explore the strategies and approaches of naive users in debugging TA rules to outline possible and valid future research. 1.1.
In this pilot study, a tangible educational IoT device prototype (Figure 1) was employed [9]. Such a device was designed to support mathematics learning in elementary school children. It has a rectangular surface with five slots for placing external pieces (digit, symbol, or operator tiles), and it is provided with a visual and acoustic feedback system. This tool makes it possible to implement different types of mathematical exercises such as comparing quantities, sorting numbers, and simple arithmetic operations.
Teachers could customize the tool's functioning through a set of language primitives for actions, states, and events implemented as part of an existing authoring interface (see Figure 2). The primitives for the actions consist of commands to control visual (lights) and acoustic feedback. At the same time, states and events describe respectively the operations that teachers and children can perform on the tool (i.e., insertion and removal of tiles) and the tool configuration at that moment. An example of an event description is "WHEN a digit tile is inserted", while an example of a state is "WHILE the position to the right of the inserted tile is empty". Finally, "Turn on blue LED SMARTER" is an example of action.
Five (5) primary school math teachers with no previous experience in programming were recruited. The study consisted of individual semi-structured interviews conducted remotely (90 minutes).
After describing the tool and the interface through videos and explanatory images, the participants were familiarized with the interface and a set of rules already prepared and related to a simple number ordering game (from the smallest number to the largest). In the first phase, participants were then asked to discuss how they would evaluate the correctness of the rules and then to indicate three specific actions by which they would test the program's functioning (debugging task).
In the second phase of the study, participants were given an error-finding task on a different set of rules in which two types of errors were intentionally included. Specifically, both inconsistent and redundant rules were introduced, that is, rules providing for two conflicting actions for the same trigger and different rules producing the same outcome, respectively.
A verbal reporting procedure was used to explore participants' strategies and mental models during the familiarization and error-finding tasks. 2.1.
As a first approach to the debugging task, all participants imagined testing the rules in the field with a hands-on approach, i.e., in the classroom with their students. Some of them proposed a trial-and-error strategy ("I would try to make a rule if I realized something was wrong, I would go back and change it"), while others (participants B and C) suggested testing directly with the children using a more functional and child-focused approach.
Specifically, some participants (such as B, D, E) proposed a “step-by-step” strategy, e.g., proposing to insert the first tile, then a correct tile, and finally an incorrect one, checking the outcomes each time a single tile was inserted. In contrast, Participant A took a different approach, preferring to place all the tiles first to observe the result. Interestingly, such an approach, focused on obtaining a direct solution, has already been observed in novice programmers and has been associated with a misunderstanding of how the system works [10][11].
Finally, participants A and C proposed the placement of a symbol tile to test the alternative case. Also, participant E evaluated this approach, but she argued it was not fundamental considering the task goal of sorting numbers progressively. Debugging task results are summarized in Table 1.
In the error-finding phase, participants exhibited two different approaches. Some assumed a ruleby-rule approach (participants B and D), directing the error search to individual rules and ignoring their relationships. Participants who adopted this approach failed to identify any errors present in the ruleset. They also state that the errors were due to the excessive number of rules, in their opinion, unnecessary and confusing. Conversely, participants who extended their error search to include the relationships between rules (participants A, C, E) could identify some errors. Eventually, only one participant was able to identify all errors correctly. Results of the error-finding task are summarized in Table 2.
Consistent with the literature [
Error search extended to rule relationship
Error search limited to individual rules Error search extended to rule relationship
Error search limited to individual rules Error search extended to rule relationship
Inconsistency
None Inconsistency
None Inconsistency, Redundancy
In this pilot study, we explored the strategies adopted by non-programmer users for debugging a TA rule set and for finding inconsistent and redundant rules. While the extant literature on end-users debugging TA rules focuses on tools to support bug identification, in our approach, we tried to focus on the mental models users initially assume in facing a debugging task.
Three different debugging strategies implemented by participants emerged from the results. These
strategies were partly similar to those already observed in novice programmers [10][11][12].
Nevertheless, to our knowledge, they have not been discussed in the context of EUD with TA rules.
Specifically, naive users might have more problems considering the entire set of rules and their
relationships. We believe that extending this approach with a more robust study will help to design
more effective tools (along the lines, for example, of the work done by Corno et al. [
This work has been supported by the Italian Ministry of Education, University and Research (MIUR) under grant PRIN 2017 "EMPATHY: EMpowering People in deAling with internet of THings ecosYstems" (Progetti di Rilevante Interesse Nazionale – Bando 2017, Grant 2017MX9T7H).
[7] Ko & Myers 2004 (Ko, A. and Myers, B. 2004. Designing the whyline: A debugging interface for asking questions about program behavior. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 151–158) [8] Manca, M., Paternò, F., Santoro, C., & Corcella, L. 2019. Supporting end-user debugging of trigger-action rules for IoT applications. International Journal of Human-Computer Studies, 123, 56-69. [9] Andrao, M., Desolda, G., Greco, F., Manfredi, R., Treccani, B., Zancanaro, M. 2022, June .
SMARTER: an IoT learning game to teach math. In International Conference on Advanced Visual Interfaces (AVI 2022), June 6–10, 2022, Frascati, Rome, Italy. [10] Chen Li, Emily Chan, Paul Denny, Andrew Luxton-Reilly, and Ewan Tempero. 2019. Towards a framework for teaching debugging. Proceedings of the Twenty-First Australasian Computing Education Conference on - ACE '19 (2019). DOI:http://dx.doi.org/10.1145/3286960.3286970 [11] Fitzgerald, S., Lewandowski, G., McCauley, R., Murphy, L., Simon, B., Thomas, L. and Zander, C., 2008. Debugging: finding, fixing and flailing, a multi-institutional study of novice debuggers.
Computer Science Education, 18(2), pp.93-116. [12] S. Fitzgerald, R. McCauley, B. Hanks, L. Murphy, B. Simon and C. Zander, "Debugging From the Student Perspective," in IEEE Transactions on Education, vol. 53, no. 3, pp. 390-396, Aug. 2010, doi: 10.1109/TE.2009.2025266.