=Paper=
{{Paper
|id=Vol-3408/short-s2-01
|storemode=property
|title=Routine Creation in Multi-User Contexts: Improving the Quality of Life through Conflict Resolution
|pdfUrl=https://ceur-ws.org/Vol-3408/short-s2-01.pdf
|volume=Vol-3408
|authors=Barbara Rita Barricelli,Daniela Fogli
|dblpUrl=https://dblp.org/rec/conf/iseud/BarricelliF23
}}
==Routine Creation in Multi-User Contexts: Improving the Quality of Life through Conflict Resolution==
https://ceur-ws.org/Vol-3408/short-s2-01.pdf
Routine Creation in Multi-User Contexts: Improving
the Quality of Life through Conflict Resolution
Barbara Rita Barricelli† , Daniela Fogli1,∗,†
1
Department of Information Engineering, University of Brescia, Via Branze 38, Brescia, Italy
Abstract
This paper presents an approach to detecting and resolving the conflicts that may arise when different
users create routines for a shared smart home. By interacting with a virtual assistant, end users can
create routines, namely sequences of actions operating on smart devices, which are triggered by a specific
event. The virtual assistant is able to detect a possible conflict between a routine under creation and an
existing routine, and guide the user to its resolution. The final aim is to prevent household conflicts and
lead to more peaceful and harmonious coexistence among smart home users living under the same roof.
Keywords
Virtual assistant, IoT ecosystem, Routines, End-User Development, Conflict resolution
1. Introduction
Virtual assistants (VAs) are more and more present in smart home environments playing an
intermediary role between home inhabitants and their Internet-of-Things (IoT) ecosystem,
which includes smart sensors and devices such as smart TV, air conditioner, smart lights, and
the like [1]. A VA is composed of a hardware component, the so-called smart speaker (e.g.,
Amazon Echo, Google Nest, and Apple HomePod), and a software component that manages the
conversation between the users and the smart speaker (e.g., Amazon Alexa, Google Assistant,
Apple Siri).
VAs typically provide a companion mobile app that can be used by end users to create routines
for managing the behavior of IoT ecosystems. Routines comprise a name, a trigger used to
activate the routine itself, and a sequence of actions that are executed when the trigger fires.
For instance, the user can define the routine ”Good morning”, select as trigger a specific time
on specific weekdays, and finally choose different actions to be performed at that time on those
weekdays on one or more available smart devices (e.g., switch on the coffee machine, raise the
blinds in the bedrooms, and play music through the VA speakers). Routine creation can thus be
regarded as an End-User Development (EUD) [2, 3, 4] activity aimed at the customization of
smart home behavior.
IS-EUD 2023: 9th International Symposium on End-User Development, 6-8 June 2023, Cagliari, Italy
∗
Corresponding author.
†
These authors contributed equally.
Envelope-Open barbara.barricelli@unibs.it (B. R. Barricelli); daniela.fogli@unibs.it (D. Fogli)
GLOBE https://barbara-barricelli.unibs.it (B. R. Barricelli); https://daniela-fogli.unibs.it (D. Fogli)
Orcid 0000-0001-9575-5542 (B. R. Barricelli); 0000-0003-1479-2240 (D. Fogli)
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
In [5], we showed that routine creation could be moved from the companion mobile app
to the VA itself. Indeed, multi-modal interaction made available by a smart speaker with a
touchscreen can be exploited for this task, fostering the computational fluency [6] of non-tech
savvy users who are neither experts in routine creation nor aware of this possibility in the
mobile app.
Interaction with virtual assistants is generally designed with a single user in mind. Today,
the plurality of users, their coexistence, their different needs, and the relationships between
them are not considered: VAs just offer ways to personalize individual interaction with the IoT
ecosystem by creating multiple personal accounts. This approach does not consider the reality
of smart homes, which are multi-user environments where conflicts regarding shared spaces
and resources may arise due to different needs, preferences, and habits.
Equitable division of resources is not always possible; it may be the case that these resources
are limited and therefore shared among multiple users in the same time interval, generating
conflicts that must somehow be detected and resolved. For example, within a smart home, it may
happen that different users disagree on the level of lighting in a room because of the different
activities they intend to perform (watching a movie vs. reading a book); or that different users,
who intend to perform different activities (listening to music and listening to an audiobook),
want to use the same playback device, such as an Amazon Alexa smart speaker, in the same
time interval.
In particular, one of the main points of conflict may be routines created by different users
and scheduled to be executed at a certain time or when a certain condition occurs.
Our research explores how conflicts between routines can be detected at routine creation
time and possibly resolved - not with an automatic/intelligent intervention of the VA itself, but
with a human-driven process.
2. Literature on Conflict Detection and Resolution
Several studies have been reported in the literature proposing suitable solutions to detect and
resolve various types of conflicts that may arise in an environment inhabited by several people
where limited resources need to be shared. The suggested approaches can be divided into three
main categories: automatic, mediated, and combined.
Automatic approaches, which are usually called rule-based approaches, do not require user
intervention; rather, the system finds the best solution by following a set of rules predetermined
by the designers and implemented in the system. There are several strategies for automated
conflict resolution: some of them are based on a fair principle, which does not prefer any user, but
instead calculates the average value, when possible, of user preferences (e.g., room temperature
is calculated as the average value of the temperature values liked by users); while other strategies
adopt a first-come-first-serve (FCFS) approach, where the first user accessing the service or
device has full control of it [7]. More sophisticated strategies base their operation on user
preferences and/or priorities to provide the most appropriate solution to the context [8, 9, 10].
Within this category, there are some systems that base conflict resolution on variables rather
than static parameters, i.e., measures that vary according to context and transient situations,
such as users’ health conditions [7, 11].
Unlike those of the first category, the mediated approaches always involve user participation
in conflict resolution [12]. This approach is close to the normal conflict resolution mechanism
that occurs between human beings, where an exchange of information and opinions useful to
find a shared solution occurs between them [13, 14]. In the case where the conflict is resolved
through user intervention, the system could play a passive role, simply waiting for the person’s
decision, or it could actively participate in conflict resolution by providing the user with possible
solutions to the problem to choose from, based on the preferences of the users involved [15, 16].
Lastly, the combined approaches include conflict resolution strategies that combine the first
two approaches [17]. Depending on the type of conflict and the users involved, these systems
decide whether to resolve the conflict independently or require user participation.
3. Types of Conflict in Routine Creation
Considering routine creation on VAs like Amazon Alexa, a conflict can occur when actions
belonging to two different routines are meant to be performed on shared devices (e.g., listening
to music vs. listening to the news through the same smart speaker) or when the same action is
included in two routines but requires to be executed with different preferences (e.g., listening to
classical music vs. listening to hip-hop).
A specific type of conflict arises when actions modify environmental variables (e.g., tempera-
ture degree, environment noise, lighting) for which users would like to set contrasting values
due to different underlying motivations. For instance, a home inhabitant interested in energy
saving would be more inclined to switch off the lights or lower the heating temperature at a
certain time, whilst another inhabitant could be paying more attention to health and might
desire that the temperature never goes below 19 Celsius degrees. Underlying motivations can
also be related to different users’ activities: e.g., reading would require the light on, whilst
sleeping would require the light off.
We can also classify conflicts on the basis of how routines overlap in terms of the time
of execution and duration and in terms of the actions included in routines that conflict with
each other. Considering time, two routines overlap partially when the two time intervals
do not completely coincide, while they overlap completely when the two time intervals are
identical. On the other hand, we speak of overlap of two routines, from the point of view of their
component actions, when some of the actions in one routine conflict with those in another one
(partial overlap) or when all the actions in the two routines conflict with each other (complete
overlap).
When discussing conflicts between routines, it is appropriate to distinguish the time at which
they are detected. In fact, such conflicts can be detected either at creation time, when the user
attempts to save a new routine, or at run time, when a routine is invoked by the user or when
it is executed on its scheduled time. Depending on the time at which conflicts are detected,
the resolution strategy must change: in the first case, it is possible to prevent the creation of
conflicting routines, whereas in the second case, it is possible to intervene only at the time of
the invocation of the second routine, partially or totally restricting the actions of one of the two
routines involved.
4. A Combined Approach to Conflict Resolution
Our combined approach was defined by considering Amazon Alexa VA and Amazon Echo
devices. Therefore, due to some technical limitations, not all possible types of conflicts have
been investigated. So far, we have focused on conflicts between two scheduled routines and
have dealt with conflict detection and resolution at creation time.
From the literature review on the topic, it emerges that it is necessary to integrate and
combine more than one strategy for conflict resolution to design an effective approach. We
point out two important aspects that have to be considered and among which the right balance
has to be found:
• Decision-making autonomy of the virtual assistant: the virtual assistant must be able
to detect and resolve all conflicts arising between scheduled routines autonomously by
applying predetermined rules. For this to happen, the conflicts must have a clear nature –
i.e., they must be caused by an explicit temporal overlap of the actions of the two routines
involved.
• User decision-making power: the user must be denied the possibility of creating a new
routine that conflicts with an existing one, but, at the same time, they must be allowed and
guided to make changes to their routine to eliminate any possible conflicts. In addition,
in all those situations where the presence of a conflict is possible but not certain, the
user must be informed of it and let free to decide whether to ignore the notification or to
confirm the presence of a conflict and proceed with the routine modification.
In our approach, to detect conflicts, the system uses a set of rules to be applied during the
creation of the second routine for checking all actions that concern the control of a device or
an environmental variable. For a conflict to exist, there must be a temporal overlap between
the actions performed in the two routines. Determining whether there is a temporal overlap
between two actions is easy when they start at the same instant (e.g., one action involves playing
music at 3:00 p.m., the other turning on the TV at 3:00 p.m. in the same room, thus affecting the
same environment variable) or when the duration of at least one of the two actions is explicit
(e.g., one action involves turning on the TV at 7:00 p.m., another action involves playing music
from 6:50 p.m. to 7:30 p.m.).
An important aspect not to be underestimated is the naturalness of the conflict resolution
process: the virtual assistant should, on the one hand, act as an impartial and fair agent, and on
the other hand, it should simulate what really happens in a domestic environment where there
is a hierarchy among the occupants. The proposed approach assigns each user a priority level
depending on their age to replicate such a hierarchy: Children (0-15), Youth (16-25), Adults
(26-70), and Seniors (71+).
However, the exclusive use of priority as a decision-making parameter would marginalize
low-priority users (starvation), who would see their requests often denied, regardless of their
motivations and needs. This turns out to be a significant problem in those cases where low-
priority users have important motivations for carrying out their actions, while the motivations
of high-priority users are superficial. Consider, for example, the case where the conflict involves
two routines programmed to control the lighting in a room, the first created by a user belonging
to the Youth category, who needs to study for a university exam, and the second created by an
adult who just wishes to relax watching TV. Clearly, in an environment where people respect
each other, considering the motivations for which the routines were created, the adult user
would spontaneously grant control of the room to the younger user.
To replicate what happens in an environment inhabited by people who respect each other, the
system should be equipped with a mechanism that also considers the relevance of users’ actions
in deciding the outcome of a conflict. Each user will therefore be asked to assign a ranking
to a set of motivations for routine creation: cleanliness, energy conservation, entertainment,
health, physical activity, relaxation, rest, study, and work. We call this ordered list of motivations
user’s preference. By having the user’s preference and motivation for why they are creating a
routine, the system can resolve the conflict with an existing routine by comparing the priority
each user implicitly assigns to their routine using the preference-motivation pair. Using such a
mechanism, the virtual assistant can also accommodate the needs of low-priority users if by
this comparison their routine results to be more relevant.
The first strategy used to resolve the conflict is based on the preference-motivation pair,
which prioritizes the routine whose motivation occupies the highest position in the respective
preference rankings of the involved users. If the two motivations occupy the same position
in their respective rankings, the system attempts to resolve the conflict by applying a second
resolution policy based on users’ priority comparison. If a further case of parity is found, the
system resolves the conflict using the last policy, first-arrived-first-served, which, between the
two conflicting routines, favors the one created first.
Once the conflict is resolved, if the routine to be retained is the one that is already existing,
the virtual assistant will notify the user who is creating the second routine that it is impossible
to finish the creation process and will allow changing the routine definition to avoid conflict to
happen. Otherwise, the virtual assistant will delete the old routine, notify the affected user of
this decision, and save the most recent routine.
There is a special case where the existence of a conflict is not clearly determined: since most
actions cannot be assigned a duration beforehand, it is necessary to decide on time thresholds
for each type of action, within which the system signals the presence of a possible conflict.
Consider, for example, the case where two routines, programmed to be performed only ten
minutes apart, involve playing music from the smart speaker and turning on the television in
the same room. Since both actions do not indicate an explicit duration, a system would not
detect a conflict. There must be defined a time threshold defining an interval of time in which
the action’s effect is still ongoing: e.g., for music playback, one could think of an interval of
thirty minutes. If the routine under creation will attempt, during this interval, to access the
same device or the same environmental variable, the system notifies the occurrence of a possible
conflict and provides the user with the opportunity to decide how to behave: either ignore this
conflict possibility or proceed to prevent it. Of course, the virtual assistant can do no more
than merely inform of the possible occurrence of a conflict without trying to resolve it. This is
because the occurrence of such a conflict is not certain but only hypothesized since the virtual
assistant cannot determine a priori the actual duration of the actions. In fact, going back to
the example above, it could be that one of the two users wants to listen to music for only ten
minutes, thus not causing any conflicts. This is just one example in which the role of users and
therefore, human intervention in the decision-making process is fundamental.
5. Discussion and Conclusion
End-user development for smart homes has been studied for about 15 years, and different
AI-based approaches exploiting natural language processing and voice-based interaction are
currently proposed in literature [18, 19, 20, 21]. Indeed, in household environments, one cannot
hypothesize neither that inhabitants with a high degree of computational literacy are present
nor that computational fluency can be taught in this context like reading and writing [22]. On
the contrary, computational fluency can be encouraged and supported by means of easy-to-use
tools that may engage users in personal and shared meaningful activities [23]. We worked in
this direction, proposing to move the routine creation activity from the mobile app on a separate
device (the smartphone) to the VA itself, which supports a multi-modal dialogue based on visual
messages, touch gestures, and voice messages. We demonstrated that, notwithstanding a lower
efficiency in terms of time for routine creation when compared with the use of the companion
app, there is an improvement in usability and user experience since the users are more engaged
by such type of interaction and find the EUD activity easier to learn [5].
In this paper, we made a step further by exploiting the VA (and its smart speaker) to manage
possible conflicts that may arise in multi-user environments, favor collaboration among family
members or housemates, and improve the quality of life in smart homes. Conflict resolution in
this context can be regarded as a wicked problem in that there is no unique and always valid
solution that can be specified a priori, but conflict resolution depends on the situation and often
may require a direct debate between users, beyond the mere system use.
The approach described in this paper takes into account the design trade-off [24] between a
totally automatic conflict resolution strategy that can be implemented through AI algorithms
and a complete user-driven approach that might require a high effort from the human even when
the situation is easy to manage or can be initially screened by an automatic system. Thus, our
approach aims to realize the so-called Intelligence Augmentation [25, 26], where the technology
does not replace humans but empowers them; specifically, in our case, the system detects if an
actual or potential conflict will occur, proposes a solution when there is a clear outcome, or
provides the user with different suggestions when the conflict is not certain and more than one
solution can be put in place. This idea follows the human-centered AI way of thinking [27],
which expands from an AI algorithm-focused view to a human-centered perspective in which
humans always keep control over the technology.
Traditional human-centered interaction design methods have been adopted to implement
an Amazon Alexa Skill for end-user development of routines with conflict resolution: iterative
prototyping allowed us to obtain a preliminary version of the Skill that has been evaluated with
a 3-member family and a group of 4 housemates. All the participants were able to complete the
assigned tasks and provided positive feedback about the proposed approach to both EUD and
conflict resolution.
Acknowledgments
The authors are grateful to Giorgio Inglima Modica for his collaboration in the development of
the approach to conflict resolution.
References
[1] B. R. Barricelli, E. Casiraghi, S. Valtolina, Virtual assistants for end-user development
in the internet of things, in: A. Malizia, S. Valtolina, A. Morch, A. Serrano, A. Stratton
(Eds.), End-User Development, Springer International Publishing, Cham, 2019, pp. 209–216.
doi:10.1007/978- 3- 030- 24781- 2_17 .
[2] H. Lieberman, F. Paternò, V. Wulf, End User Development (Human-Computer Interaction
Series), Springer-Verlag, Berlin, Heidelberg, 2006.
[3] F. Paternò, V. Wulf (Eds.), New Perspectives in End-User Development, Springer, Cham,
2017. doi:10.1007/978- 3- 319- 60291- 2 .
[4] B. R. Barricelli, F. Cassano, D. Fogli, A. Piccinno, End-user development, end-user program-
ming and end-user software engineering: A systematic mapping study, Journal of Systems
and Software 149 (2019) 101–137. doi:https://doi.org/10.1016/j.jss.2018.11.041 .
[5] B. R. Barricelli, D. Fogli, L. Iemmolo, A. Locoro, A multi-modal approach to creating
routines for smart speakers, in: Proceedings of the 2022 International Conference on
Advanced Visual Interfaces, AVI 2022, Association for Computing Machinery, New York,
NY, USA, 2022, pp. 1–5. doi:10.1145/3531073.3531168 .
[6] G. Fischer, Computational Literacy and Fluency: Being Independent of High-Tech Scribes,
Hildesheim, 2005, pp. 217–230. URL: https://l3d.cs.colorado.edu/wordpress/wp-content/
uploads/2016/04/hightechscribes-05.pdf.
[7] D. Chaki, A. Bouguettaya, Adaptive priority-based conflict resolution of iot services, in:
2021 IEEE International Conference on Web Services (ICWS), 2021, pp. 663–668. doi:10.
1109/ICWS53863.2021.00091 .
[8] Y.-H. Lee, F. J. Lin, Situation awareness and conflict resolution in smart home with multiple
users, in: 2019 IEEE 5th World Forum on Internet of Things (WF-IoT), 2019, pp. 852–857.
doi:10.1109/WF- IoT.2019.8767178 .
[9] K. Nurgaliyev, D. D. Mauro, N. Khan, J. C. Augusto, Improved multi-user interaction
in a smart environment through a preference-based conflict resolution virtual assistant,
in: 2017 International Conference on Intelligent Environments (IE), 2017, pp. 100–107.
doi:10.1109/IE.2017.21 .
[10] R. Camacho, P. Carreira, I. Lynce, S. Resendes, An ontology-based approach to conflict
resolution in home and building automation systems, Expert Systems with Applications
41 (2014) 6161–6173. doi:https://doi.org/10.1016/j.eswa.2014.04.017 .
[11] A. Erfanian, T. Zeng, Y. Hu, Dynamic strategies of conflict resolution on human perception
of equality within multi-user collaborative virtual environments, in: 9th IEEE International
Conference on Collaborative Computing: Networking, Applications and Worksharing,
2013, pp. 363–370. doi:10.4108/icst.collaboratecom.2013.254052 .
[12] B. Ospan, N. Khan, J. Augusto, M. Quinde, K. Nurgaliyev, Context aware virtual assistant
with case-based conflict resolution in multi-user smart home environment, in: 2018
International Conference on Computing and Network Communications (CoCoNet), 2018,
pp. 36–44. doi:10.1109/CoCoNet.2018.8476898 .
[13] S. M. Easterbrook, E. E. Beck, J. S. Goodlet, L. Plowman, M. Sharples, C. C. Wood, A Survey
of Empirical Studies of Conflict, Springer London, London, 1993, pp. 1–68. doi:10.1007/
978- 1- 4471- 1981- 4_1 .
[14] M. S. Poole, M. Homes, G. DeSanctis, Conflict management and group decision support
systems, in: Proceedings of the 1988 ACM Conference on Computer-Supported Coopera-
tive Work, CSCW ’88, Association for Computing Machinery, New York, NY, USA, 1988, p.
227–243. doi:10.1145/62266.62284 .
[15] C. Shin, W. Woo, Socially aware tv program recommender for multiple viewers, IEEE Trans-
actions on Consumer Electronics 55 (2009) 927–932. doi:10.1109/TCE.2009.5174476 .
[16] C. Shin, A. K. Dey, W. Woo, Mixed-initiative conflict resolution for context-aware appli-
cations, in: Proceedings of the 10th International Conference on Ubiquitous Computing,
UbiComp ’08, Association for Computing Machinery, New York, NY, USA, 2008, p. 262–271.
doi:10.1145/1409635.1409671 .
[17] C. Shin, W. Woo, Service conflict management framework for multi-user inhabited smart
home, JUCS - Journal of Universal Computer Science 15 (2009) 2330–2352. doi:10.3217/
jucs- 015- 12- 2330 .
[18] F. Corno, L. De Russis, A. Monge Roffarello, From users’ intentions to if-then rules in the
internet of things, ACM Trans. Inf. Syst. 39 (2021). doi:10.1145/3447264 .
[19] M. Chkroun, A. Azaria, Lia: A virtual assistant that can be taught new commands by
speech, International Journal of Human–Computer Interaction 35 (2019) 1596–1607.
doi:10.1080/10447318.2018.1557972 .
[20] A. S. Lago, J. P. Dias, H. S. Ferreira, Managing non-trivial internet-of-things systems
with conversational assistants: A prototype and a feasibility experiment, Journal of
Computational Science 51 (2021) 1–12. doi:10.1016/j.jocs.2021.101324 .
[21] S. Gallo, F. Paterno, A conversational agent for creating flexible daily automation, in:
Proceedings of the 2022 International Conference on Advanced Visual Interfaces, AVI 2022,
Association for Computing Machinery, New York, NY, USA, 2022, pp. 1–8. doi:10.1145/
3531073.3531090 .
[22] A. A. diSessa, Changing Minds: Computers, Learning, and Literacy, MIT Press, Cambridge,
MA, USA, 2000.
[23] G. Fischer, E. Scharff, Meta-design: Design for designers, in: Proceedings of the 3rd Con-
ference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques,
DIS ’00, Association for Computing Machinery, New York, NY, USA, 2000, p. 396–405.
doi:10.1145/347642.347798 .
[24] G. Fischer, Identifying and exploring design trade-offs in human-centered design, in:
Proceedings of the 2018 International Conference on Advanced Visual Interfaces, AVI ’18,
Association for Computing Machinery, New York, NY, USA, 2018, pp. 1–9. doi:10.1145/
3206505.3206514 .
[25] D. C. Engelbart, Toward augmenting the human intellect and boosting our collective IQ,
Commun. ACM 38 (1995) 30–32. doi:10.1145/208344.208352 .
[26] G. Fischer, A research framework focused on humans and AI instead of humans versus
AI, in: B. R. Barricelli, G. Fischer, D. Fogli, A. I. Mørch, A. Piccinno, S. Valtolina (Eds.),
Proceedings of the Sixth International Workshop on Cultures of Participation in the Digital
Age: AI for Humans or Humans for AI? Frascati, Italy, June 7, 2022, volume 3136 of CEUR
Workshop Proceedings, CEUR-WS.org, 2022, pp. 1–8.
[27] B. Shneiderman, Human-centered AI, Oxford University Press, 2022.