The adoption of smart homes is steadily increasing and users expect their homes to act in intelligent ways. One core question is how to learn from user behavior to gain smartness that can be adopted by the system. This paper has a twofold contribution: first, we propose a new approach for letting smart homes learn collaboratively. Second, we present a prototype implementation and discuss challenges learned from this version. This approach should serve as an initiator for the development of intelligent smart home systems outside isolated solutions and on the basis of human needs.
The acceptance of smart homes is steadily increasing, as statistics show that the number of active households is expected to reach 111.2 million by 2023 [Statista2019]. At the same time users expect their homes to act in intelligent ways, instead the interaction of today’s smart home systems consists of a modern variant of the ”window-icon-menu-pointer” paradigm. But an almost 40 years old paradigm ”should not mislead us into thinking that it’s an ideal interface” [Van Dam2000]. The functionality of physical switches has been shifted to mobile phones in the last decades. In this way we are able to digitize interactions, but at the same time we create a more complex multilevel action with a mobile phone – for example for a simple action like switching on the light we have to wake up the phone, switch on the desired app, and finally turn on or off a particular lamp. In the following we propose an approach with which we want to foster a discussion about collaborative ambient intelligence in the context of smart homes. The idea of this work is to capture the interaction with products digitally and on this basis to identify possible subsequent steps and to suggest them to the user. The collaborative Smart Home network is intended to promote the learning of human behavior with Smart Home systems. Patterns already learned can thus be exchanged between different systems and prevent intensive and computational new learning. 2
Cook et al. [Cook2003] present a concept predicting the behavior and habits of users. This results in certain behavior patterns that are repeated almost constantly and can be operated by the system. Two further publications followed in 2006 by De Carolis et al. [De Carolis2006B, De Carolis2006A]. The basic idea of these publications is an agent-based architecture. Here the so-called Butler Interactor Agent serves as a mediator between those agents who control the intelligent devices in the house and the user or residents. The butler should be able to learn from the user’s preferences and act according to them. In case of critical decisions the user should always have the possibility to intervene.
In [Cavone2011] Cavone et al. deal with the choice of suitable workflows on the basis of the identified context. The workflow describes all actions that lead to the desired goal or need of the user. The choice of such a workflow is made by the butler agent based on the information of sensor agents and then proposed to the user as a service. The user can accept, reject or change the choice of the butler agent. Based on the user’s acceptance, a learning behavior of the system takes place in order to be able to react better to future events.
Gupta et al. [Gupta2014] deal with the communication between vehicles to find a free parking space by means of swarm intelligence. A particle swarm optimization algorithm calculates the shortest route to a free parking space if there are several in the immediate vicinity. Context awareness is made possible by a variety of sensors attached to the vehicles. Despite the focus on ”safety and comfort measures for road traffic” [Gupta2014], the essential process sections, especially in the area of information transfer and the use of a mesh network, can be considered as a possible solution to exchange the necessary information between different smart homes in the immediate vicinity. 3
The physical environment of a smart home can be divided into a number of smaller entities – e.g. the bedroom, living room, kitchen, etc. Each one captivates with its own complexity. The human behavior can vary through every change of environment. The closer automation is to be tailored to people, the more complex it will become. For this reason, a different perspective might be advisable. Instead of tailoring automation to people, the actual functionality could be designed to serve people. So instead of letting a system learn the human behavioral repertoire, you can let the system learn how they interact with their environment. The focus is thus placed on an object that cannot be changed, e.g. a light switch. 3.1
Human Interaction with Smart Home The humans interaction with their environment can be described as a sequence of single actions. Figure 1 shows a very simple and hypothetical representation of a coming home scenario. The sequence shows the arrival at home, the switching on of the light, the making coffee as well as the closing of the windows. The behavior extends over several rooms and time and may even overlap. These individual actions are now used in the concept for the recognition of patterns and subsequent description of rules.
The recognition and applying of patterns in the context of human behavior is one of the more difficult areas. Sensors are used to detect the interaction of a user with a smart home component – or the component itself is able to verify the interaction as such. In order to make a more robust statement about the context, it is useful to aggregate the information of different sensors. For example, the duration of the switched-on light and possible movements in a room can be used to determine the time spent in that room more precisely. Let’s assume that the string ”AFCJ” from figure 2 represents a recognized pattern. Each of the individual characters stands for a stay in a certain environment (e.g. the living room). If one is now able to determine the time spent in an environment, it would also be possible to predict when the person will change environment. The aggregation of sensor data can thus make time-critical statements possible. In this way, the necessary time factor can also be taken into account when interacting with Smart Home components.
At this stage, there are several approaches to implement a system. We propose an approach in which the system determine the most likely follow-up action after each user interaction and propose it to the user. The system then learns from the user’s approval and/or rejection in order to make more robust proposals. The System has the opportunity to perform various subsequent steps without the user’s consent, provided a general consent has been obtained. Finally, the system converts the recognized patterns into executable rules. Similar to current smart home gadgets, rules would then be used to execute previously defined actions under certain conditions. These rules serve to create a collaboration between smart homes. In order to counter the cold start problem and increase the learning behavior of a system, the defined rules are then distributed to different Smart Home systems. This allows residents to use intelligent functions of their smart home from the very beginning. This eliminates the need for several weeks of training. Smart Homes thus learn collectively. For the distribution of rules, we propose the approach of a mesh network based e.g. on WLAN. Individual houses form the nodes of a larger network that can transmit information independently of the Internet. Houses further away form their own network. Network settlements can develop, which can also form larger areas up to cities. Since there is no central administration compared to the cloud service, this represents a more secure communication option for the distribution of sensitive data.
To ensure compatibility in different smart homes, we propose an ontology that can map individual smart home components to other homes. Systems should thus be able to apply foreign rules independently of the names chosen by the user. Ontology thus forms a link between smart homes and makes collaboration possible. 4
The first prototype implementation is a small scale system. A application case is used to describe how a person comes home in the evening and interacts with various components. Due to the lack of a real test environment, the prototype consists of two Raspberry Pis including GrovePi Shields from Dexter Industries to simulate two collaborative homes. Figure 3 shows the test setup with one of the used Raspberry Pi. The sensors and actuators are connected to the expansion board. This simulated environment includes that the sensors are represented by buttons and the actuators by simple LEDs. MQTT is used for the communication between the sensors, the smart home system and the actuators. A Markov chain algorithm is used to recommend possible pending interactions. For this purpose, each user interaction is appended to a sequence of behaviors as a single element. In this way, the behavior sequence is constantly extended. At the technical level, the sequence is merely a series of alphabetical characters that can be processed by the algorithm. After each user action, the system checks which is the most frequently occurring successor element in the behavioral sequence. The identified element represents a physical Smart Home device and is then proposed to the user as an automatism or executed by the system itself if a general agreement for automation has already been made.
The collaboration, i.e. the exchange of learned knowledge, was also implemented using MQTT in the first prototype. A prerequisite for the distribution of rules is, of course, that a certain interaction repertoire has already been learned by the system. The action sequence can then be sent by the user as a single string to an MQTT topic. Since the individual Smart Home components are represented as simple alphabetical characters during the first implementation, another Smart Home simply has to insert the action sequence into its own system, so that the application of externally learned automatisms works well according to expectations. In the other Smart Home, after each user interaction, the system suggests the most likely Smart Home component to follow, as it has learned this from the collective. It checks the interaction repertoire it has received from the other smart home to propose a following smart home component for the last interaction performed by the resident. This paper describes a new approach and initial prototype for a smart home that learns from interactions with its components and distributes what it learns to other smart homes for collective development. However, based on our prototype we gained additional insights for improving our approach and implementation. The result of this work opens some questions and challenges that need to be considered further.
Future considerations are still a more robust prediction of possible subsequent steps. A relevant tool for this could be machine learning including classification, forecasting, sequence modeling, or a combination of several approaches. A further point is the distribution of the learned behavior sequences or the rules derived from them. It would make sense to implement the mesh network described above and to test it in a concrete application case. This results in the necessity to consider the aspect of data protection. In the context of this work, the focus was primarily on interaction, pattern recognition and collaboration. Aspects of data protection will be considered in future work. Our prototype works for one-person households; for multi-person households we need to distinguish events generated by different individuals. In addition, human behavior can be very erratic, which must be relevant in future considerations. A not insignificant aspect, which must be considered in further work, is the collaboration of the Smart Homes. An ontology must be developed that allows semantic assignment of components from one house to another. In this way, an adequate implementation of collectively learned behavior patterns in newly networked smart homes can be ensured.
We conclude, that our approach for collaborative smart homes is a solution for learning smart home behavior among multiple stakeholders by distributing them in the collective. It is an approach to address the cold start problem. However, the questions discussed previously need to be addressed in future work.