Action Inferences from IoT Devices: a Risk Detection Case Study Applied in Smart Home Abdullah Alhejaili?? a.e.s.alhejaili@lboro.ac.uk, Department of Computer Sciences, Loughborough University, UK Abstract. Internet of Things (IoT) is a recent hot area in both academic and industrial interests, which has been proposed to address many ap- plication domains, including e-health, smart car, smart city, and smart home etc. While IoT paradigm allows to connect heterogeneous objects (e.g., sensors and/or devices), understanding their relationships can help to derive smart actions and useful predictions. In this research, we focus on action derivation from smart home by analysing Things relationships and their correlations. We consider fire risk detection problem as a case study to evaluate different critical as- pects of the approach, including precision, scalability, performance, and effectiveness in terms of cost and complexity. We are currently working on building a simulation tool as a base of the study using some of the cutting-edge technologies, such as OWL ontology, semantic web services and data mining algorithms. Keywords: Internet of Things, Smart Home, Home Automation Sys- tem, Event Recognition, Risk Detection. 1 Introduction Internet of Things (IoT) is a promising evolution in the next big wave of web and internet technologies. Under its vision, everyday objects (such as people, services, devices, and sensors) need to be interconnected and smartly able to communicate in a constructive and sensible ways to provide perfect services [12]. Concerning Smart Home (SH) as one of the hottest IoT application domains [8], different distributed devices inside and outside home (e.g., light bulb, radiator, cooker, and tab etc.) should be network-enabled (using, e.g., ZigBee or Wi-Fi) [4]. The connection of these devices as well as their relationships can be exploited to support many desire and intelligent services, including energy saving, security and safety, risk and fire protection. With respect to the latter (i.e., fire protection service), the UK-government reports [2,1] that the accuracy of the existing solutions for detecting fire inci- dences is not good enough. They discover that there is approximately 40% of ?? I’m a full time PhD candidate under the supervision of Dr Shaheen Fatima (S.S.Fatima@lboro.ac.uk). I have officially started on January 2016, and this is my second year. II false-alarm incidents, attended by fire and rescue services. In addition, their re- port mentions several key reasons behind fire-alarm failure. One of these reasons is that the detector has a limited range to cover, resulting in many true-negative cases (e.g., when a real fire incident occur and the smoke did not reach the de- tectors). These limitations have motivated us to provide an IoT based solution that can predict for some specific and potential fire incidences in advance. The idea is to analyse the behaviour of SH entities, focusing on their relationships, to infer essential actions. In this research, we are generally interested in detecting unexpected actions in smart home by analysing Things relationships and their correlations. We seek to investigate the derivation techniques of smart actions in principle, and then practically evaluate their precision, scalability, performance, and effectiveness in terms of cost and complexity. To this end, we apply our research on SH domain for detecting fire incidences as a motivating case study. Currently, we are investigating some of the cutting-edge technologies to adopt, such as OWL ontology, semantic web services and statistical data mining algorithms. The remainder of this paper is organised as follows. section 2 presents briefly work related to event based detection techniques. Sections 3 and 4 respectively present our research questions and proposal, focusing on SH application domain. Then, section 5 discusses the current status of our work and the plan for the next activities. Finally, section 6 concludes the paper. 2 Related Work Existing event-detection approaches fall roughly into one of the two categories: ontology-based [11,6] and sensor-based [9,3] approaches. For example, [6] propose an ontology-based and a rule-based reasoning (so-called SWRL) approaches for risk detection and/or service decision support in SH management. They de- veloped a prototype tool that monitors all SH environments, including specific sensor readings that describers neighbourhood behaviours, for providing real- time suggestions. Their approach is extendable, i.e., flexible for defining a new or omitting an existing SH’s entity from the system with no time restriction. This seems a good feature in general, but applying it frequently could affect detection’s accuracy. In [3], they suggest a SH with the purpose of promoting safe environment methods, relaying entirely on wireless sensor networks. One of the notable features, in their protocol, is the possibility of converting old home to be smart using sensing techniques. In SH, different detection activities are typically implemented, each to reach a specific goal. For instance, a well-SH system should provide healthcare services by monitoring resident’s movement or body condition. Whiles, other features such as safety-service would require monitoring different entities in a different mechanism such as observing daily habits (e.g., cooking) of home residents. [5] classify broadly user activities for event-detection into four types: – Single user sequential activities, where a single user performs only one ac- tivity at a time. III – Multi-user sequential and simultaneous activities, where more than one user perform the same activity, e.g. drinking tea together. – Multi-user collaborative activities, where multiple users perform different activities cooperatively to achieve the same gaol, e.g. more than one user are cooking together. – Multi-user concurrent activities, where multiple users perform different ac- tivities independently aiming to different gaols, e.g. one user is watching TV while the other is cooking. To our knowledge, there are two closely relevant proposals to ours that con- sider explicitly fire detection problems [10,7]. These proposed solutions we aware of are principally fire alarm or video based system. They are mostly devoted to address different fire accidents based on their real occurrence, but not factually beforehand. The fundamental techniques behind the current solutions are rang- ing from conventional devices, relying on, e.g., smoke or temperature measure- ments, to the high smart image processing solutions, such as flickering colours or video analysis system. Despite this, the ability of maximising home protections based on in-advance risk detection has not been addressed yet. 3 Research Hypothesis and Questions SH supports integrating different devices and systems to be managed by a single control unit. It allows automating some actions without setting a timer or making an explicit request. Rather than monitoring or controlling home environment, SH needs to be more intelligent for generating smart actions, such as detect- ing unobvious fire risk cases or notifying effectively for future expectations in a dynamic way. Our main hypothesis is that “following the IoT paradigm by us- ing the concepts and innovative technologies (such as OWL ontologies, semantic web services, statistical machine learning algorithms) would make home infras- tructure more interactive, smart and aware in providing better/robust services”. To shed lights on our objectives behind this hypothesis, we list two conceptual questions that may indicate the contributions we seek to make as follows: – What kinds of smart buildings an action prediction approach like we propose is useful for, and on which kinds of Thing’s relationships this approach is more effective? – What are the technical pros and cons of using IoT paradigm, including, com- plexity, scalability, performances, and usability? Generally, these questions can be addressed by building up SH’s simulation that allows to compare different risk scenarios with the ability of identifying the key pros and cons of the approach. Having such simulation would also help to explain whether following the IoT paradigm for smart home risk detection is beneficial as compared to the other alternative approaches. IV 4 Proposed Approach and Preliminary Work To address the research questions of our research, we intend to examine two well-known techniques: (1) Ontology based structure to represent home model in a logical way, allowing to design entities, sensors, and their complex relation- ships. And (2) Machine learning to maximise home services by enabling some entities controlling automatically another entity without explicitly defining pre- request for every action. These two methods are to fulfil the main requirements of building a framework architecture for our research. This architecture would rely principally on three main components: a workflow simulation to simulate home entities in terms of generating input/output reading, controlled by a generic in- terface; a database schema designed by OWL-Ontology for storing all data; and an adapter to integrate our tool with a machine learning tool for encoding the recoded data as well as collecting inferred actions. Fig. 1: Our informal SH model on the top and the defined top level ontology classes with (is-a) relation between them on the bottom Progress so far. Fig. 1 describes briefly a suggested model for our SH. It consists of several connected entities to the local network, and each entity has a unique id such that its status (e.g., on/off) can be checked. Sensor S3, as an example, is used to detect the presence of people inside the kitchen. Here, if the V Fig. 2: An example of a training data set on the top (consisting of 11 classes and 28 instances), and generated decision tree from it on the bottom Cooker is off and nobody insides the kitchen, no risk can be detected unless something unusual occur, which can be detected by smoke or heat alarm. Even though it is highly recommended for people to stay in the kitchen while cooking, sometime people may forget to switch off their cooker before they leave. In re- sponse to this recommendation, if SH is capable for analysing the relationships between, e.g., Cooker and S3, optimising fire risk detections in terms of alerting as early as possible for any expected risk can be achieved. Consequently, house- holders can take, in advance, further actions, e.g., going back to the kitchen or switching off the Cooker remotely, etc. We have tried out to check conceptually the validity of our proposal, using the extracted dataset, see Fig. 2. In principle, this dataset describes the learning inputs such that the classes (i.e., defined in Fig. 1) appear as columns, and each instance, representing reading data, appears as a row. In this preliminary experiment, we assume that the 28 instances (see, Fig. 2) are already extracted manually by defining the behaviours of the entities, i.e., modelled in Fig. 1. For example, (rows from 1 to 21) describe some obvious cases that no risk event needs to be generated. They cover the cases when the Cooker is either off (rows 1 - 17), or in-use as normal by someone in the kitchen, indicated by S3 (rows 18 - 21). The rest of the rows cover some expected risk cases, including the absence of householders in the kitchen while the Cooker is ON (rows 22 - 27), or somebody already in but the utilisation of the Cooker exceeds the normal average time (row 28). The latter can be a target to fainting cases, especially for elderly people, in which it increases the level of their protection. Nevertheless, we intend to develop a home simulation system to generate interactively many instances for evaluating a variety of different scenarios. The bottom part of Fig. 2 illustrates the result obtained by RandomTree algorithm, graphically visualised as a decision tree of 7 nodes. This decision tree allows to determine whether a risk event must be triggered based on the current real-time reading data. We have conducted another simple experiment on the same dataset (see Fig. 2), applied by Apriori algorithm, to illustrate a different useful type of learning output. Listing 1.1 describes the result obtained, which represents asso- VI ciation rules between some entities. Such results can help in understating many or probably all risk cases, but more importantly, it can enhance the learning decision by avoiding false-positive actions. For example, rule (Cooker=False ==> Action=no) states explicitly that no action is required if the Cooker is off. Cur- rently, this action could not be inferred by our dataset as no instance representing it. However, the association rules can be used as a validation (e.g., testing the accuracy of the inferred decision tree) by filtering out any false-positive action, generated imprecisely with the condition (e.g., Cooker==False ). Listing 1.1: Learning output which describe association rules between entities 1. Cooker=F a l s e => S e n s o r 3=F a l s e . . . 2. Ahmed smartphone=F a l s e => S e n s o r 3=F a l s e . . . 3. d u r a t i o n=none => S e n s o r 3=F a l s e . . . 4. d u r a t i o n=none => Cooker=F a l s e . . . 5. Cooker=F a l s e => d u r a t i o n=none . . . 6. Cooker=F a l s e => A c t i o n=no . . . 7. d u r a t i o n=none => A c t i o n=no ... 8. Cooker=F a l s e d u r a t i o n=none => S e n s o r 3=F a l s e ... 9. S e n s o r 3=F a l s e d u r a t i o n=none => Cooker=F a l s e ... 10. S e n s o r 3=F a l s e Cooker=F a l s e => d u r a t i o n=none ... 5 Research Plan and Current State Table 1 outlines our research progress as well as the main activities with time line that are expected to be made by December 2019. The fourth and fifth rows of the table present the current status of the work. # Description of task From To 1 Gaining some deep knowledge about IoT concepts and/or approaches to pick up completed some interesting research questions. 2 Proposing a conceptual solution, for the problem discussed in Sec. 3, that generally completed relies on the derivation of smart actions by analysing behaviours of IoT devices. 3 Investigating some supporting tools for logical structure and data mining analysis. completed 4 Completing the initial SH model using OWL as a logical database structure, and 80% completed Java/C# for implementation. We plan to use Protg 5 to design the entire entities of our suggested SH’s model and their relationships. This would promote extracting significant actions by relying on SPARQL queries. 5 Developing a discrete-event SH simulation for generating input/output data that 10% completed describes the behaviour of SH’s entities. 6 Developing an interactive graphical prototype tool that combines both the SH Jul 17 Sep 17 simulation and OWL database schema. This tool should provide a flexible way to define examples of resident’s behaviours as well as SH’s devices in a workflow manner, allowing to evaluate different fire risk scenarios. 7 Analysing and evaluating the results (obtained from step #6), and then writing Oct 17 Dec 17 up of a publishable document. 8 Optimising our prototype tool to include some data-mining algorithms for making Jan 18 Mar 18 precise decisions to the cases that are not defined in the workflow. Here, the plan is to adopt Weka 3.0, which supports an API interface for processing, evaluating and visualising all learning steps, starting from data preparation step to the analysis of obtained learning outputs. 9 Analysing and comparing the results with (from row #8) and without (from row Apr 18 Jun 18 #6) using data-mining algorithms to assess their impacts on derived actions. 10 Modifying the first version of our SH model to represent different home character- Jul 18 Sep 18 istics. This is to evaluate the impact of applying our approach on different kinds of buildings. 11 Surveying the existing non-IoT related approaches to give impression on how much Oct 18 Dec 18 adopting IoT paradigm for SH risk detection is beneficial. 12 Thesis writing up Jan 19 Dec 19 Table 1: Research progress and expected activities VII 6 Conclusion In consonance with computable existing fire-protection solutions, our IoT ap- proach will be conceptually modelled to be complementary to the recent ad- dressable risk detection devices for validation and evaluation only. Therefore, to generalise the main contributions of this approach, we will investigate empirically how our derivation technique of unexpected/smart actions can be customised to suit different IoT applications and specifications. References 1. Fire statistics data tables - statistical data sets - GOV.UK. https://www. gov.uk/government/statistical-data-sets/fire-statistics-data-tables., accessed: 2017-05-20 2. Gaught, P., Gallucci, J., Smalldridge, G.: Fire statistics england, 2014/15 (pub- lished on 2016). https://www.gov.uk/government/uploads/system/uploads/ attachment_data/file/532364/fire-statistics-england-1415-hosb0816. pdf., accessed: 2017-05-20 3. Ghayvat, H., Liu, J., Mukhopadhyay, S.C., Gui, X.: Wellness sensor networks: A proposal and implementation for smart home for assisted living. IEEE Sensors Journal 15(12), 7341–7348 (Dec 2015) 4. Gill, K., Yang, S., Yao, F., Lu, X.: A zigbee-based home automation system. IEEE Trans. Consumer Electronics 55(2), 422–430 (2009) 5. Gu, T., Wu, Z., Wang, L., Tao, X., Lu, J.: Mining emerging patterns for recognizing activities of multiple users in pervasive computing. In: 6th Annual International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Ser- vices, MOBIQUITOUS 2009, Toronto, Canada, July 13-16, 2009. pp. 1–10 (2009), https://doi.org/10.4108/ICST.MOBIQUITOUS2009.6818 6. H, X., Y, J., Z, X., C, S.: A semantic approach with decision support for safety service in smart home management. Sensors 16(8), 1224 (2016) 7. K, Y., Kim, A., J, H.: RGB color model based the fire detection algorithm in video sequences on wireless sensor network. IJDSN 10 (2014) 8. Pavithra, D., Balakrishnan, R.: Iot based monitoring and control system for home automation. In: 2015 Global Conference on Communication Technologies (GCCT). pp. 169–173 (April 2015) 9. Suthokumar, G., Tharmakulasingam, J., Johnnirajh, A., Subendran, P., Dias, D.: Smart building - towards the insight remotely. In: 2016 International Conference on Internet of Things and Applications (IOTA). pp. 395–400 (Jan 2016) 10. V, S., L, P., Van de W, R., M, B., S, B.: State of the art in vision-based fire and smoke detection. In: International Conference on Automatic Fire Detection, 14th, Proceedings. vol. 2, pp. 285–292. University of Duisburg-Essen. Department of Communication Systems (2009) 11. W, K., I, M., B, M., S, T., L, A., T, Y.: Activity recognition using context-aware infrastructure ontology in smart home domain. In: 7th International Conference on Knowledge, Information and Creativity Support Systems, KICSS, Melbourne, Australia. pp. 50–57 (2012) 12. Whitmore, A., Agarwal, A., Xu, L.D.: The internet of things - A survey of topics and trends. Information Systems Frontiers 17(2), 261–274 (2015)