Spatiotemporal knowledge representation and reasoning under uncertainty for action recognition in smart homes Farzad Amirjavid, Kevin Bouchard, Abdenour Bouzouane, Bruno Bouchard Department of mathematics and computer science 555, university boulevard, Chicoutimi, Quebec, Canada G7H2B1, University of Quebec At Chicoutimi (UQAC) farzad,amirjavid@uqac.ca Kevin.bouchard@uqac.ca Abdenour_bouzouane@uqac.ca Bruno_bouchard@uqac.ca Abstract Although uncertainty and imprecision is included always We apply artificial intelligence techniques to perform data with the action recognition field, in most of the performed analysis and activity recognition in smart homes. Sensors researches up to now [1,6,11,13,14,15,16,23,24] the embedded in smart home provide primary data to reason about existing uncertainty and imprecision in OA‟s behavior and observations and provide appropriate assistance for residents to home state is not considered and they are not robust if complete their Activities Daily Livings (ADLs). These residents activity realization models change. Furthermore, any small may suffer from different levels of Alzheimer disease. In this change in sensors network, sensors locations and sensors paper, we introduce a qualitative approach that considers number could lead to restricting all their applied models spatiotemporal specifications of activities in the Activity and all the previous training tests would not be useful any Recognition Agent (ARA) to do knowledge representation and reasoning about the observations. In this paper, we consider more. Moreover, objects movement, which provides different existing uncertainties within sensors observations and important information in activity recognition, has not been Observed Agent‟s activities. In the introduced approach if the considered. more details about environment context be provided, the less Most of the surveyed activity recognition approaches do activity recognition process complexity and more precise not tolerate relatively detailed information about the real functionality is expected. world and even they may avoid more sensors for not to receive complementary information about the activities. 1 Introduction The reason is that increase in number of applied sensors Smart home mostly addresses the health-care problem of could lead to process complexity and they would need a performing automated assessment of functional health for huge dataset for training. I contrast, the introduced elder adults and provision of automated assistance that will approach in this paper welcomes the increase in input allow people suffering from Alzheimer to remain information and in the case of change in sensors network independent [16]. In order to live independently at home, structure, and the old knowledge would be still valid. adults need to be able to complete key activities of Daily Furthermore, the increase in provided information would Living, or ADLs, however tracking of ADL even cause to decrease in process complexity. accomplishment is a time consuming task for caregivers. In this paper, we are explaining an intelligent agent that To provide automated assistance we apply Activity tries to explain the observations and detects anomalies in Recognition Agent (ARA) to reason about observations the case that there is no explanation. Applied knowledge provided by the embedded sensors in Smart Home. representation and reasoning techniques that benefit from In this paper, we deal with the activity recognition process activities temporal and spatial specifications is discussed performing in Activity Recognition Agent (ARA). Event and we introduce fuzzy contexts that can briefly indicate Recognition Agent (ERA) detects realized events and the home state and possible events that could occur in report them to the ARA. ARA provides a report for the contexts. Plan Recognition Agent (PRA) about observed and The art of ranking and classification between generated inferred activities and finally the Assistance Provision hypotheses inferred from available knowledge and present Agent (APA) would provide appropriate assistance for the observations can lead to better adjustment between Observed Agent (OA). The schema1 shows the general system‟s inference and the real world. In this way, process in the smart home. reasoning can be less complicated and so it causes less error to choose the right decision in decision-making 2.2 World state and fuzzy context process. “Fuzzy Context” is the term used to express the home state A brief explanation about general activity recognition with it. In this way, environmental parameters (called items process is that after that ERA provides ARA the current and indicated by ix) are measured and then fuzzified by home state and happened events in fuzzy context and fuzzy fuzzy membership functions. To express a general form of events frame (knowledge representation), the possible fuzzy context, we apply the following form: hypotheses through time line are generated and ranked dynamically. Then in the reasoning process, the C (i1, i2 ,..., in ) explanations about observations would be provided. Temperature is an example for item. For instance, when the thermometer indicates 37 degree it can be inferred that it belongs to “Warm” class (applying fuzzy roles and defuzzification functions) and finally warm is reported instead of 37 degree. Home state is finally formed by such this information. As a simple example for home state, consider a home that includes some embedded sensors to indicate the home state. These sensors indicate “OA location”, “door state”, “heater state”, “oven state” and “temperature”. Mentioned sensors generate continuously schema1- the general smart home process model values along time axis. The following indicates the final defuzzified home state: 2 Knowledge Representation Cobst (OA : at _ oven, door : closed , heater : off , Oven : off , temperature : warm) A knowledge representation system is applied to interpret sentences in the logic in order to derive inferences from 2.3 Events them. When we design a knowledge representation system, We define events as each meaningful change in sensors we have to make choices across a number of design spaces. generated values. ERA simply receives generated values The single most important decision to be made, is the from the sensors and checks whether the value belongs still expressivity of the KR. Our desire is to include more to a new class. A change in received values class means an effective parameters in action recognition process who event has happened and the event is reported to the ARA. may make the knowledge representation enough expressive and may make the reasoning process not so relatively 2.4 Discussion difficult. Brahman and Levesque [1984] introduced the Allen temporal logic is a famous temporal logic that mentioned desire as contradictory goals; however, we introduced thirteen temporal relations between actions. believe that applying fuzzy context can lead to more Morchen argued that Allen‟s temporal patterns are not expressiveness and simpler reasoning in an intelligent robust and small differences in boundaries lead to different agent. That is because fuzzy context holds more details at patterns for similar situations [2]. Furthermore, the one hand and at the other hand the defuzzified context complexity increases if the OA performs multiple actions prevents to generate many relatively similar contexts that simultaneously. Moreover, it does not also indicate the can make the reasoning process complicated. Here we actions beginning and terminating moments. introduce two key knowledge types and their From the mentioned problems, we have inspired the idea representation methods. that we can consider the beginning event (temporal point) instead of interval consideration and so in this way, it 2.1 Environmental parameters or context items would be necessary just to compare beginning points of actions and their durations would be justified as their components that contain fuzzy, relative and estimative Embedded sensors in the smart home provide primary data measures as value. So, in brief it can be said that only the for the Activity Recognition Agent (ARA). The received before relation would be considered and the possible data by sensors that is raw and unprocessed introduce the moments that other actions can begin on them. environmental components (such as temperature, doors state, heater state, Observed Agent‟s position, etc) that may To implement the mentioned idea we have applied the be effective on action recognition process. In fact, the possibility theory that was first introduced by mentioned components form the body of contexts and are Zadeh[7,8,9,10]. In summary, it is assumed that after named as context items. Unfuzzy context is a context that observation of an event, all the possible actions can begin is constituted from a set of items and we define fuzzy simultaneously and the most possible moments for events context as context constituted from fuzzified items. occurrence is indicated. The farther from most possible occurrence moments the less ranking value in hypotheses however, this knowledge can include some temporal ranking introduced in “3.3” section. information about next possible contexts that can possibly happen in future. We refer to the first introduced type as The result is that multiple simultaneous running actions absolute time and the second one as relative time. can be considered and it is enough flexible to consider different possible temporal relations between actions and To represent temporal dependency (absolute time), we gives an estimation (by defuzzifying the fuzzy time up to insert a new item to the fuzzy context ontology that is next Action‟s beginning moment) to predict the action called fuzzy time item. In this way, contexts for similar termination moment. conditions but different temporal conditions are made. A For example, for the action entering to the kitchen, the function is implemented to check whether the current time table1 indicates the possible events (actions beginning is adjustable to the defuzzified time item existing in the points) that are possible to occur after previously assumed fuzzy context. occurred events and their possible occurrence moments. Time elapse as a possible fuzzy event is also applicable. An example for defuzzified item of fuzzy time can be like “morning”. To represent temporal information (relative time) using a fuzzy trapezoidal digit, we indicate the possible transition moments to different possible contexts and it is implemented by a simple table containing the concerning data. This relative data is converted to the real time at the running time. Table1. Possibility distributions for relations between events for 2.6 Spatial Knowledge Representation action “entering to the kitchen” Another key knowledge that is helpful to do better reasoning is spatial knowledge that indicates the context dependency to the objects locations. As it was mentioned earlier, movement of objects in the real world provides noticeable information for the activity recognition process. There can be considered two general spatial knowledge Schema2. Schema2. Possibility fuzzy distributions Possibility temperature possibility for distributions occurrence classes distributions for occurrence moments moments for occurrence moments forms. The first one, which would be referred to as absolute position, indicates the objects positions in the real In the table1 the possibility distribution for the “before” world and the second one that would be referred to as relation is indicated by the normalized numbers (from 0 to relative positions indicates the position of objects to each 1) and in schema2 the possibility distribution for possible other. In the fuzzy context, a section is dedicated for the occurrence moments of the next event is indicated by objects positions in the home (first spatial knowledge type) t e1:e2 which is a trapezoid fuzzy number. In this digit t1 is and the second spatial knowledge type is indicated in the Event Recognition Agent (ERA). One example for the soonest moment that event2 can occur after another absolute position application in activity recognition is that, event1, moments between t2 and t3 are the most possible to infer the cooking activity it is necessary to observe the moments that event2 can occur and t4 is the latest moment pan on the oven. An example for the relative position that event2 can occur. (We have forborne to include the inference is that if approach of pot to glass be observed it necessity distributions in our calculations, which is already can inferred that OA has fulfilled the glass with the pot‟s dependent to the possibility distributions.) containing liquid such as coffee. ERA provides this The table1 is implemented as a data table in database and it information as recognized event for the ARA. indicates the effective environmental parameters to recognize the action “entering to the kitchen”. 2.6.1 Discussion 2.5 Temporal Knowledge Representation To recognize objects movements we have applied RFID We define the term temporal knowledge as a kind of tags and antennas. This process is done in ERA and we knowledge that is dependant to the time and may lead to would have a short introduction of it in here. different inferences in different temporal contexts; In a brief description, we have attached RFID tags on the 3 Reasoning objects and used RFID antennas to recognize the OA‟s activities. We have made a program in Java to recognize The reasoning process in activity recognition follows the the performed activities by the OA. Every six observation, hypothesis generation and hypothesis pruning microseconds applied RFID antennas check the steps. environment to detect the RFID tags. By having just one RFID antenna and attaching RFID tags on the objects, we are able to recognize if the object is close or far from the 3.1 Hypothesis Generation antenna. By adding the second antenna, we would be able to make four regions. The first region is the region around Hypotheses are generated only in the case of event the first antenna, the sec ond region would be around the recognition reported by the ERA. Movement of objects, second antenna, and the third region is the region in front elapse of time and a switch in controlling sensors states are of both antennas and the region that both antennas show possible observable events. In fact, the generated equal signal strength to detect the objects and the fourth hypotheses indicate the possible future contexts could region is the region that no antenna can easily detect the possibly be observed in the future. object (see schema3). The hypothesis generation process in summary is that at first hypotheses are generated based on a table named as possible fuzzy events (see table1) that could have been generated in future in the current context. At the second step, they are assigned the possible observation moments by the use of trapezoidal fuzzy digit (see schema2) and finally they are ranked or weighted (see part 3.3). the mentioned process is illustrated in schema4. Schema3. Regions defined by RFID antennas In ERA, entering and exiting a region is recognizable by the available equipments and the concerning events are reported to the ARA. In the absolute position recognition, it„s enough to find the object‟s location in one of the mentioned regions, however in relative position recognition we should find two target objects in one region. 2.7 Spatiotemporal Knowledge Representation Schema4.hypothesis generation Spatiotemporal knowledge is key environmental 3.2 Hypothesis Generation through the time line information to do activity recognition; however, there is other effective environmental information such as temperature, door‟s position and other items that are also Considering uncertainties for unrecognized but in reality useful for controlling affairs in smart home. to represent happened events (there are several reasons for it), it is such this knowledge we have divided fuzzy context into possible that it defects the reasoning process and so ARA three major sections. One section for temporal knowledge, wrongly detects normal actions or activities as anomaly. another section for spatial knowledge and third section for To improve the activity recognition efficiency we consider controling items is provided. that possible events may have happened but not observed and they are generated and pruned through the time line. A question that may arise in here is that what could be the Introduced fuzzy context let us consider different occurrence time of undetected event? The answer is that knowledge types in action recognition and the controlling the defuzzified value of the fuzzy trapezoid number can affairs (using checking functions) are done at the transition indicate the possible moment that the event has happened. moments. Transition between contexts is also indicated by In the case of anomaly detection, it would be checked the observed fuzzy events reported by ERA. whether there have been no undetected event and there is no previously generated hypotheses that can explain the occurred events. limit the pruning to a fix number of levels. Whenever a hypothesis be proved, the concerning weight for that node 3.3 Hypothesis Ranking is assigned one. When new hypotheses are generated, they are inserted as tree leafs (we can call it also decision tree) and then they are ordered by defuzzified occurrence moment from left to right. To describe briefly the ranking process, we assign each observed and proved a higher point and in contrast unobserved or not yet proved hypotheses are assigned lower points. schema5.hypothesis pruning, reasoning and explication of observation The rank and weight of generated hypotheses ( wi (t ) ) can change dynamically by elapse of time. The primary assigned weight is derived from the possibility distribution In the schema5 the sequence of C1 and C2_1 and C3_1 for occurrence of event (  e1:e2 existing in table1) and as indicate an explanation about the latest observations. the fuzzy trapezoid number affects it, so by elapse of time it can differ to the past weights (  t , schema2). The 3.5 Reasoning Process e1:e2 third parameter to affect the hypotheses rankings is the Our goal in reasoning process is to find an explication that possibility distribution of the upper node occurrence ( Wu ). can explain the observations. Observation of a fuzzy event Finally, γ affects the ranking value. γ is a value that is is a good reason to decide whether there are anomalies or resulted from a trade-off between smart home precision in not. However, the more OA be conscious the more rely on event detection and uncertainties about behaviours of unproved hypotheses. The sequence of observed events can Observed Agent (OA) or in other words Alzheimer explain the current activities and actions. Furthermore, the severity degree. At one side, the more severity in contexts can explain the precedence of home states. So, Alzheimer illness the less confidence on the OA and at the recognition of current context from the previously other side the more precision in event recognition, the generated hypotheses can well explain the observations more confidence on the reports and so it would be less and current activity(ies). Whenever no explanation for the necessary to trace the tree down to a lot of levels. The observation is found or the explanation does not include ranking formula is indicated as: minimum acceptance weight (dependent to γ), so the observed action would be recognized as abnormal action. wi (t )   .Wu .  e1:e2 . te :e 1 2 4 Implementation and Conclusion The ARA was implemented in VB.net environment and it 3.4 Hypothesis Pruning was simulated in SIMACT [27]. The activity “entering to the kitchen” was simulated in different scenarios (but the To prevent the increase in number of less possible same old embedded sensors) and some uncertainties in hypotheses, pruning is necessary. Pruning is applied in the event recognition (see picture1). Anomaly detection would case of low possibility distribution of event occurrence. In not be better than 50% done if the unproved hypotheses addition, observation of a possible event that could have grow deeper than three levels in decision tree. In spatial happened calls the pruning function1. Another way is to reasoning it can be said that the more antennas be applied, the more precise hypotheses would be generated. It can be 1 inferred that in the introduced approach, in the case of To estimate the closeness of new observation to the previously increasing the sensors number, more precise hypotheses 1 would be generated and proved. Fuzzy context at one hand generated and assumptive hypotheses, we applied the  can express well the real world state and it can decrease  reasoning complexity if it be well defuzzified. formula to check the difference between values of the observed and assumptive context items. If all the differences between all 1 the items be more than  then no explain is found.  8- Fuzzy Sets as a basis for a theory of possibility, L.A. Zadeh, Fuzzy Sets and Systems, vol. 1, pp. 3-28, 1978. 9- Possibility Theory, D.Dubios, H.Prade, Plenum Press, 1988. 10- Fuzzy sets and probability : Misunderstandings, bridges and gaps, D.Dubois, H. Prade, Proc. of the Second IEEE Inter. Conf. on Fuzzy Systems, volume 2, pp. 1059-1068, 1993. 11- Vikramaditya R. Jakkula, and Diane J. Cook, "Learning Temporal Relations in Smart Home Data",Proceedings of the Second International Conference on Technology and Aging, Canada, June 2007. 12- James F. Allen: Maintaining knowledge about temporal intervals. In: Communications of the ACM. 26/11/1983. ACM Press. S. 832-843, ISSN 0001-0782. 13- Roy P., Bouchard B., Bouzouane A., Giroux S.: A possibilistic approach for activity recognition in smart Picture1- the activity “entering to the kitchen” simulation in homes for cognitive assistance to Alzheimer's patient. In SIMACT Activity Recognition in Pervasive Intelligent Environment (Atlantis Ambient and Pervasive Intelligence), L. Chen, C. Nugent, J. Biswas, J. Hoey Editors, World Scientific Publishing Company, ISBN: 978-9078677352, pp. 1-20, September 2010. 5 Future Works 14- Roy P., Bouchard B., Bouzouane A., Giroux S.: Challenging issues of ambient activity recognition for cognitive We recommend the interested researches to survey the assistance. Handbook of research on Ambient Intelligence Activity Recognition in the case of multiple residents in and Smart Environments: Trends and Perspectives, IGI smart homes and also to introduce an optimization model global, F. Mastrogiovanni and N. Chong Editors, for fuzzy roles to decrease the activity recognition Information Science Publishing, ISBN: 1616928573, pp. 1- mistakes. 25, august 2010. 15- Roy P., Bouchard B., Bouzouane A., Giroux S: Combining 6 References pervasive computing with activity recognition and learning, 1- Bruno B., Bouzouane A., Giroux S.: A Keyhole Plan Web Intelligence and Intelligent Agents, Zeeshan-hassan Recognition Model for Alzheimer's Patients: First Results, Usmani (Ed.), ISBN: 978-953-7619-85-5, INTECH, pp. Journal of Applied Artificial Intelligence (AAI), Taylor & 447-462, 2010. Francis publisher, Vol. 22 (7), pp. 623-658, July 2007. 16- G.Singla, D. Cook, and M. Schmitter-Edgecombe. 2- Didier Dubois and Eyke Hullermeier, comparing Probability Incorporating temporal reasoning into activity recognition Measures Using Possibility Theory: A Notion of Relative for smart home residents. Proceedings of the AAAI Peakedness, International Journal of Approximate Workshop on Spatial and Temporal Reasoning, pages 53-61, Reasoning, 2007. 2008. 3- D. Dubois, H. Prade and R. Sabbadin (2001) Decision- 17- Diamond J. A report on Alzheimer disease and current theoretic foundations of possibility theory. Eur. J. research. Technical report, Alzheimer Society of Canada, Operational Research, 128: 459-478. (2005), 1-19. 4- D. Dubois and H. Prado. Possibility Theory, Plenum Press, 18- Baum C., Edwards D.,: cognitive performance in senile New York, 1988. dementia of the alzheimers type: The kitchen task 5- D. Dubois, H.T. Nguyen and H. Prade, Fuzzy sets and assessment. The American Journal of Occupational Therapy. probability: misunderstandings, bridges and gaps. In: D. 1993, Vol. 47 (5), 431436. Dubois and H. Prade, eds, Fundamentals of Fuzzy Sets. 19- Jensen, F. V., Bayesian Networks and Decision Graphs Boston, Mass: Kluwer, 343-438, 2000. (Springer 2001) 6- Cook, D.J. Youngblood, M. Heierman, E.O., III 20- M. Iosifescu, "Finite Markov processes and their Gopalratnam, K. Rao, S. Litvin, A. Khawaja, F, MavHome applications" , Wiley (1980) an agent-based smart home, IEEE Computer Society 21- N.A. Abdul-Manaf, M.R. Beikzadeh, rep esentation and Washington DC, USA, 2003. Reasoning of Fuzzy Temporal Knowledge (2006), IEEE Int. 7- Probability measures of fuzzy events, L.A.Zadeh, Journal Conferences on Cybernetics & Intelligent Systems and Math. Anal. Appl., vol 23, pp. 421-427, 1968. Robotics, Automation & Mechanics (CIS-RAM 2006). 22- Dubois, D., Prade, H. and Sandri, S. (1991). “On possibility/Probability transformations”, proc. Of the 4th International Fuzzy Systems Association (IFSA‟91) Congress, Brussels, Mathematics, PP. 50-53. 23- V. Jakkula, J. Cook. “Temporal pattern discovery for anomaly detection in a smart home”, 3rd IET International Conference on Intelligent Environments (IE 07), 2007. 24- S. Luhr, G. West, S. Venkatesh. „Recognition of emergent human behaviour in a smart home: A data mining approach“, Pervasive and Mobile Computing Volume 3 , Issue 2, 2007. 25- G. Nagypal, “a fuzzy model for representing uncertain, subjective and vague temporal knowledge in ontologies”, http://www.springerlink.com/content/atnluqe2gn8y7h27, 2003. 26- D. Dubois and H. Prade (1998) Possibility theory: Qualitative and quantitative aspects. In D. M. Gabbay and P. Smets P., editors Handbook of Defeasible Reasoning and Uncertainty Management Systems, Vol. 1., Dordrecht: Kluwer Academic, 169-226. 27- http://www.springerlink.com/content/j4g35l38913w2j0t