Combining Timed Data and Expert’s Knowledge to Model Human Behavior ∗ Laura Pomponio Marc Le Goc Eric Pascual LSIS - Laboratoire des LSIS - Laboratoire des CSTB - Centre Scientfique et Sciences de l’Information et Sciences de l’Information et Technique du Bâtiment des Systèmes des Systèmes 290, route des Lucioles, BP Marseille, France Domaine Universitaire de 209 CSTB - Centre Scientfique et Saint Jerôme Avenue 06904 Sophia Antipolis Technique du Bâtiment Escadrille Normandie Niemen Cedex, France Sophia Antipolis, France 13397 Marseille Cedex 20, eric.pascual@cstb.fr laura.pomponio@lsis.org France marc.legoc@lsis.org Alain Anfosso CSTB - Centre Scientfique et Technique du Bâtiment 290, route des Lucioles, BP 209 06904 Sophia Antipolis Cedex, France alain.anfosso@cstb.fr ABSTRACT tivities of daily living (IADL) increase with the age and, One of the major issues of monitoring activities in smart in general, are higher among people 60 years or more [29]. environments is the building of activity models from sen- These restrictions affect the autonomy and the well-being of sor’s timed data. This work proposes a general theoretical people in the last phases of life. Many older adults or people approach to this aim, based on a Knowledge Engineering with disabilities wish to remain in their home for as long as methodology and a Machine Learning process that are both possible even when their daily needs are affected. Certain funded on a general theory of dynamic process modeling, surveys indicates that nearly 75 percent of respondents age the Timed Observation Theory. In the proposed frame- 45 or older hope to stay in their homes as they age [1]. Un- work, activity recognition is an abstraction process where der these considerations, and taking into account that the the activities are conceived as entities at different abstrac- autonomy of a person depends not only of its capacities to tion levels. This paper aims at showing that prior expert’s accomplish acts of daily life, but also on the possibilities knowledge about resident activities can be compared with that the environment can provide, there is a growing inter- posterior knowledge induced from timed data. The proposed est in observing ADLs and monitoring health through smart approach is described through the database of the prototyp- environments [33, 6]. ical home of the GerHome project. A smart environment ”is able to acquire and apply knowledge Keywords about an environment and also to adapt to its inhabitants in Smart Environment, Human Activity, Dynamic Process Mod- order to improve their experience in that environment” [8]. eling, Machine Learning An example of smart environment is a smart home as Aware Home [2], EasyLiving [5], MavHome [7, 17], CUS Smart Home [31], iDorm [16], QuoVADis [15], CASAS [28, 24] and 1. INTRODUCTION GerHome [34, 35], where inhabitant behavior is recorded by Physical or cognitive functional limitations and difficulties sensors and monitored by a program in order to detect the in activities of daily living (ADL) and in instrumental ac- activity carried out (such as cooking, eating, watching TV, ∗This work is financed by CSTB. Contract no. 1256/2008 etc.). Activity monitoring consists of comparing resident behavior with activity models to determine the executed activity and to detect anomalies or behaviors that require automatic in- tervention of the environment. Nevertheless, the definition of models for human activity monitoring is one of the ma- jor issues due to the randomness of human behavior and, therefore, to the subjective notion of the concept activity. The work presented in this paper proposes a general theo- retical framework which conceptually defines the notion of activity and relates Knowledge Engineering methodologies with Data Mining techniques to define and to identify res- ident activities. The application of this approach is illus- trated through the GerHome project of Centre Scientifique et Technique du Bâtiment (CSTB, France). The aim of this project is to develop technical solutions to the problem of providing greater autonomy and better quality of life to the elderly at home; and thus, to work on the prevention of ac- cidents such as fall down originating in the frailty increase (a) Activity definition (Pick Up) [18] of the person. The hold method is to track the frailty trends by monitoring the daily activity and to compare a learned model with sensor data recorded from effective activities. This research path leads the way to detect activity mod- els and could offer an appropriate and reliable method to extract relevant and coherent daily activity patterns. In Section 2, we introduce related works and the motivation of our approach. Section 3 presents the theoretical frame- work proposed for modeling and recognizing the resident’s activities. Section 4 describes our proposal applied to the GerHome project. Finally, in Section 5, our conclusions are (b) Motion patterns from sensor data presented. (Sit and Stand ) [20] 2. RELATED WORKS Figure 1: Activity definition by using PCFG Human activity recognition in perceptual environments in- volves severe challenges due to the erratic nature of human behavior. To determine what is being done can be compli- cated if different activities are executed at the same time; e.g., to cook while watching TV. Besides, the same detected action can be associated with several activities depending on the context in which it is carried out then, to discriminate what is the right activity is not trivial; e.g., to open sink water tap can be part of cooking or washing dishes. More- over, activities can be interleaved: while washing dishes the phone rings, the activity is paused, the phone is answered and then, the activity is taken up again. Thus, to determine what a person is doing at a particular time is not a simple task. The problem lies in the meaning and the interpretation of the perceptual inputs due to the large gap that exists be- tween the low level signals, as pixels, sensor signals, etc., and that one that is inferred in a higher level, for example, washing dishes. Different works propose a characterization and a definition of human activity in smart environments. In particular, an activity can be considered in terms of space (activity loca- Figure 2: Activity hierarchical model constructed tion), of time (temporal patterns), of goals (intentions) and from MavHome data [9] in terms of ethnographic data [3]. On the other hand, as proposed in [20], activities are directly linked with human acts which can be specified by constructing a probabilis- uli, notion of causality amongst some qualitative activity de- tic context-free grammar (PCFG), whose alphabet consists scriptors and notion of context-sensitive intent. Similarly, of poses (Figure 1(a)). Thus, activity recognition is based [21] proposes three levels of abstraction as well: movements on a successive abstraction process where human activities as low-level semantic primitives, activities as sequences of are defined from the visual observation of body poses ob- states and movements and human behavioral actions as high tained from video data and, three levels of abstraction are level semantic events. conceived (Figure 1): continuous signal (optical flow), dis- crete event (body pose) and activity (sequence of discrete The MavHome (managing and adaptive versatile home) project events). In [19], activities depend on temporal, logical and [32] is focused on providing smart environments, whose goals causal constraints linked with an intention and three ab- are to maximize the comfort of the inhabitants, minimize the straction levels are also presented: low level sensory stim- consumption of resources, and maintain the safety of the Table 1: Different notions of the concept human activity in smart environment. Level MavHome CASAS [24, Semantic Lev- Abstraction PCFG [20] Conceptual Ac- [32, 9] 26, 25, 27] els [21] Levels [19] tivity level 2 activity temporal tax- behavioral activities activity activity (space taxon- onomy action (inter- (triples of con- (defined from a omy) action with text, behavior, set of primary the environ- state) activities) ment, causal relationships) level 1 event se- abstract event activity semantically discrete event primary activity quence (start level (se- (sequences meaningful (defined from a and end) quences of of states and activity- set of discrete events, start movement, descriptors events) and end) knowledge (rules of causal- related to ity, context- statistics of sensitive) temporal sequences) level 0 discrete event sensor level human move- low level sen- continuous discrete event (discrete ment (does sory stimuli signal event) not require (there is not contextual notion of time, or temporal physical states knowledge) or causality) to adapt to changes in the behavior patterns. All of these approaches include the idea of hierarchical ab- straction, and define three levels of abstraction (Table 1): discrete events, sequence of discrete events and a taxonomic classification at the highest level. Nevertheless, these def- initions about the concept activity depend mainly on the techniques used to build models for activity recognition. We propose then a general paradigm to define a conceptual notion of human activity that is not subject to a particular application and which considers three levels of abstraction: at level 0 timed observations or discrete event, at level 1 pri- mary activities as specific timed observation sequences, and finally activity as sequences of primary activities. Besides, we present a general procedure to define activity models which combines Knowledge Engineering with Data Mining. The advantage of our approach is to facilitate the definition of the principles of a general abstraction process from data. Figure 3: Example hierarchical activity model con- structed from CASAS data [26] 3. A THEORETICAL FRAMEWORK FOR MODELING HUMAN ACTIVITIES environment and its residents. In this project, once again, Our proposal is based on relating a Knowledge Engineer- three levels of abstraction are proposed (discrete events com- ing Methodology to a Timed Data Mining technique, i.e. ing from sensors, event sequence and activity) and the move the Timed Observations Modeling For Diagnosis (TOM4D) from an abstraction level to the other is based on models methodology [22, 14, 13] and the machine learning process that are produced using a process of Knowledge Discovering called Timed Observations Mining For Learning (TOM4L) from Databases (the Apriori algorithm [30] or Hiden Markov [4, 12]. Both TOM4D and TOM4L come from the math- models, Figure 2). A similar approach is used in CASAS [27, ematical Theory of Timed Observations [11] that provides 25, 26, 23], an adaptive smart home system that discovers a theoretical framework to facilitate the dynamic process and adapts to changes in the resident’s preferences in order modeling for monitoring, diagnosis and control. to generate satisfactory automation policies. In this case, a temporal point of view about the different levels of abstrac- 3.1 Human Activities As Observation Classes tion (Figure 3) is considered. It is to note that CASAS uses In this framework, a process is an arbitrary set X(t) = an algorithm of pattern adaptation miner (PAM) in order {xi (t)}i=1...r of time functions xi (t) defined on R (i.e. sig- Abstraction Level 2 Ω2 2 2 2 2 1 L = Θ(X2, Δ2) Abstraction Level 1 1 1 1 L = 1 0 Ω1 Θ(X1, Δ1) Abstraction Level 0 0 0 0 0 L = Θ'(X0, Δ0) Ω0 GERHOME Figure 4: Activity abstraction process nals provided by sensors). A timed observation is a couple for any level of abstraction ` (` ∈ N). Thus, the following (δi , tk ) which corresponds to the assignation of a predicate definitions are introduced. θ(xi , δi , tk ) where δi is constant and tk ∈ R a time stamp. When making an abuse of language, such a predicate can always be interpreted as the predicate EQU ALS(xi , δi , tk ) Definition 2. Let X ` be a set of abstract variables belong- ing to an abstraction level ` and let ∆` = x` ∈X ` ∆`x` be S (i.e. xi (tk ) = δi ). A monitoring program Θ(X, ∆) is a program Θ that analyzes the set of time functions xi (t) as- such that ∆`x` is a set of values assumable by the variable sociated to the set of variables X = {xi }i=1...r . The aim of x` . An abstract observation class at the abstraction level ` a monitoring program is to write timed observations (δi , tk ) is a singleton Ci` = {(x` , δ ` )}, with x` ∈ X ` and δ ` ∈ ∆`x` . in a database whenever a time function xi (t) ∈ X(t) satisfies some predicate θ(., ., .). Generally speaking, such a predicate is satisfied when xi (t) matches against a behavioral model Definition 3. A behavioral model M ` defined at abstrac- [10] that can be as simple as the switch of an interrupter tion level ` is a set of n-ary timed relations between obser- or, requiring complex techniques, such as signal processing vation classes defined at the abstraction level `. techniques for artificial vision. Definition 1. Let X be a set of variable names of a process The move from an abstraction level ` − 1 to the level ` is S X(t) = {xi (t)}i=1...r and let ∆ = x∈X ∆x be such that ∆x made when associating a particular set of behavioral models is a set of values assumable by the variable x ∈ X via a (at level ` − 1) to a given observation class Ci` (at level `). program Θ. An observation class Ci is a set of pairs (x, δ) Considering this as a general principle, Definition 4 specifies such that x ∈ X ∧ δ ∈ ∆x . the notion of abstraction level. In other words, an observation class Ci associates variables Definition 4. An abstraction level ` is a structure L` =< x ∈ X with constants δ ∈ ∆x . For simplicity reasons, an ob- X , ∆` , C ` , M `−1 > where ` servation class is usually defined as a singleton Ci = {(x, δ)}. This allows to formally define usual notions of events: • X ` is a set of variable names defined at level `, • A discrete event is a pair (x, δ) with x ∈ X, δ ∈ ∆x , • ∆` is a set of values assumable by the variables, denoting that the value δ is assumed by the variable • C ` is the set of observation classes belonging to the x. A discrete event corresponds then to a singleton level `, such that each observation class Ci` ∈ C ` is a observation class Ci = {(x, δ)}. singleton and, • A discrete event occurrence is a triplet (x, δ, tk ) with x ∈ X, δ ∈ ∆x , tk ∈ R denoting that the value δ is • M `−1 is a behavioral model defined at level ` − 1. assumed by the variable x at the time tk . A discrete event occurrence is then a timed observation (δ, tk ) of an singleton observation class Ci = {(x, δ)}. Consequently, the variables X = {xi }i=1...r of a process X(t) = {xi (t)}i=1...r are associated with the lowest level 0: L0 =< X 0 , ∆0 , C 0 , M −1 > where X 0 = X and, nat- The notion of observation class also contemplates different urally, M −1 = ∅ since there is not observation classes in levels of abstractions; that is to say, a particular set C ` = a previous level and so no behavioral models can be de- {C1` , ..., Cn` }, n ∈ N of observation classes Ci` can be defined fined with the timed observation paradigm. At level 1, ℓ=0 ℓ TOM4L M TOM4L ℓ Θ'(X0, Δ0) Ωℓ Ω (ElpLab) GERHOME TOM4D Knowledge ℓ=ℓ+1 (∃ Lℓ+1) Base Θ(Xℓ+1, Δℓ+1) L ℓ+1 = ℓ Expert Figure 5: Definition of activity models (Expert symbol has been taken of www.civicore.com) L1 =< X 1 , ∆1 , C 1 , M 0 > where the variables and the ob- servation classes are abstract. Each class of C 1 is associ- Knowledge ated with a behavioral model of M 0 ; that is, a set of n- Base ary relations included in M 0 . Similarly, at level 2 where Expert L2 =< X 2 , ∆2 , C 2 , M 1 >, variables and observation classes are abstract and each observation class of C 2 is associated with a sub-set of M 1 . TOM4D Process Model TOM4L The definition of these abstraction levels allows to specify: the different types of discrete events (sensor data) as obser- vation classes at level 0 (Ci0 ∈ C 0 ), each primary activity £(X,¢) Timed Observation Sequences as an observation class at level 1 (Ci1 ∈ C 1 ) and; finally, an activity as an observation class at level 2 (Ci2 ∈ C 2 ). Figure 6: Structure of logical precedence in the con- The passage of a level ` − 1 to a level `, where each class of struction of models from data (Expert symbol has been C ` is associated with a behavioral model of M `−1 , can be taken of www.civicore.com) accomplished by a program Θ(X ` , ∆` ) which analyzes the flow of timed observations at level ` − 1. In other words, Θ(X ` , ∆` ) assumes the matching of the flow of timed obser- vations at level `−1 against the models in M `−1 , and records of the methodology TOM4D or from timed observations in the corresponding timed observation (δ ` , tk ) in a database. a database through TOM4L. Knowledge can come from ex- Figure 4 illustrates these concepts in the context of the Ger- perts’ knowledge or can be new knowledge acquired from Home project where a monitoring program Θ0 registers a set the built models validated by experts. In turn, the timed Ω0 of sequences of discrete events, considered as timed ob- observations are obtained through a monitoring program Θ servations at level 0, from sensors that perceive the process which uses models to detect changes in the process and thus, ”the resident’s behavior at home”. For its part, Θ(X ` , ∆` ) it writes timed observations in a database. This structure (` = 1, 2) writes occurrences of observation classes defined of logic precedence allows to organize the available elements at level `, from a model M `−1 that allows to recognize the to carry out a procedure of model definition. behavior at level ` − 1 and interpret it as more abstract activities at the higher level. Figure 5 illustrates the process of definition of the inhabi- tant’s activities for the GerHome project, where an moni- toring agent Θ0 produces the timed observations in Ω0 (i.e. 3.2 Activity Model Definition Process coming from sensors). The application of the TOM4L pro- The abstraction process requires to establish the different cess (through the software ElpLab) to timed observations in levels; and therefore, to define behavioral models M ` . To Ω` (at level `) produces a behavioral model MT` OM 4L rep- this aim, we propose a procedure of activity definition based resentative of these observations. This model is analyzed on the combination of learning from data using Data Mining through the TOM4D methodology and a source of knowl- techniques (TOM4L process), and the use of expert’s knowl- edge (documents, data, experts, etc.) in order to define a edge through Knowledge Engineering (TOM4D methodol- behavioral model of interest M ` ⊆ MT` OM 4L and the ab- ogy). Figure 6 shows the logic-precedence structure of the stract observation classes linked to this one (i.e. activi- process of model construction where the relations are be- ties at the next level ` + 1). Thus, the abstraction level tween passive entities (as knowledge base, process model L`+1 =< X `+1 , ∆`+1 , C `+1 , M ` > can be specified in order and timed observations) and conceptually active entities (as that an agent Θ detects in Ω` occurrences of M ` and reg- TOM4D, TOM4L, the expert and the monitoring program isters occurrences of observation classes Ω`+1 . In a similar Θ). Thus, an passive entity can be obtained trough an active way, a new application of TOM4L on Ω`+1 begins the cy- entity which can require another passive entity. In the fig- cle to define the activities of the next level which are later ure, a model can be built from a priori knowledge by means validated by experts. The TOM4L process provides both a general matching pro- knowledge, the different abstraction levels of an activity can gram Θ(X ` , ∆` ) and a general algorithm to discover models be specified; and thus, to analyze if the activity is represen- M ` at any abstraction level. The next section illustrates the tative of the available data. On the other hand, to analyze application of the TOM4L process to the GerHome project. the available data to extract behavioral models and then, to define activities at different levels of abstraction. 4. APPLICATION Previous works on GerHome implement systems for moni- 4.1 From a priori Activity Definition to Data toring elderly activities from video event and environment Analysis event [33, 34, 35]. The first efforts to define activity mod- Documents, set of data, information transmitted by experts els were carried out in a manual way from scenarios de- and common sense allow to interpret sensor signals and to fined by experts. Nevertheless, the randomness of human define what sequences of events determine an activity. Once behavior leads to that the manual definition of activities is this established, each activity can be validate by experts extremely complex. Consequently, we aim to define activ- and collated with the resident’s behavior registered in a ities by means of using the TOM4L automatic techniques database. Activities in the living room will be considered combined with available knowledge interpreted trough the in order to illustrate part of the process of activity defini- TOM4D methodology. tion in which the starting point is a priori knowledge. Activity definition is accomplished from data registered by The living room of GerHome has five sensors registering sensors in the laboratory GerHome. This laboratory is an the resident’s behavior: three detecting presence and other apartment made up living room, bathroom, kitchen and two detecting use of the phone and use of the TV. Each room, where different kinds of sensors record inhabitant be- message of a timed data registered from sensors is inter- havior (Figure 7). preted through TOM4D as a variable that assumes a par- ticular value; that is to say, as an observation class. For ex- Sensors ample, the messages PRESENCE.LIVING_ROOM.CHAIR.1.true temperature, humidity, and PRESENCE.LIVING_ROOM.CHAIR.1.false are interpreted ROOM luminosity as a binary variable x0L1 (PRESENCE.LIVING_ROOM.CHAIR.1) volumetric presence that takes values true or f alse. Thus, the observation classes 0 0 occupation (bed, chair,...) C1027 = {(xL1 , f alse)} and C1028 = {(xL1 , true)} can be water consumption specified. Similarly, the other variables in the living room LIVING ROOM electricity consumption are identified: x0L2 (PRESENCE.LIVING_ROOM.CHAIR.2), x0L3 (PRESENCE.LIVING_ROOM.ARMCHAIR), x0L4 (USE.LIVING_ opening (doors, windows) ROOM.TEL) and x0L5 (USE.LIVING_ROOM.TV); and so also, the KITCHEN opening (furnitures) 0 corresponding observation classes: C1029 = {(x0L2 , f alse)}, image 0 0 0 0 0 BATHROOM C1030 = {(xL2 , true)}, C1025 = {(xL3 , f alse)}, C1026 = 0 0 0 0 0 data concentrator {(xL3 , true)}, C1037 = {(xL4 , begin)}, C1038 = {(xL4 , end)}, 0 C1039 0 = {(x0L5 , begin)} and C1040 = {(x0L5 , end)}. Figure 7: GerHome layout with sensors Considering a priori knowledge on alternative activities in the living room and a certain notion on them, watch TV is GerHome’s logs, as depicted in Figure 8, are timed data proposed as a possible activity made up of sitting down and 1 of the form ”yymmdd-hhmmss.mss/Msg” where ”yymmdd- turning on the TV. Hence, an abstract class C101 of level 1 hhmmss.mss” (like 080313-122225.825) is a time stamp tk can be specified to represent the activity watch TV and it and ”Msg” (like USAGE.KITCHEN.MICRO_WAVE_OWEN.begin), can be associated with behavioral models of level 0, which 0 0 is a constant δ associated with an observation class Ci0 . The are composed of at least the observation classes C1028 , C1030 , 0 0 ElpLab software, which implements the TOM4L approach, C1026 (linked to sit down) and C1039 (linked to turn on the uses a natural number i to identify the class. TV). Therefore, although the models in principle are not known, some relation is supposed between the observation [...] class 101 and the classes 1028, 1030, 1026, 1039 as Figure 9 080313-122225.825/USAGE/KITCHEN.MICRO_WAVE_OWEN/begin illustrates. 080313-122226.145/OPENCLOSE/KITCHEN.REFRIGERATOR/open 080313-122228.929/OPENCLOSE/KITCHEN.REFRIGERATOR/close [...] From data and given a particular observation class, the TOM4L process allows to discovery the behavioral sequences that finish in the given class, and to find the time constraints Figure 8: GerHome’s logs [0, λ2 ] where λ1 is the average times between two observation class occurrences. Then, taking in account the relations that would define watch TV (Figure 9), the study of the class In particular, a spatial taxonomy is considered for the pur- 1039 (to turn on the TV) is carried out. pose of analyzing behavior in each area of home, so logs are classified according to the different spaces (Figure 7). Figure 10 shows the behavioral model associated with the 0 class C1039 where the discovered model consists only of turn- In this section we describe how the activity definition can ing on and turning off the TV. This indicates that only the be carried out by means of complementing knowledge about variable x0L5 is involved in the behavior; that is to say, only activities with data analysis. On the one hand, from a priori the use of the TV as Figure 12(a) graphics (where true and x0L1 x0L3 x0L2 x0L4 x0L5 Level 2 WATCH_TV.true 101 Level 1 PRESENCE.LIVING_ROOM.CHAIR.1.true USAGE.LIVING_ROOM.TV.begin 1028 1039 Level 0 1030 1026 PRESENCE.LIVING_ROOM.CHAIR.2.true PRESENCE.LIVING_ROOM.ARMCHAIR.true Figure 9: Activity Definition - Watch TV Figure 13: Strength of relationship between obser- USAGE.LIVING_ROOM.TV.end vation classes This explains the previously obtained models (Figures 10, USAGE.LIVING_ROOM.TV.begin 11) and allows to suppose that the available logs are not representative of a real-life watch TV activity. Figure 10: Behavioral model associated with the class 1039 The experts validated this deduction and thus, the robust- ness of the TOM4L approach was verified. Therefore, an a PRESENCE.LIVING_ROOM.ARMCHAIR.false priori definition of activity can be proposed by experts and collated with data in order to establish the adequacy of its definition. PRESENCE.LIVING_ROOM.ARMCHAIR.true 4.2 From Data Analysis to Activity Definition Figure 11: Behavioral model associated with the For the analysis of data, behavior executed in the kitchen class 1026 is considered where there are 14 sensors and thus, 24 obser- vation classes. The study is concerned with the use of the [0,0h21m38s] stove (PRESENCE.KITCHEN.STOVE.true, classID=1024). [0,0h8m6s] x L5= 1 0 x L5= 0 0 x L3= 1 0 x0L3= 0 turn on [0,9h8m46s] turn off sit down get up [0,18h52m20s] (a) use of the TV (b) use of the armchair Figure 12: Graphical representation of behavior as- sociated with the classes 1039 and 1026 f alse are represented like 1 and 0). A similar result on the use of the armchair is obtained by 0 studying the observation class C1026 where the found be- havior consists only of sitting down and getting up of the armchair (Figure 11). Once again, there is only one variable (x0L3 ) involved in the discovered behavior as Figure 12(b) graphics. Figure 14: Model tree portion of the class observa- tion 1024 These outcomes do not represent the intuitive idea about the behavior in a living room where if the resident actuates in the environment should exist some relation between the The TOM4L process provides a set of 50 n-ary relations that different variables (or sensors). describes the occurrences of the class 1024 (Figure 14). The figure 15 shows one of these 50 n-ary relations. The proposal TOM4L defines the BJ-measure [12] that allows to establish is then to use these relations in order to define activities in how strong is the relationship between the different obser- other level of abstraction. vation classes. Figure 13 shows this measure between the observation classes of the living room, calculated from the The n-ary relations and their observation classes are ana- available data, where columns and rows identify the men- lyzed and then grouped with different criteria defined from tioned classes. Note that the relations that exist are only the mentioned analysis, according to the TOM4D methodol- those between the classes linked with the same variable. ogy. For example, the model m01 that describes the behavior OPENCLOSE.KITCHEN.REFRIGERATOR.open OPENCLOSE.KITCHEN.CUPBOARD.SINK.open PRESENCE.KITCHEN.STOVE.true OPENCLOSE.KITCHEN.CUPBOARD.SINK.close OPENCLOSE.KITCHEN.REFRIGERATOR.close Figure 15: n-ary relation (behavioral sequence) associated with the class 1024 Signature 5 5. CONCLUSION In this paper, a general theoretical framework to model and recognize resident activities was presented. Based on the Signature 6 areas of Knowledge Engineering and Timed Data Mining, this framework conceives human activities as entities at dif- ferent levels of abstraction and generalizes thus, the notion Signature 25 of activity. This generalization allows that the definitions of resident activity and the process of activity recognition are independent of any Data Mining technique or particular implementation. Besides, a general procedure to define the Signature 49 different abstraction levels and their behavioral models from data and experts’ knowledge was described. Signature 50 We applied our proposal to the GerHome’s timed data com- ing from the sensors of a home prototype, in order to show that a priori Expert’s knowledge can be collated with the timed data of a data base and, inversely, when a priori Ex- Figure 16: Models (n-ary relations) associated with pert’s knowledge is not available, behavioral models can be activity A1 found from timed data and then validated by Experts. We are now applying our approach to homes where activities are made by residents in different real-life context such as Table 2: Activity definition at abstraction level 1 hospital or nursing room. from n-ary relations Signature Activity Observation Abstract ID (model Classes Class 6. REFERENCES M 0) [1] AARP. Home and community preferences of the 45+ population. November 2010. Level 0 Level 0 Level 1 [2] G. D. Abowd, A. F. Bobick, I. A. Essa, E. D. Mynatt, A1 5, 6, 25, 1023, 1035, 1035, 111 and W. A. Rogers. Discovering expert’s knowledge 49, 50 1024 from sequences of discrete event class occurrences. In A2 ICEIS (2), pages 253–260, 2008. 2, 3, 7, 9, 1003, 1004, 1017, 112 12, 13, 14, 1018, 1024 [3] F. K. Aldrich. Smart homes: past, present and future. 15, 17 Inside the smart home. Springer-Verlag, 2003. A3 [4] N. Benayadi. Contribution à la découverte de 0, 4, 8, 17, 1005, 1006, 1007, 113 24, 26, 27, 1008, 1009, 1010, connaissances temporelles Ãă partir de données 34 1011, 1012, 1023, datées. Thesis, 2010. 1024 [5] B. Brumitt, B. Meyers, J. Krumm, A. Kern, and A4 S. Shafer. Easyliving: Technologies for intelligent 37, 43, 48 1033, 1034, 1024 114 ... ... ... ... environments. In Proceedings of Second International Symposium on Handheld and Ubiquitous Computing, pages 12–29, 2000. associated with using the stove is given in Figure 16, and [6] D. Cook. Health monitoring and assistance to support an activity identified as A1, is associated with this model. aging in place. j-jucs, 12(1):15–29, 2006. An abstract observation class, let us say 111, representing [7] D. Cook, J. Augusto, and V. Jakkula. Ambient this activity is specified and is linked with the aforesaid intelligence: technologies, applications, and model. Table 2 shows different activities and their abstract opportunities. Pervasive and Mobile Computing, classes specified defining the level L1 =< X 1 , ∆1 , C 1 , M 0 >. 5(4):277–298, 2009. ElpLab allows to record the occurrences of the classes C 1 of [8] D. J. Cook and S. Das. Smart environments: the level L1 , and the same approach can be done to define technology, protocols and applications. John Wiley & L2 =< X 2 , ∆2 , C 2 , M 1 >. Sons, Inc., 2005. [9] D. J. Cook, G. M. Youngblood, and G. Jain. Advances in Preference Handling, pages 78–84, 2008. Algorithms for smart spaces. The engineering [24] P. Rashidi and D. Cook. An adaptive sensor mining handbook on smart technology for aging, disability and model for pervasive computing applications. In 2nd independence. A. Helal, M. Mokhtari and B. International Workshop on Knowledge Discovery from Abdulrazak. John Wiley & Sons, Inc., 2008. Sensor Data, 2008. [10] L. Goc. SACHEM, a real-time intelligent diagnosis [25] P. Rashidi and D. Cook. Keeping the resident in the system based on the discrete event paradigm. loop: adapting the smart home to the user. Trans. Simulation, 80(11):591–617, 2004. Sys. Man Cyber. Part A, 39(5):949–959, 2009. [11] M. L. Goc. Notion d’observation pour le diagnostic des [26] P. Rashidi and D. J. Cook. Keeping the intelligent processus dynamiques: Application à sachem et à la environment resident in the loop. In 4th International découverte de connaissances temporelles. Habilitation Conference on Intelligent Environments, pages 1–9, a Diriger des Recherches. Unversité de Droit 2008. d’Economie et des Sciences d’Aix-Marseille, 2006. [27] P. Rashidi and D. J. Cook. An adaptive sensor mining [12] M. L. Goc and N. Benayadi. Discovering expert’s framework for pervasive computing applications. knowledge from sequences of discrete event class Lecture Notes in Nomputer Ncince, 5840:154–174, occurrences. In ICEIS (2), pages 253–260, 2008. 2010. [13] M. L. Goc and E. Masse. Towards a Multimodeling [28] P. Rashidi, S. Member, D. J. Cook, L. B. Holder, and Approach of Dynamic Systems for Diagnosis. In M. Schmitter-Edgecombe. Discovering activities to Proceedings of the 2nd International Conference on recognize and track in a smart environment. IEEE Software and Data Technologies (ICSoft’07), July TRANSACTIONS ON KNOWLEDGE AND DATA 2007. ENGINEERING, 2010. [14] M. L. Goc, E. Masse, and C. Curt. Modeling Process [29] S. D. Santos and Y. Makdessi. Une approche de From Timed Observations. In Proceedings of the 3rd l’autonomie chez les adultes et les personnes âgées. International Conference on Software and Data Études et résultats. DRESS, (718), February 2010. Technologies (ICSoft’08), July 2008. [30] R. Srikant and R. Agrawal. Mining sequential [15] J. B. H. Medjahed, D. Istrate and B. Dorizzi. Human patterns: generalizations and performance activities of daily living recognition using fuzzy logic improvements. In 5th International Conference on for elderly home monitoring. FUZZ-IEEE 2009. IEEE Extending Database Technology. Advances in Database International Conference on Fuzzy Systems, pages Technology EDBT 96., pages 3–17, 1996. 2001–2010, 2009. [31] S. H. You, H. J. Park, T. S. Kim, J. W. Park, U. Burn, [16] H. Hagras, F. Doctor, V. Callaghan, and A. Lopez. An J. A. Seol, and W. D. Cho. Developing intelligent incremental adaptive life long learning approach for smart home by utilizing community computing. In type-2 fuzzy embedded agents in ambient intelligent UCS’07: Proceedings of the 4th International environments. IEEE Transactions on Fuzzy Systems, Conference on Ubiquitous Computing Systems, pages 15(1):41–55, 2007. 59–71, Berlin, Heidelberg, 2007. Springer-Verlag. [17] V. Jakkula, D. J. Cook, and A. Crandal. Enhancing [32] G. M. Youngblood and D. J. Cook. Data mining for anomaly detection for smart homes using temporal hierarchical model creation. IEEE Transactions on pattern discovery. Advanced intelligent systems. Systems - Man, and Cybernetics. Part C: Applications Springer, 2008. and Reviews, 37(4):561–572, 2007. [18] E. Kim, S. Helal, and D. Cook. Human activity [33] N. Zouba, B. Boulay, F. Bremond, and M. Thonnat. recognition and pattern ddiscovery. IEEE Pervasive Cognitive vision. chapter Monitoring Activities of Computing, 9:48–53, 2010. Daily Living (ADLs) of Elderly Based on 3D Key [19] H. Muhammad. A computational framework for Human Postures, pages 37–50. Springer-Verlag, 2008. unsupervised analysis of everyday human activities. [34] N. Zouba, F. Bremond, and M. Thonnat. Multisensor College of Computing Georgia Institute of Technology, fusion for monitoring elderly activities at home. In 2008. Proceedings of the 2009 Sixth IEEE International [20] A. S. Ogale, A. Karapurkar, and Y. Aloimonos. View Conference on Advanced Video and Signal Based invariant modeling and recognition of human actions Surveillance, AVSS ’09, pages 98–103. IEEE using grammars. Workshop on dynamical vision at Computer Society, 2009. ICCV05. In In WDV, 2005. [35] N. Zouba, F. Bremond, M. Thonnat, A. Anfosso, [21] M. Pantic, A. Pentland, A. Nijholt, and T. S. Huang. E. Pascual, P. Mallea, V. Mailland, and O. Guerin. A Human computing and machine understanding of computer system to monitor older adults at home: human behavior: a survey. In Artifical Intelligence for Preliminary results. Gerontology. International journal Human Computing, volume 4451 of Lecture Notes in on the fundamental aspects of technology to serve the Computer Science, pages 47–71. Springer, 2007. ageing society, 8(3), Summer 2009. [22] L. Pomponio and M. L. Goc. Timed observations modelling for diagnosis methodology: a case study. In Proceedings of the 5th International Conference on Software and Data Technologies (ICSoft 2010), July 2010. [23] P. Rashidi and D. Cook. Adapting to resident preferences in smart home. In AAAI Workshop on