=Paper=
{{Paper
|id=None
|storemode=property
|title=Combining Timed Data and Expert's Knowledge to Model Human Behavior
|pdfUrl=https://ceur-ws.org/Vol-729/paper1.pdf
|volume=Vol-729
}}
==Combining Timed Data and Expert's Knowledge to Model Human Behavior==
https://ceur-ws.org/Vol-729/paper1.pdf
Combining Timed Data and Expert’s Knowledge to Model
Human Behavior ∗
Laura Pomponio Marc Le Goc Eric Pascual
LSIS - Laboratoire des LSIS - Laboratoire des CSTB - Centre Scientfique et
Sciences de l’Information et Sciences de l’Information et Technique du Bâtiment
des Systèmes des Systèmes 290, route des Lucioles, BP
Marseille, France Domaine Universitaire de 209
CSTB - Centre Scientfique et Saint Jerôme Avenue 06904 Sophia Antipolis
Technique du Bâtiment Escadrille Normandie Niemen Cedex, France
Sophia Antipolis, France 13397 Marseille Cedex 20, eric.pascual@cstb.fr
laura.pomponio@lsis.org France
marc.legoc@lsis.org
Alain Anfosso
CSTB - Centre Scientfique et
Technique du Bâtiment
290, route des Lucioles, BP
209
06904 Sophia Antipolis
Cedex, France
alain.anfosso@cstb.fr
ABSTRACT tivities of daily living (IADL) increase with the age and,
One of the major issues of monitoring activities in smart in general, are higher among people 60 years or more [29].
environments is the building of activity models from sen- These restrictions affect the autonomy and the well-being of
sor’s timed data. This work proposes a general theoretical people in the last phases of life. Many older adults or people
approach to this aim, based on a Knowledge Engineering with disabilities wish to remain in their home for as long as
methodology and a Machine Learning process that are both possible even when their daily needs are affected. Certain
funded on a general theory of dynamic process modeling, surveys indicates that nearly 75 percent of respondents age
the Timed Observation Theory. In the proposed frame- 45 or older hope to stay in their homes as they age [1]. Un-
work, activity recognition is an abstraction process where der these considerations, and taking into account that the
the activities are conceived as entities at different abstrac- autonomy of a person depends not only of its capacities to
tion levels. This paper aims at showing that prior expert’s accomplish acts of daily life, but also on the possibilities
knowledge about resident activities can be compared with that the environment can provide, there is a growing inter-
posterior knowledge induced from timed data. The proposed est in observing ADLs and monitoring health through smart
approach is described through the database of the prototyp- environments [33, 6].
ical home of the GerHome project.
A smart environment ”is able to acquire and apply knowledge
Keywords about an environment and also to adapt to its inhabitants in
Smart Environment, Human Activity, Dynamic Process Mod- order to improve their experience in that environment” [8].
eling, Machine Learning An example of smart environment is a smart home as Aware
Home [2], EasyLiving [5], MavHome [7, 17], CUS Smart
Home [31], iDorm [16], QuoVADis [15], CASAS [28, 24] and
1. INTRODUCTION GerHome [34, 35], where inhabitant behavior is recorded by
Physical or cognitive functional limitations and difficulties sensors and monitored by a program in order to detect the
in activities of daily living (ADL) and in instrumental ac- activity carried out (such as cooking, eating, watching TV,
∗This work is financed by CSTB. Contract no. 1256/2008 etc.).
Activity monitoring consists of comparing resident behavior
with activity models to determine the executed activity and
to detect anomalies or behaviors that require automatic in-
tervention of the environment. Nevertheless, the definition
of models for human activity monitoring is one of the ma-
jor issues due to the randomness of human behavior and,
therefore, to the subjective notion of the concept activity.
The work presented in this paper proposes a general theo-
retical framework which conceptually defines the notion of
activity and relates Knowledge Engineering methodologies
with Data Mining techniques to define and to identify res-
ident activities. The application of this approach is illus-
trated through the GerHome project of Centre Scientifique
et Technique du Bâtiment (CSTB, France). The aim of this
project is to develop technical solutions to the problem of
providing greater autonomy and better quality of life to the
elderly at home; and thus, to work on the prevention of ac-
cidents such as fall down originating in the frailty increase (a) Activity definition (Pick Up) [18]
of the person. The hold method is to track the frailty trends
by monitoring the daily activity and to compare a learned
model with sensor data recorded from effective activities.
This research path leads the way to detect activity mod-
els and could offer an appropriate and reliable method to
extract relevant and coherent daily activity patterns.
In Section 2, we introduce related works and the motivation
of our approach. Section 3 presents the theoretical frame-
work proposed for modeling and recognizing the resident’s
activities. Section 4 describes our proposal applied to the
GerHome project. Finally, in Section 5, our conclusions are (b) Motion patterns from sensor data
presented. (Sit and Stand ) [20]
2. RELATED WORKS Figure 1: Activity definition by using PCFG
Human activity recognition in perceptual environments in-
volves severe challenges due to the erratic nature of human
behavior. To determine what is being done can be compli-
cated if different activities are executed at the same time;
e.g., to cook while watching TV. Besides, the same detected
action can be associated with several activities depending on
the context in which it is carried out then, to discriminate
what is the right activity is not trivial; e.g., to open sink
water tap can be part of cooking or washing dishes. More-
over, activities can be interleaved: while washing dishes the
phone rings, the activity is paused, the phone is answered
and then, the activity is taken up again. Thus, to determine
what a person is doing at a particular time is not a simple
task.
The problem lies in the meaning and the interpretation of
the perceptual inputs due to the large gap that exists be-
tween the low level signals, as pixels, sensor signals, etc.,
and that one that is inferred in a higher level, for example,
washing dishes.
Different works propose a characterization and a definition
of human activity in smart environments. In particular, an
activity can be considered in terms of space (activity loca- Figure 2: Activity hierarchical model constructed
tion), of time (temporal patterns), of goals (intentions) and from MavHome data [9]
in terms of ethnographic data [3]. On the other hand, as
proposed in [20], activities are directly linked with human
acts which can be specified by constructing a probabilis- uli, notion of causality amongst some qualitative activity de-
tic context-free grammar (PCFG), whose alphabet consists scriptors and notion of context-sensitive intent. Similarly,
of poses (Figure 1(a)). Thus, activity recognition is based [21] proposes three levels of abstraction as well: movements
on a successive abstraction process where human activities as low-level semantic primitives, activities as sequences of
are defined from the visual observation of body poses ob- states and movements and human behavioral actions as high
tained from video data and, three levels of abstraction are level semantic events.
conceived (Figure 1): continuous signal (optical flow), dis-
crete event (body pose) and activity (sequence of discrete The MavHome (managing and adaptive versatile home) project
events). In [19], activities depend on temporal, logical and [32] is focused on providing smart environments, whose goals
causal constraints linked with an intention and three ab- are to maximize the comfort of the inhabitants, minimize the
straction levels are also presented: low level sensory stim- consumption of resources, and maintain the safety of the
Table 1: Different notions of the concept human activity in smart environment.
Level MavHome CASAS [24, Semantic Lev- Abstraction PCFG [20] Conceptual Ac-
[32, 9] 26, 25, 27] els [21] Levels [19] tivity
level 2 activity temporal tax- behavioral activities activity activity
(space taxon- onomy action (inter- (triples of con- (defined from a
omy) action with text, behavior, set of primary
the environ- state) activities)
ment, causal
relationships)
level 1 event se- abstract event activity semantically discrete event primary activity
quence (start level (se- (sequences meaningful (defined from a
and end) quences of of states and activity- set of discrete
events, start movement, descriptors events)
and end) knowledge (rules of causal-
related to ity, context-
statistics of sensitive)
temporal
sequences)
level 0 discrete event sensor level human move- low level sen- continuous discrete event
(discrete ment (does sory stimuli signal
event) not require (there is not
contextual notion of time,
or temporal physical states
knowledge) or causality)
to adapt to changes in the behavior patterns.
All of these approaches include the idea of hierarchical ab-
straction, and define three levels of abstraction (Table 1):
discrete events, sequence of discrete events and a taxonomic
classification at the highest level. Nevertheless, these def-
initions about the concept activity depend mainly on the
techniques used to build models for activity recognition.
We propose then a general paradigm to define a conceptual
notion of human activity that is not subject to a particular
application and which considers three levels of abstraction:
at level 0 timed observations or discrete event, at level 1 pri-
mary activities as specific timed observation sequences, and
finally activity as sequences of primary activities. Besides,
we present a general procedure to define activity models
which combines Knowledge Engineering with Data Mining.
The advantage of our approach is to facilitate the definition
of the principles of a general abstraction process from data.
Figure 3: Example hierarchical activity model con-
structed from CASAS data [26]
3. A THEORETICAL FRAMEWORK FOR
MODELING HUMAN ACTIVITIES
environment and its residents. In this project, once again, Our proposal is based on relating a Knowledge Engineer-
three levels of abstraction are proposed (discrete events com- ing Methodology to a Timed Data Mining technique, i.e.
ing from sensors, event sequence and activity) and the move the Timed Observations Modeling For Diagnosis (TOM4D)
from an abstraction level to the other is based on models methodology [22, 14, 13] and the machine learning process
that are produced using a process of Knowledge Discovering called Timed Observations Mining For Learning (TOM4L)
from Databases (the Apriori algorithm [30] or Hiden Markov [4, 12]. Both TOM4D and TOM4L come from the math-
models, Figure 2). A similar approach is used in CASAS [27, ematical Theory of Timed Observations [11] that provides
25, 26, 23], an adaptive smart home system that discovers a theoretical framework to facilitate the dynamic process
and adapts to changes in the resident’s preferences in order modeling for monitoring, diagnosis and control.
to generate satisfactory automation policies. In this case, a
temporal point of view about the different levels of abstrac- 3.1 Human Activities As Observation Classes
tion (Figure 3) is considered. It is to note that CASAS uses In this framework, a process is an arbitrary set X(t) =
an algorithm of pattern adaptation miner (PAM) in order {xi (t)}i=1...r of time functions xi (t) defined on R (i.e. sig-
Abstraction
Level 2 Ω2
2 2 2 2 1
L =
Θ(X2, Δ2)
Abstraction
Level 1
1 1 1
L =
1 0 Ω1
Θ(X1, Δ1)
Abstraction
Level 0
0 0 0 0
L = Θ'(X0, Δ0)
Ω0
GERHOME
Figure 4: Activity abstraction process
nals provided by sensors). A timed observation is a couple for any level of abstraction ` (` ∈ N). Thus, the following
(δi , tk ) which corresponds to the assignation of a predicate definitions are introduced.
θ(xi , δi , tk ) where δi is constant and tk ∈ R a time stamp.
When making an abuse of language, such a predicate can
always be interpreted as the predicate EQU ALS(xi , δi , tk ) Definition 2. Let X ` be a set of abstract variables belong-
ing to an abstraction level ` and let ∆` = x` ∈X ` ∆`x` be
S
(i.e. xi (tk ) = δi ). A monitoring program Θ(X, ∆) is a
program Θ that analyzes the set of time functions xi (t) as- such that ∆`x` is a set of values assumable by the variable
sociated to the set of variables X = {xi }i=1...r . The aim of x` . An abstract observation class at the abstraction level `
a monitoring program is to write timed observations (δi , tk ) is a singleton Ci` = {(x` , δ ` )}, with x` ∈ X ` and δ ` ∈ ∆`x` .
in a database whenever a time function xi (t) ∈ X(t) satisfies
some predicate θ(., ., .). Generally speaking, such a predicate
is satisfied when xi (t) matches against a behavioral model
Definition 3. A behavioral model M ` defined at abstrac-
[10] that can be as simple as the switch of an interrupter
tion level ` is a set of n-ary timed relations between obser-
or, requiring complex techniques, such as signal processing
vation classes defined at the abstraction level `.
techniques for artificial vision.
Definition 1. Let X be a set of variable names of a process The move from an abstraction level ` − 1 to the level ` is
S
X(t) = {xi (t)}i=1...r and let ∆ = x∈X ∆x be such that ∆x made when associating a particular set of behavioral models
is a set of values assumable by the variable x ∈ X via a (at level ` − 1) to a given observation class Ci` (at level `).
program Θ. An observation class Ci is a set of pairs (x, δ) Considering this as a general principle, Definition 4 specifies
such that x ∈ X ∧ δ ∈ ∆x . the notion of abstraction level.
In other words, an observation class Ci associates variables Definition 4. An abstraction level ` is a structure L` =<
x ∈ X with constants δ ∈ ∆x . For simplicity reasons, an ob- X , ∆` , C ` , M `−1 > where
`
servation class is usually defined as a singleton Ci = {(x, δ)}.
This allows to formally define usual notions of events:
• X ` is a set of variable names defined at level `,
• A discrete event is a pair (x, δ) with x ∈ X, δ ∈ ∆x , • ∆` is a set of values assumable by the variables,
denoting that the value δ is assumed by the variable
• C ` is the set of observation classes belonging to the
x. A discrete event corresponds then to a singleton
level `, such that each observation class Ci` ∈ C ` is a
observation class Ci = {(x, δ)}.
singleton and,
• A discrete event occurrence is a triplet (x, δ, tk ) with
x ∈ X, δ ∈ ∆x , tk ∈ R denoting that the value δ is • M `−1 is a behavioral model defined at level ` − 1.
assumed by the variable x at the time tk . A discrete
event occurrence is then a timed observation (δ, tk ) of
an singleton observation class Ci = {(x, δ)}. Consequently, the variables X = {xi }i=1...r of a process
X(t) = {xi (t)}i=1...r are associated with the lowest level
0: L0 =< X 0 , ∆0 , C 0 , M −1 > where X 0 = X and, nat-
The notion of observation class also contemplates different urally, M −1 = ∅ since there is not observation classes in
levels of abstractions; that is to say, a particular set C ` = a previous level and so no behavioral models can be de-
{C1` , ..., Cn` }, n ∈ N of observation classes Ci` can be defined fined with the timed observation paradigm. At level 1,
ℓ=0 ℓ TOM4L
M TOM4L
ℓ
Θ'(X0, Δ0) Ωℓ
Ω (ElpLab)
GERHOME
TOM4D Knowledge
ℓ=ℓ+1 (∃ Lℓ+1)
Base
Θ(Xℓ+1, Δℓ+1) L
ℓ+1
=
ℓ
Expert
Figure 5: Definition of activity models (Expert symbol has been taken of www.civicore.com)
L1 =< X 1 , ∆1 , C 1 , M 0 > where the variables and the ob-
servation classes are abstract. Each class of C 1 is associ-
Knowledge
ated with a behavioral model of M 0 ; that is, a set of n- Base
ary relations included in M 0 . Similarly, at level 2 where Expert
L2 =< X 2 , ∆2 , C 2 , M 1 >, variables and observation classes
are abstract and each observation class of C 2 is associated
with a sub-set of M 1 . TOM4D Process Model TOM4L
The definition of these abstraction levels allows to specify:
the different types of discrete events (sensor data) as obser-
vation classes at level 0 (Ci0 ∈ C 0 ), each primary activity £(X,¢) Timed Observation
Sequences
as an observation class at level 1 (Ci1 ∈ C 1 ) and; finally, an
activity as an observation class at level 2 (Ci2 ∈ C 2 ).
Figure 6: Structure of logical precedence in the con-
The passage of a level ` − 1 to a level `, where each class of struction of models from data (Expert symbol has been
C ` is associated with a behavioral model of M `−1 , can be taken of www.civicore.com)
accomplished by a program Θ(X ` , ∆` ) which analyzes the
flow of timed observations at level ` − 1. In other words,
Θ(X ` , ∆` ) assumes the matching of the flow of timed obser-
vations at level `−1 against the models in M `−1 , and records of the methodology TOM4D or from timed observations in
the corresponding timed observation (δ ` , tk ) in a database. a database through TOM4L. Knowledge can come from ex-
Figure 4 illustrates these concepts in the context of the Ger- perts’ knowledge or can be new knowledge acquired from
Home project where a monitoring program Θ0 registers a set the built models validated by experts. In turn, the timed
Ω0 of sequences of discrete events, considered as timed ob- observations are obtained through a monitoring program Θ
servations at level 0, from sensors that perceive the process which uses models to detect changes in the process and thus,
”the resident’s behavior at home”. For its part, Θ(X ` , ∆` ) it writes timed observations in a database. This structure
(` = 1, 2) writes occurrences of observation classes defined of logic precedence allows to organize the available elements
at level `, from a model M `−1 that allows to recognize the to carry out a procedure of model definition.
behavior at level ` − 1 and interpret it as more abstract
activities at the higher level. Figure 5 illustrates the process of definition of the inhabi-
tant’s activities for the GerHome project, where an moni-
toring agent Θ0 produces the timed observations in Ω0 (i.e.
3.2 Activity Model Definition Process coming from sensors). The application of the TOM4L pro-
The abstraction process requires to establish the different cess (through the software ElpLab) to timed observations in
levels; and therefore, to define behavioral models M ` . To Ω` (at level `) produces a behavioral model MT` OM 4L rep-
this aim, we propose a procedure of activity definition based resentative of these observations. This model is analyzed
on the combination of learning from data using Data Mining through the TOM4D methodology and a source of knowl-
techniques (TOM4L process), and the use of expert’s knowl- edge (documents, data, experts, etc.) in order to define a
edge through Knowledge Engineering (TOM4D methodol- behavioral model of interest M ` ⊆ MT` OM 4L and the ab-
ogy). Figure 6 shows the logic-precedence structure of the stract observation classes linked to this one (i.e. activi-
process of model construction where the relations are be- ties at the next level ` + 1). Thus, the abstraction level
tween passive entities (as knowledge base, process model L`+1 =< X `+1 , ∆`+1 , C `+1 , M ` > can be specified in order
and timed observations) and conceptually active entities (as that an agent Θ detects in Ω` occurrences of M ` and reg-
TOM4D, TOM4L, the expert and the monitoring program isters occurrences of observation classes Ω`+1 . In a similar
Θ). Thus, an passive entity can be obtained trough an active way, a new application of TOM4L on Ω`+1 begins the cy-
entity which can require another passive entity. In the fig- cle to define the activities of the next level which are later
ure, a model can be built from a priori knowledge by means validated by experts.
The TOM4L process provides both a general matching pro- knowledge, the different abstraction levels of an activity can
gram Θ(X ` , ∆` ) and a general algorithm to discover models be specified; and thus, to analyze if the activity is represen-
M ` at any abstraction level. The next section illustrates the tative of the available data. On the other hand, to analyze
application of the TOM4L process to the GerHome project. the available data to extract behavioral models and then, to
define activities at different levels of abstraction.
4. APPLICATION
Previous works on GerHome implement systems for moni- 4.1 From a priori Activity Definition to Data
toring elderly activities from video event and environment Analysis
event [33, 34, 35]. The first efforts to define activity mod- Documents, set of data, information transmitted by experts
els were carried out in a manual way from scenarios de- and common sense allow to interpret sensor signals and to
fined by experts. Nevertheless, the randomness of human define what sequences of events determine an activity. Once
behavior leads to that the manual definition of activities is this established, each activity can be validate by experts
extremely complex. Consequently, we aim to define activ- and collated with the resident’s behavior registered in a
ities by means of using the TOM4L automatic techniques database. Activities in the living room will be considered
combined with available knowledge interpreted trough the in order to illustrate part of the process of activity defini-
TOM4D methodology. tion in which the starting point is a priori knowledge.
Activity definition is accomplished from data registered by The living room of GerHome has five sensors registering
sensors in the laboratory GerHome. This laboratory is an the resident’s behavior: three detecting presence and other
apartment made up living room, bathroom, kitchen and two detecting use of the phone and use of the TV. Each
room, where different kinds of sensors record inhabitant be- message of a timed data registered from sensors is inter-
havior (Figure 7). preted through TOM4D as a variable that assumes a par-
ticular value; that is to say, as an observation class. For ex-
Sensors ample, the messages PRESENCE.LIVING_ROOM.CHAIR.1.true
temperature, humidity, and PRESENCE.LIVING_ROOM.CHAIR.1.false are interpreted
ROOM
luminosity as a binary variable x0L1 (PRESENCE.LIVING_ROOM.CHAIR.1)
volumetric presence that takes values true or f alse. Thus, the observation classes
0 0
occupation (bed, chair,...) C1027 = {(xL1 , f alse)} and C1028 = {(xL1 , true)} can be
water consumption specified. Similarly, the other variables in the living room
LIVING
ROOM electricity consumption
are identified: x0L2 (PRESENCE.LIVING_ROOM.CHAIR.2), x0L3
(PRESENCE.LIVING_ROOM.ARMCHAIR), x0L4 (USE.LIVING_
opening (doors, windows)
ROOM.TEL) and x0L5 (USE.LIVING_ROOM.TV); and so also, the
KITCHEN
opening (furnitures) 0
corresponding observation classes: C1029 = {(x0L2 , f alse)},
image 0 0 0 0 0
BATHROOM C1030 = {(xL2 , true)}, C1025 = {(xL3 , f alse)}, C1026 =
0 0 0 0 0
data concentrator
{(xL3 , true)}, C1037 = {(xL4 , begin)}, C1038 = {(xL4 , end)},
0
C1039 0
= {(x0L5 , begin)} and C1040 = {(x0L5 , end)}.
Figure 7: GerHome layout with sensors Considering a priori knowledge on alternative activities in
the living room and a certain notion on them, watch TV is
GerHome’s logs, as depicted in Figure 8, are timed data proposed as a possible activity made up of sitting down and
1
of the form ”yymmdd-hhmmss.mss/Msg” where ”yymmdd- turning on the TV. Hence, an abstract class C101 of level 1
hhmmss.mss” (like 080313-122225.825) is a time stamp tk can be specified to represent the activity watch TV and it
and ”Msg” (like USAGE.KITCHEN.MICRO_WAVE_OWEN.begin), can be associated with behavioral models of level 0, which
0 0
is a constant δ associated with an observation class Ci0 . The are composed of at least the observation classes C1028 , C1030 ,
0 0
ElpLab software, which implements the TOM4L approach, C1026 (linked to sit down) and C1039 (linked to turn on the
uses a natural number i to identify the class. TV). Therefore, although the models in principle are not
known, some relation is supposed between the observation
[...] class 101 and the classes 1028, 1030, 1026, 1039 as Figure 9
080313-122225.825/USAGE/KITCHEN.MICRO_WAVE_OWEN/begin illustrates.
080313-122226.145/OPENCLOSE/KITCHEN.REFRIGERATOR/open
080313-122228.929/OPENCLOSE/KITCHEN.REFRIGERATOR/close
[...] From data and given a particular observation class, the TOM4L
process allows to discovery the behavioral sequences that
finish in the given class, and to find the time constraints
Figure 8: GerHome’s logs [0, λ2 ] where λ1 is the average times between two observation
class occurrences. Then, taking in account the relations that
would define watch TV (Figure 9), the study of the class
In particular, a spatial taxonomy is considered for the pur- 1039 (to turn on the TV) is carried out.
pose of analyzing behavior in each area of home, so logs are
classified according to the different spaces (Figure 7). Figure 10 shows the behavioral model associated with the
0
class C1039 where the discovered model consists only of turn-
In this section we describe how the activity definition can ing on and turning off the TV. This indicates that only the
be carried out by means of complementing knowledge about variable x0L5 is involved in the behavior; that is to say, only
activities with data analysis. On the one hand, from a priori the use of the TV as Figure 12(a) graphics (where true and
x0L1 x0L3 x0L2 x0L4 x0L5
Level 2
WATCH_TV.true 101
Level 1
PRESENCE.LIVING_ROOM.CHAIR.1.true USAGE.LIVING_ROOM.TV.begin
1028 1039
Level 0
1030 1026
PRESENCE.LIVING_ROOM.CHAIR.2.true PRESENCE.LIVING_ROOM.ARMCHAIR.true
Figure 9: Activity Definition - Watch TV
Figure 13: Strength of relationship between obser-
USAGE.LIVING_ROOM.TV.end vation classes
This explains the previously obtained models (Figures 10,
USAGE.LIVING_ROOM.TV.begin 11) and allows to suppose that the available logs are not
representative of a real-life watch TV activity.
Figure 10: Behavioral model associated with the
class 1039 The experts validated this deduction and thus, the robust-
ness of the TOM4L approach was verified. Therefore, an a
PRESENCE.LIVING_ROOM.ARMCHAIR.false priori definition of activity can be proposed by experts and
collated with data in order to establish the adequacy of its
definition.
PRESENCE.LIVING_ROOM.ARMCHAIR.true
4.2 From Data Analysis to Activity Definition
Figure 11: Behavioral model associated with the For the analysis of data, behavior executed in the kitchen
class 1026 is considered where there are 14 sensors and thus, 24 obser-
vation classes. The study is concerned with the use of the
[0,0h21m38s]
stove (PRESENCE.KITCHEN.STOVE.true, classID=1024).
[0,0h8m6s]
x L5= 1
0
x L5= 0
0
x L3= 1
0
x0L3= 0
turn on [0,9h8m46s] turn off sit down get up
[0,18h52m20s]
(a) use of the TV (b) use of the armchair
Figure 12: Graphical representation of behavior as-
sociated with the classes 1039 and 1026
f alse are represented like 1 and 0).
A similar result on the use of the armchair is obtained by
0
studying the observation class C1026 where the found be-
havior consists only of sitting down and getting up of the
armchair (Figure 11). Once again, there is only one variable
(x0L3 ) involved in the discovered behavior as Figure 12(b)
graphics. Figure 14: Model tree portion of the class observa-
tion 1024
These outcomes do not represent the intuitive idea about
the behavior in a living room where if the resident actuates
in the environment should exist some relation between the The TOM4L process provides a set of 50 n-ary relations that
different variables (or sensors). describes the occurrences of the class 1024 (Figure 14). The
figure 15 shows one of these 50 n-ary relations. The proposal
TOM4L defines the BJ-measure [12] that allows to establish is then to use these relations in order to define activities in
how strong is the relationship between the different obser- other level of abstraction.
vation classes. Figure 13 shows this measure between the
observation classes of the living room, calculated from the The n-ary relations and their observation classes are ana-
available data, where columns and rows identify the men- lyzed and then grouped with different criteria defined from
tioned classes. Note that the relations that exist are only the mentioned analysis, according to the TOM4D methodol-
those between the classes linked with the same variable. ogy. For example, the model m01 that describes the behavior
OPENCLOSE.KITCHEN.REFRIGERATOR.open
OPENCLOSE.KITCHEN.CUPBOARD.SINK.open PRESENCE.KITCHEN.STOVE.true
OPENCLOSE.KITCHEN.CUPBOARD.SINK.close OPENCLOSE.KITCHEN.REFRIGERATOR.close
Figure 15: n-ary relation (behavioral sequence) associated with the class 1024
Signature 5 5. CONCLUSION
In this paper, a general theoretical framework to model and
recognize resident activities was presented. Based on the
Signature 6 areas of Knowledge Engineering and Timed Data Mining,
this framework conceives human activities as entities at dif-
ferent levels of abstraction and generalizes thus, the notion
Signature 25
of activity. This generalization allows that the definitions
of resident activity and the process of activity recognition
are independent of any Data Mining technique or particular
implementation. Besides, a general procedure to define the
Signature 49
different abstraction levels and their behavioral models from
data and experts’ knowledge was described.
Signature 50 We applied our proposal to the GerHome’s timed data com-
ing from the sensors of a home prototype, in order to show
that a priori Expert’s knowledge can be collated with the
timed data of a data base and, inversely, when a priori Ex-
Figure 16: Models (n-ary relations) associated with pert’s knowledge is not available, behavioral models can be
activity A1 found from timed data and then validated by Experts. We
are now applying our approach to homes where activities
are made by residents in different real-life context such as
Table 2: Activity definition at abstraction level 1 hospital or nursing room.
from n-ary relations
Signature Activity Observation Abstract
ID (model Classes Class 6. REFERENCES
M 0) [1] AARP. Home and community preferences of the 45+
population. November 2010.
Level 0 Level 0 Level 1 [2] G. D. Abowd, A. F. Bobick, I. A. Essa, E. D. Mynatt,
A1
5, 6, 25, 1023, 1035, 1035, 111 and W. A. Rogers. Discovering expert’s knowledge
49, 50 1024 from sequences of discrete event class occurrences. In
A2 ICEIS (2), pages 253–260, 2008.
2, 3, 7, 9, 1003, 1004, 1017, 112
12, 13, 14, 1018, 1024 [3] F. K. Aldrich. Smart homes: past, present and future.
15, 17 Inside the smart home. Springer-Verlag, 2003.
A3 [4] N. Benayadi. Contribution à la découverte de
0, 4, 8, 17, 1005, 1006, 1007, 113
24, 26, 27, 1008, 1009, 1010, connaissances temporelles Ãă partir de données
34 1011, 1012, 1023, datées. Thesis, 2010.
1024 [5] B. Brumitt, B. Meyers, J. Krumm, A. Kern, and
A4 S. Shafer. Easyliving: Technologies for intelligent
37, 43, 48 1033, 1034, 1024 114
... ... ... ... environments. In Proceedings of Second International
Symposium on Handheld and Ubiquitous Computing,
pages 12–29, 2000.
associated with using the stove is given in Figure 16, and [6] D. Cook. Health monitoring and assistance to support
an activity identified as A1, is associated with this model. aging in place. j-jucs, 12(1):15–29, 2006.
An abstract observation class, let us say 111, representing [7] D. Cook, J. Augusto, and V. Jakkula. Ambient
this activity is specified and is linked with the aforesaid intelligence: technologies, applications, and
model. Table 2 shows different activities and their abstract opportunities. Pervasive and Mobile Computing,
classes specified defining the level L1 =< X 1 , ∆1 , C 1 , M 0 >. 5(4):277–298, 2009.
ElpLab allows to record the occurrences of the classes C 1 of [8] D. J. Cook and S. Das. Smart environments:
the level L1 , and the same approach can be done to define technology, protocols and applications. John Wiley &
L2 =< X 2 , ∆2 , C 2 , M 1 >. Sons, Inc., 2005.
[9] D. J. Cook, G. M. Youngblood, and G. Jain. Advances in Preference Handling, pages 78–84, 2008.
Algorithms for smart spaces. The engineering [24] P. Rashidi and D. Cook. An adaptive sensor mining
handbook on smart technology for aging, disability and model for pervasive computing applications. In 2nd
independence. A. Helal, M. Mokhtari and B. International Workshop on Knowledge Discovery from
Abdulrazak. John Wiley & Sons, Inc., 2008. Sensor Data, 2008.
[10] L. Goc. SACHEM, a real-time intelligent diagnosis [25] P. Rashidi and D. Cook. Keeping the resident in the
system based on the discrete event paradigm. loop: adapting the smart home to the user. Trans.
Simulation, 80(11):591–617, 2004. Sys. Man Cyber. Part A, 39(5):949–959, 2009.
[11] M. L. Goc. Notion d’observation pour le diagnostic des [26] P. Rashidi and D. J. Cook. Keeping the intelligent
processus dynamiques: Application à sachem et à la environment resident in the loop. In 4th International
découverte de connaissances temporelles. Habilitation Conference on Intelligent Environments, pages 1–9,
a Diriger des Recherches. Unversité de Droit 2008.
d’Economie et des Sciences d’Aix-Marseille, 2006. [27] P. Rashidi and D. J. Cook. An adaptive sensor mining
[12] M. L. Goc and N. Benayadi. Discovering expert’s framework for pervasive computing applications.
knowledge from sequences of discrete event class Lecture Notes in Nomputer Ncince, 5840:154–174,
occurrences. In ICEIS (2), pages 253–260, 2008. 2010.
[13] M. L. Goc and E. Masse. Towards a Multimodeling [28] P. Rashidi, S. Member, D. J. Cook, L. B. Holder, and
Approach of Dynamic Systems for Diagnosis. In M. Schmitter-Edgecombe. Discovering activities to
Proceedings of the 2nd International Conference on recognize and track in a smart environment. IEEE
Software and Data Technologies (ICSoft’07), July TRANSACTIONS ON KNOWLEDGE AND DATA
2007. ENGINEERING, 2010.
[14] M. L. Goc, E. Masse, and C. Curt. Modeling Process [29] S. D. Santos and Y. Makdessi. Une approche de
From Timed Observations. In Proceedings of the 3rd l’autonomie chez les adultes et les personnes âgées.
International Conference on Software and Data Études et résultats. DRESS, (718), February 2010.
Technologies (ICSoft’08), July 2008. [30] R. Srikant and R. Agrawal. Mining sequential
[15] J. B. H. Medjahed, D. Istrate and B. Dorizzi. Human patterns: generalizations and performance
activities of daily living recognition using fuzzy logic improvements. In 5th International Conference on
for elderly home monitoring. FUZZ-IEEE 2009. IEEE Extending Database Technology. Advances in Database
International Conference on Fuzzy Systems, pages Technology EDBT 96., pages 3–17, 1996.
2001–2010, 2009. [31] S. H. You, H. J. Park, T. S. Kim, J. W. Park, U. Burn,
[16] H. Hagras, F. Doctor, V. Callaghan, and A. Lopez. An J. A. Seol, and W. D. Cho. Developing intelligent
incremental adaptive life long learning approach for smart home by utilizing community computing. In
type-2 fuzzy embedded agents in ambient intelligent UCS’07: Proceedings of the 4th International
environments. IEEE Transactions on Fuzzy Systems, Conference on Ubiquitous Computing Systems, pages
15(1):41–55, 2007. 59–71, Berlin, Heidelberg, 2007. Springer-Verlag.
[17] V. Jakkula, D. J. Cook, and A. Crandal. Enhancing [32] G. M. Youngblood and D. J. Cook. Data mining for
anomaly detection for smart homes using temporal hierarchical model creation. IEEE Transactions on
pattern discovery. Advanced intelligent systems. Systems - Man, and Cybernetics. Part C: Applications
Springer, 2008. and Reviews, 37(4):561–572, 2007.
[18] E. Kim, S. Helal, and D. Cook. Human activity [33] N. Zouba, B. Boulay, F. Bremond, and M. Thonnat.
recognition and pattern ddiscovery. IEEE Pervasive Cognitive vision. chapter Monitoring Activities of
Computing, 9:48–53, 2010. Daily Living (ADLs) of Elderly Based on 3D Key
[19] H. Muhammad. A computational framework for Human Postures, pages 37–50. Springer-Verlag, 2008.
unsupervised analysis of everyday human activities. [34] N. Zouba, F. Bremond, and M. Thonnat. Multisensor
College of Computing Georgia Institute of Technology, fusion for monitoring elderly activities at home. In
2008. Proceedings of the 2009 Sixth IEEE International
[20] A. S. Ogale, A. Karapurkar, and Y. Aloimonos. View Conference on Advanced Video and Signal Based
invariant modeling and recognition of human actions Surveillance, AVSS ’09, pages 98–103. IEEE
using grammars. Workshop on dynamical vision at Computer Society, 2009.
ICCV05. In In WDV, 2005. [35] N. Zouba, F. Bremond, M. Thonnat, A. Anfosso,
[21] M. Pantic, A. Pentland, A. Nijholt, and T. S. Huang. E. Pascual, P. Mallea, V. Mailland, and O. Guerin. A
Human computing and machine understanding of computer system to monitor older adults at home:
human behavior: a survey. In Artifical Intelligence for Preliminary results. Gerontology. International journal
Human Computing, volume 4451 of Lecture Notes in on the fundamental aspects of technology to serve the
Computer Science, pages 47–71. Springer, 2007. ageing society, 8(3), Summer 2009.
[22] L. Pomponio and M. L. Goc. Timed observations
modelling for diagnosis methodology: a case study. In
Proceedings of the 5th International Conference on
Software and Data Technologies (ICSoft 2010), July
2010.
[23] P. Rashidi and D. Cook. Adapting to resident
preferences in smart home. In AAAI Workshop on