=Paper=
{{Paper
|id=None
|storemode=property
|title=Spatiotemporal knowledge representation and reasoning under uncertainty for action recognition in smart homes
|pdfUrl=https://ceur-ws.org/Vol-710/paper3.pdf
|volume=Vol-710
|dblpUrl=https://dblp.org/rec/conf/maics/AmirjavidBBB11
}}
==Spatiotemporal knowledge representation and reasoning under uncertainty for action recognition in smart homes==
https://ceur-ws.org/Vol-710/paper3.pdf
Spatiotemporal knowledge representation and reasoning under
uncertainty for action recognition in smart homes
Farzad Amirjavid, Kevin Bouchard, Abdenour Bouzouane, Bruno Bouchard
Department of mathematics and computer science
555, university boulevard, Chicoutimi, Quebec, Canada G7H2B1, University of Quebec At Chicoutimi (UQAC)
farzad,amirjavid@uqac.ca
Kevin.bouchard@uqac.ca
Abdenour_bouzouane@uqac.ca
Bruno_bouchard@uqac.ca
Abstract Although uncertainty and imprecision is included always
We apply artificial intelligence techniques to perform data with the action recognition field, in most of the performed
analysis and activity recognition in smart homes. Sensors researches up to now [1,6,11,13,14,15,16,23,24] the
embedded in smart home provide primary data to reason about existing uncertainty and imprecision in OA‟s behavior and
observations and provide appropriate assistance for residents to home state is not considered and they are not robust if
complete their Activities Daily Livings (ADLs). These residents activity realization models change. Furthermore, any small
may suffer from different levels of Alzheimer disease. In this change in sensors network, sensors locations and sensors
paper, we introduce a qualitative approach that considers number could lead to restricting all their applied models
spatiotemporal specifications of activities in the Activity
and all the previous training tests would not be useful any
Recognition Agent (ARA) to do knowledge representation and
reasoning about the observations. In this paper, we consider more. Moreover, objects movement, which provides
different existing uncertainties within sensors observations and important information in activity recognition, has not been
Observed Agent‟s activities. In the introduced approach if the considered.
more details about environment context be provided, the less Most of the surveyed activity recognition approaches do
activity recognition process complexity and more precise not tolerate relatively detailed information about the real
functionality is expected. world and even they may avoid more sensors for not to
receive complementary information about the activities.
1 Introduction The reason is that increase in number of applied sensors
Smart home mostly addresses the health-care problem of could lead to process complexity and they would need a
performing automated assessment of functional health for huge dataset for training. I contrast, the introduced
elder adults and provision of automated assistance that will approach in this paper welcomes the increase in input
allow people suffering from Alzheimer to remain information and in the case of change in sensors network
independent [16]. In order to live independently at home, structure, and the old knowledge would be still valid.
adults need to be able to complete key activities of Daily Furthermore, the increase in provided information would
Living, or ADLs, however tracking of ADL even cause to decrease in process complexity.
accomplishment is a time consuming task for caregivers. In this paper, we are explaining an intelligent agent that
To provide automated assistance we apply Activity tries to explain the observations and detects anomalies in
Recognition Agent (ARA) to reason about observations the case that there is no explanation. Applied knowledge
provided by the embedded sensors in Smart Home. representation and reasoning techniques that benefit from
In this paper, we deal with the activity recognition process activities temporal and spatial specifications is discussed
performing in Activity Recognition Agent (ARA). Event and we introduce fuzzy contexts that can briefly indicate
Recognition Agent (ERA) detects realized events and the home state and possible events that could occur in
report them to the ARA. ARA provides a report for the contexts.
Plan Recognition Agent (PRA) about observed and The art of ranking and classification between generated
inferred activities and finally the Assistance Provision hypotheses inferred from available knowledge and present
Agent (APA) would provide appropriate assistance for the observations can lead to better adjustment between
Observed Agent (OA). The schema1 shows the general system‟s inference and the real world. In this way,
process in the smart home. reasoning can be less complicated and so it causes less
error to choose the right decision in decision-making 2.2 World state and fuzzy context
process. “Fuzzy Context” is the term used to express the home state
A brief explanation about general activity recognition with it. In this way, environmental parameters (called items
process is that after that ERA provides ARA the current and indicated by ix) are measured and then fuzzified by
home state and happened events in fuzzy context and fuzzy fuzzy membership functions. To express a general form of
events frame (knowledge representation), the possible fuzzy context, we apply the following form:
hypotheses through time line are generated and ranked
dynamically. Then in the reasoning process, the C (i1, i2 ,..., in )
explanations about observations would be provided. Temperature is an example for item. For instance, when
the thermometer indicates 37 degree it can be inferred that
it belongs to “Warm” class (applying fuzzy roles and
defuzzification functions) and finally warm is reported
instead of 37 degree. Home state is finally formed by such
this information. As a simple example for home state,
consider a home that includes some embedded sensors to
indicate the home state. These sensors indicate “OA
location”, “door state”, “heater state”, “oven state” and
“temperature”. Mentioned sensors generate continuously
schema1- the general smart home process model values along time axis. The following indicates the final
defuzzified home state:
2 Knowledge Representation Cobst (OA : at _ oven, door : closed , heater : off , Oven : off , temperature : warm)
A knowledge representation system is applied to interpret
sentences in the logic in order to derive inferences from 2.3 Events
them. When we design a knowledge representation system, We define events as each meaningful change in sensors
we have to make choices across a number of design spaces. generated values. ERA simply receives generated values
The single most important decision to be made, is the from the sensors and checks whether the value belongs still
expressivity of the KR. Our desire is to include more to a new class. A change in received values class means an
effective parameters in action recognition process who event has happened and the event is reported to the ARA.
may make the knowledge representation enough expressive
and may make the reasoning process not so relatively 2.4 Discussion
difficult. Brahman and Levesque [1984] introduced the Allen temporal logic is a famous temporal logic that
mentioned desire as contradictory goals; however, we introduced thirteen temporal relations between actions.
believe that applying fuzzy context can lead to more Morchen argued that Allen‟s temporal patterns are not
expressiveness and simpler reasoning in an intelligent robust and small differences in boundaries lead to different
agent. That is because fuzzy context holds more details at patterns for similar situations [2]. Furthermore, the
one hand and at the other hand the defuzzified context complexity increases if the OA performs multiple actions
prevents to generate many relatively similar contexts that simultaneously. Moreover, it does not also indicate the
can make the reasoning process complicated. Here we actions beginning and terminating moments.
introduce two key knowledge types and their From the mentioned problems, we have inspired the idea
representation methods. that we can consider the beginning event (temporal point)
instead of interval consideration and so in this way, it
2.1 Environmental parameters or context items would be necessary just to compare beginning points of
actions and their durations would be justified as their
components that contain fuzzy, relative and estimative
Embedded sensors in the smart home provide primary data
measures as value. So, in brief it can be said that only the
for the Activity Recognition Agent (ARA). The received
before relation would be considered and the possible
data by sensors that is raw and unprocessed introduce the
moments that other actions can begin on them.
environmental components (such as temperature, doors
state, heater state, Observed Agent‟s position, etc) that may To implement the mentioned idea we have applied the
be effective on action recognition process. In fact, the possibility theory that was first introduced by
mentioned components form the body of contexts and are Zadeh[7,8,9,10]. In summary, it is assumed that after
named as context items. Unfuzzy context is a context that observation of an event, all the possible actions can begin
is constituted from a set of items and we define fuzzy simultaneously and the most possible moments for events
context as context constituted from fuzzified items. occurrence is indicated. The farther from most possible
occurrence moments the less ranking value in hypotheses however, this knowledge can include some temporal
ranking introduced in “3.3” section. information about next possible contexts that can possibly
happen in future. We refer to the first introduced type as
The result is that multiple simultaneous running actions absolute time and the second one as relative time.
can be considered and it is enough flexible to consider
different possible temporal relations between actions and To represent temporal dependency (absolute time), we
gives an estimation (by defuzzifying the fuzzy time up to insert a new item to the fuzzy context ontology that is
next Action‟s beginning moment) to predict the action called fuzzy time item. In this way, contexts for similar
termination moment. conditions but different temporal conditions are made. A
For example, for the action entering to the kitchen, the function is implemented to check whether the current time
table1 indicates the possible events (actions beginning is adjustable to the defuzzified time item existing in the
points) that are possible to occur after previously assumed fuzzy context.
occurred events and their possible occurrence moments.
Time elapse as a possible fuzzy event is also applicable. An
example for defuzzified item of fuzzy time can be like
“morning”.
To represent temporal information (relative time) using a
fuzzy trapezoidal digit, we indicate the possible transition
moments to different possible contexts and it is
implemented by a simple table containing the concerning
data. This relative data is converted to the real time at the
running time.
Table1. Possibility distributions for relations between events for 2.6 Spatial Knowledge Representation
action “entering to the kitchen”
Another key knowledge that is helpful to do better
reasoning is spatial knowledge that indicates the context
dependency to the objects locations. As it was mentioned
earlier, movement of objects in the real world provides
noticeable information for the activity recognition process.
There can be considered two general spatial knowledge
Schema2. Schema2.
Possibility fuzzy
distributions
Possibility
temperature
possibility
for
distributions occurrence classes
distributions for occurrence moments
moments
for occurrence moments forms. The first one, which would be referred to as
absolute position, indicates the objects positions in the real
In the table1 the possibility distribution for the “before”
world and the second one that would be referred to as
relation is indicated by the normalized numbers (from 0 to
relative positions indicates the position of objects to each
1) and in schema2 the possibility distribution for possible
other. In the fuzzy context, a section is dedicated for the
occurrence moments of the next event is indicated by
objects positions in the home (first spatial knowledge type)
t e1:e2
which is a trapezoid fuzzy number. In this digit t1 is and the second spatial knowledge type is indicated in the
Event Recognition Agent (ERA). One example for
the soonest moment that event2 can occur after another absolute position application in activity recognition is that,
event1, moments between t2 and t3 are the most possible to infer the cooking activity it is necessary to observe the
moments that event2 can occur and t4 is the latest moment pan on the oven. An example for the relative position
that event2 can occur. (We have forborne to include the inference is that if approach of pot to glass be observed it
necessity distributions in our calculations, which is already can inferred that OA has fulfilled the glass with the pot‟s
dependent to the possibility distributions.) containing liquid such as coffee. ERA provides this
The table1 is implemented as a data table in database and it information as recognized event for the ARA.
indicates the effective environmental parameters to
recognize the action “entering to the kitchen”.
2.6.1 Discussion
2.5 Temporal Knowledge Representation
To recognize objects movements we have applied RFID
We define the term temporal knowledge as a kind of tags and antennas. This process is done in ERA and we
knowledge that is dependant to the time and may lead to would have a short introduction of it in here.
different inferences in different temporal contexts;
In a brief description, we have attached RFID tags on the 3 Reasoning
objects and used RFID antennas to recognize the OA‟s
activities. We have made a program in Java to recognize
The reasoning process in activity recognition follows the
the performed activities by the OA. Every six
observation, hypothesis generation and hypothesis pruning
microseconds applied RFID antennas check the
steps.
environment to detect the RFID tags. By having just one
RFID antenna and attaching RFID tags on the objects, we
are able to recognize if the object is close or far from the 3.1 Hypothesis Generation
antenna. By adding the second antenna, we would be able
to make four regions. The first region is the region around Hypotheses are generated only in the case of event
the first antenna, the sec ond region would be around the recognition reported by the ERA. Movement of objects,
second antenna, and the third region is the region in front elapse of time and a switch in controlling sensors states are
of both antennas and the region that both antennas show possible observable events. In fact, the generated
equal signal strength to detect the objects and the fourth hypotheses indicate the possible future contexts could
region is the region that no antenna can easily detect the possibly be observed in the future.
object (see schema3).
The hypothesis generation process in summary is that at
first hypotheses are generated based on a table named as
possible fuzzy events (see table1) that could have been
generated in future in the current context. At the second
step, they are assigned the possible observation moments
by the use of trapezoidal fuzzy digit (see schema2) and
finally they are ranked or weighted (see part 3.3). the
mentioned process is illustrated in schema4.
Schema3. Regions defined by RFID antennas
In ERA, entering and exiting a region is recognizable by
the available equipments and the concerning events are
reported to the ARA. In the absolute position recognition,
it„s enough to find the object‟s location in one of the
mentioned regions, however in relative position
recognition we should find two target objects in one
region.
2.7 Spatiotemporal Knowledge Representation Schema4.hypothesis generation
Spatiotemporal knowledge is key environmental
3.2 Hypothesis Generation through the time line
information to do activity recognition; however, there is
other effective environmental information such as
temperature, door‟s position and other items that are also Considering uncertainties for unrecognized but in reality
useful for controlling affairs in smart home. to represent happened events (there are several reasons for it), it is
such this knowledge we have divided fuzzy context into possible that it defects the reasoning process and so ARA
three major sections. One section for temporal knowledge, wrongly detects normal actions or activities as anomaly.
another section for spatial knowledge and third section for To improve the activity recognition efficiency we consider
controling items is provided. that possible events may have happened but not observed
and they are generated and pruned through the time line. A
question that may arise in here is that what could be the
Introduced fuzzy context let us consider different
occurrence time of undetected event? The answer is that
knowledge types in action recognition and the controlling
the defuzzified value of the fuzzy trapezoid number can
affairs (using checking functions) are done at the transition
indicate the possible moment that the event has happened.
moments. Transition between contexts is also indicated by
In the case of anomaly detection, it would be checked
the observed fuzzy events reported by ERA.
whether there have been no undetected event and there is
no previously generated hypotheses that can explain the
occurred events.
limit the pruning to a fix number of levels. Whenever a
hypothesis be proved, the concerning weight for that node
3.3 Hypothesis Ranking is assigned one.
When new hypotheses are generated, they are inserted as
tree leafs (we can call it also decision tree) and then they
are ordered by defuzzified occurrence moment from left to
right. To describe briefly the ranking process, we assign
each observed and proved a higher point and in contrast
unobserved or not yet proved hypotheses are assigned
lower points.
schema5.hypothesis pruning, reasoning and explication of
observation
The rank and weight of generated hypotheses ( wi (t ) ) can
change dynamically by elapse of time. The primary
assigned weight is derived from the possibility distribution In the schema5 the sequence of C1 and C2_1 and C3_1
for occurrence of event ( e1:e2 existing in table1) and as indicate an explanation about the latest observations.
the fuzzy trapezoid number affects it, so by elapse of time
it can differ to the past weights ( t , schema2). The 3.5 Reasoning Process
e1:e2
third parameter to affect the hypotheses rankings is the Our goal in reasoning process is to find an explication that
possibility distribution of the upper node occurrence ( Wu ). can explain the observations. Observation of a fuzzy event
Finally, γ affects the ranking value. γ is a value that is is a good reason to decide whether there are anomalies or
resulted from a trade-off between smart home precision in not. However, the more OA be conscious the more rely on
event detection and uncertainties about behaviours of unproved hypotheses. The sequence of observed events can
Observed Agent (OA) or in other words Alzheimer explain the current activities and actions. Furthermore, the
severity degree. At one side, the more severity in contexts can explain the precedence of home states. So,
Alzheimer illness the less confidence on the OA and at the recognition of current context from the previously
other side the more precision in event recognition, the generated hypotheses can well explain the observations
more confidence on the reports and so it would be less and current activity(ies). Whenever no explanation for the
necessary to trace the tree down to a lot of levels. The observation is found or the explanation does not include
ranking formula is indicated as: minimum acceptance weight (dependent to γ), so the
observed action would be recognized as abnormal action.
wi (t ) .Wu . e1:e2 . te :e
1 2
4 Implementation and Conclusion
The ARA was implemented in VB.net environment and it
3.4 Hypothesis Pruning was simulated in SIMACT [27]. The activity “entering to
the kitchen” was simulated in different scenarios (but the
To prevent the increase in number of less possible same old embedded sensors) and some uncertainties in
hypotheses, pruning is necessary. Pruning is applied in the event recognition (see picture1). Anomaly detection would
case of low possibility distribution of event occurrence. In not be better than 50% done if the unproved hypotheses
addition, observation of a possible event that could have grow deeper than three levels in decision tree. In spatial
happened calls the pruning function1. Another way is to reasoning it can be said that the more antennas be applied,
the more precise hypotheses would be generated. It can be
1 inferred that in the introduced approach, in the case of
To estimate the closeness of new observation to the previously increasing the sensors number, more precise hypotheses
1 would be generated and proved. Fuzzy context at one hand
generated and assumptive hypotheses, we applied the can express well the real world state and it can decrease
reasoning complexity if it be well defuzzified.
formula to check the difference between values of the observed
and assumptive context items. If all the differences between all
1
the items be more than then no explain is found.
8- Fuzzy Sets as a basis for a theory of possibility, L.A. Zadeh,
Fuzzy Sets and Systems, vol. 1, pp. 3-28, 1978.
9- Possibility Theory, D.Dubios, H.Prade, Plenum Press, 1988.
10- Fuzzy sets and probability : Misunderstandings, bridges and
gaps, D.Dubois, H. Prade, Proc. of the Second IEEE Inter.
Conf. on Fuzzy Systems, volume 2, pp. 1059-1068, 1993.
11- Vikramaditya R. Jakkula, and Diane J. Cook, "Learning
Temporal Relations in Smart Home Data",Proceedings of
the Second International Conference on Technology and
Aging, Canada, June 2007.
12- James F. Allen: Maintaining knowledge about temporal
intervals. In: Communications of the ACM. 26/11/1983.
ACM Press. S. 832-843, ISSN 0001-0782.
13- Roy P., Bouchard B., Bouzouane A., Giroux S.: A
possibilistic approach for activity recognition in smart
Picture1- the activity “entering to the kitchen” simulation in homes for cognitive assistance to Alzheimer's patient. In
SIMACT Activity Recognition in Pervasive Intelligent Environment
(Atlantis Ambient and Pervasive Intelligence), L. Chen, C.
Nugent, J. Biswas, J. Hoey Editors, World Scientific
Publishing Company, ISBN: 978-9078677352, pp. 1-20,
September 2010.
5 Future Works
14- Roy P., Bouchard B., Bouzouane A., Giroux S.: Challenging
issues of ambient activity recognition for cognitive
We recommend the interested researches to survey the assistance. Handbook of research on Ambient Intelligence
Activity Recognition in the case of multiple residents in
and Smart Environments: Trends and Perspectives, IGI
smart homes and also to introduce an optimization model
global, F. Mastrogiovanni and N. Chong Editors,
for fuzzy roles to decrease the activity recognition
Information Science Publishing, ISBN: 1616928573, pp. 1-
mistakes.
25, august 2010.
15- Roy P., Bouchard B., Bouzouane A., Giroux S: Combining
6 References pervasive computing with activity recognition and learning,
1- Bruno B., Bouzouane A., Giroux S.: A Keyhole Plan Web Intelligence and Intelligent Agents, Zeeshan-hassan
Recognition Model for Alzheimer's Patients: First Results, Usmani (Ed.), ISBN: 978-953-7619-85-5, INTECH, pp.
Journal of Applied Artificial Intelligence (AAI), Taylor & 447-462, 2010.
Francis publisher, Vol. 22 (7), pp. 623-658, July 2007. 16- G.Singla, D. Cook, and M. Schmitter-Edgecombe.
2- Didier Dubois and Eyke Hullermeier, comparing Probability Incorporating temporal reasoning into activity recognition
Measures Using Possibility Theory: A Notion of Relative for smart home residents. Proceedings of the AAAI
Peakedness, International Journal of Approximate Workshop on Spatial and Temporal Reasoning, pages 53-61,
Reasoning, 2007. 2008.
3- D. Dubois, H. Prade and R. Sabbadin (2001) Decision- 17- Diamond J. A report on Alzheimer disease and current
theoretic foundations of possibility theory. Eur. J. research. Technical report, Alzheimer Society of Canada,
Operational Research, 128: 459-478. (2005), 1-19.
4- D. Dubois and H. Prado. Possibility Theory, Plenum Press, 18- Baum C., Edwards D.,: cognitive performance in senile
New York, 1988. dementia of the alzheimers type: The kitchen task
5- D. Dubois, H.T. Nguyen and H. Prade, Fuzzy sets and assessment. The American Journal of Occupational Therapy.
probability: misunderstandings, bridges and gaps. In: D. 1993, Vol. 47 (5), 431436.
Dubois and H. Prade, eds, Fundamentals of Fuzzy Sets. 19- Jensen, F. V., Bayesian Networks and Decision Graphs
Boston, Mass: Kluwer, 343-438, 2000. (Springer 2001)
6- Cook, D.J. Youngblood, M. Heierman, E.O., III 20- M. Iosifescu, "Finite Markov processes and their
Gopalratnam, K. Rao, S. Litvin, A. Khawaja, F, MavHome applications" , Wiley (1980)
an agent-based smart home, IEEE Computer Society 21- N.A. Abdul-Manaf, M.R. Beikzadeh, rep esentation and
Washington DC, USA, 2003. Reasoning of Fuzzy Temporal Knowledge (2006), IEEE Int.
7- Probability measures of fuzzy events, L.A.Zadeh, Journal Conferences on Cybernetics & Intelligent Systems and
Math. Anal. Appl., vol 23, pp. 421-427, 1968. Robotics, Automation & Mechanics (CIS-RAM 2006).
22- Dubois, D., Prade, H. and Sandri, S. (1991). “On
possibility/Probability transformations”, proc. Of the 4th
International Fuzzy Systems Association (IFSA‟91)
Congress, Brussels, Mathematics, PP. 50-53.
23- V. Jakkula, J. Cook. “Temporal pattern discovery for
anomaly detection in a smart home”, 3rd IET International
Conference on Intelligent Environments (IE 07), 2007.
24- S. Luhr, G. West, S. Venkatesh. „Recognition of emergent
human behaviour in a smart home: A data mining approach“,
Pervasive and Mobile Computing Volume 3 , Issue 2, 2007.
25- G. Nagypal, “a fuzzy model for representing uncertain,
subjective and vague temporal knowledge in ontologies”,
http://www.springerlink.com/content/atnluqe2gn8y7h27,
2003.
26- D. Dubois and H. Prade (1998) Possibility theory:
Qualitative and quantitative aspects. In D. M. Gabbay and P.
Smets P., editors Handbook of Defeasible Reasoning and
Uncertainty Management Systems, Vol. 1., Dordrecht:
Kluwer Academic, 169-226.
27- http://www.springerlink.com/content/j4g35l38913w2j0t