In this paper we describe an implicit user interface for smart environment control: We make our system guess how to assist the user(s) proactively. Our controller is based on two formal descriptions: One that describes user activities, and another that specifies the devices in the environment. Putting both together, we can synthesize a probabilistic model, the states of which resemble activities performed by the user(s) and are annotated with sequences of device actions, with the latter to be executed in cases particular activities have been recognized. The resulting system is purely reactive and can be executed in real time.
As computers become smaller and smaller, the vision of ubiquitous
computing becomes true. At the same time, smart environments
contain a large number of devices and become thus more and more
complex. Thus, configuration as well as correct usage gets more
time consuming and error prone. Exploring new ways to control
these invisible devices is a challenge addressed by current research
[
As illustrating example in this paper we use the task of giving a presentation inside our smart meeting room. This environment is introduced below. The graphical representation of this task is given in Figure 1. Here the task of giving a presentation decomposes to a sequence of sub-tasks. The user starts the presentation with entering the room and moving to the front of the room. When the presentation is finished the user moves to the door to leave the room. A more detailed description of this example is given in section 3.
The controller described below is based on semantic models of the user and its environment. For our system we currently employ formal action descriptions and task models which are compiled into a probabilistic model. All necessary concepts are briefly introduced below.
Hidden Markov Models (HMMs) [
Depending on the available sensors, we can detect the current
activity of users. For example, an indoor positioning system can be used
to detect whether a user is entering the room and heading for the
presentation stage. As customary in activity and intention
recognition, we use probabilistic models. Such models can cope with
noisy and contradictory sensor data and allow nonetheless to infer
the most likely sequence of actions or complex intention. Here,
we use Dynamic Bayesian Networks [
>>
Front >>
>>
abilistic modeling. Calculating a probability distribution over the
current state with respect to the observed sensor data as well as the
previous state is known as filtering. Doing this requires a model,
that describes both, the behavior of the user and the sensor data
observable. In addition of recognizing the activity these methods
allow to predict future activities, in this case intentions, of the user.
Complex behaviors of (groups of) users can formally be described
using CTTE [
In the planning domain definition language (PDDL) [
The environment where most of our experiments take place is the
so called Smart Appliance Lab. This room is instrumented with
various sensors such as the location tracking system Ubisense [
true
false
true
TT
FT
false
TF
FF
environment as well as the middleware controlling the devices of
the environment are described in [
The task specification in Figure 1 contains a detailed description of this example. The annotated effects (illustrated as clouds) describe the desired state of the environment for the following sub-tasks. As description language for task models we use CTML, as described in section 2. The graphical representation of the task model omits the description of the observation data and the priority function. The world state in this example is given in Table 1 and only consists of the two devices. Each of them has a binary state, in case of the projector it is either turned on or turned off. The canvas can be up or down.
Our goal is now to build a controller that recognizes the current state of the user and executes corresponding device actions that makes the annotated effects come true. By executing these action sequences that system automatically assists the user in achieving his goals. (:action canvasdown :parameters (?c - canvas) : :precondition (not (isdown ?c)) :effect (isdown ?c )) (:action projectoron :parameters (?p - projector) : :precondition (not (ison ?p)) :effect (ison ?p))
This section explains how to combine formal descriptions of the environment and the user behavior into a purely reactive probabilistic model. First the description of the user is compiled into a probabilistic model allowing to recognize the user’s intentions. Then, we enrich this model by annotating the states with actions, executable by the environment. While running the system, and based on the current state of the environment one of the states will be most likely. The actions attached to the state are then simply executed, resulting in a system supporting the user while achieving his high-level goals.
In the following sections we discuss two different possibilities of enriching the model with device actions. The first is to generate different HMM states for each possible world state. The latter is to annotate the corresponding state with sequences of device actions for all possible world states. A plan for the current world state is then accessible by taking the world state as key for a lookup table resulting in the corresponding plan.
We start with the annotated task model from Figure 1, which
consists of a task model and effects annotated to sub-tasks. Each effect
is a subset of the world state. It consists of the cartesian product
of all device states. We apply the transformation given in [
We extend the original task model by annotating tasks with their effects with respect to the world state. The effect of the annotated task should be true after executing the task. This can be done by either the user or the controller. Figure 1 contains the additional effect specification for the Move Front and the Move Door subtasks. In our scenario the effect of the state Move Front is that the environment is prepared for the presentation, namely the canvas is down and the projector is on.
To ensure the effects of an annotated HMM state become true, the controller has to execute device actions depending on the current world state. The canvas has to be lowered if it is up but if the projector is already turned on we can omit turning on the projector. Therefore we have to generate sequences of devices actions for each possible situation. This is done by taking each world state as start situation for a planner and the desired subset of the world state, described by the effects of the annotated HMM state as goal. To realize this the planner takes device action specifications, as shown in Figure 2 and Figure 3. Result of this planning step is a sequence of device actions for each possible world state that has to be executed to create the effects specified in the original annotated task model in Figure 1.
The next two sections describe how to use these world state device action sequence pairs to generate the controller. Both approaches follow the workflow given in Figure 5.
In order to create the distinction of the different world states as HMM states, it is necessary to unfold annotated HMM states by using the different world states. Therefore we replace the annotated HMM state by extended HMM states that are generated from each possible world state and the state itself. In our example Move Front will be replaced by each element of the cartesian product of the world state to ensure that each observation of a world state corresponds to one HMM state. Here Move Front is replaced by four new HMM states, each representing a possible world state. Only the HMM state that covers the complete effects has a transition to the HMM state generated from the following sub-task. In our example only the Move Front TT state, that assumes that the projector is on and the canvas is down has this transition.
This allows to attach plans to the states in the probabilistic model that needs to be executed in that state as follows: Every annotated user state is combined with every world state, that is a combination of all states of the devices. This world state is used as precondition for device actions during compilation process.
The generated HMM contains so called slices, that consists of all states generated from one sub-task from the CTML specification. One slice itself was created from the cartesian product of the world state. The intra-slice states differ from each other only by the possible observations of the world state. The names of the intra-slice states illustrated in Figure 4 contain the true/false value given in Table 1. Every state of the slice has an incoming intra-slice transition with a probability given by the number of states. Only the states that create the effects given in task model description have a outgoing inter-slice transition with very high probability. It is possible ProjectorOff (projector1)
Leave Room that there are more than one state that covers the same effect due to missing effects to single devices of the world. The probabilities of the intra-slice transitions are just given by the number of target states. Inter-slice transitions are generated with respect to the temporal operator of the sub-tasks. The probability of transitions are generated by the normalized weight of the single sub-tasks.
Another approach to enrich the HMM with device action sequences for user assistance is to create a lookup table for necessary device actions and attach it to the corresponding annotated HMM state. As in the first approach the device action sequences depend on the current world state and need to be generated by a planner that uses the specified effects as goals. Attaching this device action sequence - world state pairs to the annotated HMM state provides a lookup table at runtime. Using the world state, consisting of all device states the table provides the pre-generated device action sequence that has to be executed in order to make the specified effects to the environment become true.
In our scenario the states Move Front and Move Door are extended by lookup tables for user assistance. The table annotated to the Move Front state contains device action sequences that ensure that the projector is turned on and the canvas is down. The Move Door HMM state is annotated with a table of device action sequences that ensure that given any world state the projector is turned off and the canvas is up. Figure 6 contains a graphical representation of the generated extended HMM. Probability distribution functions as well as state transition probabilities are omitted for reasons of clarity.
The previous sections describe two approaches to attach pre-generated plans to states of a probabilistic model, namely an HMM. Both approaches generate realtime capable systems, that do not have to solve planning problems such as finding a sequence of device actions to support the user. The first approach creates an HMM with Initial
Enter Room
Move Front
Move Front
Leave Room TT TF FF FT
ProjectorOn (projector1) CanvasDown (canvas1) ProjectorOn (projector1) CanvasDown (canvas1)
TT TF FF FT
CanvasUp (canvas1) ProjectorOff (projector1) CanvasUp (canvas1) ProjectorOff (projector1) very much states, because it creates states for every possible world state. Here the distinction of the world state is done at the state level. The idea of unfolding HMM states by using every possible world state is appropriate if the state of a device is not reliable or noisy.
The second approach is to move the distinction of the world states from different HMM states with attached plans to one HMM state that contains a table of multiple plans, one for each possible world state. The number of HMM states is independent from the world state, which avoids a very high number of states. This approach is applicable whenever the world state is known definitely. This means that each device state is observable without any noise or inconsistency.
We developed an execution framework for Bayesian inference that is able to perform fast online filtering of HMM’s and particle filters. By separating the model description from the implementation of the algorithms we designed a highly reusable framework. This framework enables users to embed parameterized filters into different environments. This enables us to integrate the generated controller into the software structure described above. Users of this environment only need to implement problem specific details. A probabilistic model for filtering in our framework has to be implemented in C++. One has to describe three different parts. First a specification of the state space, that is in the case of an HMM represented by a set of states. Second the transition probabilities from each state to another, represented as matrix of probabilities. Third a probability distribution of sensor observations for each state. To provide a more intuitive tool for describing HMM’s we introduced a description language that supports a simple way to describe HMM based models.
The compilation process creates the state space of our controller
from the single sub-tasks of the user model combined with each
possible device state combination. The transition probabilities are
given by the probabilities of the model generated only by the
subtasks as described in [
In this paper we showed that a context-aware controller for smart environments can be created from a combination of semantic models of the user and the environment. The controller is based on bayesian inference where the model was generated from task-based specification of users together with a precondition and effect specifications of each device forming the environment. Sensor data as well a accumulated world state serve as input, a device action sequence, that needs to be executed as output of the inference process. We introduced two ideas to merge task based user models and precondition and effects specification of the environment to create probabilistic models that assist the user.
Further research should include smart environment evaluation of the controller described here. Both approaches should be evaluated and the results should be compared for different devices and scenarios. This includes tests for maximum manageable complexity of the state space as well as minimal complexity that creates sufficient user support. Due to the compile time planning process we are able to pre-generate action sequences. This allows to find modeling problems such as deadlocks at compile time. Our approach in this paper utilizes HMM’s for inference. However, since the state space may explode and exact inference will not be suitable, we can change the inference algorithm to Monte Carlo based methods such as particle filters. These methods are already supported by the execution environment described above.
The controller introduced here is described as central service. It is possible to decentralize this approach to the usage of multiple services, each of them describing a subspace of the model. By comparing the likelihood of multiple services, it is be possible to choose either an action sequence of one agent or a combination of multiple sequences that do not disturb each other.
Another point that should be analyzed is how both systems behave if only the most probable device action sequences will be preplanned. If the system reaches a state that does not contain a device action sequence the plan has to created at runtime. The realtime behavior of this extension has to be examined.
Frank Krüger’s work in the MAXIMA project as well as Gernot Ruschers’s work in the MAIKE project are both supported by Wirtschaftsministerium M-V at expense of EFRE and ESF.