We describe rst experiments for controlling smart environments using a brain-computer interface. The graphical user interface is automatically synthesised from device models that specify e ects of device functions on the environment. Thus, the number of interactions can be reduced, and a novel way of human machine interaction is introduced: Controlling the environment instead of single devices.
Our experience so far shows that with those simple BCI
devices, interaction is kept within tight bounds due to the
limits of this communication channel: A merely small character
set is available at a low frequency, which leads to a severely
limited data rate. Applications based on low-cost BCIs thus
have to deal with these limitations and to adapt their
graphical user interfaces (GUIs). These need to optimise the
number of user interactions necessary to trigger an intended
application function. Thus, highly application-speci c GUIs
need to be implemented, and various approaches have been
developed that are aimed at grouping functions smartly.
With the vision of Ubiquitous Computing coming true,
device become more and more invisible to the user, and hence
cause the need for novel user interfaces. One speci c
application eld in this context is that of smart environments
[
One approach to provide a user interface for a dynamic
device ensemble would be the synthesis of a dynamic GUI from
formal device descriptions. Various approaches, for example
built upon UPnP [
Our approach presented in this work relies on the following working hypothesis: What users of a smart environment are interested in is not the individual device, but their e ect on the environment. One simple example: When a user switches a lamp on, he actually just wants to increase the lightness of the room. In this way, all the lamps of this room are able to increase the lightness and would therefore be redundant with respect to their e ects on the environment. With this article, we describe rst experiments to control a smart environment using the neural impulse actuator (NIA), a low-cost brain non-invasive computer interface. We use semantic models of the environment and the devices. We model the devices with respect to their speci c in uence onto the environment. We present a principle approach for the synthesis of graphical user interfaces in order to reduce the number of necessary interactions.
After presenting the neural impulse actuator, we brie y discuss our lab used for the experiments. Furthermore, we give a short introduction to STRIPS: A formalism to describe preconditions and e ects of operations.
In 2007, OCZ Technology Group Inc. [
After calibrating the NIA, it is supposed to be usable as a virtual joystick and to switch events, which can be triggered by di erent electric potentials or muscle movements. In our experiments, we found the following actions easy and stable to recognise: eye movement in general, heavy muscle movement on the forehead, or moving the jaw, light muscle movement on the forehead, heavy thinking, and relaxing, or closing the eyes.
Here, we want to use the NIA to control our lab environment, even while working on other subjects. Hence, inputs triggered by heavy thinking and relaxing are not suitable signals for a smart environment controller. However, parallel performance to compose more complex signals does not seem to be helpful, as we want to provide an easy-to-use interface. Therefore, we have at most three distinguishable signals at hand: (i) eye movement, together with (ii) heavy and (iii) light forehead muscle movement. To complicate things further, users of the NIA can perform those signals at a merely low frequency of about 10 per minute at most, leading to a comparatively low data rate.
For our experiments we utilised our SmartLab: An instrumented meeting room (cf. g. 2) equipped with a number of remotely controllable devices. It is frequently used as a room for lectures, presentations, and meetings, but also as an experimental setup for user studies.
Our lab is equipped with a couple of sensors, needed to
observe state changes in the room. There are e.g.
sensors capable of detecting whether the windows are closed
or opened, measuring the current temperature, or detecting
persons that enter or leave the room, and estimating their
number and current positions [
On the other hand there is a number of remotely
controllable devices required in typical meeting rooms: Dimmable
lamps as well as movable projection screens and sun shades,
controllable via EIB [
Our lab features a powerful middleware (as for instance
described in [
To describe the capabilities of devices and their possible
actions, we suggest to use STRIPS-operators as illustrated in
[
Due to their associated declarative semantics, they are well suited for an automatic interpretation and hence for the construction of a controller. We annotate every operator with the middleware command which needs to be executed to perform the operation. Figure 3 shows two simple operators describing how to switch a lamp l on and o .
As depicted above our user interface needs to cope with a dynamic ensemble of heterogenous devices, which is the reason we do not have the option to hard-code a certain controller for a xed environment. Therefore, we need to consider ways and means of synthesising a controller from an abstract description of the environment and the devices.
According to gure 4, we consider three di erent modelling tiers: Top left in the sketch is the layer of the background model. It speci es existing parameters of the environment and how they can be modi ed. To provide an example we have formalised two speci c parameters and their respective operations in gure 5: There exists some parameter that corresponds to the lightness of our room, and it can be modi ed by two operations named increase and decrease. The parameter temperature is handled likewise. This explicit kind of formal speci cation of environmental parameters is needed later on to describe the e ects on the environment caused by triggering device actions.
As illustrated in section 2.2 all of our devices are made accessible through our middleware, that way providing on the one hand the entire set of current properties to other components and on the other an easy-to-use interface for triggering device actions. In gure 4 this device representation layer is called device model and establishes a certain level of abstraction from the plain hardware, where every device can exist without requiring local knowledge on the existence of other devices, the middleware or even the environment. Besides properties and actions this layer holds additional information on the device such as the device type, its name, or whether the device is currently available in the system. The third modelling tier, the e ect model, now etablishes the relation between the raw device descriptions from the device model and the environmental parameters from the background model. This is done by modelling device actions with respect to their e ect on the environment. Every action is annotated with a formal description of the in uence of its execution on the speci ed environmental parameters. As depicted in gure 3 for instance the execution of the turnOn method of a lamp has an e ect on the environmental parameter lightness in the form that the latter would be increased. We suggest to use STRIPS as modelling formalism to describe the semantic meaning of actions to the environment.
Precond: Lamp(l) ^ Room(r) ^ In(r,l) ^ O (l) Effect: :O (l) ^ On(l) ^ Lightness.increase(r) Command: l.turnOn() Action(switchO (l,r))
Precond: Lamp(l) ^ Room(r) ^ In(r,l) ^ On(l) Effect: :On(l) ^ O (l) ^ Lightness.decrease(r) Command: l.turnOff() Background Model
Effect Model
Devices Device Model
Device Model
Device Model
Device Model Actions
Controller GUI
GUI Views
GUI The information aggregation mechanisms of our middleware enable applications to gather these e ect models analogously to the previously mentioned properties as well as type and status information of devices. Therefore a GUI application { called controller in gure 4 { is now able to provide itself with a list of all devices together with their descriptions, i.e. the type (lamp, sun shades, ...) of the device and all its available actions, including their particular preconditions and e ects. After collecting the operators, it can generate diverse GUIs views. Below we present some rst experimental views that shall demonstrate the descriptive power of our proposed approach.
As mentioned above, there are basically three di erent actions (keystrokes) a human can reliably perform. Based on the formal description of our devices we implemented lab controllers tailored for a limited communication between human and computer to evaluate our approach. We designed them following the Mac Finder's Column View. Two keys are used to move the focus up and down a list, the third to select the item.
Our very rst prototype contained three columns, of which the rst contained a list of device types, the second all available devices of the type selected in the rst column, and the third all the applicable functions of the device in the second column. This prototype does not involve any environmental knowledge yet. Due to its three-button-based design, this controller interface would enable users who have to rely on a NIA to take control of a complex and dynamic set of heterogeneous devices once these have been seen by the system through the middleware. But without further tweaking and tuning { which we elaborate on in section 4 { the menus would be very large if they contain every possible action for every possible device.
Operation(Lightness, increase)
Operation(Lightness, decrease) Parameter(Room, Temperature)
Operation(Temperature, increase)
Operation(Temperature, decrease)
Our second prototype now extracts advantages from our formalisation: As an alternative to a static arrangement based on the device type, we can group our devices with respect to their e ects on the environment, as we did in our second controller, which is shown in gure 6. Here, the rst column has been replaced by a column containing controllable parameters as for example the lightness, or the temperature. Then, all those devices are listed in the second column which can actually in uence the selected parameter. Finally, the third column would again contain performable actions. But our formalisation of e ects has further advantages: The previously depicted controller still displays all the devices even if they have similar in uence on the environment. This leads to a long list of devices with each of them still providing every possible action. But, with respect to their e ects they are kind of redundant and if we assume that a user is not interested in the device and its particular action itself but is interested in its e ect, we can omit all these information and simply provide control options of an environment. For this purpose, we can simply use our existing model of environmental parameters. Our third controller prototype working this way is depicted in gure 7. In the rst column it displays the environmental parameters of the background model, which can then be adjusted by selecting one of the items in the second column.
Within our research project MAike, all devices send their
descriptions to a central look-up, realised as a tuple space
[
Our investigations leave us with mixed feelings. On the one hand, it is indeed possible to evoke a limited set of actions using the NIA controller. On the other hand, the concentration required from the user (and, indeed, the level of self control regarding the facial expression), in our opinion leaves ample room for optimisation of both signal acquisition at the sensory level and signal processing at the algorithmic level. While developing the controller, we could reliably distinguish three di erent inputs only. Of course the NIA itself provides a richer interface in form of analogue joysticks, but we found that those are hard to control while concentrating on other tasks and they cannot be distinguished as reliable as needed to control the environment. Nonetheless, a better recognition of crisp events would be desirable for the future. With this work we present a mechanism that deals with the issue of operating a complex and dynamic system through such restricted channels. If we had to deal with an environment that would be static and consist of a xed number of well-known devices, we would be able to build a controller being perfect in terms of the number of interactions. Because we need an automatically synthesised controller, our approach uses STRIPS operators to apply semantic knowledge of the system's devices and their possible actions aiming at design (and synthesis) of adequate user interfaces. In the simple approaches presented above, the menus would still grow very large with every new device entering the scene if the menus still contain every possible action for every possible device. Therefore, it is desirable to show the user not the whole menu. Instead, we should o er more abstract actions. Furthermore, we would like those abstract actions to be context dependent and automatically generated. The focus of this work was not to develop the perfect-working graphical user interface for controlling tasks using the NIA controller. Our goal was to investigate several approaches to reduce the number of interactions a user has to perform to have a function executed. Therefore, the synthesised GUIs will never be the best ones one would nd through manual design. This work considers ways and means of involving formal descriptions of environmental knowledge into automatic GUI development. We have not yet evaluated which are best suited for the present application case and di erent approaches from the ones depicted in this work are imaginable.
As mentioned above, the menu structure used in our prototype, does probably not allow to control larger environments. In the following section 5, we have discussed a number of methods to create more suitable menus and higher level actions. Those have to be implemented and tested with respect to their usability.
Nevertheless, our solution seems to be a principle approach for the synthesis of graphical user interfaces in general, which could bring a signi cant bene t not only for motorically challenged users, or those who require hands-free interaction in situations, where speech control is not an option.
The controller described above is applicable for small wellde ned scenarios, but certainly not to control a complex infrastructure with hundreds of device-actions. This is due to the realised menu-based interaction. Considering a usual living room, it is not hard to think of many di erent devices which should be controllable. Just think of all lamps, shades, multi-media devices, etc. Therefore, alternatives are currently under investigation. We try to learn user preferences online and group devices dynamically. For example, we could group devices together as one virtual device if they have been used in one go a couple of times. Another grouping approach could be a Hu man-like encoding which would put the more important devices above others in the list. As mentioned above lots of user evaluation needs to be done in order to nd more intuitive user interface design methods that make use of formal descriptions of the e ects caused by device actions on the environment and of the environment itself. One approach could be another control metaphor which is especially designed for brain computer interfaces: A rectangle that periodically rolls over a list of items on the screen and each time marks the currently covered item. The user now simply thinks of the particular item he wants to choose and a signal from the brain indicates the moment when the bar covers this item. This selection mechanism can not only be utilised with lists of items, but also like shown in gure 8: First a horizontal bar goes the up-down direction and stops when the height of the selected item is reached. Second, the vertical bar starts moving and is used to indicate the particular device in this line.
The area of automatic intention analysis, that is the detection of the user's goals based on his current activities, opens further possibilities. Given the user's current goals, we could automatically re-arrange the menus to provide faster access to device which are most likely to be used in the current situation. Going one step further, high-level actions could also be added to the controller. A room detecting the start of a lecture could o er the compound action like \put my presentation on the projector", which consists of several smaller actions. For this purpose some additional strategy synthesis component would be needed.
We have not yet fully covered the possibility of the existence of multiple environments, containing di erent devices, but controllable through the same user interface. This could be the case in a smart home consisting of several smart rooms, where users possibly would like to have functions performed in other rooms remotely or trigger actions in di erent rooms simultanously. One straightforward approach would be an additional column inserted before the rst one, that speci es the particular room. After this column would be the ones depicted in the previous sections.
So far we have been relying on the feedback on an existing computer screen. This is possible for users that can carry a small display with them. E.g., for people depending on a wheel chair, this monitor can be integrated into it. Alternatives should be investigated, e.g. sounds or small displays right next to the devices which are controllable. This would enable the user to move freely around in the environment without watching a computer screen for every action.
Gernot Ruscher's work in the MAike project as well as Frank Kruger's work in the MAxima project are both supported by Wirtschaftsministerium M-V at expense of EFRE and ESF.