Controlling Smart Environments using a Brain Computer Interface Gernot Ruscher∗ Frank Krüger Sebastian Bader Thomas Kirste Universität Rostock, Albert-Einstein-Str. 21, 18059 Rostock, Germany ∗ corresponding author: gernot.ruscher@uni-rostock.de ABSTRACT have to deal with these limitations and to adapt their graph- We describe first experiments for controlling smart environ- ical user interfaces (GUIs). These need to optimise the num- ments using a brain-computer interface. The graphical user ber of user interactions necessary to trigger an intended ap- interface is automatically synthesised from device models plication function. Thus, highly application-specific GUIs that specify effects of device functions on the environment. need to be implemented, and various approaches have been Thus, the number of interactions can be reduced, and a novel developed that are aimed at grouping functions smartly. way of human machine interaction is introduced: Control- ling the environment instead of single devices. With the vision of Ubiquitous Computing coming true, de- vice become more and more invisible to the user, and hence cause the need for novel user interfaces. One specific ap- Categories and Subject Descriptors plication field in this context is that of smart environments H.5 [User Interfaces]: Graphical User Interfaces, Input [7]. These build complex sets of heterogenous devices, partly Devices and Strategies fixed to the environment and partly brought-in by the user. Thus, applications in smart environments need to base on General Terms such a dynamic ensemble of devices which are possibly un- Design known in advance. Developing user interfaces which provide control options for lots of devices with lots of different func- Keywords tions would be a tough task by itself. In addition with the BCI, smart environments, graphical user interfaces dynamics of the underlying device ensemble it quickly seems to be insolvable. 1. INTRODUCTION & MOTIVATION One approach to provide a user interface for a dynamic de- Brain Computer Interfaces (BCI) are ongoing research since vice ensemble would be the synthesis of a dynamic GUI from the 1970s [10], employing invasive technologies as well as formal device descriptions. Various approaches, for example non-invasive approaches such as EEG. Key potential of BCI built upon UPnP [4] or Jini [1], make it possible to gener- is the possibility of man-machine interaction without requir- ate GUIs, even for the control of a dynamic device ensemble. ing motor activities: Hands free, no gestures, no speech, no Unfortunately, users’ experience shows difficulties with these pointing and clicking. Recently, low-cost devices have be- approaches. come available at market targeting the gaming scene, claim- ing, at a very competitive price, to provide the capability of Our approach presented in this work relies on the following cerebral control for at least a limited set of interactions. working hypothesis: What users of a smart environment are interested in is not the individual device, but their effect Our experience so far shows that with those simple BCI de- on the environment. One simple example: When a user vices, interaction is kept within tight bounds due to the lim- switches a lamp on, he actually just wants to increase the its of this communication channel: A merely small character lightness of the room. In this way, all the lamps of this room set is available at a low frequency, which leads to a severely are able to increase the lightness and would therefore be limited data rate. Applications based on low-cost BCIs thus redundant with respect to their effects on the environment. With this article, we describe first experiments to control a smart environment using the neural impulse actuator (NIA), a low-cost brain non-invasive computer interface. We use semantic models of the environment and the devices. We model the devices with respect to their specific influence onto the environment. We present a principle approach for the synthesis of graphical user interfaces in order to reduce the number of necessary interactions. Copyright is held by the author/owner(s) SEMAIS'11, Feb 13 2011, Palo Alto, CA, USA 2. PRELIMINARIES After presenting the neural impulse actuator, we briefly dis- cuss our lab used for the experiments. Furthermore, we give a short introduction to STRIPS: A formalism to describe preconditions and effects of operations. 2.1 The Neural Impulse Actuator In 2007, OCZ Technology Group Inc. [5] introduced the Neural Impulse Actuator (NIA): A simple BCI controller, basically a headband, equipped with three electrodes cap- turing electrical potentials from the forehead. Those poten- tials include electromyogram (potentials arising from muscle control), electroencephalogram (signals from the nerves in the brain) and electrooculogram (signals coming up during eye movement). A special controller is used to connect the sensors to the computer. The NIA registers itself as a USB human interface device, which basically permits it to act like Figure 2: Our SmartLab. any other input device, e.g. a keyboard or mouse. Figure 1 shows a photo of the NIA controller. room for lectures, presentations, and meetings, but also as After calibrating the NIA, it is supposed to be usable as a an experimental setup for user studies. virtual joystick and to switch events, which can be triggered by different electric potentials or muscle movements. In our Our lab is equipped with a couple of sensors, needed to experiments, we found the following actions easy and stable observe state changes in the room. There are e.g. sen- to recognise: sors capable of detecting whether the windows are closed or opened, measuring the current temperature, or detecting • eye movement in general, persons that enter or leave the room, and estimating their • heavy muscle movement on the forehead, or moving number and current positions [3]. the jaw, • light muscle movement on the forehead, On the other hand there is a number of remotely control- • heavy thinking, and lable devices required in typical meeting rooms: Dimmable • relaxing, or closing the eyes. lamps as well as movable projection screens and sun shades, controllable via EIB [2], a computer video and audio matrix Here, we want to use the NIA to control our lab environ- switcher to connect brought-in devices with the installed ment, even while working on other subjects. Hence, inputs projectors and audio equipment, just to mention the most triggered by heavy thinking and relaxing are not suitable important. Those devices are actuators in essence, but can signals for a smart environment controller. However, paral- also be seen as specific sensors, in each case providing access lel performance to compose more complex signals does not to their respective status. seem to be helpful, as we want to provide an easy-to-use interface. Therefore, we have at most three distinguishable Our lab features a powerful middleware (as for instance de- signals at hand: (i) eye movement, together with (ii) heavy scribed in [6]) which on the one hand allows for control of and (iii) light forehead muscle movement. To complicate all existing hardware using simple commands. On the other, things further, users of the NIA can perform those signals besides triggering device actions, our middleware enables ev- at a merely low frequency of about 10 per minute at most, ery device to make its specific properties accessible to other leading to a comparatively low data rate. components in the system. 2.2 Our SmartLab For our experiments we utilised our SmartLab: An instru- 2.3 STRIPS mented meeting room (cf. fig. 2) equipped with a number To describe the capabilities of devices and their possible ac- of remotely controllable devices. It is frequently used as a tions, we suggest to use STRIPS-operators as illustrated in [9]. Those operators formalise (i) preconditions that need to hold to make the execution of the respective operation possi- ble and (ii) effects that specify the world state changes pro- voked by the execution of the respective operation. STRIPS- operators have successfully been used in the context of smart environments before [8]. Due to their associated declarative semantics, they are well suited for an automatic interpretation and hence for the con- struction of a controller. We annotate every operator with the middleware command which needs to be executed to perform the operation. Figure 3 shows two simple operators Figure 1: The NIA describing how to switch a lamp l on and off. Copyright is held by the author/owner(s) SEMAIS'11, Feb 13 2011, Palo Alto, CA, USA 3. OUR APPROACH Middleware Semantic As depicted above our user interface needs to cope with a dy- Background Model Models namic ensemble of heterogenous devices, which is the reason we do not have the option to hard-code a certain controller Effect Model Controller for a fixed environment. Therefore, we need to consider ways Device Device Device Device Actions and means of synthesising a controller from an abstract de- Model Model Model Model scription of the environment and the devices. According to figure 4, we consider three different modelling GUI GUI GUI tiers: Top left in the sketch is the layer of the background Devices Views model. It specifies existing parameters of the environment and how they can be modified. To provide an example we have formalised two specific parameters and their respective Figure 4: Integration outline of the proposed con- operations in figure 5: There exists some parameter that cor- troller within the smart environment system. responds to the lightness of our room, and it can be modified by two operations named increase and decrease. The param- eter temperature is handled likewise. This explicit kind of The information aggregation mechanisms of our middleware formal specification of environmental parameters is needed enable applications to gather these effect models analogously later on to describe the effects on the environment caused to the previously mentioned properties as well as type and by triggering device actions. status information of devices. Therefore a GUI application – called controller in figure 4 – is now able to provide itself As illustrated in section 2.2 all of our devices are made ac- with a list of all devices together with their descriptions, cessible through our middleware, that way providing on the i.e. the type (lamp, sun shades, ...) of the device and all one hand the entire set of current properties to other com- its available actions, including their particular preconditions ponents and on the other an easy-to-use interface for trig- and effects. After collecting the operators, it can generate gering device actions. In figure 4 this device representation diverse GUIs views. Below we present some first experimen- layer is called device model and establishes a certain level tal views that shall demonstrate the descriptive power of our of abstraction from the plain hardware, where every device proposed approach. can exist without requiring local knowledge on the existence of other devices, the middleware or even the environment. As mentioned above, there are basically three different ac- Besides properties and actions this layer holds additional in- tions (keystrokes) a human can reliably perform. Based on formation on the device such as the device type, its name, the formal description of our devices we implemented lab or whether the device is currently available in the system. controllers tailored for a limited communication between hu- man and computer to evaluate our approach. We designed The third modelling tier, the effect model, now etablishes them following the Mac Finder’s Column View. Two keys the relation between the raw device descriptions from the are used to move the focus up and down a list, the third to device model and the environmental parameters from the select the item. background model. This is done by modelling device ac- tions with respect to their effect on the environment. Every Our very first prototype contained three columns, of which action is annotated with a formal description of the influence the first contained a list of device types, the second all avail- of its execution on the specified environmental parameters. able devices of the type selected in the first column, and the As depicted in figure 3 for instance the execution of the third all the applicable functions of the device in the second turnOn method of a lamp has an effect on the environmen- column. This prototype does not involve any environmental tal parameter lightness in the form that the latter would knowledge yet. Due to its three-button-based design, this be increased. We suggest to use STRIPS as modelling for- controller interface would enable users who have to rely on malism to describe the semantic meaning of actions to the a NIA to take control of a complex and dynamic set of het- environment. erogeneous devices once these have been seen by the system through the middleware. But without further tweaking and tuning – which we elaborate on in section 4 – the menus Action(switchOn(l,r)) would be very large if they contain every possible action for Precond: Lamp(l) ∧ Room(r) ∧ In(r,l) ∧ Off(l) every possible device. Effect: ¬Off(l) ∧ On(l) ∧ Lightness.increase(r) Command: l.turnOn() Action(switchOff(l,r)) Parameter(Room, Lightness) Precond: Lamp(l) ∧ Room(r) ∧ In(r,l) ∧ On(l) Operation(Lightness, increase) Effect: ¬On(l) ∧ Off(l) ∧ Lightness.decrease(r) Operation(Lightness, decrease) Command: l.turnOff() Parameter(Room, Temperature) Operation(Temperature, increase) Figure 3: An annotated STRIPS-operator describ- Operation(Temperature, decrease) ing the switchOn and switchOff actions for a lamp. Every operator contains preconditions and effects of the action, and the command to be executed to per- Figure 5: Background knowledge: Two environmen- form the action. tal parameters and their respective operations. Copyright is held by the author/owner(s) SEMAIS'11, Feb 13 2011, Palo Alto, CA, USA 4. CONCLUSIONS AND DISCUSSION Our investigations leave us with mixed feelings. On the one hand, it is indeed possible to evoke a limited set of actions using the NIA controller. On the other hand, the concen- tration required from the user (and, indeed, the level of self control regarding the facial expression), in our opinion leaves ample room for optimisation of both signal acquisition at the sensory level and signal processing at the algorithmic level. Figure 6: A user interface synthesised by our second While developing the controller, we could reliably distin- controller prototype. guish three different inputs only. Of course the NIA itself provides a richer interface in form of analogue joysticks, but we found that those are hard to control while concentrating Our second prototype now extracts advantages from our for- on other tasks and they cannot be distinguished as reliable malisation: As an alternative to a static arrangement based as needed to control the environment. Nonetheless, a better on the device type, we can group our devices with respect recognition of crisp events would be desirable for the future. to their effects on the environment, as we did in our second controller, which is shown in figure 6. Here, the first column With this work we present a mechanism that deals with the has been replaced by a column containing controllable pa- issue of operating a complex and dynamic system through rameters as for example the lightness, or the temperature. such restricted channels. If we had to deal with an environ- Then, all those devices are listed in the second column which ment that would be static and consist of a fixed number of can actually influence the selected parameter. Finally, the well-known devices, we would be able to build a controller third column would again contain performable actions. being perfect in terms of the number of interactions. Be- cause we need an automatically synthesised controller, our But our formalisation of effects has further advantages: The approach uses STRIPS operators to apply semantic knowl- previously depicted controller still displays all the devices edge of the system’s devices and their possible actions aim- even if they have similar influence on the environment. This ing at design (and synthesis) of adequate user interfaces. leads to a long list of devices with each of them still provid- ing every possible action. But, with respect to their effects In the simple approaches presented above, the menus would they are kind of redundant and if we assume that a user is still grow very large with every new device entering the scene not interested in the device and its particular action itself if the menus still contain every possible action for every pos- but is interested in its effect, we can omit all these informa- sible device. Therefore, it is desirable to show the user not tion and simply provide control options of an environment. the whole menu. Instead, we should offer more abstract ac- For this purpose, we can simply use our existing model of tions. Furthermore, we would like those abstract actions to environmental parameters. Our third controller prototype be context dependent and automatically generated. working this way is depicted in figure 7. In the first column it displays the environmental parameters of the background The focus of this work was not to develop the perfect-working model, which can then be adjusted by selecting one of the graphical user interface for controlling tasks using the NIA items in the second column. controller. Our goal was to investigate several approaches to reduce the number of interactions a user has to perform to Within our research project MAike, all devices send their have a function executed. Therefore, the synthesised GUIs descriptions to a central look-up, realised as a tuple space will never be the best ones one would find through manual [6]. Furthermore, the NIA controller integrates itself as a design. This work considers ways and means of involving new modality for user interaction among other existing ones formal descriptions of environmental knowledge into auto- (speech interaction, intention analysis, ...). So far, we inte- matic GUI development. We have not yet evaluated which grated four different device types, with at most eight devices are best suited for the present application case and different and five actions, and initial experiments showed that those approaches from the ones depicted in this work are imagin- devices are easily controllable using this simple controller able. together with the NIA. As mentioned above, the menu structure used in our pro- totype, does probably not allow to control larger environ- ments. In the following section 5, we have discussed a num- ber of methods to create more suitable menus and higher level actions. Those have to be implemented and tested with respect to their usability. Nevertheless, our solution seems to be a principle approach for the synthesis of graphical user interfaces in general, which could bring a significant benefit not only for motorically challenged users, or those who require hands-free interac- tion in situations, where speech control is not an option. Figure 7: Another user interface synthesised by our third controller prototype. Copyright is held by the author/owner(s) SEMAIS'11, Feb 13 2011, Palo Alto, CA, USA 5. A ROADMAP FOR THE FUTURE The controller described above is applicable for small well- defined scenarios, but certainly not to control a complex infrastructure with hundreds of device-actions. This is due to the realised menu-based interaction. Considering a usual living room, it is not hard to think of many different de- vices which should be controllable. Just think of all lamps, shades, multi-media devices, etc. Therefore, alternatives are currently under investigation. We try to learn user prefer- ences online and group devices dynamically. For example, we could group devices together as one virtual device if they have been used in one go a couple of times. Another group- ing approach could be a Huffman-like encoding which would put the more important devices above others in the list. As mentioned above lots of user evaluation needs to be done in order to find more intuitive user interface design methods that make use of formal descriptions of the effects caused by device actions on the environment and of the environment it- Figure 8: The user interface of a potential controller. self. One approach could be another control metaphor which is especially designed for brain computer interfaces: A rect- angle that periodically rolls over a list of items on the screen Acknowledgements and each time marks the currently covered item. The user Gernot Ruscher’s work in the MAike project as well as now simply thinks of the particular item he wants to choose Frank Krüger’s work in the MAxima project are both sup- and a signal from the brain indicates the moment when the ported by Wirtschaftsministerium M-V at expense of EFRE bar covers this item. This selection mechanism can not only and ESF. be utilised with lists of items, but also like shown in figure 8: First a horizontal bar goes the up-down direction and stops 6. REFERENCES when the height of the selected item is reached. Second, [1] http://www.jini.org, OCT 2009. the vertical bar starts moving and is used to indicate the [2] http://www.knx.org/, OCT 2009. particular device in this line. [3] http://www.ubisense.de, JUN 2009. The area of automatic intention analysis, that is the detec- [4] http://www.upnp.org/, DEC 2010. tion of the user’s goals based on his current activities, opens [5] OCZ Technology Group, Inc. Retrieved from further possibilities. Given the user’s current goals, we could http://www.ocztechnology.com, November 2010. automatically re-arrange the menus to provide faster access [6] S. Bader, G. Ruscher, and T. Kirste. A middleware for to device which are most likely to be used in the current sit- rapid prototyping smart environments. In Proceedings uation. Going one step further, high-level actions could also of the 12th ACM international conference adjunct be added to the controller. A room detecting the start of papers on Ubiquitous computing, pages 355–356, a lecture could offer the compound action like “put my pre- Copenhagen, Denmark, SEP 2010. ACM. sentation on the projector”, which consists of several smaller [7] D. Cook and S. Das. Smart Environments. Wiley, actions. For this purpose some additional strategy synthesis 2005. component would be needed. [8] C. Reisse and T. Kirste. A distributed action selection mechanism for device cooperation in smart We have not yet fully covered the possibility of the existence environments. In Proceedings of the 4th International of multiple environments, containing different devices, but Conference on Intelligent Environments, Seattle, USA, controllable through the same user interface. This could be 2008. the case in a smart home consisting of several smart rooms, [9] S. Russell and P. Norvig. Artificial Intelligence: A where users possibly would like to have functions performed Modern Approach. Prentice-Hall, 2nd edition edition, in other rooms remotely or trigger actions in different rooms 2003. simultanously. One straightforward approach would be an [10] J. Vidal. Toward direct brain–computer additional column inserted before the first one, that specifies communication. In Annual Review of Biophysics and the particular room. After this column would be the ones Bioengineering, pages 157–180, 2 1972. depicted in the previous sections. So far we have been relying on the feedback on an existing computer screen. This is possible for users that can carry a small display with them. E.g., for people depending on a wheel chair, this monitor can be integrated into it. Alter- natives should be investigated, e.g. sounds or small displays right next to the devices which are controllable. This would enable the user to move freely around in the environment without watching a computer screen for every action. Copyright is held by the author/owner(s) SEMAIS'11, Feb 13 2011, Palo Alto, CA, USA