Simulation Framework in Second Life with Evaluation Functionality for Sensor-based Systems Boris Brandherm1 , Sebastian Ullrich1,2 , Helmut Prendinger1 1 National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda-ku, 101-8430, Tokyo, Japan 2 Virtual Reality Group, RWTH Aachen University, 52074 Aachen, Germany {boris, helmut}@nii.ac.jp, s.ullrich@ieee.org ABSTRACT unforeseen problems, e. g. , regarding optimization, be- This paper describes a simulation framework for sensor- fore the system is actually installed. While simulators based systems utilizing “Second Life”, a popular vir- for specific types of sensors are available, it is still diffi- tual three-dimensional multi-user online world. With cult to evaluate a heterogeneous complex system. Hence this platform, the components of a sensor-based system the visualization of all the simulated sensor data is de- can be mapped to (or, represented by) virtual counter- sirable and an intuitive interaction capability to change parts. The intuitive user interface of Second Life and its the parameters and spatial position of the devices would comprehensive visualization support evaluation tasks of be helpful to optimize the system. ubiquitous computing applications. Developers can di- rectly control and manipulate virtual counterparts of Because of these requirements, we propose a three-di- real devices. In this way, different settings of a sensor- mensional (3D) virtual environment for the simulation, based system can be tested. The main contribution of testing and evaluation of sensor-based systems. Besides our work consists of a bi-directional interface between extensive research in virtual reality, nowadays even game sensor-based systems and Second Life. engines and multi-user online worlds provide convinc- ing 3D environments. So, instead of creating a new 3D-engine, we decided to use the 3D environment of ACM Classification Keywords Second Life [14]. In Second Life (SL), 3D content in- I.6.7 Simulation and Modeling: Simulation Support Sys- cluding buildings and props can be created easily, and tems—Environments anyone can interact with the environment in the form of an ‘avatar’ (a human-controlled graphical represen- General Terms tation of the user). Design, Experimentation The rest of the paper is structured as follows. We start Author Keywords with a brief review of related work. To motivate the Simulation, testbed, sensor-based systems, evaluation, merit of our simulation framework, we then report on Second Life the experience with an existing sensor-based system. After that, we explain two different approaches to simu- lation. First, we describe a rapid-prototyping approach INTRODUCTION for SL, and discuss its benefits and shortcomings. Sec- Ubiquitous computing environments and sensor-based ond, we describe our own simulator framework [3, 16], systems are highly active fields of research. Many ex- and explain its functionality with respect to evaluation citing new devices are being developed and the amount of sensor-based systems. We give an example where of powerful and versatile sensors is rapidly increasing our system is used to evaluate the performance of an through the advances in embedded systems and tech- indoor-positioning system. The paper concludes with a nological evolution. discussion and summary. The implementation, testing and evaluation of new ubiq- uitous systems in a real environment are laborious tasks. RELATED WORK Significant time and effort has to be spent on designing Currently, the development, testing and evaluation of and testing prototypes and simulators in order to avoid new systems is realized in different ways, ranging from real-world testing and evaluation [4] and miniature mock- ups for prototyping, to software-based simulators [1, 11]. Recent testbeds are MoteLab [17] for wireless sen- sors, eHomeSimulator [1] for smart environments, and ubiBuilding Simulator [11] for large scale context-aware Copyright is held by the authors. UbiComp ’08 Workshop W2 – systems. While these software-based testbeds are far Ubiquitous Systems Evaluation (USE ’08) September 21st, more practical than physical models, all of them are 2008 This position paper is not an official publi- limited to testing in two-dimensional space. cation of UbiComp ’08. S ig nal St rengt h:  Ubireal [10] is a 3D simulator for smart environments. V irt ual ID T Ichiban imbogami (36% ) B Yet, its focus lies on systematic testing to verify rules S ig nal St rengt h:    RF  ags Niban imbogami (53% ) B Ichiban imbogami (60% ) B and user-specified programming between different smart Niban imbogami (48% ) B Pos it ioning S phere Pos it ioning S phere  devices and sensors. There is no support for interac-  Ichiban imbogami Niban imbogami B B tive exploration and testing. Another simulator, called TATUS [12], is based on the Half-Life game engine. The system focuses on human interaction with ubiquitous l per Av Dev elo t r A a a Niban imbogami computing environments rather than the setup of such B environments. V is it or Av at ar  Ichiban imbogami B While all these approaches demonstrate promising fea- tures, they either (1) do not work as testbeds for simula- tion, or (2) provide specialized (non-generic) solutions, or (3) lack 3D interaction with the simulated virtual environment. V irt ual PDA  Let us now take a closer look at solutions that feed real- world data from sensors into SL and/or SL data into the real world (e. g. to control a device). In [8] data from Figure 1. Example of the simulated positioning system in Second Life with a Visitor Avatar experiencing the a specialized power-plug based sensor network are fed system and a Developer Avatar who is interactively ad- into the virtual world by means of a (latency restricted) justing the properties of a virtual RFID tag. SL script-based implementation of a XML-RPC proto- col. The data is used for visualization but there is no support for interaction with it. [9] is a work where sen- The system described in [2] features different kinds of sors embedded in commercial mobile phones are used sensors and emitters: infrared beacons and radio-fre- to infer real-world activities (whether the user is idle, quency identification (RFID) tags. The accuracy of the walking or running), that in turn are mapped to visual positioning system depends on the good placement of metaphors in the virtual environment. [5] reports about these sensors and emitters in the environment. The a real-world control panel that can both control objects user carries a mobile device, which is equipped with the in the virtual world of SL and in turn be controlled by corresponding sensors. On this device the position of them. Changes to the knobs or pushbuttons in the real the user is computed based on the received sender data world are translated to their virtual counterparts in SL, and afterwards displayed on a 2D map on the device. and pushing the virtual buttons controls the LEDs on the real world control panel. When the system was installed in the environment, ini- tial trials to improve accuracy by adding additional Although these approaches demonstrate interesting re- emitters failed. Furthermore, in boundary regions of sults, (1) they are not generic, (2) they don’t provide a the emitter field some unexplainable artifacts appeared direct bi-directional feedback loop (e. g. if we control a in the position calculation. These open issues could be real device via the virtual counterpart from inside SL resolved easily in our SL based simulation system, as and the status of the real device changes this change is described in the following sections. immediately fed into and represented in SL again and vice versa.) and (3) they don’t take into account the context of the devices in the environment (e. g. the po- RAPID PROTOTYPING IN SECOND LIFE sition and the orientation of a sensor can be crucial for Our first approach was to investigate the simulation ca- the system behavior like a indoor positioning system). pabilities of SL. Therefore we used the official API of SL that is called “Linden Scripting Language” (LSL). This Because of the lack of interactive and generic solutions, programming language allows one to assign scripts to we have created a bi-directional simulation framework in-world objects. With over 300 library functions and for SL [3]. This system has been extended from a scripted different data and message types, scripts can control simulation within SL to a more flexible interface and the behavior of virtual objects and communicate with will be described by an example application. other objects and avatars (users of SL). Limitations of LSL include time delays for movement of objects (0.2 sec) and memory constraints for scripts (16 KB). These EXAMPLE APPLICATION constraints have a high impact on the achievable simu- Positioning systems are often used in ubiquitous com- lation accuracy, response times, and achievable simula- puting environments. As a simple motivating example tion complexity within SL. for our simulation framework, we chose an existing in- door positioning system [2]. Sensor placement for such To simulate the previously discussed positioning system a system is a non-trivial task as it depends on several we created virtual objects in SL that represent RFID factors such as the infrastructure, the amount and type tags and can be positioned within the virtual environ- of available sensors, and interferences. ment interactively (see Fig. 1). Visitors who wish to be Virtual Avatars irtual Sensors irtual O Emitter V &  ic EnvVironment Estimated  Dev es Second Lif e Position  (3D irtual orld) Virtual Message roxy V W Emitter P Signal Strength Signal Strength Twin orld c Mediator S electedDat a of  ͲW S econd fe and Mediator S ens orͲ BasLiedS yste m Virtual PDA with RFID Sensor Interface N Sensor Based Figure 2. Conceptual overview of the example applica- Ͳ  System tion: the virtual PDA reads the signal strengths of the Software Sensors & Simulators Systems  ic   virtual emitters and calculates an estimated position. Dev es positioned by the system in SL have their avatar take a copy of a PDA object. The virtual PDA embeds a LSL Figure 3. Architecture that embeds existing sensor- script that is the core of the simulated positioning sys- based systems (2) with Second Life (3) through the Twin-World Mediator (1). tem and performs the calculation of the position. The PDA communicates with the virtual emitters and takes the signal strength to estimate the position, just as the tem, and (3) the SL 3D environment. The main task of real system (see Fig. 2). In the real world application, the Simulation Framework is to provide an interface to the result is shown on a 2D map on the PDA. In SL the existing systems, sensors and simulators and to medi- calculated position is visualized in three dimensions, as ate the exchange of data with SL for visualization and a floating sphere in the virtual environment (see Fig. 1). interaction. If a user logs in to SL as a “visitor avatar”, he or she can experience and interactively test and evaluate the In the following we first describe the architecture in sensor-based system. As a “developer avatar” the user more detail. To illustrate the usage of the system, we can additionally adjust the virtual sensors and devices show how we embedded our example application. in the virtual environment. Components of the Architecture The artifacts in the example application could be expe- rienced in the simulation in SL. In both cases the signal The Twin-World Mediator (1) consists of the Mediator, range of the RFID tags has been too high compared to the Message Proxy, the Interface and a database. The the distance of each RFID tag to another. This expla- key component of the simulation framework is the Me- nation was found after interactive testing (repositioning diator which handles the data exchange between Mes- the sensors and adjusting their sender range). Conse- sage Proxy, Interface and the database of the simulation quently, the best results were achieved with just a small framework. It ensures the data exchange of the com- overlapping of the RFID signals. ponents of the sensor-based system with their virtual counterparts in SL. Benefits of this approach include the ability of rapid prototyping of coarse simulations with reduced com- A sensor-based system(2) typically consists of software plexity and no need for additional software or servers: systems and sensors & devices. Simulators are used the scripts in SL reside within the objects. Shortcom- for unavailable sensors and devices. The components ings of this approach are the aforementioned limitations of the sensor-based system register as listeners through of the scripting language. Additionally, it can be very the interface for the desired data. tedious or even impossible to map the functionality of The Message Proxy connects to Second Life(3) and gath- an existing system into LSL. A better solution is to ers data about the components in SL and sends them to provide an interface to reuse existing systems, this ap- proach is described in the following section. the Mediator. We are especially interested in the state of the avatars, the virtual sensors & devices, and the virtual environment because they represent the virtual SIMULATION FRAMEWORK WITH TWIN-WORLD MEDIA- counterparts of the sensor-based system which is to be TOR tested based on their data. Obstacles in the virtual en- Our architecture (Fig. 3) consists of three components: vironment such as walls and other objects can influence (1) the Twin-World Mediator, (2) a sensor-based sys- the simulation. virtual sensors and devices). In those objects we have embedded specific LSL scripts to send the updates to the message proxy (see below). Update Scripts for Virtual Objects The example LSL script (shown in Fig. 4) continuously sends updates about position changes and/or rotation with a predefined refresh rate (here 20 times per sec- ond). During each refresh cycle first the rotation and the position of the object are determined by the func- tions llGetRot and llGetPos. The new values are then compared with the previous values in order to deter- mine whether the rotation and/or the position of the object has changed. Only the changes are transmitted (function llSay) to keep the traffic low. It depends on the object which refresh rate is necessary and which ob- ject’s changes (e. g. rotation, position, size or color) has to be transmitted. For example, for a RFID tag only the position is important and a refresh rate of 4 times per second is sufficient. The scripts are rather easy to understand and therefore quite easy to adapt for their specific task. One of the most important differences to previous and related work is that the message proxy uses commu- nication methods of both the scripting language LSL, and the libsecondlife API [7]. (Libsecondlife is an un- official API that interfaces SL as a client and enables access to data of the virtual environment.) Thus the performance can be improved, technical limitations like Figure 4. Example of an update script for virtual objects, which informs about changes of position and/or rotation the time limitations of the XML-RPC method of the and is implemented in the Linden Scripting Language. LSL are circumvented, and SL can be interfaced more effectively. System Setup Simulation Update In the following we explain how to integrate an existing After the system has been initialized, the Twin-World system to our architecture. As an initial step the sim- Mediator synchronizes the update loops of the sensor- ulation framework has to be prepared for the specific based system and of SL to allow for development and simulation task. The developer has to register the ex- testing with 3D interaction and visualization. As said isting sensor-based system with its hard- and software, before, the static data is collected only once (to save in order to inform the simulation framework which data bandwidth) and sent to the database of the Twin-World (from SL or other components) are required for the sim- Mediator. ulation task. This is done by registration as listener through the interface for the desired data. In the second step the message proxy continuously col- lects (in an infinite loop) all the subscribed dynamic In our example the sensor-based system consists of the data and sends it to the mediator. The Twin-World PDA positioning software, and a simulator for the be- Mediator notifies data changes to the listeners of the havior of the RFID tags. The required data from SL components of the sensor-based system. Then the com- comprises the coordinates and the orientation of the ponents process this data and send the results back to emitters, sensors, and avatars. the Twin-World Mediator. Data which is meant to be visualized in SL is sent via the message proxy to SL. The Twin-World Mediator configures the message proxy according to the registered listeners. Then, the mes- In the case of our example (the indoor positioning sys- sage proxy connects to SL and continuously listens (in tem), the coordinates of the virtual RFID tags in SL are SL) for the requested data and communicates it to the sent to the RFID-simulator. The coordinates and ori- database of the Twin-World Mediator. Some objects entations of avatars are also sent to the simulator. The are static and their data will be gathered only in the simulator computes the results and sends them back to initial step (e. g. parts of the virtual environment like the Twin-World Mediator, which in turn delivers the walls), whereas other objects are potentially moving; so data to the positioning system. In our case the PDA they have to report about their changes (e. g., avatars, positioning system has been slightly modified in order Virtual RFID ag bodies) then our approach would facilitate the evalua-  T tion of a sensor-based system and it even would enable to compare two different sensor-based systems with the same data under different conditions. Positioning Sphere I In Fig. 5, the aforementioned indoor positioning sys-  Sensor Rang ge:    5m tem is tested with two different sensor ranges of the Deviation: .44 41  0 0 m RFID sensor. RFID tags are placed on a uniform grid with 6 meter spacing. The Positioning Sphere I indi- Position o ingSphereII cates the estimated position for a RFID sensor with Sensor o Range:6.5m a sensor range of 5 meters and shows a deviation of Deviation: . 4 0.4041 meters. Positioning Sphere II, which indicates 0 02 6m the estimated position for a RFID sensor with a sensor Developer o vatar Niban Bi bAoga i range of 6.5 meters, shows a better accuracy (deviation m m 0.0246 meters) than Positioning Sphere I. Thus the is- sues that have been encountered with the real system V rtual PD Vi (as mentioned before in Example Application), could be A Withh RFID Sensor evaluated in a virtual setup and lead to the conclusion   that the best results are achieved with a small over- lapping of the RFID signals. With the current RFID Figure 5. Basic evaluation of an indoor positioning sys- tag placement in the environment Positioning Sphere II tem by comparing different sensor ranges and measur- ing the accuracy/deviation (i) sensor range 5 meters and matches this condition better than Positioning Sphere I. (ii) sensor range 6.5 meters. So far, we compared the system behavior of the same indoor positioning system with different sensor ranges to be able (i) to receive the simulated sensor data of the and placements of the RFID tags in the environment. virtual RFID tags, and (ii) to provide the estimated user But obviously, modified versions of the current underly- position for the Twin-World Mediator. ing algorithm or completely distinct algorithms could be compared with each other (using their best sensor range The positioning system sends its results (the estimated and RFID tag placement in the environment). For the position of the avatar) to the Twin-World Mediator. evaluation of systems other metrics than the deviation The message proxy takes this data and visualizes it in could be desirable. The values of these parameters can SL. We use a sphere to indicate the estimated position of be fed into the system via the Twin-World Mediator and the avatar in SL. In this way, the developer can inspect displayed on the top of the positioning sphere as well the results of the positioning system in SL (see Fig. 1). (by a short LSL script). In addition to text-display, color-coding and resizing can also be realized with LSL. In the following section we show how this can be ex- tended for the evaluation of a sensor-based system. DISCUSSION AND CONCLUSIONS The paper proposes a novel simulation framework based EXAMPLE EVALUATION OF SENSOR-BASED SYSTEM on the 3D virtual environment of SL, which can be used [4] reports about the evaluation of the accuracy of a as an evaluation testbed for sensor-based systems. A installed tag-based location system (Ubisense [15]) and core feature of our approach is the bi-directional in- the influence of the human body on it based on the teraction with our Twin-World Mediator. Events from fact that many tag-based systems use communication the real world are reflected to the virtual world and vice frequencies that cannot pass easily through the human versa. body. To measure the accuracy of the system, they use a combination of definitions suggested by [6]. With the example of a positioning system we have illus- trated how our simulation framework can be used and As a simple example how our approach could be used how the virtual environment can be utilized for evalua- for the evaluation of a sensor-based system we compare tion and optimization purposes. the accuracy with different system settings. To mea- sure the accuracy of the estimated position the devia- Furthermore, the architecture is flexible and extensible tion is computed by calculating the distance between and thus ensures that new sensor types, such as tem- the real position of the avatar and the estimated posi- perature sensors, accelerometers, or light sensors, can tion. The measured deviation of the estimated position be included. Spatial characteristics of devices can be is displayed on the top of the positioning sphere (see modeled and visualized to easily identify problems and Fig. 5). Obviously, the evaluation of a sensor-based interferences, e. g. when walls or other objects in the system with our approach heavily depends on the qual- virtual environment influence the characteristics of the ity of the used simulators. If the simulator takes into devices. Sophisticated simulators can also be adapted account obstacles for the sensors (like walls or human and connected to the system. Metrics for evaluation purposes can be updated via the Twin-World Mediator 8. J. Lifton, M. Feldmeier, Y. Ono, C. Lewis, and and visualized in SL. J. A. Paradiso. A Platform for Ubiquitous Sensor Deployment in Occupational and Domestic The 3D interaction capability of SL combined with the Environments. In Proceedings of the 6th embedded simulators offers many advantages and op- International Symposium on Information portunities. Virtual sensors and devices can be moved Processing in Sensor Networks (IPSN), pages intuitively by ‘direct’ (avatar-mediated) manipulation 119–127, 2007. (Fig. 1) and their parameters can also be changed easily by editing the object properties through the user inter- 9. M. Musolesi, E. Miluzzo, N. D. Lane, S. B. face of SL. Most importantly, these changes can also be Eisenman, T. Choudhury, and A. T. Campbell. fed back in real-time via the Twin-World Mediator and The second life of a sensor: Integrating real-world affect the connected system. experience in virtual worlds using mobile phones. In Proceedings of 5th Workshop on Embedded In our future work, we plan to implement a user-friendly Networked Sensors (HotEmNets 2008). ACM, interface and toolbox for developers of ubiquitous com- 2008. puting systems. To reach a broader audience and en- 10. H. Nishikawa, S. Yamamoto, M. Tamai, sure higher flexibility, the Twin-World Mediator will be K. Nishigaki, T. Kitani, N. Shibata, K. Yasumoto, adapted to the emerging and open-source virtual worlds and M. Ito. UbiREAL: Realistic smartspace system OpenSimulator [13]. Furthermore, we intend simulator for systematic testing. In Proceedings of to use the simulation framework for running system- the 8th International Conference on Ubiquitous atic experiments of sensor-based systems. Specifically, Computing (UbiComp2006), Springer LNCS 4206, computer-controlled agents, i.e. SL “bots”, will popu- pages 459–476, 2006. late the environment, and the behavior of the sensor- based system will be evaluated in the multi-agent set- 11. Y. Oh, A. Schmidt, and W. Woo. Designing, ting. developing, and evaluating context-aware systems. In 2007 International Conference on Multimedia REFERENCES and Ubiquitous Engineering (MUE’07), pages 1. I. Armac and D. Retkowitz. Simulation of smart 1158–1163. IEEE Computer Society, 2007. environments. In IEEE International Conference on Pervasive Services (ICPS’07), pages 257–266. 12. E. O’Neill, M. Klepal, D. Lewis, T. O’Donnell, IEEE Computer Society Press, 2007. D. O’Sullivan, and D. Pesch. A testbed for evaluating human interaction with ubiquitous 2. B. Brandherm and T. Schwartz. Geo referenced computing environments. In Proceedings of the 1st dynamic Bayesian networks for user positioning on International Conference on Testbeds and mobile systems. In Proceedings 1st International Research Infrastructures for the DEvelopment of Workshop on Location- and Context-Awareness NeTworks and COMmunities, pages 60–69. IEEE (LoCA’05), pages 223–234. Springer LNCS 3479, Computer Society, 2005. 2005. 13. OpenSimulator. http://opensimulator.org. 3. B. Brandherm, S. Ullrich, and H. Prendinger. Simulation of sensor-based tracking in Second Life. 14. Second Life. http://secondlife.com. In Proceedings of 7th International Conference on 15. P. Steggles and S. Gschwind. Ubisense-a smart Autonomous Agents and Multiagent Systems space platform. Technical report, Ubisense, May (AAMAS’08), pages 1689–1690. ACM Press, 2008. 2005. 4. L. Coyle, J. Ye, E. Loureiro, S. Knox, S. Dobson, 16. S. Ullrich, B. Brandherm, and H. Prendinger. and P. Nixon. A proposed approach to evaluate Simulation Framework with Testbed for the accuracy of tag-based location systems. In Sensor-based Systems in Second Life. In USE 07: Workshop on Ubiquitous Systems Proceedings of 10th International Conference on Evaluation. UbiComp Workshop Proceedings, Ubiquitous Computing (UbiComp 2008) Demo pages 292–296, 2007. Session. ACM, 2008. 5. A. Fundinger. Real Life Control Panel for Second 17. G. Werner-Allen, P. Swieskowski, and M. Welsh. Life. MoteLab: a wireless sensor network testbed. In http://channel3b.wordpress.com/2007/01/24/real- IPSN ’05: Proceedings of the 4th international life-control-panel-for-second-life/, January symposium on Information processing in sensor 2007. networks, pages 68–73. IEEE Press, 2005. 6. J. Hightower and G. Borriello. Location systems for ubiquitous computing. Computer, 34(8):57–66, 2001. 7. libsecondlife. http://www.libsecondlife.org.