Unit: Modular Development of Distributed Interaction Techniques for Highly Interactive User Interfaces Alex Olwal1,2 Steven Feiner1 1 2 Department of Computer Science Department of Numerical Analysis and Computer Science Columbia University Royal Institute of Technology New York, USA Stockholm, Sweden {aolwal,feiner}@cs.columbia.edu Figure 1. The Unit User Interface, Figure 2. The flexible pointer select- Figure 3. The 3DOF mouse, a simple displaying the Unit Graph that ing an obscured object, without visu- composite input device consisting of specifies the interaction for the ally interfering with the occluding two wireless optical mice. A Unit three–degree-of-freedom mouse. object. graph extracts the rotational accelera- tion in the plane of the surface. ABSTRACT 1 INTRODUCTION The Unit framework uses a dataflow programming lan- Despite tremendous improvements in computer systems guage to describe interaction techniques for highly interac- over the past several decades, designing and developing tive environments, such as augmented, mixed, and virtual interaction techniques is still a difficult task, especially for reality. Unit places interaction techniques in an abstraction highly interactive immersive 3D environments, such as layer between the input devices and the application, which Augmented Reality (AR), Mixed Reality (MR) and Virtual allows the application developer to separate application Reality (VR). Interaction in immersive environments in- functionality from interaction techniques and behavior. volves many different types of user input and many devices with which that input is provided, such as position and ori- Unit's modular approach leads to the design of reusable entation trackers, voice input, and haptic devices, in addi- application-independent interaction control components, tion to conventional mice, trackballs, touch screens, and portions of which can be distributed across different ma- keyboards. While there is an increasing number of well- chines. Unit makes it possible at run time to experiment known metaphors for immersive interaction, such as “the with interaction technique behavior, as well as to switch virtual hand” [3], “ray pointer” [3], and “flashlight pointer” among different input device configurations. We provide [12], there is still much variation in how these metaphors both a visual interface and a programming API for the are implemented. specification of the dataflow. To demonstrate how Unit works and to show the benefits to the interaction design Interaction techniques involve the mapping of data from process, we describe a few interaction techniques imple- input devices to application semantics. Therefore, we find mented using Unit. We also show how Unit’s distribution it particularly attractive to use a dataflow approach to the mechanism can offload CPU intensive operations, as well design of interaction techniques, in which data is processed as avoid costly special-purpose hardware in experimental through a customizable network. We introduce the Unit setups. framework [16], which allows users to specify 2D and 3D interaction techniques as dataflows and to modify them in KEYWORDS: interaction techniques, dataflow program- running programs. We use our framework to abstract inter- ming, visual programming, augmented reality, mixed real- action techniques from applications that use them, as well ity, virtual reality. as from input devices that control them. This allows users to flexibly configure and dynamically change interaction technique behavior, independent of both input devices and applications. Furthermore, in our framework, a dataflow Figure 4. a) Traditional input device interface for an interactive application. Input devices are mapped to functionality on the application level, making it hard to change devices and behavior. b) Unit places interaction techniques in an abstraction layer between input devices and the application. This layer can be changed for different combinations of interaction devices and behavior. c) Unit’s modularity also makes it possible to abstract interaction techniques from each other. can be easily distributed over multiple machines to create components are dataflow graphs, which are assembled into distributed interaction techniques, as well as distributed customized interaction techniques. In contrast to InTml, applications. As shown in Figure 1, we use a direct- whose developers emphasize their XML-based specifica- manipulation, visual-programming representation to spec- tion language, we have chosen to focus on an interactive ify the behavior of the dataflow in the Unit User Interface design process in which interaction techniques can be (Unit UI), which itself is implemented with Unit. modified at runtime. We have also been more concerned with design issues that are typical for highly interactive In the remainder of this paper, we first present related work distributed environments, such as AR/MR/VR, which dis- in Section 2, followed by brief introductions to the Unit tinguishes our framework from systems targeted for a user framework and the prototype Unit UI in Sections 3 and 4. on a single computer, such as ICON. While BodyElectric To explain how Unit can be used, we describe some exam- also addressed 3D virtual environments, its dataflow oper- ple interaction techniques that we have developed with it in ated on only a single machine (a Macintosh that controlled Sections 5, 6 and 7: a novel flexible pointer for selecting one or more SGI workstations). In contrast, Unit's dataflow objects in 3D environments (Figure 2) [15], an experimen- graph can be spread across multiple machines. tal setup for analyzing non-verbal features of speech [17], and a quickly prototyped rotationally sensitive mouse (Fig- Our approach allows the development of flexible interac- ure 3), created from a pair of conventional mice. We de- tion techniques, portions of which can be distributed, as scribe our implementation in Section 8, and present our well as replaced and remapped at runtime, and we provide conclusions and future work in Section 9. a user interface for visual dataflow programming of these behaviors, as well as a programming API. While the direct- 2 RELATED WORK manipulation creation of 3D widgets is an appealing ap- proach [25], we have chosen a dataflow language for its Data flow programming and directed-graph–based visual clear depiction of the mapping between device outputs and programming languages have been used together by a application inputs. While our work on Unit primarily ap- number of researchers to design 2D UIs and interaction plies this approach to the design of experimental 3D inter- techniques. Projects that take this approach have included action techniques, we also believe it is of significant rele- Smith’s InterCONS [19], Borning’s ThingLab [2], and Ma- vance to the design of new input devices (e.g., loney and Smith’s Morphic user interface framework [14] [7,8,20,13,18,9]). for Self [21]. A key issue here is the recognition that inter- action techniques essentially map the outputs (and inputs) 3 THE UNIT FRAMEWORK of interaction devices to the inputs (and outputs) of applica- tions; this observation has long been an underlying theme The Unit framework uses the concept of units to represent of work on building formal models of abstract graphical the nodes in the data flow. Each unit has any number of input devices, which in turn can be composed together in properties and any number of connections to properties in graphs to create hierarchical input devices [1,4]. other units, as shown in Figure 5. When a property is up- dated, a special method is called, which by default updates Most 3D interaction techniques can be conceptualized this all connected properties. Customized units typically over- way, and 3D toolkits that embody these techniques (e.g., ride this method with their own data processing, and when [10,22,23,24]) typically use abstract input devices. We done, typically redirect to the default method. These con- have chosen to abstract the interaction technique compo- nections can also be made over the network, allowing each nents in a similar fashion to BodyElectric [11], ICON [5] unit in the dataflow to be able to share its properties and and InTml [6] (See Figure 4.) As in these systems, our listen to property changes, anywhere on the network. two units are typically accomplished by reference, with the data pointer of the source property initially copied to the target property. When an update is made, the target unit is notified that the data has changed. We also provide the ability to make connections by value, in which each update replicates the data in the source unit; however, connections by reference are typically preferred for efficiency. In addi- tion, connections can be created through a chain of refer- ences to fields and methods in the property value, provided that the resulting source and target values are of the same data type. These references are more expensive since they require dereferencing, evaluation, and replication. Figure 6 shows a simple example in which two mice are Figure 5. Two units with two connected properties in the Unit UI. As the text property changes, the width field of the di- used to provide six–degree-of-freedom control of an appli- mension property of the label is updated. cation. These simple rules allow the design of flexible and custom- Distribution izable units that specify the desired behavior through their Connections can be transparently distributed over Java combination into Unit Graphs, much like electrical circuits. Remote Method Invocation (RMI), with the addition of the hostname in the specification of the connection. Following Units the peer-to-peer approach, each unit is individually distrib- As mentioned above, the key components of a unit are its uted through the RMI registry and directly accessible to properties and the ability to maintain and update connec- other units. tions to properties in other units. This distribution approach allows parts of Unit Graphs to Properties be distributed over an arbitrary number of applications run- Properties are attribute-value pairs, in which the value can ning on an arbitrary number of machines. A common prob- be a pointer to any Java Object. lem in immersive environments is the use of hardware, such as six–degree-of-freedom trackers, that have a perma- Connections nent physical connection to a single machine. Unit not only Connectivity is peer-to-peer, where only the involved units allows cross-platform access to platform-specific devices, are aware of their connections and are solely responsible but also simplifies the sharing of machine-specific devices. for administering their relationships. Connections between Our framework allows several Unit Graphs running on dif- Figure 6. A screenshot from the Unit UI, in which a dataflow graph is being specified for direct control of six degrees of free- dom through the use of two mice. (We use the scroll wheel to provide a third degree of freedom from each mouse.) ferent machines or within different programs on the same 4 THE UNIT USER INTERFACE machine, to communicate as a single graph, providing transparent access to data and flow control from anywhere. We created the Unit UI, shown in Figure 6, to allow users to design, manipulate, and visualize the dataflow in a Unit Core components Graph. Unit, the core class, which is the superclass of all units, The Unit UI lets the user add, modify and delete units, provides the general unit functionality, which most impor- properties and connections, as well as load and save Unit tantly is the handling of properties, connections, and distri- Graphs. Besides visually displaying the dataflow as di- bution. Two units are connected by specifying the source rected graphs with units, properties and connections, live unit, source property, target unit, and target property, and data propagation is visualized by highlighting a property an optional host name (for remote connections). for a short time after it is updated. The user can also switch to live views of values of interest. We have also implemented a set of core units that provide additional functionality. These include units for flow con- The Unit UI was implemented with units, using the same trol, such as switches and multiplexers, units for scalar and dataflow approach as the interaction techniques it manipu- vector operations, and units for I/O (multiple mice, key- lates—demonstrating that our framework does not restrict boards, six–degree-of-freedom sensors, speech recogni- itself to interaction technique specification, but also applies tion/synthesis). to traditional application logic. Unit UI is a 3D application, The units are arranged in a class hierarchy under the Unit and can thus coexist in the immersive environment, side- superclass. It is easy to implement new units, which typi- by-side with the interaction techniques whose behavior it controls. However, because of the limited field-of-view and cally involves overriding the changeProperty method that is called on every property update. Most core units have a low (800×600) resolution of our head-worn displays, we set of reserved property names that are used for their spe- find it more productive to interact with the Unit UI in 2.5D cific input and output properties. on high-resolution (1920×1200) 24” desktop displays. In the following sections, we describe our experience with Unit by presenting some of the experimental interaction techniques that we have developed with it. Figure 7. The dataflow for the tweaking code that manipulates the parameters in the flexible pointer through a mouse with a thumb-controlled joystick. Clicking the second mouse button alternates the parameter to be modified, while the movement of the mouse changes the value. The result is accessible in the unit named “parameters” at the bottom, which is remotely con- Figure 8. The Flexible Pointer Interaction Technique provides easier selection and clearer indicative pointing in collaborative environments in addition to the ability to select fully or partially obscured objects. 5 DESIGN OF AN INTERACTION TECHNIQUE FOR The problems that we address with the Unit framework are IMMERSIVE ENVIRONMENTS how users can control the pointer, and how we can interac- tively modify and tweak this mapping, at runtime and dur- We have implemented a novel interaction technique, called ing the design phase. the Flexible Pointer [15], which is an extension of existing ray-casting techniques for selection in immersive environ- Implementation ments. The flexible pointer allows the user to point around First, we have to decide on a representation for the geome- objects, with a curved arrow, for selection of fully or par- try of the flexible pointer. We choose a Quadratic Bézier tially obscured objects, as well as to more clearly point out spline, where position, length, and curvature of the pointer objects of interest to other users in a collaborative envi- are controlled by three points in space. ronment. The flexible pointer, shown in Figure 8, reduces ambiguity by avoiding obscuring objects, which would Secondly, we implement a corresponding, customized unit have been selected with traditional ray-casting techniques. that listens to changes in its position, end point, and control The flexible pointer also has a visual advantage in situa- point properties, and updates the geometry accordingly. We tions in which it is easy to point out an object, without ob- now have a mechanism for listening to, and updating the structing the object of interest, while still providing a con- values of this unit, both locally and over the network. Any tinuous line from the user to the target. component in our framework is thus able to listen to changes or update the geometry of the pointer, by accessing Figure 8. Overview of the curved pointer interaction technique during the design phase. From right to left: The application implements the geometry for a Bézier curve pointer and uses a distributed unit to listen to changes in its properties. The Unit UI is run on a second computer on the network. The graph manipulated by the Unit UI outputs the three parameters to the Bézier curve, while taking the five curve parameters as input. The position properties are provided by two six–degree-of-freedom track- ers. The other parameters can either be determined by a graph that calculates them based on tracker orientation, or by a graph that lets the user manually modify the trackers with a mouse-compatible device. Interaction technique abstraction occurs at sev- eral stages here, most clearly between the application, the Unit UI, and the sensors. these properties. For increased precision, our prototype Running CPU-intensive speech recognition on multiple flexible pointer utilizes a two-handed approach, where the microphones hands are tracked with two six–degree-of-freedom trackers, the distance between the hands map to the length of the First, we needed a mechanism for getting input from multi- pointer, and the relative bending of the hands determines ple microphones, so we considered the following ap- the curvature characteristics of the pointer. We implement proaches: this control behavior as a separate Unit Graph that updates 1) Using multiple general-purpose sound cards on one the properties of the above-mentioned unit that is control- computer. One would have to be careful to not run into ling the geometry. hardware conflicts, since an ordinary PC is not de- Design Process signed to have many simultaneously active sound cards. One of the hardest tasks in interaction technique design is the assignment of appropriate values to constants, and as 2) Using a special-purpose sound card with multiple au- with most interaction techniques, there are several such dio inputs. One of these cards would be too expensive constants for the flexible pointer (e.g., the scale factor for for our low-budget experimental setup. the mapping of the distance between the user’s hands to the length of the pointer). 3) Using a special-purpose array microphone for audio localization, where the signal processing is done in Thanks to Unit’s modularity, separate Unit Graphs can be hardware. The few such inexpensive consumer-level used for interactive tweaking and debugging of the running microphones we found did not provide programming interactive technique. We constructed a new graph that API access to inferred positional data. These micro- takes input from a small handheld presentation mouse with phones also put restrictions on the setup, limited by the a thumb-controlled joystick. A button click alternates be- characteristics of the microphone, and we found it nei- tween the constants that are modified and pushing the joy- ther feasible nor cost effective to build our own micro- stick up/down increases/decreases the value of the current phone. constant, as shown in Figure 7. Although we could place the graph in the same program as the flexible pointer, Second, speech recognition is CPU intensive, and running avoiding the mix of interaction technique and tweaking several instances of speech recognition software on the code seemed reasonable, and we found it more advanta- geous to run it in a separate program. In fact, the ease of distribution made us place it on a separate machine, which gave us an exclusive environment for developing the tweaking code, as shown in Figure 9. The behavior of our interaction technique can be modified in real time as soon as the graph is connected to the flexible pointer. More im- portantly, we can have the flexible pointer running con- stantly, while modifying, recompiling, and restarting the tweaking code. When satisfied with the behavior of the interaction technique, the tweaking code is removed, sim- ply by not running it. This example shows how we can use Unit to abstract the interaction techniques from the input devices and the application, and also how two interaction techniques (the flexible pointer and the tweaking code) can be abstracted from each other. 6 DISTRIBUTED SPEECH RECOGNITION, ANALY- SIS, AND LOCALIZATION We found Unit very useful in a recent experimental setup for a user interface based on speech analysis and audio lo- calization [17]. We intended to explore the use of non- verbal features of the user’s speech for implicit or explicit program control. Additionally, we planned to use multiple microphones to approximate the user’s head position, by comparing the audio from the different microphones. Figure 10. The Unit graph for the experimental non-verbal speech and audio localization setup. same machine used for the visualization would signifi- plementation supports conventional pointing devices (e.g., cantly affect the frame rate. mice, trackballs, touchpads, trackpoints, and touchscreens) and keyboards, as well as several six–degree-of-freedom A distributed approach sensors (Ascension Flock of Birds, InterSense IS600 Mark 2 Plus, and InterSense IS900) and speech recognition and Realizing that we had many available machines in our lab, speech synthesis (through the Java Speech API and IBM equipped with standard sound cards, we decided to take ViaVoice). RMI is used for distribution over TCP/IP. We advantage of Unit’s distribution mechanism to offload the have used a heterogeneous machine pool during the devel- CPU-intensive speech recognition to other machines on the opment, with machines ranging from a Celeron 400 MHz, network. Each of these machines could then support one with 192 MB RAM, running Windows 98, to a Dual Xeon microphone, without the need for any special-purpose 2.8 GHz, with 1 GB RAM, running Windows XP. The low- hardware or alteration of the hardware configuration. end machines can be used for running Unit Graphs and input device handling, while the more powerful machines We designed our Unit graph such that the speech is ana- with 3D acceleration hardware are needed for 3D graphics. lyzed locally on each speech server, with the recognized speech and the extracted speech features communicated 9 CONCLUSIONS AND FUTURE WORK over Ethernet to the application server. The Unit dataflow in the application server fuses the input and adjusts the be- We have presented Unit, a system that uses a visual data- havior of the application accordingly. Our experimental flow programming language to support designing and ex- setup is shown in Figure 10. perimenting with 2D and 3D interaction techniques. We are actively using Unit to develop interaction techniques, and It might sound contradictory that we find it more cost- have demonstrated its utility through a set of examples cre- efficient and convenient to use a separate computer, instead ated with the system. of a special-purpose sound card, to host a microphone. However, the important point here is that Unit allowed us As we have showed, Unit allows the flexible specification to use our currently available general-purpose hardware for of interaction techniques, while effectively avoiding prob- rapid prototyping of an experimental user interface, without lems related to specific hardware setups in experimental having to deal with the hardware-related issues that would systems through a peer-to-peer distribution mechanism. play a central role in designing a practical product. While Besides abstracting interaction techniques from input de- Unit made it possible to easily develop a distributed data- vices and applications, Unit’s modularity has also proven flow for our purposes, its transparent distribution mecha- convenient, since it allows debugging components to be nism also makes it straightforward and simple to reconfig- developed in a stand-alone fashion outside the interaction ure the application to run on a single machine (e.g., with technique of interest. multiple sound-cards or a multi-input sound card). ACKNOWLEDGMENTS 7 COMPOSITE INPUT DEVICES This research was funded in part by Office of Naval Re- The Unit framework has made it easy for us to develop search Contracts N00014-99-1-0249 and N00014-99-1- rudimentary prototype input devices, assembled from ar- 0394, NSF Grant IIS-00-82961 and IIS-01-21239, and gifts rangements of two or more input devices. Figure 3 shows from Intel, Microsoft Research, and Alias | Wavefront. one of the simplest examples of a composite input device: a three–degree-of-freedom mouse created from two off-the- REFERENCES shelf wireless optical mice that are rigidly attached to pro- 1. Anson, E. The Device Model of Interaction. Proc. SIG- vide an additional degree of freedom (rotational accelera- GRAPH ’82 (ACM Comp. Graph., 16(3), July 1982), Bos- tion in the plane of the surface on which they are used). ton, MA, July 26–30, 1982, 107–114. Unit provides simple means for specifying the relations 2. Borning, A. The Programming Language Aspects of Thin- between the two mouse sensors, and thus allows the behav- gLab, a Constraint-Oriented Simulation Laboratory. ACM ior of this composite input device to be visually pro- Trans. on Prog. Langs. and Sys, 3(4), October 1981, 343– grammed, completely in software, as shown in Figure 1. 387. 3. Bowman, D., Hodges, L.F. An Evaluation of Techniques for Unit makes it possible to build composite input devices that Grabbing and Manipulating Remote Objects in Immersive consist of hierarchies of different input devices and interac- Virtual Environments. Proc. Symp. on Interactive 3D Graph. tion techniques, while providing unified application-level 35–38. 1997. APIs to these devices. 4. Duce, D., van Liere, R., and ten Hagen, P. An Approach to Hierarchical Input Devices. Comp. Graph. Forum, 9(1), 15– 8 IMPLEMENTATION 26. 1990. The Unit framework is implemented with Java and Java3D, and thus runs across multiple platforms. Unit’s current im- 5. Dragicevic, P. and Fekete, J.D. Input Device Selection and- 15. Olwal, A and Feiner, S. The Flexible Pointer—An Interac- Interaction Configuration with ICON. Proc. IHM-HCI 2001. tion Technique for Selection in Augmented and Virtual Real- Frontiers, Lille, France, Springer Verlag. 543-448. 2001. ity. To appear in Extended Abstracts of ACM Symp. on User 6. Figueroa, P., Green, M., Hoover, H.J. InTml: A Description Interface Software and Tech. (UIST ’03), Vancouver, BC. Language for VR Applications. Proc. 3D Web Technology. 2003. 53-58. 2002. 16. Olwal, A. Unit—A Modular Framework for Interaction 7. Fitzmaurice, G.W., Ishii, H., Buxton, W. Bricks: Laying the Technique Design, Development and Implementation. MS Foundations for Graspable User Interfaces. Proc. Human Thesis, Dept. of Num. Anal. and Comp. Sci., Royal Inst. of Factors in Comp. Sys. (CHI '95). 442–449. 1995. Tech., Stockholm, Sweden. 2002. 8. Greenberg, S. and Fitchett, C. Phidgets: Easy Development 17. Olwal, A. and Feiner S. Using Prosodic Features of Speech of Physical Interfaces Through Physical Widgets. Proc. ACM and Audio Localization in Graphical User Interfaces. Tech- Symp. on User Interface Software and Tech. (UIST ’01), Or- nical Report CUCS-016-03, Department of Computer Sci- lando, FL, 2001, 209–218. ence, Columbia University, New York, NY, June 26, 2003. 18. Resnick, M. Behavior Construction Kits. Communications of 9. Hinckley, K. and Sinclair, M., Touch-Sensing Input Devices, Proc. Conf. on Human Factors in Comp. Sys. (CHI ’99), the ACM, 36(7). 64–71. July 1993. 223–230. 19. Smith, D.N. Building Interfaces Interactively. Proc. ACM 10. Kessler, G.D., Kooper, R., Verlinden, J.C. and Hodges, L., SIGGRAPH Symp. on User Interface Software, Banff, Al- The Simple Virtual Environment Library, Version 2.0, User's berta, October 17–19, 1988, 144–151. Guide, http://www.cc.gatech.edu/gvu/vir- 20. Suzuki, H., Kato, H. AlgoBlock: A Tangible Programming tual/SVE/docV2.0/sve.book_1.html. Technical Report, Language, A Tool for Collaborative Learning. Proc. 4th Graphics, Visualization, and Usability Center, Georgia Insti- European Logo Conf., August 1993, 297–303. tute of Technology, 1997. 21. Ungar, D. and Smith, R. Self: The Power of Simplicity. 11. Lanier, J. Grimaud, J-J, Harvill, Y., Lasko-Harvill, A., Proc. OOPSLA ’87, Orlando, FL, October 1987, 227–241. Blanchard, C., Oberman, Mark., Teitel, M. Method and sys- 22. Virtual Realty Consulting (VRCO) Inc., CaveLib. Chicago, tem for generating objects for a multi-person virtual world IL. http://www.vrco.com/products/cavelib/ cavelib .html using data flow networks. United States Patent 5588139. 1993. 23. Virtual Reality Peripheral Network (VRPN), UNC, Chapel Hill, http://www.cs.unc.edu/Research/vrpn. 12. Liang, J., Green, M. JDCAD: A Highly Interactive 3D Mod- eling System. Comp. and Graph., 18(4). 499–506. 1994. 24. VR Juggler—Open Source Virtual Reality Tools, Virtual Reality Applications Center, Iowa State University, 13. MacKenzie, I. S., Soukoreff, R. W., Pal, C. A Two-ball http://www.vrjuggler.org/. Mouse Affords Three Degrees of Freedom. Extended Ab- stracts Human Factors in Comp. Sys. (CHI ’97). 303–304. 25. Zeleznik, R. C., Herndon, K. P, Robbins, D. C., Huang, N., 1997. Meyer, T., Parker, N., Hughes, J.F. An Interactive 3D Tool- kit for Constructing 3D Widgets. Proc. SIGGRAPH '93. 81– 14. Maloney, J. and Smith, R. Directness and Liveness in the 84. 1993. Morphic User Interface Construction Environment. Proc. ACM Symp. on User Interface Software and Tech. (UIST ’95), Pittsburgh, PA, 1995, 21–28.