Actuated Peripherals as Tangibles in Desktop Interaction

Abstract

TUIs use everyday physical objects to interact with digital information. With decades of usage, computer peripherals became everyday physical objects. We observed that users manipulate them for other purpose than input and output devices. For example users turn their screen to avoid sun reflections, or move their keyboard ans mouse because they need space on their desk. In this work we see computer peripherals as everyday objects, and use them as TUIs. This paper presents two levels of tangible interaction with desktop computers: the first one is a keyboard with actuated keys. The keys can raise from their initial position, which can be used to represent interaction or extend interaction with keyboards. On the second level we actuated a mouse, a keyboard and a screen so that they can move around on the desk. We present scenarios showing how it extends interaction with a desktop computer setup.

-

INTRODUCTION In the early days of tangible interaction, Ullmer and Ishii described Tangible interaction this way: “TUIs will augment the real physical world by coupling digital information to everyday physical objects and environments.”[ 18 ]. The idea is to break the barrier between the physical and the digital world. With this paradigm, any object can either represent digital information or be a proxy for manipulating digital information. Similarly to input and output devices, these objects are instrumented with sensors or actuators, which are the links with the digital world.

The question whether a computer mouse is a TUI is interesting. On one hand it complies with Ullmer and Ishii’s definition. Since the introduction of this definition, computer peripherals became everyday objects. On the other hand, the computer mouse was specifically designed for interaction with digital information, and would not exist otherwise. Now, consider the situation of a user having a talk with a slideshow. He holds the mouse in the hand and just use the buttons to move to the next slide, the same way he would do with a remote control. In this case he is not using the mouse as it was designed for. Is it sufficient to consider the computer mouse as a TUI in this specific scenario? Further, now imagine an actuated computer mouse so that it can move around on the table. This mouse moves around on the desk to give the user notifications when he is not watching the screen. In this situation, the mouse is clearly not used as the mouse was designed to be used. In this work we explore actuation and motion as a way of interacting with computer peripherals in the way they were not designed for. We discuss scenarios in which computer peripherals are tangible objects for interaction with digital information.

RELATED WORK We describe below evolutions of the desktop interface, the use of motion as an output modality, and shape changing interfaces.

Rethinking desktop interaction The way we interact with computer peripherals has not changed much since their invention. Mice, keyboards and screens have not changed much on an interaction point of view.

Studies showed that some design choices are questionable. For example Pietrzak et al. studied the impact of the mode delimiters for keyboard shortcuts by replicating the CTRL and SHIFT on the thumb buttons of the mouse [ 12 ]. They observed similar performance for keyboard shortcut entry than with the keyboard. This means it makes sense to revisit design choices made decades ago.

Research explored additional dimensions to extend the capacities of computer peripherals. Rekimoto et al. added capacitive sensing to the keys of a keyboard [ 16 ]. It enables sensing whether the user touches a key or not. They propose scenarios in which they use this information to display feedforward, and other scenarios which take advantage of this extended vocabulary to enhance interaction.

Beyond rethinking desktop devices, Bi et al. used the desk itself for interaction [ 3 ]. They extend the peripherals capabilities with interaction with the desk, both for multi-touch input and a projected display. At the opposite, Gervais et al. use everyday objects as viewports, which share or extend computer screens real estate [ 6 ]. These systems explore tangible properties of the desktop environment to extend interaction. Motion output Motion is a property of the interaction with an object. It is commonly used as input values, but we are interested in motion of a physical object as an output modality. Motion as output produces visual and haptic feedback.

Löffler et al. designed insect-like desk companions [ 11 ]. These companions can move around on the desk to give the user notifications through the visual channel. Authors focused on their affective effect on the user. Interestingly, motion based interfaces can take advantage of both the visual and haptic aspects of movements. Zooids are small robots which cooperate to achieve a particular task [ 9 ]. In some situations they represents points on a graph. In other situations they move an object on a table.

Actuating an object enables dynamic force feedback when the user touches it. For example Roudaut et al. explored the idea of actuating a layer over a touchscreen to guide the finger touching the device [ 17 ]. It makes it possible to teach users gestures, such as gesture keyboard symbols. Other studies use motion to encode information. Either the system controls the movement [ 5, 13 ], or only constrains the movements of the user [ 14 ]. Similarly to Tactons [ 4 ], information is coded by mapping pieces of information to signal parameters such as amplitude, size or shape. Below, we explain how we use motion to extend interaction with computer peripherals. Shape changing interfaces Actuating objects also make it possible to change their shape, and therefore their affordances. Knobslider is an example of interactive object which is either a button or a slider, depending on its shape [ 7 ]. This object was specifically designed to behave this way. At the opposite, Kim et al. designed an inflatable mouse [ 8 ] which can either give notifications, or be used as an elastic input device for continuous rate control. ACTUATED PERIPHERALS In this project we use both the concepts of motion as output, and shape changing interfaces to redesign computer peripherals. We discuss design rationales on the device level, desktop level, and envision extending the concept to en entire room or a house.

Device level Motion is an essential aspect of interaction with peripherals. Pointing devices rely on movement measurements. Keyboards use binary key positions as input data. In the Métamorphe project [ 1 ], we actuated the keys so that they can either be up or down (Figure 1). Keys can still be pressed, whether the key is up or down.

This shape changing keyboard has new properties compared to regular keyboards. When a key is up, the use can push it in four directions, or even pinch it (Figure 2). With a touch sensor all around it, the key could be used as an isometric pointing device such as a trackpoint.

Our previous studies showed that raising keys eases eyes-free interaction with the keyboard. Specifically we observed that users can easier locate raised keys and surrounding ones.

The possibilities of such a keyboard go beyond text typing and keyboard shortcuts. Similarly to Relief [ 10 ], it is a shape changing device which can be used to display information. Desktop level We observed people when they use a desktop computer, and identified situations in which they move their peripherals besides interaction with the computer. For example we observed people turning their screen to avoid sun reflexions. Other users turned their screen either to show visual content to somebody, or to show something in the room in a video conference with the camera affixed to the screen. It is also frequent to move the mouse and keyboard to make space on the desk for something else.

In the Living Desktop project[ 2 ], we actuated a mouse, keyboard and screen (Figure 3):

The mouse can translate in the x; y plane directions. The keyboard can rotate, and translate in the x; y plane directions.

The screen can rotate, and translate in the x axis direction. With these capabilities, devices can move on their own without requiring the user to move them. The interesting question here is the degree of control the user has over his devices. There is a continuum between full control and full automation, in which we identify some particular degrees:

Telekinesis: the user moves the devices with distant controls.

Tele-operation: the user suggests movements, the device decides to which degree it complies.

Constraint: the user defines the constraints of the devices movements.

Insurrection: the user has no influence on the device movements.

We implemented a couple of scenarios, which illustrate the concept.

Peephole display Even with a large screen, the interactive screen estate is limited. We propose to use the screen as a peephole display in a larger working space. In this scenario, the screen moves on the x axis and the pixels show the content in this area in space. The screen is like a moving physical window. In this scenario the user controls the screen position.

Video Conference When video-conferencing with a desktop computer, the camera is usually affixed to the screen. The problem is when the user would like to show something he manipulates outside the camera range. He has to move the screen at the same time he is manipulating. In this scenario the screen follows the user so that he can always see the video conference, and show what he is doing to his collaborators. The user does not control the screen position in the strict sense of the term. However he can activate or deactivate this behavior and still control the screen position manually or with another interface.

Ergonomic coach Being well seated is essential for healthy office work. It reduces fatigue and pain. It is however difficult to pay attention to our posture all day. In this scenario, devices move away if we are not seated correctly on the chair. The user has no control over the devices in this situation.

Going further Looking at the office environment, there are many other objects involved. They can be actuated to provide other interactive scenarios. Probst et al. presented a prototype of chair they use for input[ 15 ]. These chairs are not actuated, but equipped with sensors. However, Nissan designed parking chairs1 which can move around. In their scenario the chairs move back under the table to tidy the room. But we can envision other scenarios. Pull-down beds are other examples of existing moving objects, which are easy to actuate. In a larger scale, the concept of moving walls makes it possible to have many rooms in small flats2. Each wall has specific equipment, suitable for a particular room. If we keep think bigger, rotating houses is another example of actuated environment3. The obvious application is to maintain sunlight at a specific location in the house. But there may be many interesting interactive scenarios to study with such a building. CONCLUSION The early studies about TUIs used to consider everyday objects for interaction. Nowadays, computer peripherals became everyday objects. As such, they can also be considered as TUIs as long as they are not used as the device they are designed to be. We discussed how actuating computer peripherals enables new interactions. We presented a prototype of keyboard with actuated keys which can move up and down. We also presented a concept of moving computer peripherals, which enable new interactions. We envision this concept can apply to many other objects in our environment. The question we must always keep in mind is the degree of control we would like to keep over these wandering TUIs. 1https://youtu.be/O1D07dTILH0 2https://vimeo.com/110871691 3https://youtu.be/dIHUwp9x8Fg

References

1. Bailly , G. , Pietrzak , T. , Deber , J. , and Wigdor , D. Métamorphe: Augmenting hotkey usage with actuated keys . In Proc. CHI ' 13 (Paris, France, May 2013 ), 563 - 572 .

2. Bailly , G. , Sahdev , S. , Malacria , S. , and Pietrzak , T. LivingDesktop: Augmenting desktop workstation with actuated devices . In Proc. CHI ' 16 (San Jose, USA, May 2016 ), 5298 - 5310 .

3. Bi , X. , Grossman , T. , Matejka , J. , and Fitzmaurice , G. Magic desk: bringing multi-touch surfaces into desktop work . In Proc. CHI ' 11 (Vancouver, Canada, May 2011 ), 2511 - 2520 .

4. Brewster , S. A. , and Brown , L. M. Non-visual information display using tactons . In E.A. CHI' 04 (Vienna, Austria, Apr. 2004 ), 787 - 788 .

5. Enriquez , M. , and MacLean, K. E. The hapticon editor: A tool in support of haptic communication research . In Proc. HAPTICS 2003 (Los Angeles, USA, Mar. 2003 ), 356 - 362 .

6. Gervais , R. , Roo , J. S. , and Hachet , M. Tangible viewports: Getting out of flatland in desktop environments . In Proc. TEI ' 16 ( Funchal , Portugal, 2016 ), 176 - 184 .

7. Kim , H. , Coutrix , C. , and Roudaut , A. Knobslider: design of a shape-changing device grounded in users' needs . In Proc. IHM 2016 ( Fribourg , Suisse, Oct. 2016 ), 91 - 102 .

8. Kim , S. , Kim , H. , Lee , B. , Nam , T.-J., and Lee , W. Inflatable mouse: volume-adjustable mouse with air-pressure-sensitive input and haptic feedback . In Proc. CHI ' 08 ( Florence , Italy, Apr. 2008 ), 211 - 224 .

9. Le Goc , M. , Kim , L. H. , Parsaei , A. , Fekete , J. -D. , Dragicevic , P. , and Follmer , S. Zooids: Building blocks for swarm user interfaces . In Proc. UIST ' 16 (Tokyo, Japan, Oct. 2016 ), 97 - 109 .

10. Leithinger , D. , and Ishii , H. Relief: a scalable actuated shape display . In Proc. TEI ' 10 (Cambridge, USA, 2010 ), 221 - 222 .

11. Löffler , D. , Kaul , A. , and Hurtienne , J. Expected behavior and desired appearance of insect-like desk companions . In Proc. TEI ' 17 ( Yokohama , Japan, 2017 ), 289 - 297 .

12. Pietrzak , T. , Malacria , S. , and Bailly , G. CtrlMouse et TouchCtrl : Dupliquer les délimiteurs de mode sur la souris . In Proc. IHM 2014 ( Lille , France, Oct. 2014 ), 38 - 47 .

13. Pietrzak , T. , Martin , B. , and Pecci , I. Affichage d'informations par des impulsions haptiques . In Proc. IHM 2005 (Toulouse, France, Sept. 2005 ), 223 - 226 .

14. Pietrzak , T. , Martin , B. , and Pecci , I. Information display by dragged haptic bumps . In Proc. Enactive /05 (Genova , Italy, 2005 ).

15. Probst , K. , Lindlbauer , D. , Haller , M. , Schwartz , B. , and Schrempf , A. A chair as ubiquitous input device: exploring semaphoric chair gestures for focused and peripheral interaction . In Proc. CHI ' 14 (Toronto, Canada, May 2014 ), 4097 - 4106 .

16. Rekimoto , J. , Ishizawa , T. , Schwesig , C. , and Oba , H. PreSense: interaction techniques for finger sensing input devices . In Proc. UIST ' 03 (Vancouver, Canada, Nov. 2003 ), 203 - 212 .

17. Roudaut , A. , Rau , A. , Sterz , C. , Plauth , M. , , and Baudisch, P. Gesture output: eyes-free output using a force feedback touch surface . In Proc. CHI ' 13 (Paris, France, May 2013 ), 2547 - 2556 .

18. Ullmer , B. , and Ishii , H. Tangible bits: towards seamless interfaces between people, bits and atoms . In Proc. CHI ' 97 (Atlanta, USA, 1997 ), 234 - 241 .