Text Entry in VR and Introducing Speech and Gestures in VR Text Entry

-

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

Copyright held by the owner/author(s).

MobileHCI, 2018 Barcelona, Spain.

Biography

I am a Computer Science PhD student at Michigan Technological University, Houghton, Michigan, USA. I have a Bachelor of Science degree in Computer Science and Engineering from University of Dhaka, Bangladesh. My current research focuses on designing interactive systems for text entry in midair and virtual reality (VR) Currently I am working under the supervision of Dr. Keith Vertanen who is a renowned researcher in the field of text entry.

Summary of Related Past Works

A lot of works have been done on text entry techniques in personal computers and mobile devices. However, works related to text entry in midair and VR environments are barely sufficient. This is mainly because there have always been challenges to design and implement text input surfaces in midair or VR, to track or sense user’s actions and to map the interaction between the user’s actions and the input surface. We have been working on designing an interface for entering text in the virtual environment for a year. Text entry in VR environment is different from computers or mobile devices because there is no physical keyboard or touchscreen in this environment to interact with. Entering text in VR environment can be achieved by speech, gestures or virtual keyboard. Our work focuses on entering text in a virtual keyboard. We have been able to create a prototype of a virtual keyboard that enables user to input text in the virtual environment. This prototype senses finger movements of the user along the keyboard by using a Leap Motion Sensor. We have also incorporated a sentence based decoder named VelociTap[ 1 ] with this prototype for auto correction and the prototype provides audio feedback for better interaction. We plan to extend the prototype to a usable interactive system and investigate its limitations and benefits by conducting user studies.

Speech and Gestures in VR Text Entry

While working with our prototype, we implemented a single gesture (a thumbs up) as a delimiter of an interaction in virtual environment. For example, the thumbs up gesture could be used to mark the end of a sentence. For our current prototype, it does not necessarily require many gestures to fulfil our main objective but it will really be interesting to explore how multiple gestures or even speech can be introduced in virtual reality text entry.

Although it seems exciting to incorporate speech and gestures in VR text entry, there are a few limitations. To make a gesture in VR it would necessarily require a body part to make a gesture. For example, a user wearing a head mounted display device in VR might move his head to make a gesture or if he is wearing a hand tracker then he can make a gesture using his fingers. However, these kind of head or finger interactions need users to move their head or upper arm of the hand frequently which may result in pain and fatigue.

Too few works have explored gesture based text entry in VR. Chun et al. [ 2 ] investigated the feasibility of headbased text entry for HMDs. In their work they used head rotation to control a point on the keys of a virtual keyboard. They investigated three techniques: TapType, DwellType and GestureType. TapType resembled tap typing in smartphones and users moved a pointer with head rotation and selected a button by tapping a key. In DwellType, users dwelled over a key to select that key and in GestureType users performed a word-level input using a gesture based typing style. They achieved the best entry rate of 24.73 WPM in GestureType by improving the gesture-word recognition algorithm and incorporating head movement pattern which was recorded during the study.

While there have been a few works in mid-air text entry using hand gestures, text entry in VR using gestures is also rare. AirTap[ 7 ], Wilson et al. [8] and Vogel et al.[ 9 ] used tap and pinch gestures to simulate button clicking in virtual environment for text entry purposes. Vulture[3], AirStroke[ 4 ] and Feit et al.[ 5 ] used hand gestures to enter text in midair. The ideas described in these works can be incorporated to enter text in VR as well.

A mid-air word-gesture keyboard has been proposed in Vulture[3]. The idea of a word-gesture keyboard is that a user can draw a pattern formed by the letters of a word in a touch surface rather than typing the letters. Swype, SlideIT and ShapeWriter are examples of word-gesture keyboards. In Vulture, the idea of implementing wordgesture keyboard in midair instead of touch or stylusbased surface was introduced. It used a large highresolution display and users wore a glove with reflective markers. AirStroke[ 4 ] and Grafitti[ 10 ] are two examples of stroke based text entry techniques. Stroke based techniques are mainly used in stylus based interfaces. In these interfaces users need to use the stylus to make a distinctive gesture to mark the entry of a character.

In comparison to the gesture based works in midair and VR, text entry using speech has remained unexplored and to my knowledge there has not been a single work related to this idea. McGlashan et al.[6] investigated technical and design issues in manipulating virtual reality with a speech interface and proposed a prototype that provides users to control specialized functions using speech.

In conclusion, gesture and speech based text entry is still in the budding process. Fortunately, the invention of effective sensing devices and motion trackers (e.g. Leap Motion Sensor, VICON tracker and HMDs etc.) is attracting researchers and is paving the way to design and implement new interactive systems. Hopefully, in the next few years this line of research will flourish and we will have better speech and gesture based text entry systems in VR.

Markussen, Anders, Mikkel Rønne Jakobsen, and

Kasper Hornbaek. "Vulture: a mid-air word-gesture

keyboard." Proceedings of the 32nd annual ACM conference on Human factors in computing systems.

ACM, 2014.

McGlashan, Scott, and Tomas Axling. "A speech

interface to virtual environments." In Proc.,

International Workshop on Speech and Computers.

1996.

Wilson, Andrew D. "Robust computer vision-based

detection of pinching for one and two-handed gesture input." In Proceedings of the 19th annual ACM symposium on User interface software and technology, pp. 255-258. ACM, 2006.

References

1. Keith Vertanen , Haythem Memmi, Justin Emge, Shyam Reyal, and Per Ola Kristensson. 2015 . VelociTap: Investigating Fast Mobile Text Entry using SentenceBased Decoding of Touchscreen Keyboard Input . In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15) . ACM, New York, NY, USA, 659 - 668 .

2. Yu Chun , Yizheng Gu, Zhican Yang, Xin Yi, Hengliang Luo, and Yuanchun Shi . "Tap, dwell or gesture?: Exploring head-based text entry techniques for hmds." In Proceedings of the 2017 CHI Conference on Human 3.

Factors in Computing Systems , pp. 4479 - 4488 . ACM, 2017 .

4. Ni , Tao, Doug Bowman , and Chris North. "AirStroke: bringing unistroke text entry to freehand gesture interfaces . " Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM , 2011 .

5. Feit , Anna Maria, Srinath Sridhar, Christian Theobalt , and Antti Oulasvirta . "Investigating multi-finger gestures for mid-air text entry." Korea( 2015 ).

7. Vogel, Daniel, and Ravin Balakrishnan . "Distant freehand pointing and clicking on very large, high resolution displays." In Proceedings of the 18th annual ACM symposium on User interface software and technology , pp. 33 - 42 . ACM, 2005 .

9. Bowman , Doug A. , Chadwick A. Wingrave , J. M. Campbell , V. Q. Ly , and C. J. Rhoton . "Novel uses of Pinch Gloves™ for virtual environment interaction techniques." Virtual Reality 6 , no. 3 ( 2002 ): 122 - 129 .

10. Castellucci , Steven J., and I. Scott MacKenzie . "Graffiti vs. unistrokes: an empirical comparison." In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , pp. 305 - 308 . ACM, 2008 .