GazeFlick: A Practical Text Input Method for AR-HMD Combining Eye Gaze and Flick Gesture⋆

Abstract

Although research has been conducted for many years on text input while wearing a head-mounted display (HMD) for augmented reality (AR), a de facto standard has not yet been established. Many HMDs recently have begun to adopt GazePinch, a method for selecting distant objects by pinching with a finger while the user's gaze is fixed on the object. On the other hand, “flick input” is mainly used for inputting Japanese and other languages on smartphones, and it is known to be a fast text input method once the user gets used to it. In this study, we propose GazeFlick, a combination of GazePinch and flick input, as a new text input method for AR/VR-HMD. Preliminary experiments have confirmed a significant improvement in input speed compared to a hand ray method with a common virtual keyboard.

Keywords

• eol>Augmented reality
• virtual reality
• text input
• eye gaze

1. Introduction

input focuses on the fact that each Japanese character (hiragana and katakana) can be grouped into five chunks, Text input while wearing a head-mounted display (HMD) and only the primary keys for each group are displayed for augmented reality (AR) and virtual reality (VR) has on the main screen. When the user touches each primary been a longstanding issue in the field. When using key, the other keys in the group are displayed around it AR/VR-HMDs while standing, traditional keyboards are and the key at the position where the user takes his/her dificult to use, so alternatives are needed. As of 2024, ifnger of the screen is input. In other words, to input the many AR/VR-HMDs have virtual keyboard functionality surrounding keys, the user must translate his/her finger (a common QWERTY layout) as a standard text input either up, down, left, or right, which is called a “flick” feature. Input is performed by performing a decision action. This is known to be a fast text input method once operation (i.e., pressing a button on the controller or tap- the user gets used to it. This method can also be easily ping a finger) with the ray from the VR controller or extended to other languages input, as long as some keys actual hand against each key. For the majority of users can be grouped. accustomed to the general QWERTY keyboard layout, In this research, we focus on GazePinch and flick input, there is little major confusion, but there is a problem that and propose a new input method for AR-HMD that comthe operation does not become much faster even if the bines them. Specifically, the appearance of the UI follows user gets used to it. Many researchers and developers that of the UI for flick input, and only the main keys of have made numerous proposals over the past decade in each group are displayed. When the user looks at one of search of better operating methods (e.g., [ 1, 2, 3, 4 ]). How- them and performs a pinch motion (same as GazePinch), ever, as of 2024, there is still no text input method that the other keys of that group are displayed above, below, can become the de facto standard. left, or right of that main key. With the user’s thumb and

The GazePinch [ 5 ], which determines the selection of index finger closed, move two fingers in any of the four distant objects by pinch movements of the finger while directions (up, down, left, or right), and the key that was the user’s gaze is directed toward the object, has begun to selected when the fingers were separated will be input. be used in many devices (e.g., Microsoft HoloLens2 with We implemented the proposed method on HoloLens2 and MRTK3, Apple Vision Pro, etc.). On the other hand, “flick conducted preliminary experiments comparing it with input” is mainly used to input Japanese and other lan- the general method of using a hand ray and QWERTY virguages on touchpad devices such as smartphones. Flick tual keyboard, and obtained promising results showing a significant improvement in input speed.

APMAR’24: The 16th Asia-Pacific Workshop on Mixed and Augmented Reality, Nov. 29-30, 2024, Kyoto, Japan * Corresponding author. $ [e-mail] (T. Kishi); 2. Related Work [e-mail] (Y. Fujimoto) € https://yfujimoto.cfbx.jp/ (Y. Fujimoto) There have been many methods proposed for text input 0000-0002-8270-2609 (Y. Fujimoto) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License on AR/VR-HMDs. In this section, we will discuss the Attribution 4.0 International (CC BY 4.0). methods that are particularly relevant to this research, focusing on recent years (from 2020 onwards).

Streli et al. proposed TapType, which allows any flat surface to be used as a pseudo-keyboard [ 6 ]. In this method, the user’s tapping motion on the flat surface is decoded by an inertial sensor built into a wristband, and is associated with the layout of the conventional QWERTY keyboard. Lu et al. devised a method for entering text without using the hands using an AR-HMD with three types of gaze input (dwell, trajectory, and blinking).

Interestingly, this method does not display a virtual keyboard, but instead has the user operate it while imagining a standard QWERTY keyboard. By combining these, it Figure 1: Overview of GazeFlick. The user is looking at it, and overcomes the problem of arm fatigue during input and the key that has been pinched is changing to red. The sub-keys the problem of the virtual keyboard blocking the real are expanding in the upper, lower, left and right directions. environment [ 7 ].

Several methods have been proposed to reduce the efort required for direct input by introducing a proba- 3. Method bility model. Fashimpaur et al. proposed the PinchType method, which uses a standard QWERTY keyboard and 3.1. Idea pinch gestures [ 4 ]. All the keys are grouped into eight types, corresponding to the eight fingers of the hand, The proposed method GazeFlick combines GazePinch [ 5 ] including the thumb. Since there are more than eight and flick input (Figure 1). One factor that contributes letters in the alphabet, there is some ambiguity, but this to the complexity of text entry is the number of inputs. method dynamically resolves this ambiguity during input Especially in Japanese, it takes, on average, two Roman by introducing a language model. Adhikary and Vertanen alphabet characters to input one character of the smallest proposed a text input method that enables fast and error- unit (i.e., hiragana) in the most common Roman input free text input by adding a function for automatically method. While this is not a major problem for physical suggesting correction candidates using text context to a keyboards that allow high-speed input, it is fatal for virgeneral virtual keyboard, and supporting these functions tual keyboards. On the other hand, there is a direct input using speech recognition [ 8 ]. Dudley et al. proposed a method that displays all 45 types of characters in all the keyboard that uses a probability model that allows for smallest units and allows the user to input them once, the uncertainty of the imperfect hand tracking function but few Japanese prefer this method because it takes time of consumer VR-HMDs. Using this, they compared the to find a large number of characters. On the other hand, peak performance of the general input method of poking in smartphones, which contain similar problems, “flick with the index finger and the long-distance input of ray input” solves these problems at the same time. casting, and analyzed them in detail [ 9 ]. It is assumed that the HMD worn by the user is a device

Takahashi et al. have also introduced flicking as a that can perform eye-gaze measurement and hand trackmethod of inputting Japanese for a VR-HMD [ 10 ]. They ing. Each key is grouped by no more than five and has an use a VR controller with a trackpad for flick input, in- ordering relationship among them. Only the youngest stead of the GazePinch idea, which is diferent from us. key in the group is displayed on the screen (this is called By using a trackpad, the operation is closer to flick input the primary key). First, as with GazePinch, the collision on a smartphone, so when used by people with experi- between the gaze and the main key is determined by meaence in flick input, it was shown that text can be input suring the gaze (Figure 2). When Gaze collides with a faster than with a QWERTY keyboard using a ray. In specific primary key and a pinch operation (index finger June 2024, Apple Vision Pro was launched in Japan, and and thumb in contact) is performed, other keys (called a method similar to the proposed method was adopted subkeys) in the group are displayed in order in the four for inputting Japanese. However, due to its special cam- locations to the left, above, right, and below the primary era arrangement, Apple Vision Pro is capable of hand key. With the index finger and thumb in contact, moving tracking over a much wider range and at a much faster the hand up, down, left or right from that position highspeed than HoloLens2 and other HMDs. Furthermore, it lights the corresponding subkey. When the index finger is known to have a fairly high level of accuracy in eye and thumb are released, the highlighted key is entered tracking. This research is mainly based on the assump- and all sub keys disappear once. Alternatively, if the user tion that HoloLens2 will be used as the device, and it will pinches the main key and then separates their thumb be tuned as a text input method suitable for limited hand and forefinger without sliding their hand, the character tracking and eye tracking performance. on the main key will be entered. The user repeats this 3.2. Implementation Microsoft HoloLens2 was used for this implementation in this paper. However, the proposed method can be easily applied to other devices that are capable of eye measurement and hand tracking. Unity2022.2.18f and MRTK3 (3.2.0) were used as the software for the implementation.

As a use case, we have implemented hiragana input for Japanese, but it can easily be replaced with other languages as well. Hiragana is based on units of 46 characters, and is divided into 11 groups containing 5 or fewer characters (Figure 3). In addition, there are characters with voiced consonants and semi-voiced consonants derived from them. After selecting each character using pinch operation, the user can input the corresponding voiced consonants by pushing his/her finger forward, and he/she can input the corresponding semi-voiced consonants by pulling his/her finger back.

Pinching was based on the vertical plane of the HMD position posture at the time of tapping, and was recognized as an up/down or left/right movement when the ifnger was moved beyond a certain distance (currently 50 mm) from it, respectively. The key that the user is looking at is set to change to a color with higher brightness.

During the pinch, the transparency of the other keys and the back panel was increased to 10 % to make the key that is being focused on stand out. After pinching, the currently selected key changes to red to make it more noticeable. The window in which the characters are displayed is fixed at the world coordinate system position that called it, and only follows when the user moves a lot (i.e., Billboard display provided by MRTK). The detailed parameter settings for implementation are as shown in Table 1. method) took 15.2 s. For (b) hiragana set (with voiced or semi-voiced consonants), no alphanumeric characters), Hand-ray took 21.2 s, and GazeFlick (proposed method) took 16.9 s.

The results showed that when used by experienced users, the proposed method has the potential to enable faster character input than hand-ray. In particular, when entering characters that include voiced and semi-voiced consonants, it was expected that the input speed with GazeFlick would be somewhat slower because the additional pinch gesture would need to be moved in the depth direction, but the speed was almost same. In addition to Japanese, the input of additional elements for certain basic characters can be seen in multiple other languages (Thai, Arabic, French, etc.). The fact that the proposed method showed that it was possible to input voiced and semi-voiced consonants smoothly without significantly reducing the movement speed is expected to be beneficial in the use of these multiple languages as well.

5. Summary and Future Work

In this study, we proposed GazeFlick as a method of entering text that is an extension of GazePinch, which uses eye movement and flicking. Preliminary experiments (with only one person) showed that, with practice, this method could be 20 % to 30 % faster than the conventional hand-ray method. In the future, we plan to conduct formal user studies and clarify the applicability of the system to multiple languages (particularly English).

Acknowledgments

This work was partially supported by JSPS KAKENHI Grant Number JP22K12127.

References

[1] J. J. Dudley , K. Vertanen , P. O. Kristensson , Fast and precise touchbased text entry for head-mounted augmented reality with variable occlusion , ACM Transactions on Computer-Human Interaction 25 ( 2018 ).

[2] S. Ahn , S. Heo , G. Lee, Typing on a smartwatch for smart glasses , in: Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces , ISS '17 , ACM , 2017 , pp. 201 - 209 .

[3] W. Xu , H.-N. Liang , Y. Zhao , T. Zhang , D. Yu , D. Monteir , Ringtext: Dwell-free and hands-free text entry for mobile head-mounted displays using head motions , IEEE Transactions on Visualization and Computer Graphics 25 ( 2019 ) 1991 - 2001 .

[4] J. Fashimpaur , K. Kin , M. Longest , Pinchtype: Text entry for virtual and augmented reality using comfortable thumb to fingertip pinches , in: Extended Abstracts of the 2020 Conference on Human Factors in Computing Systems, CHI EA '20 , ACM , 2020 , pp. 1 - 7 .

[5] K. Pfeufer , B. Mayer , D. Mardanbegi , H. Gellersen , Gaze + pinch interaction in virtual reality , in: Proceedings of the 5th Symposium on Spatial User Interaction, SUI '17 , ACM , 2017 , pp. 99 - 107 .

[6] P. Streli , J. Jiang , A. Fender , M. Meier , H. Romat , C. Holz , Taptype: Ten-finger text entry on everyday surfaces via bayesian inference , in: Proceedings of the 2022 Conference on Human Factors in Computing Systems, CHI '22 , ACM , 2022 .

[7] X. Lu , D. Yu , H.-N. Liang , J. Goncalves, itext: Handsfree text entry on an imaginary keyboard for augmented reality systems , in: The 34th Annual ACM Symposium on User Interface Software and Technology, UIST '21 , ACM , 2021 .

[8] J. Adhikary , K. Vertanen , Text entry in virtual environments using speech and a midair keyboard , IEEE Transactions on Visualization and Computer Graphics 27 ( 2021 ) 2648 - 2658 .

[9] J. J. Dudley , J. Zheng , A. Gupta , H. Benko , M. Longest , R. Wang , P. O. Kristensson , Evaluating the performance of hand-based probabilistic text input methods on a mid-air virtual qwerty keyboard , IEEE Transactions on Visualization and Computer Graphics 29 ( 2023 ) 4567 - 4577 .

[10] R. Takahashi , S. Shirai , J. Orlosky , Y. Uranishi , H. Takemura , A japanese character flick-input interface for entering text in vr , in: 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct ) , ISMAR2021 , IEEE, 2021 , pp. 261 - 263 .