Abstact-In this article, we investigate the usability of user interfaces based on hand gestures which are implemented using Kinect II and “Leap Motion” devices. In order to investigate the usability, we propose gesture usability testing framework and perform experiments using three widespread gestures dedicated for option selection. The experiments were carried out using mentioned devices. The experiment results show that “Grab to select” gesture for “Leap Motion” gives best results in terms of execution time for every button placement on UI.
I. INTRODUCTION
In recent, the usage of Kinect II and “Leap Motion” devices can be observed in many areas such as gaming, business, or remotely controlled robotics. The usability of natural user interfaces which can be developed using such devices as “Kinect-II” and “Leap Motion” has important role for determining the quality of the product.
Although usability plays important role, recent studies show
that, not all user interfaces developed for “Kinect-II” and
“Leap Motion” are user-friendly due to their lack of gesture
controls [
In this study, we present a solution to test the effectiveness of common gesture motions for “Kinect-II” and “Leap Motion”.
II. RELATED WORKS
As paper states, interface buttons using Kinect, tests
Natural User Interface controls in terms of Execution time by
creating interaction types as comparison points. The research
aims to increase usability of Kinect when controlling NUI
elements on an application [
Another experiment compares two type of gesture
recognition system using multiple devices such as Kinect,
“Leap Motion” with comparison to Mouse and Keyboard. The
paper [
A research [
Kinect Kiosk User Experience Evaluation lets users to be
able setup their own profiles and store the information of user
interface preferences for an individual user. This lets users to
have their own flexible set of controls when using Kinect to
control Graphical User Interfaces [
The research [
To measure the effectiveness of motion types using Kinect -II and “Leap Motion”, we focused on three main gesture types that are used commonly today. These are stated as, “Grab to select” where user makes fist motion of hand to select specific menu button on interface. “Wait to select”, the gesture type that provides users to select when the hand gesture stands on specific menu item after a time lapse and “Push to select” that user does forward and backward hand motion while on a button to trigger click.
To determine usability of “Kinect-II” and “Leap Motion” gestures a testing procedure will be followed. For this purpose, an application is developed to find execution time of hand gestures with commonly used UI design. This will allow us to get precise data from user interactions. Later, the data is printed to Txt file to be able to note it down for our research goal.
The system is developed using Unity Engine, it has 3 scenarios with 9 randomly highlighted clickable buttons (see fig. 1) which are placed on the canvas in such a manner that they cover uniformly all the window.
The scenarios in the system represents 3 hand gesture types. These are, “Wait to select”, “Grab to select” and “Push to select” for “Kinect-II” and “Leap Motion”. Buttons are in a square formation 3 at top, 3 at middle and 3 at the bottom placed side by side. The purpose of this formation is to find out the effectiveness of each hand gesture in differently located buttons on user interfaces.
After user completes the click sequence for all buttons, system prints execution time in txt file with specific gesture type and button that are clicked.
The system indicates user with begin button to start testing procedure. After the button is clicked, user makes necessary selections on supposedly highlighted button. Each time a highlighted button triggered, system once again indicates user with count down timer starts from 5 seconds. When the time reaches 0, user attempts to click another highlighted button. With each button click system prints execution time of buttons pressed by the user. This helps us to see performance of user for each button on Interface.
IV. EXPERIMENT
The goal is to measure gesture performance of the user interface implemented using “Kinect-II” and “Leap Motion” devices.
In our experiment, we use “Kinect-II” and “Leap Motion” devices connected to the Personal Computer. Unity’s 5.0 was used for the development and testing of the system.
Also, original firmware SDKs for both devices are used during the development of the system.
Table 1 – “Wait to select” Gesture – Kinect II device USERS execution time given in
As we can see from Table 1 buttons 5, 6, and 9 have 3 lowest execution times compared to other buttons. During the test, we observed that, most users perform faster executions if the buttons are closer at the side of hand that controls “Kinect-II”. In “Grab to select” data from Table 2, buttons 3, 5, 6, 8 have faster execution times than others. During the test, it is determined that users find and grab the button at center and at the side where they control “Kinect-II” easier. When buttons placed at corners users grab is more unstable than other button placements. “Push to select” Gesture values from Table 3 shows us that, 5 at the center and 2 at the center top buttons have lower values compared to other buttons. According to test reviews, when users do push movement, it is observed that they partly loose the control of hand movement towards the buttons standing at sides, this causes users to get out of button area and cause “Kinect-II” to not register a click trigger. Users find easier to push buttons standing at the center and at top center, this way, they do not lose control of the hand movement. When users stay on “Leap Motion” sensors, they centralize their hand on “Leap Motion” and this lets them to trigger button 5 faster than the other buttons.
Table 5 – “Grab to select” Gesture – “Leap Motion”
USERS
Despite “Kinect-II”’s “Grab to select” gesture (easier to click center or top-center button Table 2), “Leap Motion’s” “Grab to select” gesture makes interactions easier if the buttons stand at the center or bottom-center of the screen. Because users mostly lower their hand in testing session while keeping their hand on “Leap Motion”. It gets easier to trigger click on the center-bottom (button 8) and at the center (button 5).
Table 6 – “Push to select” Gesture – “Leap Motion” USERS 6 3.67 2.31 4.92 3.99 2.91 3.78 3.61 3.65 3.15 6 In “Push to select” Gesture for “Leap Motion”, it is the same situation here as in “Grab to select” Gesture of “Leap Motion”.
By comparing “Wait to select” gesture for “Kinect-II” and “Leap Motion”. We see that users spend more time on moving hand on “Kinect-II” device than the “Leap Motion”. Since users need to wait for 2 seconds to trigger button click all the values for this specific gesture stands above 2 seconds.
Even if in some button placements they get close values, “Leap Motion’s” execution time performs 11.22% faster than “Kinect-II” for Wait and Select Gesture.
Fig. 3. Comparison of execution time for “Grab to select” Gesture on
Kinect II and “Leap Motion”
When we look at the Grab and Select Gesture on both devices the gesture of “Leap Motion’s” execution time performs 60.38% faster than “Kinect-II”.
In “Push to select” gesture type for both devices, “Leap Motion” performs 58.64% faster than the Kinect II in terms of execution time.
V. DISCUSSION
During this experiment, we measured the performance of known gesture types for “Kinect-II” and “Leap Motion”. After the tests on 10 random users, “Grab to select” gesture for “Leap Motion” gives best result at all locations for interacting on UI between all gesture types. Also, “Leap Motion’s” effectiveness is higher than “Kinect-II” in all gestures for controlling UI. We observed that “Wait to select” gesture for “Kinect-II” device performs slow if button placements are at the opposite side of hand that controls the “Kinect-II”. “Leap Motion” performs fast in all conditions but we see that users can make even faster executions if the placements are in the middle up, center and bottom of the screen. It is observed that Bottom corner sides for the placements of buttons reduces interaction effectiveness in all gesture types. This depends on “Kinect-II”’s and “Leap Motion’s” detectable range and correct user movements. In “Push to select” and “Grab to select” gestures, “Leap Motion” has top 3 fast execution times for the buttons placed at the middle up, center and the bottom of the screen.
In this study, we have investigated 3 gestures (“Wait to select”, “Grab to select”, “Push to Select”) which are widely used in “Kinect-II” and “Leap Motion” applications. The system can be used as guidance to determine the effectiveness of gestures and devices.
Results in Fig. 3 show us that, “Grab to select” gesture for “Leap Motion” performs the best in terms of execution time.
In the future work, we are planning to study more gestures which can be applied in natural user interface.