Modern smartphones allow for gestural touchscreen and free-form user interaction such as swiping across the touchscreen or shaking the device. However, user acceptance of motion gestures in recommender systems have not been studied much. In this work, we investigated the usage of gestural interaction patterns for mobile recommender systems. We designed a prototype that implemented at least two input methods for each available function: standard on-screen buttons or menu options, and also a gestural interaction pattern. In a user study, we then compared what input method users would choose for a given function. Results showed that gesture usage depended on the specific task. In general, users preferred simpler gestures and rarely switched their input method for a function during the test.
Recommender systems recommend movies, restaurants or other
items to an active user based on ratings of items or other
information about users and items. Recently, the focus in
recommender systems research has been changing from
investigating algorithms to studying the user experience [
The aim of this project is to study gestural interaction patterns for mobile recommender systems on smartphones, such as swiping across the touchscreen, or shaking the device. The specific goal of the work described in this paper is to map recommender functions - such as initiating a search for recommended items or rating an item - to reasonable gesture and motion interaction patterns. We designed a prototype to allow comparing user interface options and conducted a user study to find out which interaction patterns users would select when given a choice.
Saffer [
Single Tap is a brief one-finger tap on the screen and used in virtually every application to interact with on-screen buttons and similar interface objects. Double Tap means to tap the screen twice in rapid succession with one finger. Pinch/Spread is a twofinger gesture. The user places two fingers on the screen and moves them together (Pinch) or away from each other (Spread). This is most commonly used for zooming in (Spread) and out (Pinch). One-Finger-Hold Pinch is a more complex two-finger gesture. In this case, one finger rests on the screen, while a second finger moves on the screen to adjust a slider or other numerical value, for example.
Slide means to move a single finger over the screen in a continuous motion. Slide is generally used for dragging objects like sliders and slowly scrolling through views exceeding the screen’s dimensions. Fling is a quick, long movement of one finger in one direction and can also be used for quickly scrolling through list views. Flick (or Swipe) is a shorter gesture similar to the longer Fling and commonly used as Swipe-To-Delete in file systems: a Flick gesture performed on an item generally deletes this item from a list. Another usage is moving to the next screen, resembling turning pages in a book. Shake Device and Tilt Device (along x, y or z axis) are free-form motion gestures with no screen interaction required.
Technically, any touch pattern can be drawn on the screen using one or more fingers, e.g. a rectangular pattern. However, finding the balance between gesture detection precise enough to distinguish different patterns, and vague enough to allow for user errors when drawing the patterns is difficult. In addition, explaining complex patterns to the user is challenging and therefore, complex patterns are rarely used in mobile applications.
Previous research on the usage of gestures in mobile scenarios
focused on the user acceptance of motion gestures in general and
hardly applied these techniques for the interaction with
recommender systems. In own previous work, we designed a
minimalistic user interface for a map-based recommender based
on gestural interaction, but for the larger screens of tablets [
We implemented the prototype application for Android 2.2 (Froyo) and tested it on a Google Nexus One smartphone with Android. The goal of the test application was to provide different input methods for functions typically found in recommender systems to test which interaction patterns the user would chose in the successive study. The selection of functions in our application is not really specific to mobile recommenders and considers recommenders in a wider sense, i.e. taking also "search" applications into account. The scenario for the prototype is a movie search and recommendation application that resembles the Internet Movie Database (IMDb) mobile application (see http://www.imdb.com/apps).
We provided at least two different input methods for each application function, either • • • on-screen buttons, menu options (the user has to select a specific "menu" option1 to show additional buttons), or gestural interface options (cf. Section 2.1).
The next subsection describes considerations for mapping gestures to application-specific functions.
Single Tap is commonly used for interaction with on-screen interface objects and should not be used for other applicationspecific purposes. The same applies to Slide and Fling for scrolling screens or dragging objects. Contrariwise, Double Tap is not bound to any standard features and thus application-specific features can be mapped to it. As Pinch/Spread is generally used for zooming, mapping it to other application features may be confusing as well. However, the One-Finger-Hold Pinch (OFHP) variation of this gesture is applied in our application.
Since no screen interaction is necessary for the free-form gesture Shake Device, this gesture may be used independently from any interface restrictions, for example for application-wide functions. An application-wide function can be called at any time, regardless of the current screen of the application, e.g. the "home" button on most mobile systems. Functions depending on viewing items onscreen may not be viable for use with Shake Device, since shaking the screen makes focusing on displayed objects on the screen harder. The nature of the other motion gesture, Tilt Device, suggests either a use for simple actions like a binary +/- rating (making use of the left-right or front-back movements of Tilt Device), or for any navigation function along two or three axes. Tilt Device is not applied in our test application, because the application does not use binary ratings.
In the test application, the user can use a search interface to select among movie genres and find items. The search interface can be reached from the start screen, main menu or through the options menu. After searching, a list of corresponding items is shown (Fig. 2, left). Users can scroll up and down the list, remove items from the list or select an item to display more details by using Single Tap. The item details screen (Fig. 2, right) shows information for the selected movie and allows for bookmarking and rating the item. In addition, an options menu is available on every screen to return to the search screen or main menu of the application (Fig. 2, right). The following functions are available and implemented by at least two input options each: • • • •
Bookmark: The user can bookmark an item by using onscreen or options menu buttons (Fig. 2, right), or by using the Double Tap gesture in the item details screen Find Random Item: Accessible application-wide through the options menu or by using the Shake Device gesture Save Search Parameters: This function is available in the search screen via an on-screen button or by a Double Tap in this screen Find Similar: The item details screen shows three movies similar to the selected one ("similar to this 1 On most systems, a dedicated software or hardware button opens up the options menu movie" part in Fig. 2, right). The user has the option to find more similar items by using an on-screen button or the Flick gesture Exclude Item: Available in the list view as an on-screen button (Fig. 2, left) or via the Flick gesture Rate Item: Users can rate items in the item details screen by selecting the "Rate" on-screen button (Fig. 2, right). Then, a rating scale of 1 to 10 stars appears. The user can set his or her desired rating by either using the rating scale as an on-screen button or applying the OneFinger-Hold Pinch (cf. Section 2.1) gesture.
We have conducted a user study to find out what input method for a given function is preferred by the test users. The evaluation was performed with each of the participants individually. To start, each user was given an explanation of the application and was then allowed to practice navigating the different functions and input methods for about ten minutes. The participants then had to perform a set of 18 instructions in the application in a certain order. The list mentioned the required tasks only; the input method to perform them was not specified. By doing so, we tested which input method the test persons found more intuitive to use for a certain task. The beginning of the sequence of instructions read as follows: (1) Find Random Item, (2) Find Similar Item, (3) Rate Item, (4) Open Main Menu, (5) Open My Recommendations, (6) Exclude Item from Recommendations, and so on. Some of the requested functions appeared several times in the list, for example Find Random Item was requested three times. This was used to test whether participants would change their preferred input method for a particular function during the experiment. We recorded every user action in a log file. After a test user completed the scenario, he or she had to fill out a survey concerning his or her opinions about the input methods for the requested instructions and about the handling of the gestures in particular.
16 persons with mixed backgrounds participated in the study. Other than a few users skipping a few tasks from the instruction list, all subjects completed the given scenario. We first analyzed the log file to understand which input options the users chose to complete a given task.
Out of a total of 44 recorded usages, the Find Random function was initiated 26 times using the Shake Device gesture, and 18 times using the options menu button (see Fig. 2, right). This represents a 59.1% usage rate for the implemented gesture. Interestingly, only one out of the 16 users elected to use both available input methods; every other user exclusively used either the gesture or the button for the three instances of Find Random in our instruction list.
The Bookmark Item function is represented three times in the scenario. The users chose to use the Double Tap gesture 27 out of 46 times (58.7%). However, at one instance in the scenario, the activity in focus is the item list, which only implements bookmarking via double tapping. In this case, 11 of 16 users (68.8%) chose the Double Tap gesture, while the rest of the users elected to take additional time to first open an item’s details page and bookmark there. While the users were on an item’s details page, they called only 16 of 35 (45.7%) instances of Bookmark Item using the Double Tap gesture. All differences to 100% in this paragraph are due to the uses of the on-screen bookmark button – the options menu button was never used.
The use of the Save Search Parameters function was requested only once in the scenario and can be called using Double Tap or an on-screen button. This is the function with the clearest favorite among the input methods: 15 out of 16 users (93.8%) chose the on-screen button.
The scenario contained two instances of the Exclude Recommended Item function, operable via Flick gesture or an onscreen button. 18 of 32 (56.3%) calls were made using gestural interaction. A relatively high number of users used both input methods for this task: 4 out of 16 participants (25%). This is even though the two instances of the Exclude Recommended Item task occurred directly after each other in our task list.
Rate Item and Find Similar Item each occur two times in the scenario. For both, a clear preference towards the standard input method of an on-screen button can be seen: for Rate Item, only 10 of 32 instances (31.3%) were operated with the One-Finger-Hold Pinch gesture. Even more one-sided, the Find Similar Item function was only initiated using Flick in 3 of 32 cases (9.4%). The remaining percentages represent instances of functions called via on-screen button.
In the first part of the survey we asked the participants how intuitive they find the input methods for the six functions on a scale from 1 to 5. Figure 3 illustrates the results with a higher number meaning "more intuitive". In general, the results correspond to the log file very well: input methods that were actually preferred and used by the participants received higher grades for intuitivity. For example, the participants find the onscreen buttons for Save Search and Find Similar very intuitive. On the other hand, the Shake Device for Find Random Item, Double Tap for Bookmark and Flick for Exclude Item gestures received higher grades in comparison with on-screen or option menu buttons.
The next question was whether inclusion of an on-screen button was worth the necessary screen space for it. Our users mostly were in favor of it: the majority of users denied this question for Exclude Item only (Fig. 4). Interestingly, this is the only on-screen button in the list view (Fig. 2). The goal of the next part of the survey was to determine the user’s favorite input method for each function. The distribution of choices for each function is shown in Fig. 5 and is comparable to the grades for intuitivity: interaction patterns that users perceived as intuitive were chosen as favorite input method. We also asked the test users about their prior experience with touchscreen devices and analyzed whether it would relate to differences in the results. The most significant difference was that 62.5% of the users with more prior touchscreen experience rated the Shake Device gesture as intuitive, while only 12.5% did so among the users with less experience. We noted a similar difference regarding the Flick gesture.
Concerning the ease of handling of the four gestures, the participants considered all gestures, except One-Finger-Hold Pinch (OFHP), as easy to handle in general. One of the problems with OFHP was that lifting a finger while adjusting the desired rating for item ends the rating process. In addition, the calibration for the rating scale of one to ten stars was difficult. So this gesture might be more suitable for simpler tasks with fewer options.
The results of the study presented in this work may be used to improve the design of user interfaces for mobile recommender systems and other similar applications. Our study showed that users preferred the simpler, easier to handle gestures over the more complex ones. Complex gestures like One-Finger-Hold Pinch must be carefully calibrated for ease of handling. Omitting on-screen buttons is only an option in activities where content space is rare, in our case the overview list of items. For the item detail screen, simply touching a button was the favorite input method most of the times. The options menu was not very popular in any of the used cases. This is likely due to the fact that opening the options menu is an extra effort that users do not tend to make when other input methods are available.
While Double Tap for bookmarking items was received very well, the Double Tap gesture for Save Search Parameters was not very popular and received low grades for intuitivity. This may be due to the layout of the corresponding screens because users might have the fear of accidently tapping on other interface elements. In essence, the use of gestural interaction patterns seems to depend on the actual screen and function detail. Interestingly, users did not change their preferred input mode much during the test: they mostly used the same method for the same task throughout the scenario. Users with more experience with touchscreen devices were more open towards gestures than users with less experience. Future work includes studying in more detail how more complex gestures can be introduced in mobile recommender systems to improve user interaction. Moreover, a long-term study would be interesting because user acceptance might change if smartphone users get more and more used to complex motion gestures.