As smartphones' capabilities are more and more similar with those of computers, the demand for high-end multimedia content is increasing. In this paper we present two new visually rich mobile guide applications and then we compare them with the popular Google Maps and Nokia Maps applications. We present and analyze the results of an empirical study made with over 20 users in which we have compared these applications in the context of a game event at a big stadium. We have obtained direct feedback from participants as well as implicit feedback using a tracking system to analyze the interaction with the applications and infer activity metadata.
Over the past years we have seen an incredible growth of mobile applications, especially caused by the advances in technology from both sides: mobile manufacturers and mobile operators. Devices have many capabilities, many smartphones have become much more powerful than two-year old PCs.
A very important functionality brought by the advances in technology is the location capability of these devices. Today, a smartphone is not just able to show your agenda, or to run a small application, but it is also able to give your position (more or less accurate, depending if you are using GPS, Wi-Fi or GSM-based positioning [DR07, CK02]). It can analyze and react on your movements (using accelerometer), store the position of your car and show you the path to follow to return to it, share your photos/videos and other personal items with your family or friends and much more. The smartphone has become the digital personification of ourselves.
Applications and services that use the location capabilities of mobile devices are commonly called Location Based Services (LBS)[BKH08] and applications that use the context of the user (position can be included here together with profile, mood, orientation, social networks, etc...) are so called context-aware applications [MZ07]. A particular type of LBS and context-aware applications are the mobile guide applications [AAH+97]. These applications use context information (position, environment, user information, social network, personal interests, etc...) to present the user a map centered on his current position as well as activities (fan, cultural events, concerts) or places of interest (bars, restaurants, theaters, metro station) that are near the user. Popular examples are Google Maps [Mapa] and Nokia Maps [Mapb].
The advances in mobile technology (both network and devices) lead to a greater demand for better and richer content. Current mobile data-speeds exceed 10Mb/s (using HighSpeed Downlink Packet Access) and will continue to grow in the future as LTE (Long Term Evolution) [3GP] and WiMax [16006] become fully deployed. As for the hardware, we can already see GPU, GPS, Wi-Fi, dual core and 5 mega-pixel camera in a single device (e.g. Nokia N95 [N95]). It is clear that we can deliver great rich multimedia content to mobile users such as interactive 2D images and videos, interactive 3D and Augmented Reality [BC05, Nur06, HMB07].
We think the content on mobile guide applications can be greatly improved and we have created two applications to prove this. In order to check what is the user preference and what is the most popular visual content for mobile guiding applications we have made a user study. In this study we have compared our applications with two of the most popular mobile guide applications: Google Maps and Nokia Maps. We have also created a test framework in order to record the user interaction with these applications. We used this framework to analyze the activity of the user while using these applications. As it will be presented in the following sections, the recorded information is very important in order to get an implicit feedback on the usefulness and simplicity to use of the applications. The paper is organized as follows. Section 2 describes the motivation of our work. In Section 3 we present our framework and the applications tested. In Section 4 we detail the user study and we show and interpret the results. In Section 5 we summarize our work and discuss about future research. On big events like football games, before and after the match there are plenty of attractions and interesting activities around the main venue (the stadium in our case). Example of such activities are small football contests, car auctions or rotating cups (see Figure 1). These attractions can be very hard to observe and locate, especially due to the large number of people that participate to such events. Very often people don’t know how to locate these activities or they don’t know of their existence. In this paper we analyze the particular case of a football game at the TFC Stadium in Toulouse.
Mobile devices can be of great help in the task of locating and showing the interesting activities around a stadium.
Currently used mobile guide applications lack rich-media support to help users in visualizing these activities or locating them. Existing applications such as Google Maps or Nokia Maps don’t show at all these small activities around the stadium, and for those places of interest that are marked (like bars or restaurants near the stadium) there is generally a great level of interaction needed from the user side. Moreover, these commercial applications don’t offer enough visual content to the user and they make very little use of multimedia capabilities in current devices. However, future and beta versions of these applications show great evolution in terms of multimedia content on mobile devices.
We created two different prototype applications in order to expose the interesting activities around the stadium to the user. We show the position of these activities and we also use a rich visual content in order to let the user know better what are these activities about. The first prototype uses fake 3D images, simulating a 3D model because the user can move around the stadium and zoom in and out as if he/she were inside the stadium area. This prototype is based on our recent work regarding fake 3D images to guide users in stadiums [CCG08]. The second prototype uses video sequences recorded directly on site. Each video sequence represents an activity that is going on at the stadium on the occasion of a big game.
We want to test the four different applications (our two prototypes, Nokia Maps and Google Maps) in the context of our stadium. As it is not fair to compare the four applications using only one criteria (for example our prototypes have clearly more attractive POI around the stadium because we have created them), we have investigated multiple characteristics of these mobile guide applications, including the level of interaction, visual content, user input and application reaction. We have conducted an empirical study to test the applications, presented in Section 4.
In order to save the results from the test and analyze them further, we have created a test framework which logs all interactions between the user and the tested applications. Besides the implicit feedback obtained from the logged events we have also created a questionnaire (explicit feedback) that was given to each test user. This questionnaire contains multiple questions that are relevant to the usability of the application as well as to the background of the test user. We describe the test framework and the feedback questionnaires more in detail in the following sections. Using the results from both implicit and explicit feedback we are able to derive some interesting conclusions.
3.1
In this section we will present the mobile device, the four applications and the test framework used in our user evaluation.
For our research we are using a Nokia N95 device. This smartphone has multiple features, including A-GPS, GPU (graphic processing unit), dual processor (ARM11), Wi-Fi, 5 Mega-pixel camera, 240x320 screen resolution and the possibility to play MPEG4 video files. The N95 runs on the Symbian OS, version 9.2 and it provides a rich sets of APIs to access most of the phone’s features. We have used the native Symbian C++ to develop our prototypes. In order to get interesting results from the empirical evaluation we have created a test framework to log and analyze the interactions between the user and the tested applications. All tested applications contain the same points of interest around the TFC stadium (the common area tested in our evaluation). For each user we start a new session and then for each application tested we start a new application session. We send all input events (see Figure 3) from the mobile applications to our server in order to log the interaction and extract useful information on a per user and per application bases. We define an event as one simple interaction between the user and the mobile application (e.g. pressing a key). Normally the user tracking service works only with HTML pages : our platform includes a proxy that modifies the HTML page returned to the user in order to add some JavaScript listeners intended for tracking user interactions (mouse-click, mouse-move, key-input, etc...). In order to make our mobile applications work with the platform, we have created a bridge (Bridge) which listens to HTTP requests from the mobile application and then forwards them to the actual service (Observer and Database, see Figure 3). 3.3
The four applications that we have used for our tests are the two popular applications Google Maps and Nokia Maps and our two prototype applications using fake 3D images and video sequences. 3.3.1
Since its release in February 2005, Google Maps has been one of the most popular web guiding applications. Besides the web application, Google has created Google Maps for mobile devices. One of the most popular features of the mobile Google Maps is called My Location which can give the approximate location of the user by using only the information from the mobile network without any constraint on the GPS capabilities of the phone. The last version of the Google Maps for Symbian (2.03) also offers the Street View feature, allowing users to actually see real images from the places they choose. Although, this last feature is still limited to major cities in a few countries.
The Google Maps application interface can be seen in the Figure 4. The stars represent the points of interest -POI- (the activities around the stadium in our case). The user can move through the map using the navigation keys (up, down, left, right), zoom in and out using the 1 and 3 keys, select a POI using the middle key in order to see more information about it, see a list of the POI, center the map on the selected POI and switch between a simple 2D map or a satellite view. The position of the user (if available) is marked in the interface by a small blue dot. It is also possible to use the Street View feature to see images from the Point Pierre de Coubertin which is close to the stadium. (target on POI) (Street View) (Options Menu) Nokia Maps is developed by Nokia and it is one of the most popular mobile guide applications [Res]. Its success is mainly due to the fact that it is targeted only to mobile phones and its features and user interface have been thought especially for these devices. This application is only available for Nokia phones, running the Symbian OS and using the S60 or S40 user interface. The current version (2.0) only has 2D and satellite images, but the 3rd version [3.0] which is already available for some models running Symbian 9.3 or later (soon available on Symbian 9.2 and thus on the N95) allows a 3D navigation of the maps, including 3D landmarks (the TFC stadium has a coarse 3D model in these maps). In the Figure 5 we can see the user interface of Nokia Maps for both versions 2.0 and 3.0. The blue balloon-like icons represent our POI. The navigation through the map and selection of POI is done in a similar manner as in the Google Maps application. The main differences are in the map and POI interface, the way to display the information of the POI and the options of the application. The position of the user is presented by the rounded dashed rectangle. When the user is navigating through the map and the rectangle position is close to a POI its border line transforms to a continuous line. 3.3.3
Our first prototype uses a fake 3D model based on images rendered from a 3D model of the stadium. One of the main differences of this approach when comparing it to the 3D landmarks from Nokia Maps is that we use a fine and high quality model of the stadium. The application allows the user to navigate around the stadium using simple images and it gives the sensation of a realistic navigation in the stadium.
(perspective view) (perspective view) The interface of the application can be seen in Figure 6. The application presents a perspective view of the stadium around which the user can move using the left and right keys. On top of the image we have overlaid numbered icons representing the POI. When the user presses the key corresponding with the number on the icon a new image is presented to the user, containing a visual representation of the POI as well as a text containing more information regarding the activity or place. Using the and # keys of the phone the user can zoom in and out in the images in order to see better where each activity is exactly located. A similar approach of using images and text as advertisements can be seen on the Smart Rotuaari project [TJM+03]. 3.3.4
In the second prototype we use video sequences recorded from the stadium to show exactly what interesting activities or places exist around the stadium just before the actual game. As it can be seen in Figure 7 we have used a top view of the stadium taken from Google Earth to display the stadium area to the user. On top of this image we have added overlay icons representing the interesting activities and places. The user can zoom in or out and move around the top image to locate the position of the icons. When he/she presses the key corresponding to the icon a video sequence is played in the application, showing the real activity. The user has the possibility to adapt the volume using the volume keys of the N95 and he can stop playing the video using the middle key. The points of interest (POI) used in the four applications represent different activities that exist around the stadium with the occasion of a game (see Figure 1). We call these POIs advertisements interchangeable throughout the text. The same POIs are used in all the four applications, even if they are represented in different ways: default markers on Nokia Maps and Google Maps, images and videos in our prototypes. We use a tracking web service in order to get and analyze user interactions with the mobile applications. As shown in Figure 3, we were compelled to make a bridge in order to use the actual service. Nevertheless, whereas the user session identification is made automatically using the JavaScript listeners and the web service, it has to be handcrafted in the presented case : the generic listener on the mobile phone manages sessions and adds their ID to each request done to the server. 4
In this section we explain the organization and the results of our empirical evaluation. We did this user test to compare the four applications presented in order to learn what is the preferred multimedia content, what is the best user interface, what is the most time consuming application and to learn what is the level of interaction required by each application. 4.1
After preparing the applications (basic sanity checks) we made two kinds of tests (see Figure 8). Firstly, we made half of the tests with unknown people in a crowded area in down town Toulouse. A few days later we made another test with colleagues from our lab (IRIT) and students from our university (ENSEEIHT).
(Lab Test)
(Student Test) We needed an Internet connection to connect with our user tracking system. For the tests made in the lab we used our wireless LAN and for the tests done in the city and in the university we used the 3G connection of a mobile network.
For each test user we followed these steps: brief the user with an overview of the test, explain why we are doing it and present the four applications start a new test session so we know that all the following recorded events belong to this test user (we note the session number so we can compare the implicit feedback with the explicit one) present shortly each individual application and let the user test them. The order was: Google Maps, Nokia Maps, First Prototype and Second Prototype. This order was chosen arbitrary but it was kept for all the participants. give the questionnaire to the user to be completed collect the questionnaire and reward the user with a pen that has the institute label on it Each user tested each application for a maximum of two minutes. The scope of all the tests was to visit all the activities around the stadium within the two minutes by interacting with the application: moving the display screen around the stadium, zooming in and out, selecting POI, watching a video or using some special feature like StreetView on Google Maps. We have chosen the limit of two minutes based on preliminary experiments. We had a total of 21 users, 11 were women and 10 were men. 5 of them were young (15 to 20 years old), 11 adults (between 20 and 30 years) and 5 mature people (30 to 50 years old).
The questionnaire comprised 27 questions, organized in three categories: user reactions using the tested applications, technical problems and user background. The questionnaire usually took about five minutes to complete. 4.2
Using the information from the completed questionnaires we compiled multiple statistics. The charts of these statistics can be seen in Figure 9.
From chart (a) we can observe that the Fake 3D application had the best visual content, while Nokia Maps had the worst content. In chart (b) we see that people liked the interaction with the Google Maps application. Most of the people decided that these type of applications will not definitely make them visit the presented places but at least the information was useful and they liked the applications as they would recommend the application to others (see charts (c) and (e)). From chart (d) we see that men preferred the Google Maps application while women voted for the Fake 3D application. Chart (f) is also representative for the difference of perception between men and women as men remembered nine activities while women only five. 4.3
We made many statistics based on the recorded information. We had over 7000 user events recorded in our database and we created different scripts in order to obtain the different statistics. In Figure 10 we present the results of our implicit analysis.
Based on chart (a) we can observe that most of the users used the 2 minutes to test all the applications except the Prototype 2. On this application users only had to press a key to watch a video which was about 8 seconds long. This fast interaction allowed users to test the application in less than one minute. Chart (b) reveals one of the most interesting results from our research, the number of events divided by the number of visited advertisements in each application. This result tell us, in average, the number of events (key pressed) done by the user before actually seeing an advertisement (video, image, text, etc...). The Fake 3D application required the highest number of events before visiting a POI but this can be explained also because the application offered a simulated 3D view of the stadium and most of the users where interacting extensively with the application in order to see the exact location of the advertisements as well as the stadium itself. Chart (c) shows the average number of events done by users in each application while chart (d) shows the average time spent looking at the information from the different advertisements. As expected the Video application required the least amount of interaction while the time visiting the advertisements is the largest because each advertisement was represented by a video about 8 seconds long. Chart (f) confirms that users explored the visual content in the Fake 3D application as they have been zooming in and out extensively in order to see better the stadium and the position of the advertisements. 4.4 In this section we analyze the results from both implicit and explicit feedback and we try to observe the best and the worst in each application based on the user opinion. The fastest access to information was provided by Google Maps, as it can be seen in Figure 9g. It was also one of the preferred applications and the most easy to use application, despite the small problems in use caused by our logging system. Google Maps has provided a simple but attractive user interface and a fast access to POIs. Based on Figure 10f we can see that users didn’t have to zoom in or out to visit a POI and it was the application with the most visited advertisements (see Figure 10e). The StreetView feature of the application had a very positive impact as most of the users were amazed by this feature. Nokia maps was not able to perform in any category based on our results. Even if most of the functionality is similar to Google Maps the users preferred the later or the two prototypes in all the aspects presented. The main reasons for this were the lack of a visual image of the stadium and the latency in selecting POIs. Nokia Maps features a satellite view as well as Google Maps but its resolution is much lower and our stadium was not well visible. This is why we have used the 2D map which contained only a brown shape of the stadium. To select a POI users had to wait for a few seconds for the cursor to detect the POI under it and change its state. This was clearly a bad influence on the application usage. We are confident that the next version of Nokia Maps (3.0) which is already available for some devices will bring a much better user experience as it provides 3D landmarks and what we hope a better navigation.
The Fake 3D application (Prototype 1) has clearly the best visual content based on Figure 9a. This success is caused by the use of multiple images rendered from a nice 3D model of the stadium. Looking at Figures 10b and 10f we could think that this application involved too much interaction caused by the nice but possibly hard to use model. Contrary to this assumption, users actually liked this kind of content, fact proved by the result in Figure 9d. This means that mobile guide applications could benefit a lot from nice visually attractive models as users will enjoy them.
In the Video application (Prototype 2) we’ve tried to use video sequences as advertisements for two reasons: to reduce the interaction needed and to show real content directly from the stadium. Based on Figures 10a, 10b and 10c we can see that the degree of interaction is the lowest between the tested applications. Even if we expected that users will prefer this type of application, based on Figure 9 it seems that the Video application was not the best in any category, even if some users preferred this kind of application. Some people argued that the video sequences presented scenes with many people and this made the presented place hard or impossible to observe. Other comments refer to the impossibility to locate yourself or the interesting activity within the stadium while watching the video scenes. Based on comments and results we think that video and other type of rich multimedia content could greatly improve mobile guide applications but this content has to be made in a professional manner and it should present the POIs as clear as possible. In Table 1 we compare the answers to some of the questionnaire items between the lab and the public tests. Based on the results the only common opinion is the preference of a nice visual model to represent the environment and clear images and descriptions to represent the POIs. In this paper we have analyzed four different mobile guide applications: Google Maps, Nokia Maps and two prototypes created in our lab. We added some missing features of the two commercial applications into our prototypes: fake 3D images rendered from a 3D model and real video sequences.
We made an empirical evaluation, testing the applications within the lab but also in a public place with people not related with the lab. We monitored the user activity using a tracking system and we recorded all the actions of the users. After the test we have asked the users to complete a questionnaire.
Based on the results from both the questionnaires and the recorded data we were able to identify the good characteristics of each application as well as their weaknesses. Both Nokia and Google have already been making improvements in the multimedia content of their applications. The last version of Nokia Maps (3.0) contains 3D landmarks, including the TFC stadium that was used in our test and the last version of Google Maps (2.03) has included the Street View feature which allows the user to see real images from the city. The main drawback of these applications is the limited number of 3D landmarks in the case of Nokia Maps and the limited number of real images in the case of Google Maps. It is clear that users want richer multimedia content and commercial applications are already making the steps necessary to improve this very important part of mobile guide applications.
Next, we would like to concentrate on the context of the user, in order to show more relevant content according with the user preference. We want to take in consideration all the available context information, including position, time, profile and mood as well as personally selected preferences. Also we would like to research the mobile services that adapt themselves based on the environment conditions: noise, light, etc. On a close direction we also want to investigate the usefulness of augmented reality for mobile applications like mobile guides or museum applications. [16006] [3.0] [3GP] [BC05] [BKH08] [CCG08] [CK02] [DR07] [HMB07] [Mapa] [Mapb] [MZ07] [N95] [Nur06] [Res] [TJM+03]
IEEE Standard for Local and metropolitan area networks Part 16: Air Interface for Fixed and Mobile Broadband Wireless Access Systems. IEEE Std 802.16e and IEEE Std 802.16 Cor 1-2005, pages 1–822, 2006.
Nokia Maps Beta 3.0. http://www.nokia.com/betalabs/maps.
Long Term Evolution System Architecture 3GPP. http://www.3gpp.org/article/lte.
(a) (b) (c) (d) (e) (f)
(g) Figure 9: Statistics based on feedback questionnaires (a) (b) (c) (d) (e) (f) Figure 10: Statistics based on events logged using the user tracking system