=Paper=
{{Paper
|id=Vol-1307/paper4
|storemode=property
|title=Identifying Approaches to Usability Evaluation
|pdfUrl=https://ceur-ws.org/Vol-1307/paper4.pdf
|volume=Vol-1307
|dblpUrl=https://dblp.org/rec/conf/gsr/KoletsisCC14
}}
==Identifying Approaches to Usability Evaluation==
https://ceur-ws.org/Vol-1307/paper4.pdf
GSR_3
Geospatial Science Research 3.
School of Mathematical and Geospatial Science, RMIT University
December 2014
Identifying Approaches to Usability Evaluation
Ms. Erin Koletsis, Prof. William Cartwright and Prof. Nicholas Chrisman
RMIT University Australia
Email: erin.koletsis@student.rmit.edu.au
Abstract
Maps are ubiquitous, and created by people of all skill levels. However, many users still struggle to effectively
interpret spatial information. Such users have been defined as ‘functionally map illiterate’ (Clarke 2003). This
inability, or perhaps, more accurately, disability, to interpret spatial information can lead to negative
experiences. This can be of great concern when these users need to undertake navigational tasks such as way-
finding. As a result, map users may experience anything ranging from unnecessary anxiety to sometimes tragic
consequences, due to them being geographically disoriented or completely unaware of their location. To
improve the map reading experience for inefficient map readers, it is argued that maps need to be created that
allow them to interpret spatial information more easily and effectively.
Various methods have been used to evaluate maps for their potential to correctly convey spatial information to
specific user groups. It is therefore necessary to determine which methods are the most applicable for evaluating
maps created for functionally map illiterate individuals. We note that there are certain problems related to all
usability testing situations. Here, we believe that familiarity of landscape needs to be eliminated in order for
testing to be effective. This paper reports on early findings to identify possible approaches for evaluating this
type of map for this user group.
Keywords: usability; map reading; navigation
1. Introduction
It has been identified that many map users experience anxiety when attempting to read maps for way-finding
purposes (Khazravi & Karimipour 2012). These users are unable to complete such tasks and as a result, cannot
navigate and way-find to the fullest potential. This paper reports initial findings of research that seeks to
determine a new type of map or map-related object which has the potential to eliminate the ambiguities of many
map designs which may cause problems for everyday users of geospatial information communication artefacts.
The research is generally focused on users who cannot effectively interpret spatial information from existing
maps available to them, such people have been labelled ‘functionally map illiterate’ (Clarke 2003). It seeks to
determine the most appropriate course of action to determine the most effective design.
By creating a new map or alternative map-related objects, the gap between those who can use maps effectively
and those who cannot may decrease. This would afford more people to more easily and efficiently complete map
reading tasks and navigate using maps or map-related objects. Such tasks would not be daunting or associated
with negative feelings.
A potential problem identified is that any map evaluation tests undertaken to determine the usability of an
alternative map or map-related object must be done in an environment unfamiliar to the participant (user). The
results from tests undertaken in environments that are familiar to the participant cannot be regarded to be
accurate, since the user’s dependency on the map can vary or they may not need to use the map at all. To be
deemed useful, the map needs to prove to be effective for navigating and way-finding in territories unfamiliar to
users.
To determine the best approach to adopt for this research, a number of approaches were identified and compared
for their ability for use as an effective approach for evaluating map usability of general use maps.
2. Approaches for Evaluating Map Design
After completing a thorough search, twenty-one relevant papers and other documents were reviewed to identify
the approaches currently in use. The papers were chosen based on their focus on specific user groups, the
presence of an evaluation technique and/or the creation of a new product. Whilst reviewing literature of
population and/or purpose specific maps (The term ‘population specific’ here refers to users with a common
need. In the context of the overall research project, a map is to be made which will focus on the specific
population of map users who are functionally map illiterate) a variety of approaches were identified that could be
employed in this research. As well, similarities in the actual processes used in evaluations were identified. Of
the papers reviewed, all except the statistical and interactive maps, involve the evaluation of maps used for
navigation. This is due to the overall goal of the research project which aims to create a map for the purpose of
navigation. Therefore, the review of statistical and interactive maps were undertaken to gauge the differences in
usability evaluations between maps of these types and maps for navigation. The remaining reviews have been
categorised by either their relation to a specific user group, or specific map type, since not all maps created for
the purpose for navigation are engaged with in the same way, under the same conditions or in the same
environment. Furthermore, some of the maps listed could fall in to several of the categories thus it needs to be
clarified that the maps have been categorised based on their primary focus.
The focus of the approaches reviewed was on maps that have been created for specific populations and/or
purposes. This was decided so that the approaches found were standardised, as it is thought that user specific
maps would have the most potential as a test vehicle. The aspects of map usability evaluation identified in the
approaches include:
• Use of human test subjects representative of the end user;
• Think aloud protocols;
• Questionnaires;
• Focus Groups;
• Participant feedback/formal and informal interviews;
• Completion of map reading tasks;
• Use of real and simulated environments; and
• Statistical analyses for interpretation of results.
The specific map user groups addressed in the methodologies reported in papers reviewed include: users of
nautical maps (Porathe 2006), users of pedestrian maps (Delikostidis 2011; Ishikawa et al. 2008; Crampton
1992), users of You-Are-Here (YAH) maps (Marquez, Oman and Liu 2004; Klippel, Freska and Winter 2006),
helicopter pilots (Harwood and Wickens 1991), electronic navigation display users (Lavie and Oron-Gilad 2011;
Yeh and Chandra 2006; Hsu, Lin and Chao 2012 ), blind and visually impaired users (Kostopoulos et al. 2007;
Rice et al. 2005; D’Atri et al. 2008; Heuten et al. 2008), users of statistical maps (Pickle 2003) and users of
interactive maps (Andrienko et al. 2002).
The following section provides summaries of the methodologies found.
2.1 General Case
Looking generally at mapmaking and map use, Suchan and Brewer (2000) and Board (1978) recommend using a
qualitative method to form the basis of research, and evaluating findings using a quantitative method. Citing
examples, Suchan and Brewer provide reasons for questionnaires, interviews, focus groups, and verbal protocols
being highly beneficial in this type of research situation. In the 1970s there was high interest in studying how
paper maps work, hence Board’s diagram refers to the evaluation of paper maps. Board suggests a number of
map reading tasks that should be performed when analysing the effectiveness of a map. He states that the map
reading tasks should be dependent upon the intended use of the map and that the evaluation must utilise an
empirical approach. A number of questions are provided for which the researcher should answer before
conducting any experiment, in an attempt to effectively determine the most appropriate map reading tasks. Here,
the questions asked are What type of map? For whom is the map intended? Under what conditions will it be
used?, What map reading tasks are the most appropriate for the purpose?
These questions facilitate creative thinking and by addressing them, the researcher can maximise the return of
meaningful and useful results. The process of determining the appropriate map reading tasks are also provided in
a diagrammatic form (Figure 1).
Figure 1 - Procedure for identifying appropriate experimental map reading tasks (Board 1978)
It is only after establishing the map reading tasks that Board (op. cit.) recommends deciding on the types of
questions to be asked of participants. Upon the completion of data collection Board (op. cit.) advises that
appropriate methods of ‘scoring’ must be implemented. These methods take the form of a statistical analysis.
Whilst there have been great technological advances and therefore new ways of interacting with and reading
maps, Board’s proposed process remains valid since it provides a foundation for developing tasks for evaluating
map usability. Since the development of Board’s diagram, the use of computers has become ubiquitous and as a
result, the focus of evaluating maps has expanded to include digital web maps on computer screens, smartphones
and tablets. Fortunately, Board’s process can accommodate other map display formats. The questions Board
poses in his diagram are still relevant and can be taken in to consideration when developing methodologies for
map evaluations regardless of the platform used.
2.2 Nautical Maps
Maps and charts have almost always been used as navigational tools. Their formats vary from paper to digital
and they can be used in air, sea and land navigation. Here, an example of a new sea navigation system is given to
provide some insight into the types of map designs that may be evaluated.
Porathe (2006) investigated how to reduce disorientation at sea and thus make sea navigation and travel safer. To
do this, a software system was custom designed and tested participants for their ability to complete navigational
tasks in a simulated environment. He collected further data in the form of interviews and questionnaires and
analysed the results using statistical analyses. The results of the initial experiments were used to develop
prototype navigation systems that were then tested in a real-world environment.
2.3 Pedestrian Maps
Pedestrian maps are one of the most commonly used maps in our society. Therefore, there is a great need for
them to be useful and user-friendly. With the introduction of smartphones, almost everyone can access
pedestrian maps. This section will discuss three studies that have involved the evaluation of pedestrian maps or
user skills using such maps.
Delikostidis (2011) identified a number of issues with pedestrian maps accessible by smart phone devices. In an
effort to eradicate these issues, Delikostidis developed a prototype pedestrian navigation system. The researcher
recruited twenty four human test subjects who were chosen based on their unfamiliarity with the pre-selected
testing locations. Evaluation of the navigation system involved engaging participants in navigation and map
reading tasks in two locations that differed in their spatial layout. The participants were required to complete the
navigation and map reading tasks whilst being video and audio recorded, using the new navigation system, as
well as the Google Maps interface. The researcher had the participants use think-aloud protocols to determine
any difficulties the participants may experience. The interfaces were alternated between the two testing
locations, using only one interface per location by each individual. The participants were given cue cards that
provided scenarios for them to undertake. These scenarios included navigating between two locations,
identifying their own location, reorienting the display, landmark identification and searching for nearby transport
access points. Once the map reading activities had been completed, the researcher conducted semi-structured
post-session interviews where he was able to gain more information about issues that arose during the
experiment. The qualitative data collected was analysed using research software Atlas.ti. For quantitative data,
graphs were created for comparisons between the usability of the prototype system and the Google Maps
interface.
A comparison of usability of a GPS-based navigation system, paper maps and direct-experience was conducted
by Ishikawa et al. (2008) to determine which method was the most effective for pedestrian way-finding. The
researchers split participants in to three groups depending upon the three formats being tested. The participants
were required to complete a way-finding task and to determine where their start point was once they had found a
‘goal’ location. The participants provided feedback relating to the perceived difficulty of the way-finding task.
From this study, it was found that the participants using the GPS-based navigation system took the longest to
complete the tasks and had less ability to determine the direction of their starting location.
A study conducted by Crampton (1992) analysed the differences between novice and experienced way-finders.
The participants were required to look at a map and decide how they would get between two specified locations.
From here they were required to act as if they were at the actual location the map depicted and used a ‘think
aloud’ protocol to document their journey. By collecting participants’ verbal statements the author was able to
create behavioural graphs and morphograms. The behavioural graphs represented the ‘structure of the subjects’
problem-solving processes and the morphograms were used to plot the ‘number of morphological references in
the protocol’. In doing so, the author was able to gain insight in to how the participants went about solving way-
finding problems. The creation of behavioural graphs and morphograms allowed the researchers to identify the
most commonly used and also the most effective strategies that participants used when way-finding.
2.4 You-Are-Here Maps
Historically, Marquez, Oman and Liu (2004) investigated the use of You-Are-Here (YAH) maps on board the
International Space Station (ISS) (Zhu et al. 2011). This was done due to the fact that astronauts were often
disoriented and had trouble navigating within the ISS. The map designers identified key landmarks that
astronauts would most likely use to determine their location and added these elements to the map. The usability
of the map was not tested; instead suggestions were made as to how it should be tested. These suggestions
included using human test subjects to describe how they would find their way from their current location to
another using the map. In another study, Klippel, Freksa & Winter (2006) evaluated pre-existing YAH maps to
establish whether their clarity was effective enough to find their way inside a building, and thus be able to exit a
building efficiently in an emergency. The researchers obtained three YAH maps and assessed them against a
number of criteria: completeness; perceptibility; semantic clarity; ambiguity; consistency; placement;
correspondence; alignment; architectural cues; and the effectiveness of the YAH symbol. From this research it
was determined that a number of elements of a typical YAH map require improvement in order for them to be
effective in the event of a building evacuation.
Those with smartphone devices would be well aware of users’ ability to determine location instantly through the
use of the maps loaded on to the device. Currently, in the case of Google Maps, a blue dot with an arrow shows
the user their location and orientation. The need for users to always know their location and have the map
egocentrically designed is obvious. This technology, is not however, fail safe. The blue dot showing the user
location can often be in an incorrect position, especially when the user is on the move. Furthermore, it has been
discussed that presenting a map in an egocentric reference frame is not the best method to employ for all way-
finding tasks (Harwood and Wickens 1991; Porathe 2006) and, as a result, it has been proposed that users should
be able to switch between ego and exocentric reference frames depending on the task and the user’s preference.
2.5 Maps For Helicopter Pilots
To determine how to better create and design maps for helicopter pilots, Harwood and Wickens (1991)
conducted a series of experiments with twenty helicopter pilots. These pilots completed a series of tasks using
two different map displays (north-up and track-up). The researchers evaluated the pilots’ abilities to complete
the tasks effectively. The tasks were completed in a simulation environment, on landscapes unfamiliar to the
participants, thereby ensuring no prior knowledge of the location. Following this exercise, the pilots were asked
what they thought of the map displays. The researchers found no benefit in one map display over the other
however they did determine the need for creating a ‘configurable map display’ that the pilots can use to change
to suit their needs or preferences.
2.6 Electronic Map Displays
When analysing the usability of electronic map displays, Lavie and Oron-Gilad (2011) tested subjects for their
ability to effectively complete navigation tasks. They also assessed users for their perception of map display
usability. The electronic map display was evaluated for its usability in a driving situation. The users were
required to complete navigational tasks using the electronic navigation display in a simulated and unfamiliar
environment. A number of different maps were analysed and the results compiled using an ANOVA (Analysis of
Variance) (Sprinthall 2012). These researchers state that simply testing for actual usability or perceived usability
on their own is not sufficient and that both need to be tested for in order to fully determine map usability. This
research was carried out on twenty participants, with the authors stating that for more reliable results, more
participants would be required.
Hsu, Lin and Chao (2012) conducted a series of experiments to determine the effects of difference maps on
driving performance. The study engaged human test subjects who were required to complete a series of
navigation tasks in an unfamiliar real environment using 2D and 3D electronic maps, as well as a paper map.
The time it took for the participants to complete the tasks and the routes they took were recorded and analysed
using a statistical analysis. In a similar study, Rhodes and Gugerty (2012) conducted experiments to establish the
effects of electronic map display and individual ability differences on users’ navigation performance. In this
study, participants used an unmanned aerial vehicle (UAV) simulator to perform ‘route-following, map
reconstruction, and cardinal direction judgement’ tasks with track-up and north-up maps. The participants were
required to follow a predetermined path and answer a series of questions relating to the tasks completed. After
each ‘mission’ the participants completed the ‘NASA Task Load Index workload measure’ (Hart 2006) and then
attempted to sketch a map of the locations of each of the landmarks they came across on their mission.
Following from this, participants completed spatial tests and questionnaires. The results of the data collected
were analysed using a series of statistical analyses appropriate to each task.
After identifying variations in the symbols used on electronic map displays used by aviation pilots, Yeh and
Chandra (2006) set out to standardise these symbols in an effort to minimise ambiguity. The study involved
experienced pilots who were required to determine the meaning of a series of symbols. The pilots’ responses
were used to ascertain which meanings were most commonly associated with particular symbols. From this, the
researchers were able to begin designing a standard intended to be distributed amongst electronic map display
producers, to ensure all displays comprised of consistent symbology.
2.7 Audio And Haptic Maps
Blind and visually impaired users bring about a whole new aspect of navigating tools. Due to their lack of sight,
audio and haptic/tactile maps are commonly used to assist these users in way-finding. In two of the methods
reviewed, people who fell into this category were also engaged as the human test subjects (Kostopoulos et al.
2007; Rice et al. 2005). Kostopoulos et al. (2007) had users explore the map with exocentric reference frames
within a virtual environment. This reference frame was chosen due to the fact that better results could be
obtained in navigation and object recognition tasks. The paper written by Rice et al. (2005) discusses the ‘Haptic
Soundscapes’ research project. This project involved the use of “nearly one hundred” human testing subjects and
‘the development of cartographic interfaces that use auditory and haptic cues” (Rice et al. 2005, p. 381). The
researchers engaged the participants to use existing interfaces with their own added element, ‘a haptic frame’,
which guided the user and informed them of their location on the screen. They recorded and tracked the
movements of the user’s mouse across the computer screen and collected comments from the participants on
their thoughts about the interface. By collecting data in this way, they gained an insight into the way the
participants actually used the interface and also what their feelings towards it were. Using this information
allowed the researchers to effectively evaluate the actual usability and the perceived usability of the interface.
In a contrasting study, D’Atri et al. (2008) engaged mainly experts to test their proposed navigation system. The
experts undertook the experiments to initially identify the aspects of the system that required most modification
and the following experiment was conducted using one blind participant. By using a blind participant, the final
experiment was conducted so as to evaluate the true usability of the navigation system, since the participant
reflected the proposed type of end user of this map type.
Audio and haptic navigation methods have also been tested for use by both visually impaired and users with
normal eyesight (Heuten et al. 2008). In their study, Heuten et al. engaged visually able users to assess the
usability of a tactile belt. Experiments were conducted in both controlled and uncontrolled environments and
required the user to state the angle they felt the belt was indicating. The authors of the study note that they
regretfully did not record the participants’ feedback whilst undertaking the experiments.
2.8 Statistical Maps
In researching the most effective way to display geo-referenced statistical data, Pickle (2003) devised a set of
recommendations for this type of map. These recommendations were determined through the use of human
subjects. Participants formed focus groups to discuss preferences of various aspects of map design and also
completed a series of map reading tasks. Seven aspects (which had previously been identified) were analysed in
these experiments. Each aspect was discussed and tested individually. After testing the participants, the result
having the highest rate of usability was selected.
2.9 Way-Finding In Virtual Environments
The strategies of users when way-finding in a large scale virtual world were analysed by Darken and Sibert
(1996). In this research, the authors had their participants complete several way-finding and navigational tasks,
whilst speaking their thoughts aloud. In doing so, the researchers were able to collect data about the behaviours
of the participants when completing tasks. The routes/paths that the participants took to complete the tasks were
also recorded and analysed. The approach for this research was a combination of qualitative data collecting
followed by an analysis of the data using a quantitative method, in the form of an ANOVA. Completing an
ANOVA allowed the researchers to collate and analyse their data in an easy manner and to derive meaningful
results.
2.10 Interactive Maps
In 2002 when interactive maps were still a relatively new concept, Andrienko et al. (2002) assessed the usability
of tools in a particular GIS. These tools allowed the user to engage and interact with the data being displayed.
The main goals of this research were to determine how well users could understand the tools, whether or not
users could remember how to use them after a period of time and if the users would like the tools after becoming
familiar with them. The researchers first created a prototype that was tested for flaws by prospective users and
usability experts and remedied accordingly. Five interactive techniques were chosen for testing; outlier removal,
visual comparison, dynamic classification, dynamic query and dynamic linking. Three phases of testing were
carried out, all of which involved human test subjects. The first two phases took place in situ and were carried
out on the same group on two separate occasions. The first phase consisted of nine participants. Of those nine,
only six participated in the second phase. The group was required to undertake the same tasks for each phase.
The third phase involved collecting data via email. A larger number of participants were involved in the final
testing phase due to the remote collection of data. The phase three group were also required to complete the
same tasks as in phases one and two. The data collected contained information regarding the successful
completion of tasks, the ability to answer questions correctly and the participants’ feelings towards the usability
of the system. The data was analysed using statistical analyses, where the results of each phase were compared
against one another and results within phases compared between each another.
3. Comparison of Approaches
Figure 2 - Comparison of Approaches
The table above (Figure 2) provides a visual overview of the methods that are most commonly used in the
usability evaluation of different map types. From this table, it can be seen that maps used for navigation
encompass the greatest variety of methods and that the use of human test subjects, the completion of map
reading tasks, and the use of simulated environments are the most commonly used elements of the papers
reviewed.
4. Approaches Applicable to this Research
Of the methodologies reviewed, those that could be applied to this specific research project include:
• Human test subjects completing map reading tasks in conjunction with think aloud protocols;
• Human test subjects completing map reading tasks in conjunction with questionnaires;
• Human test subjects completing map reading tasks in simulated or real environments;
• Engaging human test subjects in interviews; and
• Analysing results data using statistical analyses.
Using this review, a rudimentary proposed usability evaluation procedure has been developed for the evaluation
of the new map design for functionally map illiterate individuals:
1. Source/develop appropriate unfamiliar environments for experiments.
2. Recruit functionally map illiterate participants (Using Santa Barbara Sense of Directions Scale) (Auger
and Maguire 2013) and invite to join a focus group for the evaluations.
3. Undertake heuristic evaluations – part 1.
4. Conduct simple navigational and way-finding activities with participants using ‘think-aloud’ protocols.
(Suchan and Brewer 2000).
5. Conduct map reading activities that involve feature identification and mental map replication.
6. Undertake heuristic evaluations – part 2.
7. Conduct post-test interviews.
8. Analyse results using statistical analysis.
9. Reflect, refine and undertake a number of iterations, where necessary.
Whilst still at an early stage of development, this proposed method utilises all five applicable methods that were
identified in the review process. It allows for the analysis of perceived usability versus actual usability through
the use of heuristic evaluations and the completion of map reading activities.
It also provides solutions to the problem of testing in environments familiar to the participants. In the papers
reviewed, the issue of familiar environments was solved by locating a real world environment that no
participants had prior experience in or by testing in simulated environments. A number of authors (Harwood
and Wickens 1991; Crampton 1992; Darken and Sibert 1996; Rice et al. 2005; Porathe 2006; D’Atri et al. 2007;
Kostopoulos et al. 2007; Lavie and Oron-Gilad 2011; Rhodes and Gugerty 2012) included virtual or simulated
testing environments. Such environments allow for greater control of geographic elements and can be purpose
built allowing for customisation. The possibility of customisation provides a solution for the problem of
familiarity of landscape, because an unfamiliar environment can be created. Virtual or simulated environments
could be used for initial testing of a product before validating the results in a real world environment (Porathe
2006). For the best results in using a virtual environment it must be as close to reality as possible. The Oculus
Rift (Oculus VR 2014) is an immersive technology that allows for 360° head tracking, a stereoscopic 3D view
and an ultra-wide field of view. Oculus VR (2014, para. 2) state that this technology allows the user to
‘seamlessly look around the virtual world just as [they] would in real life.’ Technologies such as this provide
inexpensive alternatives to extensive real world testing.
5. Conclusion
This paper identifies the various approaches available for evaluating user and/or purpose-specific maps and
further highlights which approaches are applicable to the overall goal of the research project. It also provides
potential solutions to the issue of testing participants in familiar environments.
The next step in this research will involve identifying alternative communication technologies. This information
will be used in conjunction with the approaches to evaluating usability as outlined in this paper, to develop a
precise methodology for evaluating the new map that is yet to be produced.
6. References
Andrienko, N, Andrienko, G, Voss, H, Bernardo, F, Hipolito, J & Kretchmer U 2002, ‘Testing the Usability of
Interactive Maps in CommonGIS’, Cartography and Geographic Information Science, vol. 29, no. 4, pp.325-342.
Board, C 1978, 'Map reading tasks appropriate in experimental studies in cartographic communication',
Cartographica: The International Journal for Geographic Information and Geovisualization, vol. 15, no. 1, pp. 1-
12.
Clarke, D 2003, 'Are you functionally map literate', paper presented to Cartographic renaissance. Proceedings
21st International Cartographic Conference (ICC 2003), Durban, South Africa.
Crampton, J 1992, 'A cognitive analysis of wayfinding expertise', Cartographica, vol. 29, no. 3-4, pp. 46-65.
D'Atri, E, Medaglia, CM, Serbanati, A & Ceipidor, UB 2007, 'A system to aid blind people in the mobility: A
usability test and its results', paper presented to Systems, 2007. ICONS'07. Second International Conference on.
Delikostidis, I 2011, 'Improving The Usability of Pedestrian Navigation Systems', University of Twente.
Hart, SG 2006, 'NASA-task load index (NASA-TLX); 20 years later', paper presented to Proceedings of the
Human Factors and Ergonomics Society Annual Meeting.
Harwood, K & Wickens, CD 1991, 'Frames of reference for helicopter electronic maps: The relevance of spatial
cognition and componential analysis', The International Journal of Aviation Psychology, vol. 1, no. 1, pp. 5-23.
Heuten, W, Henze, N, Boll, S & Pielot, M 2008, 'Tactile wayfinder: a non-visual support system for wayfinding',
paper presented to Proceedings of the 5th Nordic conference on Human-computer interaction: building bridges.
Hsu, SH, Lin, CH & Chao, CJ 2012, 'The effects of different navigation maps on driving performance', Percept
Mot Skills, vol. 115, no. 2, pp. 403-14.
Ishikawa, T, Fujiwara, H, Imai, O & Okabe, A 2008, 'Wayfinding with a GPS-based mobile navigation system:
A comparison with maps and direct experience', Journal of Environmental Psychology, vol. 28, no. 1, pp. 74-82.
Khazravi, A & Karimipour, F 2012, 'Cognitive Readability Enhancing of Cartographic Maps for Pedestrian
Navigation', International Journal of Brain and Cognitive Sciences, vol. 1, no. 3, pp. 11-7
Klippel, A, Freksa, C & Winter, S 2006, 'You‐are‐here maps in emergencies–the danger of getting lost', Journal
of Spatial Science, vol. 51, no. 1, pp. 117-31.
Kostopoulos, K, Moustakas, K, Tzovaras, D, Nikolakis, G, Thillou, C & Gosselin, B 2007, 'Haptic access to
conventional 2D maps for the visually impaired', Journal on Multimodal User Interfaces, vol. 1, no. 2, pp. 13-9.
Lavie, T & Oron-Gilad, T 2013, 'Perceptions of electronic navigation displays', Behaviour & Information
Technology, vol. 32, no. 8, pp. 800-23.
Marquez, JJ, Oman, CM & Liu, AM 2004, 'You-are-here maps for international space station: Approach and
guidelines', SAE Technical Papers.
Montello, DR 2010, 'You are where? The function and frustration of you-are-here (YAH) maps', Spatial
Cognition & Computation, vol. 10, no. 2-3, pp. 94-104.
Oculus VR 2014, Oculus Rift Next-Gen Virtual Reality, Oculus VR, viewed 12 November 2014,
Pickle, LW 2003, 'Usability testing of map designs', paper presented to Proceedings of Symposium on the
Interface of Computing Science and Statistics.
Porathe, T 2006, ‘3-D Nautical Charts and Safe Navigation’, PhD Thesis, Mälardalen University, Västerås,
Sweden
Rice, M, Jacobson, RD, Golledge, RG & Jones, D 2005, 'Design considerations for haptic and auditory map
interfaces', Cartography and Geographic Information Science, vol. 32, no. 4, pp. 381-91.
Rhodes, W & Gugerty, L 2012, 'Effects of electronic map displays and individual differences in ability on
navigation performance', Human Factors: The Journal of the Human Factors and Ergonomics Society, vol. 54,
no. 4, pp. 589-99.
Sprinthall, RC 2012, Basic statistical analysis, 9th ed. edn, Pearson Allyn & Bacon, Boston.
Suchan, TA & Brewer, CA 2000, 'Qualitative methods for research on mapmaking and map use', The
Professional Geographer, vol. 52, no. 1, pp. 145-54.
Yeh, M & Chandra, D 2006, 'Pilot stereotypes for navigation symbols on electronic displays', paper presented to
Proceedings of the International Conference on Human-Computer Interaction in Aeronautics.
Zhu, L, Yao, Y, Xu, P & Bian, Z 2011, 'Study on space station design elements for intra-vehicular navigation: A
survey', paper presented to Electronic and Mechanical Engineering and Information Technology (EMEIT), 2011
International Conference on.