One of the signi cant challenges facing the development of web portals is optimising the usability of such e-infrastructures for diverse, yet specialised uses. This paper presents a usability framework for supporting the iterative testing and development of the Australian Urban Research Infrastructure Network (AURIN). The AURIN portal serves as a securitised environment for data sharing, and as a repository of open-source analytical routines to support urban decision-making. Harmonisation, security, accessibility and discoverability are critical concerns at the data level as many di erent kinds of federated data are being referenced by the system. Furthermore, urban researchers are accessing this infrastructure framework across Australia. A broad spectrum of digital data relevant to human settlements needs to be not only discoverable, but also comprehensible and usable by non-experts in the area. Managing the complexity across many areas and supporting a spectrum of nave to expert users are signi cant challenges. In this research we introduce a usability framework, which elicits feedback from nave users through to domain experts. This feedback is captured through handson workshops with participants and one-on-one user testing sessions. In this paper we report some of the preliminary ndings from early workshops with end-users and some of the one on one testing sessions. We analyse this feedback to understand the current strengths and weaknesses of the system. The end-user feedback provides a dual purpose. Firstly, it is fed-back to the technical development team as part of a cycle of continuous improvement. Secondly, the workshops also provide a vehicle for engaging end-users and raising awareness of the AURIN data search, interrogation and visualisation capabilities. We conclude the paper by discussing the next steps in developing and implementing the usability framework in relation to the AURIN portal.
User-centric design is of critical importance to design of web-based portals and to ensure success as indicated
by the end-users returning (Tatnall 2005). The AURIN portal provides access to data and tools to support
Copyright c by the paper's authors. Copying permitted only for private and academic purposes.
evidence-based decision-making in the context of urban settlements in Australia. In concert these data and
analytical tools provide the backbone to sophisticated Spatial Decision Support System (SDSS) capability. SDSS
are essentially iterative, computer based systems designed to support decision-makers in solving semi-structured
spatial problems
An identifying feature of a Decision Support System (DSS) is the employment of information systems to process
data into a form that assists knowledge sharing and communication between experts and non-experts, informing
a community of users (decision-makers) focussed on solving a common problem
Typically embedded within SDSS are expert tools and models, which aim to be e ective in processing data into
information that is comprehensible by non-experts in a particular domain. For example, urban planning is a eld
where researchers, planners, policy-makers and communities need to be involved in the decision-making process.
When this information is combined, a better understanding of the structure of the problem and potential
solutions is attainable. The e ectiveness of these tools can be compromised by discipline-speci c approaches that
only highlight one aspect of the greater challenge. As such, SDSSs that support multi-disciplinary collaborative
approaches to solving urban issues need to be designed with accessibility in mind across a range of experience
and expertise to best allow diverse (and often divergent) approaches to be combined as seamlessly as possible.
The usability of such a system's front-end is of pivotal importance in determining the uptake and e ective
deployment of a SDSS. This paper examines the reiterative testing and development cycle of an online SDSS known
as the AURIN portal, which is being developed to support researchers focused on solving problems pertaining
to the urban settlements across Australia. The AURIN portal provides a gateway to 100s of geospatial datasets
and a suite of analytical and visualisation tools to support this broad user community. Areas of endeavour
span across social demographics, housing supply and demand, transport, population health and well-being and
innovative urban design applications. These discipline-speci c clusters are referred to as 'AURIN lenses' as they
conceptually focus on a particular area of national signi cance and provide an interface between the technical
development, the expert community and end-user engagement
The challenges presented by urban areas increasing in both size and density, coupled with greater demands on
infrastructure, socio-economic policy and sustainable design and planning have given rise to many innovative
solutions internationally as to how to best manage these areas. Many groups have formed around the concept
of 'smart cities'
There is a general direction for governments internationally to move toward an open data sharing model
Typically SDSS are designed by experts for experts; for instance, Planning Support Systems (PSS) aim in
engaging an end-user community (in this case planners) in the use of computer assisted decision support tools
After an initial period of technical development, AURIN has begun a series of workshops, user surveys and
interviews to introduce the system to end-users from a range of urban disciplines. As part of this engagement
process, detailed user testing is being undertaken by a small group of typical end-users. This is to achieve
a better understanding of how people are interacting with the system and, importantly, to provide feedback
to the technical development team to address usability and functionality issues identi ed by the group. The
AURIN development process follows the agile method
Respondents were predominantly involved in urban research at a university level. Half of the respondents were University Academic (Teaching and/or Research) and a third were University Post-graduate students (PhD or Masters). The remainder were involved in Local Government. A factored gap analysis of the self-assessed level of expertise showed the user groups to rank themselves as more adept in urban issue than technical skills, with one-third of users classifying themselves as 'expert-level' and another third as 'inexperienced'. This is to be expected as the workshops were run in-conjunction with two conferences the State of Australian Cities (SOAC) (2 workshops) and The Australia New Zealand Regional Science Association International (ANZRSAI) (1 workshop) and attracted a participants interested in urban issues and ranging from student-level to senior academics.
Between beta releases, it is important to gather as much information as possible to identify usability issues and feed these back into the development process. AURIN is using several ongoing instruments to gauge the performance of the system as a whole:
Figure 2 illustrates the temporal framework of the user testing process. Beta 3 was the rst public release inviting user feedback, the system having been developed internally only for the rst two beta releases. Preceding each subsequent beta release is a User Advisory Group (UAG) session followed by a series of workshops. The process employs screen capture software and an unguided speak-out-loud interview process when conducted in a usability lab. Feedback received by these instruments is fed-back into the development process via the existing bug reporting and feature request framework.
Workshops are conducted between beta releases to give participants hands-on experience in a collaborative environment. These typically last 3 hours. At the conclusion of these workshops, participants are surveyed to gauge usability, con dence, focus areas/scales, self-assessed discipline and technical experience (via Likert scales) and open-ended questions prompting good points, bad points and suggestions for future development.1 The UAG has been established to test pre-release functions and provide feedback on the usability and functionality of the system at each beta phase of the portals development. The UAG comprises of professional users from the urban research community, which spans the disciplines encompassed through the AURIN Lenses to form a small group of invited participants for each beta testing phase. The membership of this group may change depending on the functionality associated with a particular Beta release.
Individual sessions have been undertaken with an individual users and at least one AURIN member to take notes and help in session facilitation. Each session lasts 1.5-2 hours. The sessions are tailored to speci cally address the new functionality planned for a particular Beta release. Each user session consists of ve components: 1. Blue sky: Users would convey their understanding of what the AURIN portal is and what it potentially could provide. 2. Unguided test: Without any assistance, participants are asked to log into the AURIN portal and a simple exercise such as preparing a chart. 3. Guided test: Any di culties encountered in pt.2 would be addressed and the user guided through a more di cult task with the support of user manual and tutorial documentation where available. 4. Collaborative brainstorming of features and functionality: discussing user stories describing what/how a user might use data/tool/visualisation resources with basic interaction ows and desired outputs. 5. Debrie ng: Finishing this process the user will be given the survey with additional questions focussing on issues of access, interface and task based interactions.
AURIN has undertaken a parallel survey as part of a Monitoring and Evaluation process harmonised with the beta releases to gauge the perceptions of the broader user community, particularly stakeholders that are involved in the broader project who have not had any speci c training in the operation of the system. 4 4.1
In December 2013, three workshops were facilitated by AURIN introducing a total of 24 participants to the AURIN portal. Participants were invited to ll out a paper survey on completion of each workshop. In total 15 responses were collected. 4.1.1
Respondents indicated an interest in all scales of urban research ranging from an individual to a National level. Predominantly, 60% of respondents indicated and interest in the precinct/neighbourhood level, with 43% indicating interest in cities and towns, and 30% indicating interest at building scale. 70% of respondents were interested in Urban Housing, and nearly 60% interested in Urban Transport, although no responses were indicated for Urban Logistics. 4.1.2
Respondents indicated their main interest was in data download (4.36 on the Likert scale) and Data search and discovery (4.31) while Analytical tools indicated the weakest demand (3.67). This result could con rm that the portal is primarily seen as a data acquisition tool, however still registering a demand for more specialised analytical routines. Only 4 respondents (26%) indicated that they were already familiar with the Australian Urban Research Infrastructure network (AURIN) project prior to using the AURIN portal, however, respondents strongly agreed that the information on the AURIN portal is valuable and that they were likely to visit the AURIN portal in the future. A score of only 3 was registered by respondents indicating that they were neutral as to being able to nd what they need quickly on the AURIN portal. This points towards issues associated with usability of the portal that need to be considered. 4.2
Over the period of a month concluding mid-September 2013, Six (6) individuals were interviewed, the rst two interviews used the Beta2 release and the subsequent four used Beta3. Participants were from academia (3) and Local Government (2) and State Government (1). This initial series of interviews was aimed at gleaning detailed user feedback from a small number of users in order to identify common issues at an early stage. It is important to note that each of the participants had been to an introductory AURIN workshop, or had been recommended by someone who had attended. Consequently, these users had a professional interest in retrieving, analysing and presenting urban information. The interviews were conducted at participants desks in order to identify any access issues via rewalls, software, hardware or bandwidth constraints. Participants were asked to 'Think out loud' to register their perceptions as two AURIN representatives notated responses.
The unguided testing phase replicated the experience of a user logging into the AURIN portal, navigating the interface, loading data and creating a choropleth map without any guidance beyond what was accessible online. Testing revealed that the users all had speci c eld of interest and were motivated to retrieve data speci c from their respective elds- these broadly covered statistical demography, urban planning/con ict issues, urban transport analysis and well-being/social development issues. Once successfully logged in, each of the users were able to explore, nd and retrieve relevant data and all were able to create a choropleth map within 30 minutes.
Common issues in unguided tests were revealed: at the log-on phase when users needed to correctly enter user names and passwords. If a password had been forgotten, a lengthy process of retrieval and resetting ensued- in one case up to 54 minutes before successful login. This poses a signi cant barrier to accessing the system with ease; at the interface interaction stage of trying to decipher the portal-speci c iconography and work ow. Often users saw the system as being analogous to similar systems they had had experience with, however most were reluctant to refer to manuals or help documentation, preferring to intuitively navigate through the system; when users followed what appeared to be an intuitive work ow, goals were only revealed when the entire process was carried out (for example, interactive brushing), or if an incorrect path was followed, error messages were not revealed until several steps into the process. Users could not modify input parameters after loading; this was identi ed as an important part of re ning a trial-and-error process. During the collaborative brainstorming and de-brie ng stage, users volunteered techniques to remedy problems or improve e ective use of the portal including: re ning the front end for intuitive interaction for rst-time users, or with an interpretive layer to guide the user; avoiding custodian-speci c acronyms, codes or jargon in preference of verbose, human readable script, with subsequent opportunities for revealing speci c detail; pre-built use-cases, parameterised queries and industry-standard reports accessible via themes grouped around discipline; programmatic access for query and retrieval to bypass the portal.
Users trusted/had no concerns over the curation or reliability of data, and recognized the federated data structure to be a timesaving and trust-worthy feature of the portal. Users liked the fact that federated data was straight from the source, and not derived or static. This also created perceived opportunities for participating in forming and adding to the data network itself.
Diverging opinions were revealed between users- some saw it mostly as a data discovery and retrieval tools, while others saw value in the ability generate a visualisation layer over existing datasets. This is a usability issue that warrants further investigation. 4.3
During October 2013, a series of surveys were conducted from a total sample of 106 AURIN stakeholders as part of an ongoing Monitoring and Evaluation process. Stakeholders are represented by the project's communitydevelopers, project leaders, senior researchers and collaborators. Table 1 shows the responses to open-ended questions prompting respondents for three good points ('good'), three bad points ('bad') and three points for improvement ('todo'). These were clustered around common key themes.
The rst round of user testing has been an important step in empirically identifying and developing a pro le of the system's users. Broadly, users are interested in urban research; this group are mostly members of the academic research community, and the government sector. Their skills range from nave to expert. This suggests a focus on developing the user experience aspects of the portal to accommodate rst-time users, as well as o ering guides to more advanced features.
The usability framework and approach will be extended for progressive beta releases to focus in more delity
on the details of user interaction using a dedicated usability lab where usability testing can be undertaken with
a triangulation of approaches including Speak-out-loud, video recording and pre/post questionnaires. Usability
inspection through
Subsequent User Advisory Groups will invite a larger sample of participants, rst to con rm and elaborate on established personas, and secondly, to continue identifying issues encountered in the user experience. The cycle of testing at the beta phases enables newly implemented features to be tested, and further related feature requests to be registered. The nal UAG meeting and testing will represent an end point to gauge the project's progress and best ensure the continuity of the project into another phase of development.
The usability of portals and advanced SDSS tools has been identi ed as a major barrier for the use of such tools
Overall, the response from the feedback acquired through the usability framework has been positive, with endusers generally citing AURIN as a valuable initiative lling a gap. Whilst AURIN is currently at the Beta3 phase, more work is required to 'harden' the portal. This will in turn enable a viable comparative analysis to similar portal products that will further assist in positioning and improving this online SDSS. Accessibility, ease of use, functional robustness and an outcome-based focus are crucial factors to the overall usability of the AURIN portal and ultimately its success.
AURIN is an Australian Government project conducted as part of the Super Science initiative and nanced by the Education Investment Fund. The University of Melbourne has been appointed the lead agent by the Commonwealth of Australia, Department of Innovation, Industry, Science and Research.
AURIN would like to thank members of the User Advisory Group, Workshop attendees and survey respondents.
Tatnall, A. (2005). Web portals: the new gateways to Internet information and services. Igi Global.