In this paper, we aim to analyse the current level of usability on ten popular online websites utilising some kind of reputation system. The conducted heuristic and expert evaluations reveal a number of deficiencies on the overall usability of these websites, but especially on how the reputation information is currently presented. The low level of usability has direct consequences on how accessible and understandable the reputation information is to the user. We also conducted user studies, consisting of test tasks and interviews, on two websites utilising reputation information. The results suggest why the currently provided information remains under-utilised and, to a great extent, goes undetected or gets misinterpreted. On basis of the work so far, we propose ways to overcome some of the current problems by changing, rearranging and grouping of the visual elements and visual layout of the reputation information offered on the sites. The enhanced visualisations create “visual nudges” by enhancing the key elements in order to make users notice and use the information available for better and more informed decisions. .
As Internet services and peer-to-peer systems currently are
lacking in the traditional indicators of trustworthiness [
Reputation systems have grown into a prominent means to gather and provide such information about the quality of the offering and its seller for the end user. A reputation system 2Nokia
P.O.Box 407 00045 Nokia Group, Finland +358 71 800 8000
olli.immonen@nokia.com
operates by computing reputation scores for some set of objects,
such as services or items on sale, within a certain community or
domain. The scores can typically be computed on basis of a
collection of opinions – usually ratings – that other entities hold
about the objects, by employing a reputation algorithm to
calculate reputation scores based on the received ratings, which
are then published. Reputation information typically represents
users’ opinions about a particular product, service or peers [
Reputation information can be textual (e.g. descriptions,
reviews) or visual (e.g. images, symbols, statistical
visualisations), or, usually, a combination of the two. However,
currently the reputation information is often presented in such a
way that may make it hard to notice and to interpret. To make
things worse, according to our heuristic and expert evaluations,
the overall level of usability on the sites offering reputation
information is often bad enough to stop users from effectively
having the reputation information at their disposal, as it goes
undetected: if the user cannot find the functionality, the
functionality is not really there [
Which parts of the reputation information is presented visually
needs to be carefully selected: Our user studies [
To further investigate the described issues we have evaluated ten more websites of different categories (news, shopping, social networking etc.) that employ some kind of reputation system.
The main objective of the usability evaluations was to evaluate
the current level of usability of these services, and how well the
standard set of heuristics from Nielsen [
In the expert evaluations, we were focusing on the reputation information and how it is visualised in order to understand what works, what fails and how things could be improved.
As the visual prominence seems key for better utilisation of the reputation information, we introduce the idea of visual nudging for improving the usage and production of reputation Copyright © 2010 for the individual papers by the papers' authors. Copying permitted only for private and academic purposes.
This volume is published and copyrighted by its editors: Knijnenburg, B.P., Schmidt-Thieme, L., Bollen, D.
information to enable better and more informed
decisionmaking. “Nudging”, a term introduced by Thaler et al as a way
to enhance decision-making [
We will first present the background for the current study, the previously conducted user studies together with the earlier work done in this area. We will then proceed with the usability evaluations for the additional websites and discuss the findings.
We will conclude by summarising the lessons learned on what kind of usability issues we currently see as most pressing on the websites utilising reputation systems, and how they could be improved on, especially focusing on the key role of the visual elements and their prominence for the overall usability of such websites.
Reputation information is typically presented by both visual and textual means.
Currently, the most common way to present visual reputation information is to use star symbols to represent the current rating of the item under scrutiny (Figure 1). Other symbolic icons commonly used for visual reputation information include “thumbs up” or “thumbs down” and a scale consisting of circle symbols (Figure 2).
Most common representations of reputation information are
used to communicate the popularity rate of the product or
service based on users’ votes. Usually, the user is able to see the
amount of votes given describing the popularity or how much
the product is “liked”. However, this information is not
revealing the scale of the information, and the user may be left
with confusion: What is the difference between three or four
stars? How many stars a good product usually gets? How many
ratings can be considered “a lot of ratings” in this service?
Because of this ambiguity, the quality of the reputation
information is experienced as questionable: What do the ratings
actually mean (to me)? How credible are the ratings? How are
the ratings calculated? For the users, the transparency of the
information [
Possibly, partly due to all of these problems in the visually
presented reputation information, the textual information is
currently considered more important for the users: Reliance on
peer reviews has become everyday news. For example,
USAToday has recently reported the growing importance of
peer reviews, stating that “customers are increasingly vocalising
their experiences online for other travelers to read” [
Online reviews have indeed become increasingly popular as a
way to judge the quality of various products and services
[
In the context of downloading, trust and risk perception also
become an issue. For the online user, the perceived credibility of
a website or a service has a strong impact on the trust level and
risk perception [
In our earlier work [
In the study for the WidSets website [
As a direct continuation of the WidSets study, we conducted
another study focusing on Ovi and how the online reputation
information currently offered in Ovi is understood and utilised
by its users [
Our results again showed that the reputation information
available was not efficiently utilised. According to our
interpretation, the lack of cohesion between the reputation
elements hinders the understandability and use of the
information available. Users also reported that they found the
credibility and quality of the reputation information to be
questionable, which may be the result of the inconsistent and
ambiguous way of presenting the information. Users were
currently not able to find the relevant information and thus also
not able to form an overall view or an understanding about the
content and the message of the reputation information.
Based on the results from these studies we suggested [
The previous studies showed that there is a lack of visual prominence and cohesion between the different reputation elements, and the reputation information was under-utilised. The findings led to the formulation of the following hypotheses: • • • • •
The websites offering reputation information had problems with usability; More specifically, the reputation information provided has bad usability; Visual prominence of the reputation elements is guiding the decision-making process on these sites; The visually prominent elements on the websites are “wrong”; Visual nudging is not working on the websites to enhance the decision-making process.
The basic research question behind the study is: “Why is the reputation information underutilised?” By addressing this research question, and armed with an initial understanding about the importance of the visual elements, we aimed at analysing how the reputation information is currently displayed across the selected sites.
Among the various methods available in the field of Human
Computer Interaction (HCI), heuristic evaluation based on
Nielsen’s heuristics [
Heuristic evaluation is a form of usability inspection where
usability specialists or other evaluators judge how the object of
study, e.g. a website, passes on an itemised list of established
usability heuristics [
The websites chosen for the usability evaluation were wellknown sites, and selected on basis of their general popularity1: 1http://www.google.com/adplanner/static/top1000/#, http://www.alexa.com/topsites, • • • • • • • • •
eBay (shopping), www.ebay.com
www.tripadvisor.com and vacation reviews),
either with
AppStore (Apple’s store for iPhone applications). www.apple.com/iphone/apps-for-iphone/ The evaluations were performed by four evaluators: one senior HCI expert (> 10 years of experience), 2 expert (>2 years of experience) and one non-expert (< 1 year of experience). The expert evaluation focused on how the reputation information was presented on the selected sites.
Table 1 summarises the outcomes of the usability evaluations against Nielsen’s heuristics. We will now present the findings of the expert evaluations on the reputation information website by website, focusing on the main findings. The findings are marked (negative) or
(positive).
The different pieces of information are presented similarly, as if having the same value (e.g. product details and important information). This makes retrieving information for the decision-making a hard task. (Figure 3). The website presents the rating’s information through a chart with detailed information about how many users rated the item and how, as well as a direct access to their reviews.
eBay
Information about the overall purpose of the website is hard to find even when registering (statement of purpose).
The user cannot sort other users' reviews about a seller by any other category except “date”, the default category. In case a seller has both positive and negative reviews, the user will have to scroll through all the reviews to find the negative ones. This might be very time-consuming (Figure 4).
Both the ratings about the seller and the way the feedback is calculated are clearly presented to the user.
The visualisation of the rating system is ambiguous. A novice user might be confused by the two different ways of showing the ratings 1) thumbs and 2) circles. The actual meaning of the symbols becomes clear only by the time the user writes a review: thumbs are associated with a separate question "would you recommend this to a friend?" (Figure 5); circles represent the rating.
The number of reviews is not consistent. The addition of all the ratings provides a number, which is different than the one presented along with the written reviews and still different from the one obtained when the user clicks the "clear filters" option. This might jeopardise trust in the reputation system.
Information provided is not clear. For example the rating information provided for hotels consists of three different ratings (Figure 6).
The different elements of information are presented as having the same value, and without a clear structure to guide the user, which makes retrieving information a time consuming task.
The target of the reputation and the reputation elements were not easily distinguishable.
While reading the reviews, the user can see the reviewer profile with just a mouse hover, which provides an easy access to the information, prevents the disruption of the task and adds quality to the user experience.
The UI does not provide a clear guidance of what are the goals of the website, how it should be used and what is the order of importance of the content. This information is hidden behind an unnoticeable link, which makes it hard for the novice user to detect.
The users' own recommendations are listed, enabling comparison between recommendations, and adding transparency to the system.
After having rated a video as negative or positive, the user is not allowed to undo the action. This adds unreliability to the system especially as it is possible to click on the rating accidentally.
User is not allowed to delete a video previously rated as "Liked" from the "liked videos" view (Figure 7). The only actions allowed are adding it to a playlist or to a list of favorites. In order to delete a video previously rated as "liked" the user has to perform too many steps. First, the user has to open the "liked videos" view, add the selected video to a playlist or to favorites and only then remove the video. This is time consuming and counter intuitive as the user has to perform a contradictory operation – “add to favorites” - to the one they actually intend to perform.
The system does not provide a confirmation or an option to undo the action of reporting another user. This might generate unreliability in the reputation information as users can report and be reported by accident.
There is specific statistical information about the history, popularity and spread of the videos, which contributes to the transparency of the website.
Information provided under "views" shows a detailed pictorial and statistical representation of activity frequency over time and per location.
The users have access to the amount of reviews for a specific place but cannot see the relationship between other reviewed places. Even if all the reviews are positive and the place has a certain number of stars it does not provide information about its quality when compared to other places in the same area.
After rating a review as useful, funny or cool, the user is provided with feedback and the number of ratings is immediately updated, which evokes reliability in the system.
The system provides the option to undo the ratings to other users' reviews, which allows the user to correct potential mistakes and adds more trustworthiness to the ratings.
The website provides a graphical and clear explanation of ratings and ratings over time. It clearly details how the overall ratings are obtained.
The basic review contains plenty of information about the reviewers’ reputation, making the relevant information immediately available to the user and the reputation of the review itself can also be seen.
By presenting diverse information about the reviewed target and the reviewer community on the first page the website guides the novice users and keeps their interest in exploring the website.
The main page does not provide information about what is “Digg” or how it works. The lack of directions might make the novice user confused about the purpose of the website.
Advertisements were presented as having the same value as the information the user was looking for.
The system does not allow the user to delete a previously provided comment.
The scale of the “Top” is ambiguous. The user is not able to distinguish the timeframe of the “tops” and might get confused.
When clicking the icon corresponding to the number of “diggs”, the user is directed to a page presenting the comments. This is counter-intuitive since the user expects to see a list related to the number of “diggs”, instead of the comments regarding the news. The “how many diggs”- icon is the most prominent element of the page, hence it should provide the expected information.
After digging an article the system provides good feedback and updates the results immediately, which contributes to the overall reliability of the system.
The site enables users to evaluate one another’s comments, which might contribute to establish or strengthen the community feeling.
If the user rates the same movie more than once the system provides a feedback message saying the vote was counted, which might be misleading.
The user profile, accessed through the username link, only contains a list of the reviews that the user has made. The more informative user profile is accessible through an additional link on the page presenting the users’ reviews. This jeopardises the system’s consistency.
The reputation information and the links to reputation information are presented among the general information about the movie. The information is mainly presented in the form of text. The first link on the page dedicated to the reviews is blended among the general textual information and the links, which requires an extra effort from the user in order to find relevant information and differentiate between different types of information provided.
User cannot distinguish the relationships between popularity and rating of the movies. The info button on MOVIEmeter (question mark) gives some additional information but does not resolve the issue as the users may have a hard time understanding how the percentages are formed and how to interpret them.
The website provides detailed user ratings, and allows the user to access information about the voting trends for specific categories.
The website uses weighted average for unbiased ratings, which eliminates the ratings that are only intended to change the overall rating in their benefit, adding reliability to the reputation information.
The website also provides links to external reviews, which contributes for the feeling of transparency.
Information elements and advertisements are hard to tear apart. The small boxes of information and advertisements create a cluttered look for the UI and the vertical page structure does not support a natural flow of information retrieval.
The "recommend" icon does not provide clear information about if the user is recommending the other member or their posts. This might affect the results, in case the users do not understand what is recommended (Figure 8).
The website provides a guidance pop-up window for novice users as a starting page, which gives immediate information about the purpose and usage of the website.
The website provides detailed and clear information about getting promotion by points and an explanation about the meaning of the user ranking.
The members are given points according to different categories of posts. This motivates contribution as it might be seen as recognition.
The ranking status of the members, based on their individual points, is presented visually and in a clear way.
An option to read more information in the reviews - expand text – is provided, but the user cannot go back to the condensed text, which can make the page cluttered.
The site does not offer access to more details about the star ratings or all customer reviews unless the user uses the iTunes software to view applications.
The user has no information about the way the ratings are formed except for the fact that they are based on the reviews.
The user can easily sort the reviews by several categories that are provided on the left column. This adds efficiency and transparency to the presented information, as the user is able to easily find both positive and negative reviews.
The website provides a list of accessories rated and suggested by staff, which makes it easy for a first time user to navigate through what is available in the store.
When user clicks on a product, all information is provided in three sections – 1) a description with snapshots, 2) ratings and reviews by users and 3) Q&A section, with questions asked and answered by other users. This provides a complete and detailed overview of the products, contributing for transparency.
The website offers visibility for the developer, which may enhance both the willingness to contribute and the trustworthiness of the contributions.
A general problem found in most of the analysed websites was a cluttered UI and the fact that the all available information was presented in a similar fashion as if having the same value, which may cause confusion and mislead the user: The nudge to look at information that is relevant is missing. The elements available are presented in a way that does not guide the users’ attention to the relevant information while making decisions. Another main problem was related with the lack of interrelation between the different reputation elements. This has a negative effect on the information credibility provided by these elements. It may also affect the users’ willingness to contribute as it is unclear how the contribution will affect the offering.
On basis of the usability evaluations, the current level of usability on the studied websites has general usability problems that are big enough to jeopardise the use of the sites altogether. Moreover, when it comes to how reputation information is currently offered, the level of usability can be described as remarkably low. Improvements in distinguishing and understanding different types of information available and visual nudges for how they should be utilised by the user in the decision-making process can easily be suggested: • • •
Clearly distinguish between distinct sources of information: the service provider, the reputation system, advertisements, other users and what is actually meaningful – highlight the relevant information and guide the users task-flow; Tie together the different instances of reputation information to form a coherent set of information where different elements support each other; Promote transparency: clearly show where the reputation information comes from and how it is formed.
There are also social aspects related to understanding, or
accepting the information. The results of our earlier studies and
those by others have indicated that reputation information
available in textual format, in form of peer reviews in writing,
has a big importance in online decision-making [
Some ways to take all the above-mentioned aspects into account
and enhance the utilisation of all reputation elements conjointly
is likely to include creating visually prominent, real-time links
between the users. When users are exposed to appropriate
amount of social data about one another, it tends to increase the
activity of giving contributions [