In this paper, we report on a usability study of the IntrospectiveViews interface for visualizing semantic user models in social networks. In the first part of this paper, we describe how the interface can be used in a social network for visualizing and editing ontology-based user interest models. In the second part, we describe the study we conducted to evaluate the usability of this visualization. We describe the results of four evaluation methods: task-based experiment, eye tracking, interviewing, and a questionnaire. The contributions of this paper are three-fold: First, it describes an application of the IntrospectiveViews interface to user model visualization in social networks. Second, it describes and exemplifies a methodology to usability evaluation of interactive visualizations of user models. Third, it reports on the outcomes of such an evaluation: It identifies what visual techniques and interaction patterns users deem to be usable and attractive for working with models of their interests, but it also reports on the crucial usability problems of the interface and describes possible solutions to these problems. We believe that the latter two contributions to be of interest for researchers working on similar interfaces for visualization of user models.
Nowadays, a number of social networks personalize their content and navigation based on the information about individual users and groups. Personalization effects in a social network can take various forms. This can be, for example, the recommendation to connect to other users you may know or jobs you may be interested in. In addition to that, the personalization can take the form of content filtering or sorting. For instance, to help the user to keep updated with the activities of her close friends, the friends’ posts can be sorted or filtered based on the user’s individual interests – the posts of friends she is most interested in will appear more prominently than the ones from friends the user does not have close contact to. User models play an essential role for achieving such personalization effects. In a social network, they can provide such information as what friends and groups the user is most interested in, what games and applications she likes, and what her general interests are (e.g., meeting new people, keeping in touch with classmates, job search, and so on). Some of this information can be explicitly provided by the user, whereas some is inferred by the system automatically based on the user’s interaction.
In many cases, the system-generated user models are hidden from
the user. This, however, may lead to grave usability problems and
may well cause the user to not accept the system. For instance,
it violates two of Nielsen’s ten usability principles [
In order to avoid the above mentioned problems, user models need
to be scrutable. This means that the user needs to be able to view
and adapt the information contained in her user model [
However, opening the system-generated user models may be very challenging in case of complex and large models. For instance, in a semantic ontology-based user interest model the items can be interconnected through a number of semantic relations, which in their turn can be used for propagating interest among the items to compensate the scarcity of information about the user. For an average user it may be difficult to comprehend these semantic relations and their effects on the interest propagation and the end personalization effect. The large models containing a big amount of items and relations make it even more difficult.
In this paper, we describe the application of the IntrospectiveViews
interface presented in our prior work [
The rest of this paper is organized as follows: Sect. 2 provides a short overview of the previous research related to visualization of user models. In Sect. 3, we describe how IntrospectiveViews can be used for visualizing and editing semantic user models in a social network. In Sect. 4, we elaborate on the usability study of the interface, which included four methods: task-based experiment, eye tracking, interviewing, and a questionnaire. Then in Sect. 5 we summarize the most critical usability problems identified in the evaluation and outline a number of possible solutions to these problems. Finally, Sect. 6 provides a summary of this paper and an outlook into the future work.
A number of approaches have been proposed to visualizing scrutable
user models. PeerGlass architecture [
Also, the literature reports on a number of approaches to
visualizing the structures of online communities for end-users. For
instance, Vizister [
2http://touchgraph.com relations among them on spirals in a 2D space. However, unlike the IntrospectiveViews described in this paper, both interfaces support only visualization of user models. They do not address the issue of providing the user edit access to her model. Another distinguishing feature of IntrospectiveViews is its ability to visualize ontologybased user models providing rich semantics about user interests.
In this section, we describe the application of IntrospectiveViews
for visualizing an ontology-based user interest model in a social
network. This model provides information about items users in a
social network can be interested in, including people, groups,
organizations, countries, cities, and topics. The model is represented
as an overlay model, i.e., user interests are defined as a weighted
overlay of the ontology instances represented in the domain model.
The user model is defined as a set of tuples (U, T, I, V, L), where:
² U - user ID
² T - instance of a term from the domain ontology
² I - degree of interest
² V - validity of interests
² L - ID of the last update log entry (reference to the user model
log)
The semantic description of instances is defined in the domain
ontology formalized in OWL3. For each instance, the ontology
provides such information as the instance’s class, attributes, and
semantic relations to other instances. For more information on the
user model and domain ontology refer to our prior work [
IntrospectiveViews follows Shneiderman’s visual information
seeking mantra [
The entire collection of interests can be zoomed in and out using either the zoom slider present in the Navigator window or the mouse wheel. This feature allows quickly switching from the overview to 3Web Ontology Language - http://www.w3.org/TR/owl-features/ 4Screenshots and screencasts available at http://www.minervaportals.de/research/introspective-views/ the detailed view. The last can be especially useful in large models consisting of hundreds items. The user can navigate through the collection (e.g. in an enlarged view) by dragging the surface in the corresponding direction. The interface supports two filtering options. Items can be filtered by type and by interest degree. For instance, the user can display only groups and people that she is most interested in by selecting the corresponding checkboxes in the Interest Groups and Types windows located on the right side. Also, the user can obtain additional information about items. Right click on an item will display the item’s context menu (Fig. 2), through which the user can access the item’s description, display the semantic relations to related items (Fig. 3), and get justification of her interest. The last explains how the system determined the interest: it can be based on the user’s frequent communication with the entity, propagated from other items using the semantic relations, and specified by the user explicitly through IntrospectiveViews. Through the context menu, the user can also quickly access the information relevant to the selected items. For instance, by clicking a friend’s name, the user can access the chat history, posts, pictures, and other information posted by the friend on the network. In addition to viewing, the interface enables the user to edit her interest model. The user can change interest degree for a certain item by simply dragging it on the surface: Moving items closer to the circle center increases the interest and moving towards the edge decreases it. New interests can be easily added by making right click in any place on the surface and selecting the Add New Interest item from the context menu. Items can be removed by dragging on to the recycle bin or through the item’s context menu. Finally, through the item’s context menu (Fig. 2), the user can set the item’s privacy and personalization settings. This provides the user a better control on her privacy and the personalization effects. For instance, the user can define which interests may be used for which personalization effects.
In our previous study [
The user study took place at the Usability Lab of Erfurt University of Applied Sciences. It was conducted with 10 test persons, 5 male and 5 female, in age between 20 and 23. All subjects had medium or high level of technical knowledge. While we are aware that this group is rather homogenous, we believe that it represents typical users of many social networks.
In the study we used seven techniques, namely task-based experiment, audio and video recording, mouse and eye tracking, interviewing, and a questionnaire. We chose the combination of these techniques in order to find as many usability problems as possible, to better understand why users experience these problems, and to find solutions to them. The task-based experiments reveal how successful users can achieve their goals with the interface and what difficulties they experience. Filming, eye tracking, and click stream recording during the task completion help the usability experts to perform a more detailed analysis of the user’s interaction. Recording the user’s eye movement can be especially helpful for answering why the user experiences problems while completing a certain task, e.g., if the user is searching for the required control element on a wrong toolbar. Interviews and questionnaires are helpful for getting the user’s subjective reaction and opinions. They help to find what the users like and what they do not like. Also, in interviews the participants can provide valuable insights into how to improve the system.
The course of the study was organized as follows: After a short introduction, the subject was asked to sign a consent form and fill out a personal details form. Then the person was asked to complete eight tasks using the interface. During the completion of tasks, the subject was filmed using audio/video recording equipment. Also, we recorded the participant’s gaze and mouse movement. Afterwards, the subject was interviewed and asked to fill out an evaluation question. The rest of this section provides a more detailed description of each method and its results.
In general, task-based experiments are conducted to test how intuitive an interface is. To measure this, the participants are given specific tasks to solve using the interface. For our experiment we identified eight tasks representing the most common goals that users might want to achieve working with IntrospectiveViews. During the experiment each user had to complete each of these tasks using the interface.
Task 1. Acquaint yourself with the interface. Guess what purpose it has and speak out on it. The goal of this task was to determine whether the user can understand the purpose and function of IntrospectiveViews without any introduction or manual. To describe the first impression, the participants used such words as: globe, tag-cloud, portal, diagram, eye, and mindmap. Having worked with the interface for a little, most subjects noticed that items are radially positioned on the circle by interest and grouped into slices by type. Also, they relatively quickly understood the functions of the three panels located on the right side, i.e., Navigator, Interest Groups, and Types. With respect to the purpose of the interface, the subjects either had no idea at all or expressed themselves very vaguely about it. However, it is important to take into consideration that the interface was not launched from a real social network, which makes it more difficult to guess its purpose. Nevertheless, an important observation here is that most participants understood the meaning of the way the items are displayed on the surface and noticed the affordability to interact with them. After the subject had completed this task, the purpose and possible applications of the interface were explained by the evaluator.
second task was to determine whether the user comprehends the filtering options of the interface, namely, filter by interest degree and filter by type. After having acquainted themselves with the interface, most participants completed this task without problems.
termine whether the user can easily find out how to access the privacy settings of items. Most subjects had problems with this task. As the first attempt, many clicked the Settings button on the toolbar, which was supposed (was not implemented at the time of study) to show the global settings of the system. After failing to complete the task that way, almost all participants tried to find the item on the surface by minimizing the number of items on the screen using the filter by type function. Once the item was found, all participants except one made right click on the item and selected the appropriate menu item from the popped-up context menu (Fig. 2). One person tried to access the settings by making double click on the item. From this task, we have learned that performing a task on a specific item might be complicated by the difficulty to find the item on the surface.
Task 4. Find out how the system determined your interest in an item. As the first step, the subjects searched for the given item, for which all but two subjects filtered out the items of the irrelevant types and in the reduced number of displayed items found the given one. Then they opened its context menu (Fig. 2) by making right click. However, in the context menu, only two subjects selected the right item, Your Interest. All others could not find it from the first try, but simply were checking all menu items one after another. This means that the Your Interest label is not self-explanatory and understandable, hence should be renamed or else the information about the user interest should be accessed through some other means.
Task 5. Display relations of an item. Most participants successfully completed this task by making right click on the given item and selecting the corresponding command from its context menu. Visualizing semantic relations among items (Fig. 3) was appraised by most subjects as a useful feature. However, a number of them complained that the connecting arcs disappear when an item is moved. This means that the interface should provide a better control on visualizing the relations and allow the user to control not only the turning on, but also the turning off of the lines.
Views, this task can be completed by dragging items on the surface. The task was successfully completed by all participants. Many appraised the changing font size of items during the dragging. However, we observed two difficulties with this task. First, for some subjects, it took quite a long time to find the necessary item. Second, some subjects had problems to seize the item when it was overlapped by other items.
Task 7. Delete an item. All subjects completed this task successfully. Four subjects deleted the item by dragging it onto the recycle bin and the rest did it through the item’s context menu. Task 8. Add a new item in the model. All subjects completed this task without serious problems since they had already (during the completion of previous tasks) noticed the Add Interest item in the surface’s context menu that can be opened by right click on the surface. Many subjects appraised that the new interests are added in the place from which the context menu was opened.
During the task-based experiment, the participants’ gaze was recorded using the eye-tracking equipment. The results of eye-tracking show the series of gaze fixations the participants made while looking at the interface and the density of fixations. Fig. 4 shows a plot of the series of gaze fixation one participant has made during the first ten seconds with the interface. The analysis of gaze fixations for all the participants shows, that the user makes the first gazes at the labels in the red area of the surface. This is a good result since this area contains the items the user is strongly interested in, which is the most important information here. The next gazes were made either at the side windows, e.g., Navigator, or the toolbar. Also, many participants spent a considerable amount of time on reading the type labels identifying the circular sectors.
The cumulate gaze density in the first twenty seconds (Fig. 5) shows that during this period of time the participants looked more at the control elements (toolbar and side windows) than at the labels. This, however, can be caused by the novelty of the interface and the lack of knowledge about its purpose and function. Having looked at the labels on the surface, the participants could not fully understand their meaning and tried to find it using the control elements. Based on this finding, we can conclude that the meaning of labels and shades should be made more visible and self-explanatory, so that the user could quickly notice and comprehend it.
After the task-based experiment, the subjects were asked to speak out on the positive and negative characteristics of IntrospectiveViews as well as on their suggestions to further improve the interface.
Positive Characteristics. The participants spoke out on the interface as novel, innovative, cool, and easy-to-use. Many said that it would be a very useful and helpful tool in social networks. With respect to the hot-and-cold color metaphor, half of the participants said that it is an appropriate and comprehendible use of colors. The changing font size of labels on dragging and the highlighting of circular sectors on mouse over were also rated as positive. The interaction concept for changing the interest degree by dragging the label on the surface was rated as intuitive and easy-to-use. The visualization of semantic relations among items was rated especially positive, useful, and novel.
Negative Characteristics. One of the most often criticized problems was the overlapping of items, which makes it difficult to read and manipulate them. Often, in order to read the names of overlapped labels, the participants moved them in different directions, which caused unintended change of interest degree. Also, some participants said that the interface has too many control elements (especially on the toolbar) and this may confuse the user for the first time. However, another important critique was the poor visibility of some control elements: Some participants discovered the context menus of items and the surface by chance after having used the interface for a while. In the beginning, it was not clear for them that the items and the surface can be manipulated through the context menu.
Suggested Improvements. During the interview, the participants made a number of interesting suggestions to further improve the interface. One of them was the suggestion to provide a short tutorial explaining the important functions of the interface. The explanation can be provided in the form of callouts displayed next to the corresponding controls. This tutorial could be automatically displayed upon the first time with the interface and invoked later upon request. Another suggestion was to change the zoom function to show different amount of details at different zoom levels, so that in a zoomed out view only the most interesting items would be shown, but by zooming in the interface would be incrementally showing more of the less interesting items. Apart from that, it was suggested to allow the user to define own types and organize the items according to the self-defined types. Finally, a number of participants said that the interface should allow changing the color scheme of the circular surface.
To evaluate the attractiveness and joy-of-use of IntrospectiveViews, we used the AttrakDiffTM tool 5, which includes an online questionnaire and a number of diagrams visualizing the collected feedback. The AttrakDiffTM questionnaire addresses the subjective attractiveness of a product as a composite characteristic influenced by four qualities: ² Pragmatic quality (PQ): the inherent usability of a product that indicates how successful the users can achieve their
² Hedonic quality - identity (HQ-I): the ability to develop the identity and help the user to establish personal connection with the product. ² Hedonic quality - stimulation (HQ-S): the ability to stimulate the need for further use. ² Attractiveness (ATT): the general outward appearance of a product.
The questionnaire consists of 28 word-pairs (Fig. 6) organized according to the four qualities. For each pair, the subject has to cast a vote on a seven-value likert scale. 5 out of 10 participants completed the questionnaire online. Fig. 7 shows an overview of the received feedback with respect to the pragmatic (x-axis) and hedonic (y-axis) qualities. As it can be seen from the average value V, the interface was deemed to have a good hedonic quality and an average pragmatic quality. With respect to the hedonic quality, the confidence rectangle shows that the rating is relatively consistent, whereas for the pragmatic quality, it shows a significant deviation. The possible reasons for the significant disagreement on the pragmatic quality might be the small number of respondents and the prototypical state of the interface.
Fig. 9 provides a detailed view on the ratings of each of the four qualities by showing the mean values for the all 28 word pairs. As it can be seen from the diagram, all values are in the positive range. The most interesting information here is the extreme values, which show the most problematic characteristics as well as the characteristics that have been especially well resolved. Considering the extreme positive values, it can be concluded that IntrospectiveViews is a very practical, inventive, innovative, and novel interface. Considering the extreme low values, we see the following problems: First, in the pragmatic quality dimension, two word pairs received relatively low ratings. These are “cumbersome - straightforward” and “confusing - clearly structured”. This is partially caused by such problems as the relatively large number of control elements displayed within the interface, overlapping labels, and poor visibility of important control elements (e.g., controls for adding new interests and obtaining detailed information about a specific interest). Another cause of the low ratings in this dimension is the prototypical state of the interface. At the time of evaluation, some functions had not been implemented or were malfunctioning.
The rather average rating of the attractiveness is partially caused by the difference in the opinion regarding the color scheme. Half of the participants liked the hot-and-cold metaphor, but the other half found it unpleasant and even dangerous. Another reason is the relatively big number of control elements being displayed, the small size of the toolbar buttons, and the Java Swing look-and-feel of the interface elements. In the interview, some participants spoke out on the Java look rather negatively.
In this study we have identified a number of critical usability problems. We believe that researchers working on similar interfaces for user model visualization can derive benefits from our findings. This section provides a summary of the most serious problems and outlines possible solutions for resolving them.
² Difficulty to understand the main function of the interface at the first time of use. From the observations we made during the completion of Task 1 we found that most participants could not understand the main function and purpose of the interface in isolation by just using it. Since the concept of user model is new to the average social network user, the interfaces representing user models should provide a clear description of their purpose and function. One option to achieve that is to clearly show the user the connection between her user model and the adaptation/personalization effects achieved using the model’s content. Also, as it was suggested during the interview, the interface’s elements can be augmented with a short description of their function. Such description could be displayed by default for the first time use and invoked later upon request. ² Difficulty in finding a specific item. During the task-based experiment (Tasks 3-7) and the interview we found that the most difficult and time-consuming task while performing a certain action on a specific item was to locate the item on the surface. This problem becomes especially serious in large models consisting of several hundreds items. To avoid this problem the interface should provide a mechanism for searching items by name. This can be done by means of a search box, which could be either always displayed in one of the interface’s corners or invoked by clicking the search button on the toolbar or hitting the conventional [Ctrl + F] on the keyboard. Having typed the first several characters of the item’s name in the search box, the user would be able to see the matching items highlighted on the surface and easily locate the item she is looking for. ² Excessive amount of control elements. During the interview many subjects complained that their interaction was complicated by the relatively large amount of control elements displayed in the interface, namely, the buttons on the toolbar and the dialog windows for filtering displayed on the right. To reduce the amount of elements we plan to redesign the filtering mechanisms. Instead of selecting the types in the checkbox list of the Types window, the sectors containing the items of a certain types can be minimized/expanded by clicking the type’s label. In a similar way the filtering by interest group can be achieved by clicking the corresponding ring. E.g., clicking the blue ring in its normal state will minimize the ring and hide the items of the group “not interested” and clicking this ring once again will turn it to the normal state. ² Poor visibility of control elements for important functions. Another difficulty that we observed during the completion of tasks was that many subjects did not know about the context menu of items and the background, which contain such important actions as adding and deleting items, setting privacy settings, viewing semantic relations, etc. One of the reasons of being not aware of the context menus is that these menus are considered to be a common feature for desktop applications, but is not common on the Web. One option to let the user know about all the functions available for an item without putting more buttons on the toolbar is to display the list of available actions on the mouse over event. ² Overlapping items. As we observed during the task-based experiment, a number of participants experienced problems in grasping an item when it was overlapped by other items. This problem was also mentioned by many subjects during the interview. In the next version we are going to improve the layout algorithm to minimize the number of overlapped items. However, since it is difficult to ensure 100% overlapping free layout for visualizing large models on small displays, the interface should have a mechanism allowing the user to grasp an item in an overlapped bundle. A possible solution to achieve that is repositioning of overlapped items on the mouse over event. I.e., when the user moves the mouse over a bundle of overlapped items, they are automatically moved in different directions to ensure the appropriate distance between them. ² Different perception of the hot-and-cold metaphor. From the interview we learnt that half of the participants found the hot-and-cold metaphor appropriate for the color scheme of the circular surface, but the other half found it unattractive. We believe that attractiveness is relatively important characteristic of a visualization in the context of a social network. Therefore, the interface should allow the user to set the color scheme for the rings that she/he prefers. We plan to add the traffic light color scheme in addition to the hot-and-cold and let the user switch between them.
In this paper, we have shown how IntrospectiveViews can be used for visualizing and editing semantic user interests models in social networks. Through the interface, the user can view the information about her interests that the social network collects and uses for personalization. In addition, the interface allows the user to correct and update this information and define how the social network should use it for personalization effects.
Also, we have reported on a usability study of IntrospectiveViews as a visualization of user interests for social networks. We have described the results of four evaluation methods, namely, task-based experiment, eye-tracking, interviewing, and a questionnaire. The results show that users deem the interface as a useful and novel tool for social networks and that its way to present and manipulate user interest models is understandable and easy-to-use. They also reveal the existing usability problems of the interface and provide insights on solving them and further improving the interface. In the future work, we plan to solve the identified problems and implement the received recommendations in a newer version of IntrospectiveViews. We also plan to add new social features in the interface, such as the one allowing the user to compare her model with the models of other members or even groups on the social network. This can become the means for presenting the community the user’s personality as an IntrospectiveViews-based visualization of her interests. It would allow the community members to quickly obtain an overview on the user’s interests with respect to groups, topics, locations, and other types of information the user is willing to share. Finally, we plan to integrate and evaluate the interface in a real social network or at least in its prototypical implementation.
We would like to thank all the participants who took part in the usability evaluation. We also want to acknowledge Christoph Ohl and Alexander Herz for their contribution to this study. The development of IntrospectiveViews was carried out in the Minerva Context-Adaptive Portals project funded by the IBM Deutschland Research & Development GmbH.
A patent application describing the visualization and interaction concept of IntrospectiveViews has been filed by the IBM Deutschland Research & Development GmbH.