Nowadays almost all retrieval algorithms (for text, images, drawings, etc.) are mainly concerned in achieving good system-centered measures, such as precision and recall. However, these systems are used by users, who try to achieve goals through the execution of tasks. To better satisfy the users' needs we must involve them in the development process of the retrieval systems. In this paper, we argue that a user-centered approach, where users are included in the development cycle of the overall retrieval system, can lead to improved retrieval algorithms and also to a better user satisfaction while using the system.
The majority of the retrieval algorithms, whether they are for text, images, drawings, 3D objects, audio, video, etc., are mainly interested in performing well for system-centered measures, like for instance precision and recall. However, these systems are used by users who want to perform speci c tasks and achieve speci c goals. We can develop a good retrieval system, that performs well against a prede ned ground truth, but when we delivery it to users they may Copyright c 2011 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by the editors of euroHCIR2011.
EuroHCIR ’11 Newcastle, UK . not be able to nd what they want or they may not even be able to submit a query to the system.
For illustration purposes let us consider the following hypothetic scenario: \We developed a system for retrieving generic complex vector drawings, like for instance technical drawings, architectural plants or clipart drawings. We evaluated it using query-by-example and a set of prede ned drawings, achieving a good precision and recall measure. Afterwards, when we delivered the system to users, we noticed that they were not able to use it, because they could not nd the ( rst) drawing that they must use as query to nd the desired drawing. Moreover, users do not want to search for the complete drawing, but only by a subpart of the drawing."
This scenario could be avoided if before we developed the retrieval system we asked users what were their needs, what did they want to perform on the system and how they want to do it. To collect all this information we need to apply a user-centered approach where users are involved in the development of the retrieval system and algorithms.
In this paper we defend an user-centered approach as a way to create better retrieval algorithms and improve the overall retrieval system. We start by shortly describe the user-centered approach and the iterative cycle used in the user interface design. In Section 3 we describe our application of the user-centered approach in the development of retrieval algorithms. Finally, we present some conclusions. 2.
The user-centered design (UCD) is a design methodology, where the needs, skills and limitations of the users are taken into account during all stages of the development of the system. The key premise of the user-centered design is that the active involvement of the users in the development process as well as in the evaluation of the interactive products can lead to well-designed systems that best meet the desired usability goals. These systems will take advantage of users skills, will be relevant to their work and activities, and will help them rather than constrain their actions.
One of the principles from the UCD [
!"#$%&'(%)&"*%
+'&,-"."%
Typically when we want to develop a new retrieval approach, we look at the media to retrieve (text, audio, video, drawings, images, etc.), identify the features that better describe the media, create a matching algorithm and nally we compute precision and recall. Although this methodology allows us to create retrieval systems, we believe that by including the user in the development cycle will allow us to deliver better and more usable retrieval systems, that will allow users to achieve their goals and not only systems that have a good precision and recall performance.
Moreover, we should not develop retrieval systems, and that includes descriptor computation, matching algorithms and presentation of the results, without rst identifying a set of user needs and functional requirements ( rst step in the user-centered design). We need to know our users, their skills, their background, their pro le. We must identify their needs and requirements, their goals and how they achieve them. In summary, we need to do an user and task analysis before we start developing our retrieval system. User and task analysis should not only in uence the design of the user interface, but also the design of the retrieval approach or algorithm.
For instance, users could use various strategies to perform a search in a drawing retrieval system. They could use a drawing that they already have, in a le, to search for similar drawings using query-by-example, or they could draw a sketch of the drawing that they want to nd. As we can see, the retrieval solution (feature extraction, indexing and matching algorithms) will be di erent on each case. While in the rst case we only need to compare two drawings of the same complexity and with the same characteristics (sets of lines and polygons), in the second case we need to compare complex drawings with sketches (typically simpler and with less elements). Thus, the way users perform the task to achieve their goal in uence the retrieval approach that we should develop.
After developing the retrieval solution based on the user requirements, we should evaluate the retrieval system, using not only system-centered measures, but also user-centered measures, such as time to complete tasks, error rates, satisfaction, etc. As in the user-centered design of interactive systems, results from the evaluation of the retrieval system (system and user centered measures) should be used to improve the system and to re ne the user and functional requirements of the retrieval system.
One of the things that we observed in one evaluation session with users, was that users did not care about where in the order of retrieval the intended drawing appears, the important fact being that it was there. One of the users produced this comment \It [the system] found it [the drawing]! That is what counts!" However, when we evaluate retrieval systems, the majority of the existing measures and ground truth datasets privilege precision. Of course this systemcentered evaluation is important, but we should also take into account the users perspective, where they privilege recall. 3.1
Involving the users can a ect the way we develop the
retrieval algorithms. In recent years we developed a generic
approach for complex vector drawing retrieval, based on the
topology and geometry of the elements present in the
drawing. These two features were used to describe the content
of the drawings, and during matching, we rst compare the
drawings using topology and them we compare the
geometry of those with similar topologies, giving the same weigh
to both features (for more details see [
Before we developed this solution and the two retrieval systems, we performed user and task analysis to understand how users wanted to make queries to this type of systems. We notice that they prefer to draw sketches of the drawing that they were looking for than to submit an existing drawing to perform a query-by-example. Moreover, most of the times they do not have a drawing similar to the one that they are looking for.
The two systems were both evaluated with users, and from
those evaluations we observed that the way users search for
technical drawings was di erent from the way they search
for clipart drawings [
guration, as illustrated in Figure 2; for clipart drawings, users produced simpler sketches, with fewer elements and with a poorer topological description (see Figure 3).
Due to this observation during tests with users, we re ne
our retrieval algorithm for retrieving clipart drawings [
We can not develop our retrieval algorithms without involving our users into the development cycle. As in the design of interactive systems, also in the development of retrieval systems we must involve the users.
They must be involved in the initial phase, so we can understand how they search for the information, what are their knowledge, what are their limitations and what is their pro le. With this we are able to identify users needs and functional requirements.
Later on, during the development of the algorithms we should take into account this input and adapt the algorithms to provide \good results" for "our" users, and not for the users in general, or for the system.
Finally, during the evaluation stage, besides computing the traditional system-centered measures, for a set of datasets de ned as ground truth, we should also involve users in the evaluation to collect quantitative and qualitative measures. Information gather during evaluation should be used to improve the retrieval algorithms and the overall retrieval system, in the next iteration of the iterative cycle of the usercentered approach.
In this paper we defended a user-centered approach for the development of retrieval systems. As in the case of user interfaces design, also for retrieval systems is important to know our users, adapt the algorithms to them, and involve the users in the evaluation of the system.
We believe, and we had con rmed, that the involvement of the user in the development cycle of retrieval systems can conduct to better systems that satisfy users needs and are more adapted to them.
This work was supported by FCT through the PIDDAC Program funds (INESC-ID multiannual funding) and the Crush project, PTDC/EIA-EIA/108077/2008. 5. 6.