A Model of Relevance for Reuse-Driven Media Retrieval

Abstract

An often criticized fact in multimedia retrieval is, that user needs are not appropriately taken into account. Both knowledge about how end users search and how they assess the relevance of retrieved multimedia objects can provide invaluable hints for the design of multimedia retrieval systems. This paper reports on an end user study on multimedia retrieval behavior of media professionals who intend to reuse media objects in media productions. We present a conceptual model which contains empirically validated information on how users in the media production domain search for content to be reused and how relevance is assessed by them. Finally we sketch how this information can be used to improve ranking of media objects in multi-faceted retrieval scenarios.

Introduction

The amount of multimedia content available on the Web and the amount of professionally produced content stored in local or commercial databases grows every day: While there is a steady growth of professionally produced content available on the Web, a continuous blurred shift happens between consumers and producers of content, which share huge amounts of user generated content. This ever growing amount of content o ers a great potential for reuse.

Reuse of multimedia content, i.e., every kind of use of content which has been used in a certain context before, is an ongoing challenge and is mostly not very well supported by existing tools and approaches. Supporting reuse can however provide signi cant improvements in the way how content is created, including increased quality and consistency, long-term reduced time and costs for development, maintenance or adoption for changing needs [ 25 ]. As our recent observations in the domain of media production reveal, only approximately 30 percent of the produced content is based on already existing content. We furthermore revealed barriers leading to this low gure which include reasons such as \relevat content cannot be found", that \it is sometimes faster to build content from scratch", that \content is not adaptable to new situations", or that \the legal situation is either unclear or does not allow reuse".

One of the identi ed barriers of reuse includes the problem of ndability of content which maps the problem of reusability to the solution space of multimedia retrieval. One ongoing problem there is the gap between the research done and the practical end user needs in di erent contexts as many approaches take a system-centric approach focusing on technical aspects of multimedia indexing and retrieval [ 11 ] and lack a theoretical background of the characteristics of users and their needs for the design of these systems [ 24 ]: In order to support e cient retrieval, matches to a request have to be presented in an appropriate order, minimizing the distance between actual features of the content and expected features by the user.

To bridge the aforementioned gap, user-oriented studies were conducted which analyzed the indexing practices and retrieval needs of typical end users. Some of these studies resulted in analytic models which formalized the characteristics of user requests and search patterns of these users (cf. Section 2). While the main aim of these studies was to conceptualize and bridge the Semantic Gap [ 4, 5 ], the judgement of relevance for the selection of media objects has so far not been researched to a great extent. In order to overcome this situation, we present a conceptual model which contains empirically validated information on how users in the professional media production domain search for content to be reused and how relevance is assessed by them.

In this paper we examined a typical retrievel task in the studied environment: A media professional is engaged in a design task and intends to search for images to reuse in his current production. He starts with formulating his needs in an image request and receives a result set of images. After that, he checks the topicality of the images in the retrieved result set and starts browsing. If either the topicality of the returned images does not match his needs or if he is unsatis ed with the results investigated during browsing, he reformulates his query. Otherwise he applies his relevance criteria and nally selects an image which he uses in his design task.

Understanding how and why users search for and select multimedia content to be reused can provide invaluable hints for the design of multimedia retrieval systems. Therefore the research leading to this paper aimed to address the following research questions: 1. \Which factors do users use to search for reusable media objects?" and 2. \Which relevance criteria do users apply when searching for media objects to be reused?" In order to answer these questions, we built a basic model containing factors used in search and relevance assessment. To do so we analyzed prior literature and conducted interviews with design professionals. Subsequently we empirically validated the model through an end user survey and assessed the validity and importance of the factors in both tasks.

The remainder of this paper is structured as follows: Section 2 presents the background and motivation for the model. Subsequently the model is discussed in detail in Section 3: We present insights from prior literature and the basic mode. Section 4 details the validation of the model and presents the results of the conducted survey. Finally, Section 5 concludes the paper.

Research Background and Motivation

Since the 1960s research has been reported which analyzed the indexing practices and retrieval needs of typical end users. The work in this area can be divided into conceptual frameworks of image indexing (cf. [ 5, 9, 14, 22, 27 ]), which are mainly situated in cognitive psychology and models of user's multimedia retrieval needs (cf. [ 1, 2, 10, 15, 16, 19, 29 ]). The intention of these conceptual frameworks was to provide groundings for the manual and automatic image indexing and the description of the semantics of multimedia content in general and images in particular.

The earliest model was provided by Panofsky [ 22 ] who recognized three types of subject matter, for instance, primary subject matter which requires no interpretative skills, secondary subject matter which necessitates an interpretation, and tertiary subject matter (\iconology") demanding high-level semantic inferencing done by the user. Subsequent work by Shatford [ 27 ] simpli ed the three levels of Panofsky into generic, speci c and abstract. Additionally Shatford introduced the distinction between \of-ness" and \aboutness" of a picture. A simpler model was provided by Greisdorf [ 9 ] who recognized three levels which correspond to visual primitives (e.g., color or shape), logical features (e.g., objects or events) and inductive interpretation (e.g., abstract features). Joergensen et al. have further re ned the model by Shatford which resulted in the so-called visual indexing pyramid [ 14 ]. A newer model by Enser et al. builds on Joergensen's notion of semantic facets of images and furthermore takes the combination of semantic content of an image and its context into account [ 5 ].

Besides the development of these models, studies were conducted which investigated user retrieval needs. Their intention was to inform other research strands which type of semantics can be extracted from multimedia content. Most of these studies revealed, that a user is typically interested in high-level semantics which are hard to derive based on automated approaches and which are often highly subjective. An analysis of early studies in this area by Jorgensen revealed a wide variation in subject foci and terminological speciality and also that the majority of requests were for speci c events or objects, especially for speci c, named features [ 16 ]. This observation was also made in [ 1, 19 ]. Other studies reported an emphasis on generic or a ective visual features (cf. [ 2, 10, 15 ]). Validations and comparisons of these studies can be found in [ 1 ] and [ 29 ].

Especially in multimedia retrieval, uses and needs vary considerably, as media objects are used in a variety of domains (e.g., media production, art, journalism, or medicine) for di erent purposes. Furthermore, needs of professionals and needs of end users are in many cases di erent: End users are motivated by leisure, while professionals search for images for inspiration, reuse, or other reasons. As relevance di ers considerable based on the situation of the user and his needs, domain speci c investigations have been made: User needs in domain speci c collections and for speci c user groups have been conducted, e.g. for web images (cf. [ 6, 7, 15, 23 ]), for historical images (cf. [ 1, 2 ]), for medical images (cf. [ 17 ]), or for image retrieval in a journalistic context (cf. [ 11, 12, 18, 19, 29 ]). User needs of media professionals such as graphic-, or game- designers, and especially the in uence of the intention to reuse, were, however, rather unexplored up till now. Our aim was therefore to develop a conceptual model that dimensions relevance in reuse-driven multimedia retrieval in which people search for content to reuse. 3

A Model of Relevance in Media Reuse

This section presents a conceptual model which re ects factors which in uence the relevance of multimedia content for end users in the particular situation in which they look for content to reuse. The model is based on insights from existing literature, on motivation and barriers for reuse, and on user studies which investigated multimedia retrieval in professional domains. Prior insights were validated and supplemented with expert interviews conducted with media professionals. 3.1

Insights from Existing Literature

Relevance is a central concept in information retrieval and there used as a measure for retrieval and to judge the e ectiveness of an information system. Ingwersen and Jarvelin suggest that relevance is a multidimensional cognitive concept whose meaning is largely dependent on searcher's perceptions of information and their own information need (cf. [ 26 ] as cited in [ 13 ]). While content features are in most situations the most appropriate indicators for relevance, non-content features of documents can give valuable hints, too. This is especially true for multimedia retrieval in professional environments in which relevance is not only based on topicality but also on visual, qualitative, situational and other contextual factors as reported in previous studies (cf. [ 1, 2, 15, 18, 19, 21, 28, 29 ]): Markkula investigated retrieval of images in a journalistic context [ 19 ]. His observations clearly indicated the diversity of relevance criteria which were applied and the situational nature of their relevance judgements. The primary criteria which is applied by journalists to assess relevance is topicality. Secondary criteria are technical properties, technical quality and biographical criteria. Images which are technically good, are current or were not recently published are being considered as relevant in this domain. The cost of images has also been identi ed as an important criterion. Even though expressive and also aesthetic criteria, such as color and composition, were used for search by journalists, they played the most important role in the nal selection phase. The critical criteria to reject or to accept an image depended on earlier selections: An image already chosen for a page and nearby pages or used recently in a di erent or the same newspaper restricted the possibility to use other similar images. According to the journalists, the goal was to make the illustration of the page attractive, balanced, and dynamic. This was achieved by using images of di erent types (e.g., horizontal and vertical photos, portraits, group photos, action, or themes) and with di erent visual features.

Another study was conducted by Choi and Rassmussen who investigated relevance criteria in image retrieval of historic art images [ 2 ]. They identi ed nine relevance criteria together with 8 non-visual (descriptive) attributes for highly relevant images: { Time frame: The time period of the image. { Accuracy: The image accurately presents what the user is looking for. { Topicality: The image is related to the user's task. { Completeness: The image contains the necessary details. { Accessibility: The availability of the image, as in the ease of obtaining the image and the means by which image information can be accessed. { Appeal of information: The image is interesting and appealing to the user. { Novelty: The image is new to the user. { Suggestiveness: The image generates new ideas and insights for the user. { Technical attributes: These attributes include mood, emotion, point of view, or color.

Furthermore the following descriptive attributes were identi ed as being important in the assessment of relevance in image retrieval in a historical art context: creation date, notes, subject descriptors, the title of the image, the source repository, the source collection, the medium, and the name of the creator.

Eakins' study done in 2004 investigated features used in image search but not how these features a ect relevance [ 3 ]. His results showed that despite of topicality, technical quality is the most important criterion in search. Besides that, he identi ed the following features as the most relevant: { Low level features such as colour, texture or shape. { Technical quality features such as sharpness. { Semantic content containing general and speci c semantic terms. { Abstracted features such as contextual abstraction which refers to nonvisual information derived from the knowledge of the viewer, cultural abstraction which refers to aspects which can only be inferred based on a cultural background, Emotional abstraction which refers to emotional responses triggered by the image, and technical abstraction which refers to aspects requiring speci c technical expertise to interpret. { Metadata such as the image type (e.g., photographic, painting, or scan).

Othman was the rst to investigate image retrieval in a creative media-related context [ 21 ]. The relevance criteria she discovered for the domain of media production were similar to the ones by Choi and Rasmussen [ 2 ] but with a di erent mean importance of each criteria: Technical attributes were considered the most important, followed by completeness and topicality. Furthermore relevant images had to be quali ed for processing and should not require any authentication. Most images in her study involved analysis and image manipulation, and thus the majority of users rated technical attributes as the most important relevance criteria. Technical criteria included resolution, size, color, and dimension. Topicality and completeness ranked second and third which indicated that images must be right on the topic and have all the objects speci ed. Time frame was an important criterion for speci c tasks. A further novel insight from her study was that the images which were judged as relevant and met their intended use ranged from one object in the image retrieved to the whole image itself.

The most recent study which reports on end user needs in image retrieval in a journalistic context was published by Westman and Oittinen [ 29 ]. It featured 47 criteria for image selection which were partially based on insights from Markkula [ 19 ]. The criteria were grouped along the following dimensions: { Information and content (e.g., information content or story of the photograph) { Visual and compositional features (e.g., visual features, composition, or lighting ) { Technical features (e.g., technical error, sharpness, or physical size) { Abstract and a ective factors (e.g., movement and dynamicity, mood, expression of the person) { Metadata and associated information (e.g., recentness, source, or importance) { Publication context (e.g., compatibility with the headline, publication section, or importance of the article) { Work ow and other actors (e.g., timetable or possible print quality ) { Practices and feedback (e.g., image selection practices or feedback from readers) According to Westman's and Oittinen's studies, several types of criteria were used in the relevance assessments made. Contextual factors (such as publishing section or layout of the page) formed a selection frame for suitable images. Topicality was identi ed as a necessary but insu cient criterion for relevance, used mostly as a starting point. Compositional and informational criteria followed in later stages of the process. The nal selection criteria were dynamic, activated by comparisons of retrieved images and based on the characteristics and di erences between them such as dynamic elements or sharpness. Final selection criteria also were preferential or reactive in some situations; selections were based on personal impressions of images being, for instance, more interesting than others. Furthermore several implicit criteria were employed in the image selection process. Unless otherwise asked, the image retrieved was as recent as possible and, if search is carried out across multiple archives, retrieval from the own archive was preferred. Constraints such as price, previous publication, recentness and presence of other images in a product also in uenced the selection. Their results revealed that the most important relevance criteria were related to the informational content of the image. Several abstract and a ective criteria also in uenced the selection strongly. Least important were feedback and reactions from others. Various factors related to the eventual publication context of the image were considered important which means that often not the best matching image according to a query was used but the one matching the context most. A large number of individual criteria a ected image selection strongly. Technical factors were identi ed as not being as crucial as previously thought by Markkula and Sormunen [ 19 ]. 3.2

Basic Model

Our conceptual model, which is depicted in Figure 1, contains factors which are typically used by media professionals for search and/or to assess relevance of a media object. To create it, we re ned and extended prior literature, validated and supplemented it with context speci c interviews: 1. First interviews with designers, art directors, and researchers with experience in multimedia content creation and reuse were conducted. 2. Secondly, the qualitative data gathered from the expert interviews and prior observations were analyzed by transcribing and coding into meaningful expressions which were then classi ed. 3. Thirdly the results were systematically analyzed according to statistical theory. 4. Finally the analysis resulted in a cluster of themes, each containing a grouped set of factors that in uence reuse. All of these factors were constantly emphasized by media professionals.

The derived factors are arranged into clusters which are partly based on the categorization schema from Westman [ 29 ] which we further extended and tested in our survey. The clusters are further arranged into intrinsic and extrinsic features and environmental factors. The following clusters are part of the model: { Intrinsic Features 1. Information and content (I) which includes features such as topicality, completeness, or conveyed message. 2. Technical Quality (TQ) which contains features such as sharpness, or technical error. 3. Technical Properties (TF) which contains intrinsic technical features such as resolution or size. 4. Visual and compositional features (V) contain features such as composition, color, angle, or other visual features. { Extrinsic Features 5. Abstract and a ective features (A) include expression, dynamicity, eye catching ability, or mood 6. Provenance and bibliographic metadata (PM) includes features capturing previous uses of the media object such as popularity, recentness, previous publishing, etc. 7. Rights and licensing (RM) includes information regarding the rights holder(s), permitted use, etc. { Environmental Factors 8. Production work ow (PDW) captures features related to the overall production and its requirements. 9. Processing work ow (PCW) bundles features regarding the actual processing of the media objects, such as if it is adaptable or how it can be processed. 10. Publication context (PC) refers to the concrete location into which the media object should be integrated to and by that provides additional constraints such as available space, consistency with the layout, or publishing history of the item to be reused. 11. Practices and feedback (PF) refers to work related factors such as typical habits of the designer itself, typical guidelines from the company, or social recommendations from colleagues, customers or experts. To test the proposed research model and to gain insights on the degree of in uence of the di erent factors, we adopted the survey method for data collection, M1 [ 19 ] M2 [ 2 ] M3 [ 21 ] M4 [ 29 ] MR x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x validated it using statistical methods, and compared our insights to the results derived from previous related work in this area. 4.1

Methodology and Data Collection

We conducted an end user survey to collect data and designed a questionnaire re ecting the factors in order to assess the conceptual model. The resulting questionnaire was rst tested in a small group of members of a media design online forum. After analyzing the results from the test phase one item from Abstract and a ective factors has been dropped because it has not been used by any of the participants. A further re ned version was then sent as a self-administrated questionnaire to 150 media design professionals in Europe. The data was collected via an online survey in two languages (German and English)1. The back translation approach was applied in order to ensure consistency between both language versions of the questionnaire [ 20 ]. The questionnaire consists of ve parts: The rst part contains general questions regarding reuse such as how much content is reused on a personal, company- and production-oriented level, and asked for reasons and barriers for reuse. The second part contains questions regarding factors used in search for media objects to reuse and the third part contained questions regarding selection criteria of content for reuse. In parts two and three the participants were asked to indicate the frequency of the use of the factors in search and for the assessment of the relevance of a media object on a ve-point scale. The fourth part includes questions about the actual use of reused content (e.g., if it is adapted, or used as is). The fth part contains concluding questions regarding the demographics of the participants.

31 responses which makes a response rate of 21 percent were returned, from which two responses with incomplete data were eliminated from further analysis. The gathered data re ects habits of media professionals spanning the domain of print and Web design over game design to the design of learning material. The majority of the participants had an experience from 3 5 years in their domain (32:14%), followed by 25% which had more than 10 years of experience and 21:34% which had 5 10 years of experience. The remaining respondents had up to 3 years of experience.

In order to check the internal consistency of the model, it was assessed using factor analysis [ 8 ]. Further structured relationships between the variables were examined. 4.2

Survey Results

The gathered data revealed several interesting insights on barriers and motivations for reuse of media, how people search for and assess the relevance of media objects in a particular situation (cf. Section 4.2 and 4.2), and how they nally use the selected media objects (cf. Section 4.2). 1 The online questionnaire used in the survey is available at http://www.

tobiasbuerger.com/reusesurvey/ Factors A ecting Search In this section we provide answers to the question "Which factors do users use to search for reusable media objects?" based on insights from the conducted survey.

In the rst part of our questionnaire, participants were asked to indicate the importance for each feature on a ve-point scale: 1 means that the factor is never used, 2 that it is used very infrequently, 3 that it is used infrequently, 4 that it is used frequently, and 5 that the factor is used very frequently. Based on that, Figure 2 shows the mean importance of the factors used in search by media professionals grouped into the relevant clusters:

Not surprisingly, the cluster with the highest frequency is Information and Content (I) meaning that users use keywords or classi cation information to search for content very frequently. Following this cluster are six clusters having almost equal frequency. The rst one is Technical Quality (T) which includes factors such as clarity of structure, sharpness, brightness, or technical error. This is followed by Practices and Feedback (PF) including feedback from experts and colleagues which has the highest impact. After that, Rights and Licensing (RM), Technical Features (TF), Visual and Compositional Features (V) and nally Provenance and Bibliographic Data (PM) follow which are used rather infrequently. Users are typically not using a ective factors, context or work ow information for search. A further observation from related work which has been con rmed by the expert interviews is, that media professionals typically start with a keyword query and then extensively use browsing facilities. This con rms earlier insights from [ 21 ]. Furthermore it seems to be appropriate to present images with rather di ering visual properties in some situations.

The results from this part of the survey are in line with results from Eakins [ 3 ] who identi ed topicality and technical quality as the most important criteria used in search.

Factors A ecting Selection and Assessment of Relevance Our survey

revealed, that users use di erent factors from all clusters of the model to assess the relevance of media objects. Figure 3 shows the mean importance of the factors used for the assessment of relevance grouped into the clusters of the model:

Our results show only small di erences to the study done by Westman et al. [ 29 ] in which the authors reported similar mean values for di erent factors in relevance assessment. Their study revealed that the cluster Information and Topicality (I) is the most important one, followed by Abstract and A ective Factors (A) and by Visual and Compositional Features (V). Our results however suggest that Abstract and A ective Features (A) are most important even before Information and Topicality (I). This can be explained by di erences in the domain that we investigated, as a media object in media production only seems to be relevant if the aesthetics and other abstract features are compatible with the intended usage. This comes even before Information and content (I). This observation can partly be explained by the fact how media professionals reuse media objects; media professionals retrieve media objects for inspiration very frequently (in 30% of all cases) meaning that they may create media objects which re ect their thought topicality based on an aesthetically pleasing artwork. The other clusters had a similar mean importance as reported in previous work: Technical Features (TF) and Technical Quality (TQ) is followed by Publication Context (C) and other metadata. The smallest mean value is assigned to Workow Related Issues (W) and Practices and Feedback (PF).

It should be noted that some of the factors from clusters which are ranked lowest such as Rights and Bibliographic Metadata (RM) are in the top-10 of most important factors such as price or usage rights (cf. Table 2.)

Factor (Cluster) Mean Aesthetic compatibility (A) 4.67 Topical compatibility with the usage context (I) 4.46 Mental associations (I) 4.39 Technical quality (TQ) 4.32 Price (RM) 4.15 Technical adaptation possibilities (PCW) 4.11 Consistent layout (PC) 4.11 Usage rights (RM) 4.08 Technical format compatibility (TF) 4.04

Adaptation e ort (PCW) 4.00

This explains the di erence to the clusters from Westman [ 29 ] in that category. The importance of rights can be explained by the fact that the Internet is the most frequent source for reusable media objects followed by the local harddisk as our study indicated; on the Internet stock image sites are used most frequently followed by specialized image search engines such as Google image search2. Company wide content management systems are ranked even after social media sharing sites such as Flickr3. The importance of adaptability can be explained by the di erences in the domains investigated. In the journalism domain, which was investigated by Westman, images or photos are typically used as is and only marginally adapted, whereas in media production aesthetics and other abstract features have to be compatible with the intended usage. Furthermore novelty of created media objects is a very important criterion especially in games, animation or lm production, which makes the need for bigger adaptations evident (cf. Section 4.2).

Usage of Selected Media Objects The fourth part of our study revealed interesting insights into how people reuse media objects that they select. The types of reuse can be grouped according to the de nition provided in Section 1: 2 http://images.google.com 3 http://www.flickr.com (i) content is either reused as is, (ii), only parts of it are reused, (iii) it is reused after being adapted, or (iv) it is only reused for \inspiration".

In most cases, media objects are only retrieved for inspirational purposes which can be explained by the fact that work has to be original in the investigated domain. A media object as is is only reused infrequently, parts of media objects are in contrast to that reused frequently. Results of our study furthermore reveal that content is being adapted very frequently before use. The adaptations range from basic features like resolution, contrast, brightness to the extraction of the background or parts of the content such as objects or areas. 5

Conclusions and Future Work

In this paper we reported on a conceptual model which dimensions relevance assessment in multimedia retrieval scenarios in which people search for content to reuse. The model is grounded on prior literature, completed with insights gained from expert interviews and validated based on empirically gathered results from an end user survey. The model captures factors used for search and relevance assessment, proposes a clustering of these factors, and assigns a mean importance value to each factor based on the results of the reported survey.

Our next steps contain the realization of a hybrid image search engine which integrates content based search with semantic search and which takes the results reported in this paper into account in order to re-rank the fused result lists from both search engines. We believe that the values assigned to the factors can be used to rank results which were retrieved in multi-faceted search including keywords related to the topic of images but also metadata such as rights, pricing information, or visual features. We plan to perform a second validation and calibration of the model based on end user experiments using the search engine.

References

1. H. Chen . An analysis of image queries in the eld of art history . Journal of the American Society for Information Science and Technology , 52 ( 3 ): 260 { 273 , 2001 .

2. Y. Choi and E. M. Rasmussen . Users' relevance criteria in image retrieval in american history . Information Processing & Management , 38 ( 5 ): 695 { 726 , 2002 .

3. J. P. Eakins , P. Briggs , and B. Burford . Image retrieval interfaces: A user perspective . In Image and Video Retrieval, LNCS , pages 628 { 637 . Springer, 2004 .

4. P. Enser . The evolution of visual information retrieval . Journal of Information Science , 34 ( 4 ): 531 { 546 , 2008 .

5. P. Enser , C. Sandom , J. Hare , and P. Lewis . Facing the reality of semantic image retrieval . Journal of Documentation , 63 ( 4 ): 465 { 481 , 2007 .

6. A. Goodrum and A. Spink . Image searching on the excite web search engine . Information Processing & Management , 37 ( 2 ): 295 { 311 , March 2001 .

7. A. Goodrum , M. Bejune , and A. C. Siochi . A state transition analysis of image search patterns on the web . In Image and Video Retrieval , Springer, 2003 .

8. R. L. Gorsuch. Factor Analysis . Lawrence Erlbaum Publ., 1983 .

9. H. Greisdorf and B. O'Conner. Modelling what users see when they look at images: a cognitive viewpoint . Journal of Documentation , 58 ( 1 ):6{ 29 , 2002 .

10. L. Hollink , A. T. Schreiber , B. J. Wielinga , and M. Worring . Classi cation of user image descriptions . Int. J. Hum.-Comput . Stud., 61 ( 5 ): 601 { 626 , 2004 .

11. T .-Y. Hung. Search moves and tactics for image retrieval in the eld of journalsim . J. of Educational Media & Library Sciences , 42 ( 3 ): 329 { 346 , 2005 .

12. T .-Y. Hung, C. Zoeller , and S. Lyon . Digital Libraries: Implementing Strategies and Sharing Experiences, chapter Relevance Judgments for Image Retrieval in the Field of Journalism: A Pilot Study , pages 72 { 80 . LNCS. Springer, 2005 .

13. P. Ingwersen and K. Jaervelin . The Turn: Integration of Information Seeking and Retrieval in Context. Springer, 2005 .

14. C. Joergensen , A. James , A. B. Benitez , and S.-F. Chang . A conceptual framework and empirical research for classifying visual descriptors . J. Am. Soc. Inf. Sci. Technol ., 52 ( 11 ): 938 { 947 , 2001 .

15. C. Joergensen and P. Joergensen . Image querying by image professionals . J. of the American Society for Information Science and Technology , 2005 .

16. C. Joergensen . Image Retrieval: Theory and Research . The Scarecrow Press, 2003 .

17. L. Keistler. User types and queries: Impact on image access systems . In R. F. et al., editor, Challenges in Indexing Electronic Text and Images , pages 7 { 22 . Medford , NJ: Learned Information Inc., 1994 .

18. M. Laine-Hernandez and S. Westman . Image semantics in the description and categorization of journalistic photographs . In Proc. 69th Annual Meeting of the American Society for Information Science and Technology , Austin (US) , 2006 .

19. M. Markkula and E. Sormunen . End-user searching challenges indexing practices in the digital newspaper photo archive . Information Retrieval , 1 ( 4 ): 259 { 285 , 2000 .

20. M. R. Mullen . Diagnosing measurement equivalence in cross-national research . Journal of Intemational Business Studies , 26 ( 3 ): 573 { 596 , 1995 .

21. R. Othman . A model for an image retrieval tasks for creative multimedia . Performance Measurement and Metrics , 6 ( 2 ): 115 { 131 , 2005 .

22. E. Panofsky. Studies in iconology . Harper & Row , New York., 1962 .

23. H.- T. Pu . An analysis of failed queries for web image retrieval . J. Inf. Sci. , 34 ( 3 ): 275 { 289 , 2008 .

24. E. M. Rasmussen . Indexing images . Annual Review of Information Science and Technology , 32 : 169 { 196 , 1997 .

25. A. Rockley . Managing Enterprise Content. New Riders , 2002 .

26. T. Saracevic . Relevance reconsidered . In Information Science: Integration in perspectives . Proceedings of the Second Conference on Cepnceptions of Library and Information Science , pages 201 { 218 , 1996 .

27. S. Shatford . Analyzing the subject of a picture: a theoretical approach . Cataloging & Classi cation Quarterly , 6 ( 3 ): 39 { 62 , 1986 .

28. S. J. Westerman et al. Creative industrial design and computer-based image retrieval: The role of aesthetics and a ect. In A ective Computing and Intelligent Interaction, chapter Creative Industrial Design and Computer-Based Image Retrieval: The Role of Aesthetics and A ect , pages 618 { 629 . Springer, 2007 .

29. S. Westman and P. Oittinen . Image retrieval by end-users and intermediaries in a journalistic work context . In IIiX: Proceedings of the 1st international conference on Information interaction in context , pages 102 { 110 , New York, NY, USA, 2006 .