A recent trend in multimedia information retrieval systems is the integration of users, by their preferences and interests, in the retrieval process. Generally, such systems consider the user only after the query's execution, while the results' presentation. We propose to consider the user as a source of metadata, by exploiting his behaviour and to enrich the document's metadata with a usage metadata. We introduce the concept of temperature, associated to each metadata descriptor, which denotes the popularity of the multimedia document's metadata. An algorithm for the computation, the increase and the decrease of this temperature is described in details. We present also how this algorithm can be used for the enrichment of each metadata descriptor according to the user's interactions with the multimedia content and the metadata.
Nowadays, we are constantly surrounded by multimedia contents and devices. Thus, we are continuously creating and consuming multimedia data. Usually, before creating a multimedia document, the user has an idea of which kind of information he wants to include in his document and then he searches the multimedia contents that correspond to his needs [1]. Hence, the management of multimedia documents, which includes their storage, indexation and retrieval processes, is very important.
A recent trend in the information retrieval domain is the user’s integration in the retrieval process. Thus, the user’s preferences, interests and behaviour are analysed and modelled in order to improve the performance of the system. This improvement is realised by providing better results to a user query and by recommending him other interesting documents accessed by other users which have similar profiles [2].
In this context, we focus on the user’s integration in the metadata management process. We want to provide a solution for the metadata enrichment through their usage and through the user’s interaction with the multimedia document to which they are associated. This enrichment is accomplished through the concept of temperature which is associated to each metadata descriptor related to the multimedia document and to the multimedia document itself. This temperature can be considered as a popularity metadata that is updated each time the document or a part of it is consumed. Thus, more a document is consumed, the hotter it and its metadata get. In this paper we focus on the presentation of: (1) an algorithm that exploits this concept by specifying the manner in which the temperature can be increased or decreased, and (2) the algorithm’s application in several scenarios.
This kind of metadata can have several utilizations in: the recommendation systems of a certain document or only a part of it; the execution of the user’s query, by taking into account the document’s temperature in the computation of its score; the creation of the document’s resume to be displayed in the results list; the selection of video’s key-frames according to the user’s profile.
The remainder of the paper is structured as follows. We begin with an overview of multimedia metadata and the user’s interaction in the multimedia information systems, in Section 2. Then, in Section 3, we present a metadata framework that includes the concept of temperature. The proposed solution for the metadata enrichment according to their usage is described in Section 4. Finally, some preliminary results and conclusions are given. 2
From our daily experience, we can deduce that the best way to find certain desired information from a huge collection of documents is to look not at the information itself but rather at a much smaller and more focused set of data. In the context of multimedia retrieval systems, this concise information is the metadata.
The metadata can be classified in: (1) content metadata (low-level, high-level, structure, life-cycle, identification and localization and management metadata) and (2) user metadata (user interaction and user context) [3]. During the last years, the number and the heterogeneity of metadata formats increased steeply. The majority of these standards are content centred, e.g., Dublin Core, XMP, MPEG-7, TV-Anytime. In general, an information system in charge with managing and retrieving multimedia contents is composed of [4, 5]: (1) a multimedia collection which contains several multimedia contents; (2) a metadata collection which contains information about the media characteristics (e.g., size, name) and their contents; (3) an indexation engine which includes several indexing algorithms to be applied on the multimedia collection in order to enrich the metadata collection. The indexing algorithms automatically applied on the multimedia contents produce metadata encoded into different standards and formats. These metadata are further employed in the retrieval process. This makes the management of the metadata and the query execution a very important task to be realised by a multimedia information retrieval system.
In [6], the metadata is presented in the centre of the multimedia document
lifecycle, which makes the metadata creation and management a very important issue
in the handling of multimedia documents. In addition, the metadata is consumed and
produced at every stage of the document lifecycle [
In order to obtain more information from users, some other strategies have been developed. One of them is to analyze the user’s behaviour and to infer his interests [9] and his intentions [10]. These interests are used, for example, to adapt the presentation of the multimedia documents [11] and of the query results list [12] or to enrich the user query [13].
Apart from the implicit and explicit metadata we can consider also the attention [14] and usage metadata. This information is associated with the document and not with the user, as for the interests. In [15], the authors propose an algorithm for determining such metadata. The authors determine the popularity of multimedia documents in accordance with the number of users that access the documents. The authors attach this popularity information to entire documents, and not to parts of documents. Also, this information is computed in function of the number of users that access the document, and users’ interests and preferences are not taken into account.
The behaviour of the user is also used in other domains, such as the adaptive hypermedia domain [16], where the presentation of the documents is modified according to the user, and the user-centric multimedia databases [17], where the user behaviour is captured through the analysis of the query logs.
As could be noticed, the research fields where the user is taken into account are very different and vast, from the presentation’s adaptation to the multimedia information retrieval. The user’s behaviour is studied in order to adapt the documents or the query’s results, but the metadata associated to the multimedia contents are not enriched. Before presenting our approach for the metadata enrichment, we will describe in the next section the metadata framework developed in order to incorporate the notion of temperature. 3
In the domain of metadata interoperability many studies were carried out [18, 19, 20] in order to provide the possibility to use in the application the different and heterogeneous metadata standards and formats, and also to allow the exchange of metadata between systems and applications. All these approaches are focused on the interoperability problem, and they do not offer any possibility to enrich the metadata in function of their usage.
In this paper, we do not focus on the metadata model, but rather on the temperature concept. In order to illustrate this concept we present a preliminary metadata framework that allows the integration of existing metadata models and provides the possibility of enriching them through the usage. Our approach takes into account the users and their behaviour regarding the consumption of the retrieved documents and their associated metadata. The notion of temperature can be applied to any hierarchical metadata model.
In our model, Fig 1, we couple each multimedia content with a unique metadata file, that contains the whole set of metadata related to that document. The link between the two documents is done through the documentSrc attribute from the Meta_Document metadata. As a multimedia document can be composed by different media types, its metadata can be formed by many Meta_Documents, each one corresponding to one media from the multimedia content. Each Meta_Document is divided in two parts: (1) General_Metadata, which corresponds to the general metadata, such as the life-cycle and the identification metadata (e.g., the creator, the description); (2) Media_Metadata, which corresponds to the media specific metadata.
In order to be as generic as possible and to allow the integration of different existing metadata standards, we decomposed the two parts presented above in Units. Each Unit represents a metadata element, e.g., the author. It has as attributes the name of the metadata, its type and, eventually, a definition or a reference to its definition that is provided into a thesaurus. Depending on the application’s needs, a Unit can be decomposed in one or more Units. The actual value of each Unit is specified in a different element, Value, which has as attribute the source of the value, e.g., the metadata standard that provided the metadata element.
The usage metadata, the temperature, is associated to each element of the metadata format presented above. More precisely, every metadata element from the proposed framework has associated two kinds of temperature: (1) one computed for each group of users that interacts with the multimedia content, and (2) an average one for each metadata element, that is computed in function of each groups’ temperatures.
In this paper, we do not focus on the determination of the users’ groups that we use in our approach. We consider that these groups are already established and that they can evolve over time. In our work, the different groups can be disjoint or not, a user can belong to at least one group and over time he can migrate from one group to another. An approach for the creation of such groups, based on the users’ interests, is defined in [21]. The advantage of using users groups is that in this way the temperature can be used for personalisation purposes. The algorithms presented in the reminder of the paper work regardless the number of user groups defined; it works as well for single users. 4
We consider the user as an important source of implicit metadata, because he can produce metadata by interacting with the multimedia documents he obtains as results to his query. In our proposal we focus on exploiting the user’s behaviour.
In order to be able to respond to as many users’ queries as possible, in an information retrieval system many different indexing algorithms are applied. Thus, the multimedia metadata obtained are heterogeneous, from simple low-level features to more complex semantic high-level features. Usually, not all the generated metadata are used in the retrieval process. There are some metadata that are used more often than others. For this reason, we propose to enrich the metadata obtained after the indexation process with the concept of temperature. Thus, the more the documents or their associated metadata are used, the hotter they are.
We have attached the temperature to (1) the multimedia document (at the Meta_Document level in the metadata framework presented in the previous section) and also to (2) their associated metadata (the temperature attached to each metadata element in the proposed framework). This popularity metadata can be used, for example, in the query process. In the execution of a query, the popularity metadata is taken into consideration in the computation of the results’ score. This way the popular documents and segments of documents are better ranked.
Fig. 2. User’s actions in an information retrieval system
The above picture resumes the actions that a user makes when interacting with an information retrieval system. Based on these considerations, we propose to realise the metadata enrichment by taking into consideration the user’s interaction with the metadata associated to query results (step 3 in Fig. 2) and with the multimedia document (step 6 in Fig. 2). First, we describe in Section 4.1 the metadata enrichment algorithm and then, in the next sections we present its concrete application based on the user’s interaction with the results list, Section 4.2, and with the multimedia document, Section 4.3. 4.1
Independently of the manner the decision of the increasing of the temperature is taken, the temperature is computed for each period of time ∆t and it depends on the number of users that have consumed the metadata in that period. The temperature is defined as t in with 0≤ t ≤ 100. The initial value of the temperature of all the documents and of their associated metadata is 0. The algorithm used for the increase of the temperature is presented in Table 1.
The parameters of the proposed algorithm are: the metadata whose temperature has to be increased, the number of users that consumed the metadata and the identifier of the group these users belong to. The first step of the algorithm is the computation of the metadata’s temperature corresponding to the user group received as parameter. Afterwards, the average temperature of the metadata element is computed as an arithmetic mean of the temperatures associated to this metadata, corresponding to each user group in the system. For the computation of this average temperature can be use also weighted mean.
Each time the temperature of a metadata is modified using the increaseTemperature method, this modification is propagated to all its children metadata. The propagation method is presented in Table 2. It follows the same steps as the first algorithm. The temperature of each child metadata is changed with a value that is directly proportional with the variation of the temperature at the first level and with the level in the metadata hierarchy where the current element is. This propagation can be limited to a certain level in the hierarchy, specified by the MAXLevel constant
We apply the same reasoning for the propagation of the temperature to all the ancestors of the metadata element that initiated the process of temperature increasing. The propagation method is presented in Table 3. In the computation of the new temperature we follow the same rules as for the propagation to the child elements. md ← setAvgTemperature(md, avgTemp); if MD has children then
└ md ← propagateTemperatureDown(md, gID, Δtemp, 1); if MD has parent then
└ md ← propagateTemperatureUp(MD.parent, gID, n, Δtemp, 1); return md; Algorithm 2: propagateTemperatureDown
Input: The metadata, MD, for which we want to increase the temperature of the children; the identifier of the group, gID, for which the temperature has to be increased; the temperature Δtemp, that is used for the computation of the new temperature; the level of the recursive call Output: the metadata with the temperature of all its children increased inc ← computeTemperature(Δtemp); foreach child of MD do │ md ← md U setGroupTemperature(child, gID, inc); │ setHistory(child, gID, inc); │ avgTemp ← computeAvgTemperature(child); │ md ← md U setAvgTemperature(child, avgTemp); │ if level<MAXLevel then └ └ md ← md U propagateTemperatureDown(child, gID, inc, 1evel+1); return md; Algorithm 3: propagateTemperatureUp
Input: The metadata, MD, for which we want to increase the temperature of the ancestors; the identifier of the group, gID, for which the temperature has to be increased; the temperature Δtemp, that is used for the computation of the new temperature; the level of the recursive call Output: the metadata with the temperature of all its ancestors increased if MD ≠ null then │ inc ← computeTemperature(Δtemp); │ md ← md U setGroupTemperature(MD, gID, inc); │ setHistory(MD, gID, inc); │ avgTemp ← computeAvgTemperature(MD); │ md ← md U setAvgTemperature(MD, avgTemp); │ if level<MAXLevel then └ └ md ← md U propagateTemperatureUp(MD.parent, gID, inc, 1evel+1); return md; │ │ dec = getLastHistoryValue(child, history, gID); │ │ history ← removeLastHistoryValue(child, history, gID); │ └ md ← md U setGroupTemperature(child, gID, dec); │ avgTemp ← computeAvgTemperature(child); │ md ← md U setAvgTemperature(child, avgTemp); └ md ← md U decreaseTemperature(child, history, N); return md;
If the temperature is augmented all the time, then after a certain period, the temperature of all metadata will attend the maximal value. In order to avoid this, the temperature of the unused metadata is reduced with a Δtemp proportional with the number of users who has consumed them in the period before. This is done for all the metadata elements and for all the users’ groups. In order to realise this operation at each recalculation of the temperature of a metadata element, the variation is stored in a history file. The algorithm used for decreasing the temperature is illustrated in Table 4. This decreasing process can be applied after each time period Δt, or after a certain number of intervals Δt. 4.2
In a classical information retrieval system the user sends his/her query to the system and retrieves some results. These results are ranked in function of the score they have obtained after the query execution over the metadata collection. In a typical results list, each result is composed of a link to the multimedia content and the metadata associated to this content. The system’s human interface displays the results as a list. The metadata associated to each result is presented as a collapsed tree.
In order to find the most relevant document for him, the user examines first the metadata associated to the documents in the result list. This action consists, en fact, in the expansion of the displayed metadata tree, until a certain level. Another way of collecting this kind of information, in a less intrusive manner, could be the gaze tracking [22]. It can be considered as a metadata consumption and it has an influence on the metadata’s temperature.
In order to illustrate the temperature’s computation we take into consideration fist scenario of utilisation. Suppose that only a part of the metadata related to the results are displayed (e.g., the General_Metadata in the proposed framework) and that the user has the possibility to access the rest of the result’s metadata. If for the same document, in a certain time interval Δt, several users, belonging to the same group, have accessed the same additional metadata by expanding it, then the temperature of the expanded metadata is augmented with a value Δtemp proportional with the number of users (n) which have consumed them. Only the temperature corresponding to the group to which the users belong will be recalculated. More precisely, the Temperature element with the userGroupID equal to the users’ group ID will be modified for all the metadata elements displayed. According to the algorithm previously described, this change in temperature will be propagated to all the children of the expanded elements and to their ancestors as well.
In order to illustrate in more details this enrichment process, we consider the following situation: a multimedia information retrieval system where we have identified several groups of users. For readability reasons, in the examples we present in this paper we will consider only two groups. In this system, a user belonging to the first group obtains the image DSC_2249.jpg as a result to a certain query. The system displays the metadata description in the form of a collapsed tree, as the one presented in Fig.3 a). This user expands the General_Metadata element until a certain level, as displayed in Fig.3. b). In the same time other 9 users from the same group access the same metadata. Thus the metadata associated to this image will be increased. The function increaseTemperature is applied for the following metadata elements: <Value source=”DC”>; <Value source=”EXIF”> and <Unit name=”creationDate”>. For the last two elements, the temperature will be increased with a smaller value than the first one because they were not expanded until the last leaf. b) Fig. 3. a) Metadata displayed with a query result; b) The same metadata after the user interaction with it 4.3
After the study of the results list, the user chooses a multimedia document and begins to interact with it: he explores the document, he studies in more details a part of the multimedia content, he spends an important period of time examining the document, etc.. This behaviour illustrates his interest in the document and in its compounds.
We augment the temperature of the metadata which correspond to the multimedia document’s compound the user is interested in. When the temperature of a component is changed the temperature of the document and of the other metadata elements that describe the component are modified as well. The information is also propagated to the higher levels in the metadata hierarchy.
In order to illustrate this metadata enrichment, we can consider the SMIL presentation from Fig. 4. This presentation is composed of a video and an audio content and the presentation’s slides as images. The organization in time of the presentation and the eventual audio and video segments are presented in Fig. 4.
For this example, we also consider an information retrieval system where two users’ groups were identified. Several users belonging to the same group have used the system in the same time and they obtained the same SMIL presentation as a result to their different queries. They all have selected the presentation and have watched it from the 1’20’’ until de 4’50’’. In this case, the temperature of the metadata associated to all the multimedia contents displayed in this period of time will be modified. From the timeline presented in Fig. 4 we can deduce that the video segments seg_Video1 and seg_Video2, the audio segments seg_Audio1, seg_Audio2 and seg_Audio3 and the images img2.jpg and img3.jpg are candidates for having the temperature modified. At this point several strategies can be established for choosing the segments to use for the metadata enrichment. For example, if the compound was watched for at least half of its length, then its temperature will be modified. In this case, the temperature of the audio segments seg_Audio1 and seg_Audio3 will not be modified, because they were listened for less than their length. Another possibility would be to increase the temperature for all the segments that were displayed, with a value proportional with the time that they were watched.
Through the proposed algorithm for the temperature increasing, the temperature of the entire presentation will be increased, as a consequence of the consumption of a part of it. We can note that the more a document is watched, hotter it gets.
In the next section, we present some possible utilizations of the temperature concept in the context of a broadcast use case. 5
In order to validate our proposal, we have applied the concept of temperature to a web site. In this case we used the algorithm in function of the users’ interaction with the multimedia content. We consider a page of the site as a document and we associate to it metadata. The users’ interactions with the web page (e.g., clicks) are collected into a database. For the tests effectuated we considered only a group of users. We have instantiated the metadata framework by using the XML and XSD technologies and the algorithm was implemented in Java.
The Fig. 5 shows the obtained results for a web page temperature computation. The results show that the time granularity is very important in the application of our algorithm. For the same users interactions with the page the temperature obtained for the whole page is different in function of the strategies employed: compute the temperature each 24 hours, each 12 hours, each hour or less than an hour. The decrease strategy is also important when the time granularity chosen for the computation of the temperature is small. These choices are use case dependent. The curves in the Fig. 5. show that these considerations have an influence on the evolution of the temperature. Thus, making the good choice is important in the progress of the temperature. In this paper, we have presented a modality of multimedia metadata enrichment based on the users’ interaction with the multimedia content and with their associated metadata. This enrichment is done in two steps: (1) in function of the users’ interaction with the metadata and the results list and (2) in function of the users’ behaviour with the multimedia document.
We intend to implement and test our proposal in the context of the LINDO project (Large scale distributed INDexation of multimedia Objects) (http://lindo-itea.eu/) in order to determine the best parameters of the algorithm (e.g., time granularity, decrease strategy, the level of propagation). These parameters cannot be set without the intervention of the user, thus we will realise some qualitative interviews with a set of volunteers. In a first time we will implement the second scenario for the computation of the temperature (presented in Section 4.3). After the specification of the parameters we will take the experiments a little further, by using the temperature for the metadata management in a distributed system [23].
Acknowledgments: This work has been supported by the EUREKA Project LINDO (ITEA2 – 06011). 23.