Collaborative tagging systems have become rather popular for annotating any kind of resources ranging from electronic documents to real world objects. In current tagging systems resources as a whole are annotated with and referenced by user defined tags. For multimedia data, as e. g. for video data, single scenes can be identified and annotated by using MPEG-7 metadata. We propose a collaborative tagging system that is combined with an automated annotation system for synchronized multimedia presentations. MPEG-7 metadata are used for the annotation of single scenes with user compiled tagging information in combination with metadata provided directly by the author or by other annotation systems. Thus, we propose a system being able to search within multimedia data that can further be extended to search within any kind of (partial) document to achieve a more tightly focused and personalized search.
Online Social Networking enables collaboration relationships and allows exploiting these relationships for automated information distribution and classification. In particular, collaborative tagging systems (CTS) have become increasingly popular for annotating any kind of electronic documents (e. g. web pages, images, videos) or even real world objects (e. g. books, consumer goods, people). In a CTS the users assign freely chosen terms (i. e. tags) to specific resources with the purpose of referencing those resources later on with the help of the assigned tags.
By considering also other users’ tags serendipitous
discovery of new, previously unknown resources is possible via
so called tag browsing, i. e. all resources that are annotated
with the same tag(s) as a decisive resource will be referenced.
For an overview of CTS see [
On the other hand, there are manual multimedia data
annotation systems that enable the user to connect personal
annotations to single scenes of a video recording [
CTS enable personalized annotation of resources that can be utilized collaboratively by all users. YouTube [2] is a rather popular system for the collaborative annotation of video data. But, YouTube only allows the annotation of the video data as a whole and not the annotation of single parts of a video document. The majority of available video clips in YouTube is rather short and most times those clips only cover a single subject. Thus, for YouTube it is probably not necessary to provide a possibility for partial document annotation. Our system is focused on lecture recordings, where most lectures cover a variety of different topics. By providing partial document annotation facilities the user is able to annotate single video scenes that are related to a specific topic according to his own interests. By considering also those annotations that have been provided by other users, the system enables the discovery of related (similar) video scenes by tag browsing.
This section describes how MPEG-7 metadata can be used to maintain collaborative tagging information. MPEG-7 is an XML based markup language for the description of multimedia metadata. Besides various standard metadata information MPEG-7 enables the identification and annotation of distinct spatial and temporal segments within multimedia data. For our purpose, the description of temporal decomposition of video data is essential. Thereby, MPEG-7 allows the identification and annotation of overlapping temporal segments, which is a prerequisite for storing collaborative tagging information that is provided by different users. Video segments can be annotated with various information by utilizing the <TemporalDecomposition> tag of the MPEG7 metadata description scheme. Each video segment is identified and annotated with the <VideoSegment> element (see Fig. 1). Within each <VideoSegment> the elements <MediaTimePoint> and <MediaDuration> specify the segment’s temporal location within the video stream (see Fig. 2). For textual annotation MPEG-7 provides the tags <KeywordAnnotation>, <FreeTextAnnotation>, and <StructuredAnnotation>. The information connected to these tags can be utilized for a keyword based search within the video data facilitating a fine-grained access. <Mpeg7 xmlns=” . . . ”> <Description x s i : t y p e=” Cont e nt Ent i t y T y pe ”> . . . <MultimediaContent x s i : t y p e=” VideoType”> <Video> <MediaInformation> . . . <TemporalDecomposition> <VideoSegment> . . . </VideoSegment> <VideoSegment> . . . </VideoSegment> . . .
</TemporalDecomposition> </Video> </MultimediaContent> </ Description> </Mpeg7> where a set of tags is supplemented by user, date, and auxiliary (optional) rating information. Therefore, instead of the <Keyword> element we use the <MediaReview> element, which allows a video segment to be annotated with user specific textual information including also a rating indicator (see Fig. 3). The tagset denotes the set of all tags that a distinct user has employed to annotate a video segment. It is <CreationInformation> <C l a s s i f i c a t i o n> <MediaReview> <Rating> <RatingValue>9 . 1</RatingValue> <RatingScheme s t y l e=” h i g h e r B e t t e r”/> </Rating> <FreeTextReview>
tag1 , tag2 , tag3 </FreeTextReview> <ReviewReference> <CreationInformation>
<Date> . . . </Date> </CreationInformation> </ReviewReference> <Reviewer x s i : t y p e=” PersonType” >
<Name>Harald Sack</Name> </Reviewer> </MediaReview> <MediaReview> . . . </MediaReview> </ C l a s s i f i c a t i o n> </ CreationInformation> represented as comma-separated list of tags and is encoded in the <FreeTextReview> element. The date of the last modification of the tagset is encoded with the <CreationInformation> element. The user identification is encoded in the <Reviewer> element, which is derived from the MPEG-7 agent type. Furthermore, an optional rating indicator can be included to enable the ranking of video content. Thus, the <MediaReview> element provides the possibility to store all necessary collaborative tagging information. The <MediaReview> element is embedded inside the <CreationInformation> and <Classification> elements of a video segment. Within the <Classification> element several different <MediaReview> elements can be combined that each represent annotations from different users. 3.2
For collaborative tagging of video segments the design of an efficient user interface is mandatory. Thus, we define three distinguished areas in the browser’s user interface: the video display area (1), the tag display area (2), and the tag/segment definition area (3) (see Fig. 4 for an overview of the user interface). The tag display is organized as a tag cloud (2). The single tags are ordered alphabetically while their font size indicates additional information that can refer to frequency of usage or tag rating (according to the relevance indicator). We consider different display modes: either personal or popular tags can be displayed, while a static view includes all tags for the entire video in difference to a dynamic view that refers to tags used at a distinct point in time within the video. By pointing at a tag with the mouse device a list of video segments annotated with that tag will be displayed in a separate window (4). There, the video segments are represented by a miniature screen shot and by their starting time and end time. The user can select a particular video segment from the list for playback. On the other hand, the user has to get an overview of all (non disjunctive) segments that have already been annotated in the video. This information is displayed within a coordinate system with the x-axis representing the timeline and the yaxis representing overlapping sequences (5). By pointing at a video sequence within the coordinate system all tags referring to that segment are displayed. Besides user annotation, we also consider annotations provided by the author of a video resource. These annotations can include structural informations (cut points) as well as semantic information (tags, headings, comments). The interface provides the possibility to use the annotation given by the author as a default starting point for user dependent annotation. Alternatively, the video can be pre-cut at fixed time intervals that can be fine-tuned by the user. For selecting a new video sequence to be annotated, the user is able to mark starting time and end time simply by clicking special buttons in the video display during playback or/and by adjusting those cut points in a separate timeline display (6). After selecting a video sequence the user is able to add his tags in a separate tag definition window (7). For faster processing it is possible to place tags just at a specific point in time during the video playback without denoting an entire segment. Then, starting point and end point of a sequence being annotated with that tag is chosen using predefined or author-given cutpoints. To consider the most important parts of a video a rating index is displayed along a separate timeline (8). 3.3
CTS enable different ways of searching the system’s resources that can be adapted to our multimedia search: Personalized Search By utilizing his own set of tags the user is able to perform a search based on his personal information needs. These tags can be descriptive or functional by nature, i. e. they either describe a resource in general – and thus, are also useful for other users – or they draw the focus on a certain aspect that (most times) is only relevant for the user who supplied it. Esp. the functional tags are suitable to extend a general search according to personal information needs. As e.g., the user might tag several sequences of a lecture video that are relevant for an examination with the tag exam.
General Search By considering the (descriptive) tags of all users in combination with the original MPEG-7 annotations of the resource’s author, a general keyword-based search can be performed.
Tag Browsing Here, we refer to the retrieval of all resources that are annotated with the same tags as a specific resource under current consideration. Now, esp. those resources become important that have been annotated with the same tags, but by other users. In that way the user is able to discover new resources that are considered to be similar to the original resource.
Social Networking Additionally, in CTS the inherent social network of users can be considered. To participate in a CTS the user has to register which often includes the delivery of a personal profile. Thus, a social network can be defined connecting users that are considered to be similar according to their profiles. On the other hand, users that have annotated the same resource (probably even with the same tags) can be considered to be similar. Thus, by browsing resources that have been annotated by similar users, new relevant resources can be discovered. 4.
We have shown how to integrate collaborative tagging information within a MPEG-7 framework to facilitate a search function on multimedia data that is able to deliver distinct parts of interest within a multimedia document. In difference to current CTS our approach allows the annotation of partial documents which is important esp. for timedependent media, as e. g., video data. A prototype of the proposed system for collaborative video scene tagging and retrieval is under current development.
The concept of collaboratively annotating partial video
documents can be extended for other types of media, as e. g.,
for large text documents (textbooks). There, the users
(document readers) should have the possibility to annotate
distinct sections of the text document and to benefit from these
annotations in a personal or collaborative way. One way to
facilitate the identification of distinct sections within any
type of document can be realized with the help of the
document object model (DOM) [
[1] Ricoh movie tool,
http://m7itb.nist.gov/M7Validation.html. [2] YouTube - video sharing and tagging system, http://www.youtube.com/.