Current search technologies can only harness the ever increasing amount of multimedia data when su cient metadata exists. Several annotations are already available, yet they seldom cover all aspects. The generation of additional metadata proves costly; therefore e cient multimedia retrieval requires automated annotation methods. Current feature extraction algorithms are limited because they do not take context into account. In this article, we indicate how Linked Data can provide information that is vital to create an interpretation context. As a result, advanced interactions between algorithms, information and context will enable more advanced interpretation of multimedia data. Eventually, this will re ect in better search possibilities for the end user.
A tremendous increase in multimedia content production and consumption
characterized the last decade and will continue to shape the Web for generations
to come. The biggest challenge is to serve consumers the content they need in
a convenient format and in a seemingly instantaneous way. The availability of
di erent search and browsing options (e.g., keyword search, faceted search, and
content-based search) enables e cient retrieval, yet these techniques require rich
metadata [
Metadata generation remains a tedious and often manual task that fortunately can be assisted by automatic algorithms. However, these algorithms produce inexact results with an often unknown reliability. In this paper, we argue that Linked Data can play a crucial role in the annotation and interpretation of multimedia data by assisting algorithms. Additionally, the results of this annotation task can be published back to the Semantic Web, contributing to the growth of the Linked Data Cloud. This feedback loop proves important for annotating future data, as depicted in Fig. 1.
The success rate of current search techniques strongly depends on the availability of correct annotations, as illustrated below. Multimedia
Item
Linked Data
Cloud Annotation
Process
Annotated Multimedia
Item
Keyword search Current search engines determine the content of an image by
that of the surrounding text, which is inadequate for two reasons:
1. Although mostly a strong correlation between image contents and surrounding
text exists, this is not always the case. Trivial examples account for this
(e.g., try searching for caption or alt), as well as more realistic examples (e.g.,
a search for two people names often also returns other people).
2. Even a strong correlation does not imply understanding of the image contents.
[
Another major bene t of rich annotations is that they enable us to link imagery of subjects (such as George Washington and the White House) to their corresponding entities in the Linked Data Cloud, for example by using their DBpedia identi ers. Therefore, related images can show up on any search that results in entities from the Linked Data Cloud. That way, it would be possible to automatically provide keyword and faceted search on images, using the combined available knowledge about the entity.
Not all annotations have to be generated: some are already present upon acquisition of the multimedia item, others are added by consumers or people in a social network situation. 2.1
The EXIF data format, describing metadata in digital photographs, is widely
known [
Another common source of metadata, are annotations added by consumers upon
publication of their material. Moreover, in some social networks, users are
allowed to tag the content of others. Well-known examples are tags describing a
variety of topics (place, time, scenery, people. . . ). These tags however, su er from
the problem that they are informal. Therefore, several e orts to enhance their
quality exist [
Importantly, there is a growing tendency that stimulates users { perhaps unknowingly { to provide more formal tags. As examples, we cite the practices of person tagging, which creates a semantic link between a person and a photograph, and place tagging of photographs. Modern user interfaces have rendered this task intuitive for the majority of Internet users. Of course, the responsibility for the accuracy of these tags lies with these (sometimes anonymous) users, which is therefore arbitrary. It is tempting to assume that the formality of these tags is an indication of their reliability, which is a fallacy. In fact, their ease of creation can quickly lead to many formal but incorrect tags. 2.3
In production environments, there is an increasing tendency to generate metadata
as part of the content creation process [
Automated multimedia analysis research has accomplished several milestones over the past decades in domains such as image feature extraction. While advanced techniques for various tasks exist, such as face recognition algorithms, most of them remain error-prone.
Another problem is that the output of these algorithms is often not formalized.
For instance, a face recognition algorithm may recognize a certain face in an
image, but does not output an entity that is linked to its corresponding Linked
Data counterpart. A solution to this issue has been proposed in [
Still, the main issue remains the missing ability of feature extraction algorithms to collaborate on the annotation of a given multimedia item. Each algorithm is highly specialized, which is both its greatest strength and its greatest weakness. This degree of specialization comes indeed at the expense of losing an overview on the item under annotation. Humans, in contrast, possess the remarkable ability to shift rapidly between di erent levels of abstraction. This is why we can recognize faces in context, while we are not able to do so without.
Fig. 2 shows a clear example of a photograph of a face with and without context. It is evident that no human and no algorithm now and in the future will be able to recognize the person depicted in the photograph on the left with an acceptable certainty, given no additional information. Our human ability to unconsciously zoom in and out between di erent detail levels, provides us exactly with this information, given the photograph on the right. Upon recognizing one person as Hillary Clinton, we instantaneously realize the other person is Bill Clinton with a fairly high certainty.
This clearly the importance of context-awareness for feature extraction
algorithms, a task that is hard because of the involved complexity. After all, it
is impossible for algorithms to anticipate on all possible context parameters.
Therefore, we should look into platforms that are able to integrate algorithms
and context by means of knowledge and reasoning [
Information that connects various concepts in di erent ways is readily available on several semantic data sources, including DBpedia. We can imagine an annotation platform consulting these data sources to retrieve information to either complete or validate found results. The annotation process could in fact entirely take place on the Semantic Web and consist of: { general and speci c knowledge from the Linked Data Cloud; { intelligent services such as feature extraction algorithms; { a Semantic Web reasoner applying rule-based knowledge to entities. The general knowledge could drive the process, deriving concrete knowledge about a speci c multimedia item.
Linked Data knowledge On the one hand, automated interpretation requires
general knowledge about annotations, consisting of both ontologic and rule-based
knowledge. In the case of images, an example of the former is \images can contain
regions that depict a face", and an example of the latter \if an image contains
a region that depicts a person's face, then the image depicts the person". This
general knowledge helps the platform decompose the task of interpreting an image
into smaller subtasks. On the other hand, speci c knowledge about concrete topics
is essential to provide a context for the item under interpretation. Their presence
proves vital for complex reasoning schemes. Examples include ontological and
instance-based knowledge about people and interpersonal relations.
Intelligent services Automated annotation algorithms could interact with
Semantic Web knowledge if we approach algorithms as Semantic Web services [
As an example, we consider a face recognition algorithm that operates on Fig. 2. The visible features of Bill Clinton alone will be insu cient to achieve successful recognition. The nearby presence of Hillary Clinton could substantiate the assumption that the other person relates to her in some way. A search on the Linked Data Cloud for relatives, friends and co-workers would yield a list of possibilities which we can pass in a semantic way to the recognition service. As a result, the algorithm could take these suggestions into consideration and limit its search space, signi cantly increasing the odds of arriving at the correct conclusion. We could then further cross-check the solution by checking with available metadata, such as GPS location and time, and maybe retrieve the event at which the photograph was taken. Reasoning Semantic Web reasoning plays a crucial role in this application domain. In the above example, we intuitively assumed that the simultaneous depiction of two persons implies some kind of connection between them. This knowledge, which can be derived for instance by statistic methods, needs to be available formally. A reasoner is then able to instantiate this knowledge on concrete entities, e.g., to retrieve people connected to Hillary Clinton.
An important topic here is dealing with imperfections. Most data in the Linked Data Cloud are represented as absolute facts without association about possible vagueness or uncertainty. While this is sometimes applicable and even desired, automated interpretation inherently needs to deal with predictions and assumptions. As a consequence, these imperfections need to propagate when reasoning on intermediate data. 4.2
Furthermore, results from an annotation process can also be pushed back to the Linked Data Cloud. As indicated, multimedia searches are an interesting and important application. Feedback enables other useful applications as well, e.g.: { New data could be inferenced based on the generated annotations. For example, the fact that several people were at a given place at a given time could indicate their attendance of an event. { Feedback data could enhance future annotation tasks, for example by means of statistics of which people appear together frequently.
In general, feedback will potentially link many di erent concepts, contributing to the coherence of the Linked Data Cloud while at the same time complementing it with multimedia data. 4.3
Fig. 3. Use of linked data in a context-aware interpretation process
In this article, we outlined how Linked Data forms an cornerstone of a contextbased multimedia interpretation process. The incorporation of Linked Data creates a holistic view that integrates the di erent aspects of annotating. Several important topics for future research emerge, including: { the interaction of feature extraction algorithms with Linked Data; { the representation of uncertainties associated with multimedia interpretation; { the feedback of the interpretation process towards the Linked Data Cloud. All these topics illustrate the enormous potential of Linked Data for advanced automated multimedia interpretation.
The research activities as described in this paper were funded by Ghent University, the Interdisciplinary Institute for Broadband Technology (IBBT), the Institute for the Promotion of Innovation by Science and Technology in Flanders (IWT), the Fund for Scienti c Research Flanders (FWO-Flanders), and the European Union.