-aceMedia content analysis capabilities are centered around the concept of the ACE. The ACE is composed of a content layer, a metadata layer and an intelligence layer. In this paper we show one application of the ACE Intelligence layer and how its proactive conduct can help in the complex task of adding semantic metadata to multimedia content.
Index Terms—multimedia content analysis, proactive content, self-annotation
Digital multimedia content management is a very complex task. Huge efforts in research and development are being carried out in industry and academia to alleviate this complexity and bring solutions to help end-users and professionals to easily manage their collections of multimedia content.
The aceMedia project tries to help tackle this problem with
a wide range of technologies, from multimedia knowledge
representation [
The intelligence layer is envisaged to help in the complex problem of multimedia content management by enabling the content items to perform actions on behalf of the user, wherever they reside.
This paper briefly describes one of the applications of content proactiveness enabled by aceMedia technologies, namely, the creation of self-annotating content.
Content autonomy is not limited to self-annotation, other
activities based on the ACE intelligence layer are also carried
out [
Purely automatic annotations have a long way to go to
provide the user with accurate semantic annotations. The
semantic metadata associated with the content can be
improved with the help of the user. Some of our studies,
contrary to some common beliefs, showed that users are
willing to "help the system” with their manual annotations [
To incorporate users’ manual annotations we will create proactive content that analyzes its own automatic semantic annotations and asks the user pertinent questions to solve some ambiguities or add some unknown information, e.g. a face that has been detected is not known to the face recognition module and the ACE asks the user “Who is the person whose face is inside the bounding box?" The user, always in control, can obviously ignore these questions as the system is not strictly depending on them.
In the previous section we have seen some motivations for giving autonomy and proactiveness to multimedia content when we try to add manual or automatic semantic annotations to an ACE. It is important to emphasize here that the whole process of self-annotation and the ultimate decision to add semantic annotation to an ACE resides in the ACE itself.
We will not go into detail about the software architecture
that enables ACEs to run their programmable Intelligence
layer in order to give them autonomy; a brief description of
this can be found in [
Content is analyzed by different content analysis modules
that produce semantic metadata which in turn is added to the
ACE metadata layer. A typical, application driven, annotation
process is described in [
In our case of self-annotation, it is important to clarify that the ACE intelligence layer is in charge of starting/stopping the annotation process and decides which, if any, content analysis needs to be run. The ACE programmable intelligence layer does not include the analysis algorithms that analyze and produce new metadata.
As explained in the previous section, the modules in charge of analyzing the content and adding new metadata can differ from one device to the other and are offered to the ACE intelligence layer through a common framework called the Annotation-Manager.
This AnnotationManager interacts with the ACE intelligence layer and runs the requested analysis modules in the appropriate order. The AnnotationManager also deals with dependencies, e.g. a face recognition module may depend on a face detection module.
Once the AnnotationManager has called the analysis modules requested by the ACE, it will always run the Multimedia Reasoning module to ensure metadata consistency, remove ambiguities and derive new annotations if possible.
In this section we will describe a typical self-annotation process. As explained in the previous section, the SelfAnnotating ACE is in control of the annotation process but it does not perform the analysis nor the annotations itself. This way, the ACE can benefit from the different capabilities offered by different devices and contexts, see section II.
When an ACE is transferred to a different device, its selfannotation process is started.
The self-annotating Intelligence layer, asks the device what semantic annotation capabilities are present in the device. This request is received by the AnnotationManager.
The AnnotationManager will analyze the kind of content stored in the ACE, i.e. whether it is a still image, a video clip, or any other type of media. Based on this analysis the AnnotationManager decides what analysis capabilities it can offer, e.g. face detection, face recognition, speech recognition, knowledge-assisted analysis, etc.
The ACE checks if this type of annotation has already been performed, and creates a list of the missing annotation categories it is interested in. The ACE sends this list to the AnnotationManager that calls, in the appropriate order, the analysis modules and solves dependencies if needed. The resulting semantic annotations are then added to the ACE metadata layer and finally, as explained before, the Multimedia Reasoning module is called.
We have just outlined a very simple implementation of a self-annotation ACE. The ACE self-annotating intelligence layer can indeed be programmed to perform more complex tasks and take other decisions such as raising questions to the user (see Section II) or prevent certain analysis to be performed (because of privacy issues for example).
One of the objectives of aceMedia is to explore advanced content management techniques through the concept of the ACE and its Intelligence layer. aceMedia has successfully created a framework for the deployment of Autonomous Content Entities (ACEs). These ACEs can have proactive behavior that helps users in their digital media management. We have briefly outlined in this paper one of the applications of the ACE Intelligence layer, the creation of self-annotating content. We have presented the motivation which led us to make ACEs self-annotating as opposed to being annotated passively. Finally we have outlined the process and workflow of self-annotation.
Within aceMedia we are investigating other applications of the ACE Intelligence layer that are outside the scope of this paper such as self-organizing ACEs and self-governing ACEs.