The quality of content-based recommendation depends to a very high degree on the quality of the metadata available. We propose a work ow that combines novel cross media analysis platforms with linked data analysis to generate recommendations. The focus is set on an \editor user story" that combines live analysis of currently created content with a stored data backlog to select suitable content for article enrichment. Nowadays, a high percentage of (online) computer systems can be supported by recommender systems, assisting the decision taking process of customers. One of the biggest and well known domains of recommenders is the eld of shopping. Customers are presented with masses of choices to buy, while facing the problem of nding their (preferably perfect) t item. Recommenders work in the background in order to facilitate the process of nding relevant items for a given user. The recommender gathers information in order to generate a pro le for every user (and possibly every item), which in terms can be used to determine which items to recommend. Multimedia items are very similar to shopping items in a sense of supply and demand, so recommenders can also be used to facilitate various processes that involve the nding of multimedia. For example, knowledge discovery poses similar problems, as users are looking for content items that t their current needs and elds of interest. A main issue of this domain is not only content discovery, but it is also more di cult to generate so called features of items { characteristics and properties of the item { which are needed in order to compare them amongst each other. Extracting those features manually is a cumbersome task and too expensive in terms of time, even more aggravated by the sheer amount of data. Automated processes can help to overcome this step, but need careful setup in order to generate useful results. To overcome the aforementioned shortcomings of recommendation in knowledge discovery, the contributions of this paper are the design of a work ow that: { is based on linked data analysis results, possibly allowing the reuse of data from other problem domains, { is placed in an autonomous platform, enabling automated analysis of inserted content, { can be altered and adjusted to given use cases, and { is kept cross-modal, allowing recommendations being generated and consumed across di erent multimedia types.
In order to convey this, the paper is structured as follows. Section 2 discusses related work in the elds of recommendation and multimedia platforms. After positioning the work, section 3 describes identi ed use cases and their requirements, then section 4 speci es the recommendation work ow in detail. Section 5 will conclude this position paper depicting ongoing future work. 2
The two main foundations of this work are the elds of Recommender Systems and Media Analysis Platforms. The next two sections show how the presented work ts inside the vast amount of proposed recommender systems and introduces two possible analysis platforms. The actual analysis components are considered part of the platform.
Ricci et al. describe recommender systems in [
The majority of recommender systems and the research on recommender systems is divided into two categories or two sources of data: The rst approach is to observe user behaviour and draw conclusions from that, a technique called Collaborative Filtering. This assumes that within a certain user group, the user behaviour is similar. The second approach, Content Based Recommendation, relies on analysing the recommendation items, extracting certain features, and recommending similar items, based on that analysis.
The approach discussed here is clearly content based: Multimedia items will be processed by
an analysis platform, and suggested according to their semantic similarity. However, collaborative
ltering techniques can be used to reorder the list or prioritize found items. A recent survey of
recommender systems is provided by Bobadilla et al. in [
The work proposed in this paper was conducted within the scope of the MICO project3, a project with the aim to develop a multimodal analysis platform for various kinds of data. However, the proposed setting is not limited to MICO, but can be used by alternative frameworks as well.
The MICO project provides a platform that helps orchestrating di erent combinations of
registered extractors. As every extractor might produce its results in di erent varying formats, it
is desired to nd one common denominator in order to make full use of the combined metadata
and even enhance the degree of information by its recombination. One approach to achieve this
is posed by the Semantic Web and its technologies to produce Linked Data (LD). Among those,
the Resource Description Framework (RDF) [
We will extend this context in order to design a recommendation engine as part of a MICO work ow pipeline. Consequently, the recommendation process and especially its results can be stored as metadata, making it possible to generate cross-media recommendations based on metadata results of various extractor combinations.
A similar semantic web-based approach is proposed by De Meester et al. [
Their proposed framework meets state-of-the-art standards and analysis techniques and, in order to operate with an input base that is as broad and diverse as possible, is kept domain-agnostic by including multimedia analysis procedures as web services. The output follows Semantic-Web standards, enabling it to be understood and interpreted at various di erent locations. When \injecting" a multimedia item, the platform makes use of the (possibly) already present metadata background. Then, it induces a three step algorithm that can be re-iterated as often as needed, until no more change in metadata and an associated increase of quality is achieved. At rst, the algorithm combines the results of di erent analysis processes that are registered at the platform with the already present metadata. Then it can determine which results need improvement before deciding a new analysis plan for the given multimedia item. This results in a rich and robust metadata background. 3
As discussed in the previous section, content based recommendation depends to a large amount on the quality of the content analysis. Cross media analysis, i.e., combined analysis on di erent media types on the input side (e.g., video and corresponding user responses) is one promising approach to obtain the needed metadata quality.
The main user story we focus on in this report is the enrichment of journalistic articles with tting media items, the so called editor support usecase. Within MICO, there is more: for example, a showcase partner, Zooniverse4, has a related use case in which crowd-sourced discussions about a given item can generate recommendations or pointers to other discussions and consecutively other items. In this case, both the items and user discussions are analysed.
The proposed recommendation work ow is designed to deal with both problems, the main di erence are the used analysis components inside the platform.
In today's web, it is vital practice, to add (linked) content to an article for reasons of giving the reader further information and commitment to the site, but also for search engine optimization. Independent of the motivation, an editor pro ts highly from a recommendation system, that suggests tting items while the new article is written. On a high level, the related user story is: As an editor, while I create or edit articles using WordPress, I want to automatically get related articles and videos that I might link to the article.
Editing Platform
Analysis Platform
5 LD Matching 2 6
LD Cache 7
Recommendation As a rst step for generating recommendations, we implemented a pipeline designed to recommend content videos or text-comments to a given user writing her or his own comment. We will base this on the general \meaning" or topic of both, which is done by a named entity recognition (NER) component. The recommendation exposes a REST API, that can be used by a WordPress plugin to acquire the recommendation data. 4
1 (Continuous process) A crawler feeds the domain speci c media items (news, videos, ...) to the MICO analysis platform 2 (Continuous process) Analysis results are stored as RDF data, e.g., inside MICO's Marmotta5, a Linked Data platform 3 An item gets into focus, e.g., someone is writing an article on a speci c topic or a new user post is published somewhere 4 The editing platform feeds the item to the MICO platform 5 The analysis results are preprocessed by the LD Matching component 6 The LD Matching component queries the stored annotation results 7 The matching component calculates a similarity score and feeds relevant items back to the editing platform as a recommendation to the editing platform Editing Platform The term Editing Platform stands for a component that produces content that has to be analysed and matched to the cached analysis results. A good example for this is a content management system like WordPress or TYPO3. All major systems allow to integrate plugins, therefore the communication with the new service is assumed to be a plugin that communicates with the REST endpoint of the recommender service.
Content Crawler This work ow assumes that every use case has a de ned subset of content that is supposed to be recommended. This might be an internal archive of media les, e.g., a in-house video collection or public databases, e.g. YouTube or Wikimedia Commons. Licensing issues are out of scope of this paper, however it should be noted that the task of storing and evaluating license information can be accomplished within the linked-data model as well. For the prototype implementation, the Copyright Aware Crawler 6, developed within the CUbRIK7 project is used. All the crawled content is forwarded to the analysis platform for semantic indexing and further use in the recommendation process.
Analysis Platform & LD Cache As described in section 2, a central part of the proposed work ow is an analysis platform that is able to extract the desired features needed for a content based recommendation and is able to output its result in a linked data format, to pro t from the additional semantics inside the following LD Matching step. The LD Cache component emphasizes the need of the Matching component to access precomputed analysis results. Depending on the used architecture it can be integrated into the analysis platform, as it is the case for MICO. LD Matching As described in section 2, current recommender systems apply metrics to describe the similarity of concepts, behaviours, users, items or related item data. We adopt this kind of content-based approach, motivating the similarity of multimedia items on the NER linked data analysis produced by the MICO platform. Envisioned is an implementation of a component capable of computing a similarity score of stored videos towards written comments. By looking into the \meaning" of given analysis results the LD Matching component will try to identify the major topics of the associated resource. These will be used by a matching logic to gather relevant resources stored in the backing RDF store. To attain this goal it will try to group named entities into categories weighted by importance. Finding tting categorisations of the initial input can be interpreted as a mapping of the given named entities into a semantic type hierarchy. The cooccurrence of relations of instances of these types as well as shared instances and similar graph structure properties form features for the similarity computation of types and their contained instances.
By interpreting this hierarchy a semantic distance metric will be derived as foundation for the recommendation system. This method will try to make use of non-textual characteristics implied by the RDF graph. Hereby it can be considered to apply information of newly analysed resources to expand the knowledge graph.
To clarify the idea of the matching process, consider following example in gure 2, showing an exemplary RDF subclass hierarchy: Such an hierarchy will be used in our approach to match the results given by the NER analysis. After that, the semantic similarity of the given classes can be calculated to ultimately compute the similarity of two given items. For example, considering three items i ower, ipanda, and ilion with the extracted meanings of \ ower", \panda", and \lion" respectively, one would assume that ipanda is much more similar to ilion than i ower because they are much \closer" in the hierarchy. As a result, a user writing a text about owers can receive recommendations about items related to animals, in case there are no tting videos about plants or trees that would be more similar. 4.1
The proposed work ow was showcased as a demo inside the MICO project. During the remainder of the project it will be fully implemented for two show cases: An integration into WordPress 6 http://www.idmt.fraunhofer.de/en/projects/expired_publicly_financed_research_projects/ cubrik.html#tabpanel-2 7 http://cordis.europa.eu/project/rcn/100872_en.html
Resource Plant
Animal Flower
Tree
Mammal
Bird Panda
Lion
Penguin
Fig. 2. Exemplary RDF class hierarchy. Solid arrows depict a direct subclass relationship, while dashed
arrows symbolise a path of (possibly multiple) subclass relationships towards the top class. In RDF, every
class is subclass of rdf:Resource, however, transitive edges derived through inference are not considered,
as this would break the assumption of having a semantic distance between two classes, as every class
reaches Resource with one \hop" to the Resource class.
that suggests related videos to the editor, based on speech-to-text results on the video, and NER
on the draft post. The second user story is about analysing user discussions and linking it with
content available for the named entities. A snapshot on the related activities for recommendation
inside the MICO project can be found in [
This paper describes a work ow which is used in order to generate recommendations in a work ow driven environment. The process makes use of linked metadata that is produced inside the MICO platform, generating the recommendations based on a named entity recognition extractor that excerpts the meaning or trend of a written user comment towards a given video. The platform then uses that information in order to nd (already categorised) tting similar videos to recommend. As the work ow as well as its underlying utilised analysis process is kept crossmodal, other use cases, dealing with other various multimedia formats that can be analysed by NER, can easily be supported.
Next steps will also include ways of increasing the quality or reliability of the generated recommendations. Using the design of the platform, a feedback loop is envisioned, in which users can rate the received recommendations. This information will then be used in the analysis process as well as the recommendation generation process.
The matching algorithm gives room for extensibility as well. Especially in the text analysis, for example textual feature-based approaches that use Word2Vec9, a framework designed by Google to compute vector representations of words, can be used in order to further interpret the similarities of extracted named entities. This promises further improvement of the overall results, as written posts can be analysed more explicitly.
Please write an email to the authors to get full access to the recommendation and platform repository. 9 https://code.google.com/archive/p/word2vec/
This work has been partially funded by the European Commission 7th Framework Program, under grant agreement no. 610480.