Providing rich and accurate metadata for indexing media content represents a major issue for enterprises ofering streaming entertainment services. Metadata information are usually exploited to boost the search capabilities for relevant contents and as such it can be used by recommendation algorithms for yielding recommendation lists matching user interests. In this context, we investigate the problem of associating suitable labels (or tag) to multimedia contents, that can accurately describe the topics associated with such contents. This task is usually performed by domain experts in a fully manual fashion that makes the overall process time-consuming and susceptible to errors. In this work we propose a Deep Learning based framework for semi-automatic, multi-label and semi-supervised classification. By integrating diferent data types (e.g., text, images, etc.) the approach allows for tagging media contents with specific labels. A preliminary experimentation conducted on a real dataset demonstrates the quality of the approach in terms of predictive accuracy.
Nowadays, entertainment industry represents one of the most profitable and widespread business sector, with a constant growth in terms of number of users. With estimated revenues amounting to about 2 trillion of dollars worldwide, providing efective research services is a crucial task for the companies operating in multimedia content delivery. In particular, the rise of streaming services and on-demand contents fostered the interest for AI-based solutions capable to facilitate the research and identification of contents matching the user interests. Just as an example, Recommender Systems (RS) are technologies widely adopted by big players (e.g., Netflix, Disney+, Amazon, etc.) to suggest items of their catalogues able to arouse users’ interest.
Besides them, the technologies that allow for enriching content metadata with informative labels (or tags) act a key role as they can exploited to improve the RS performances and simultaneously enable a more efective research by means of the traditional research engines. Basically, these labels are used to group contents exhibiting common features and provide aggregated views for the users. However, the labelling task is a time-consuming and prone to the error process since it is manually performed by domain experts. Indeed, the lack of a common shared taxonomy can lead to yield repeated labels describing the same concept. Moreover, the assignment of a label to a content is subjective and depends on the skill and perception of the expert.
In this scenario, Artificial Intelligence (AI) techniques represent a valuable tool to automate such a process by limiting the human factor and, as a consequence, reducing the classification error. Anyway, efectively addressing this problem requires the development of specific approaches able to cope with diferent hard issues, i.e., unbalancing of the classes, lack of labelled data, capability of the models to process diferent types of data (e.g., text, images, etc.) and providing multi-class predictions on an high number of labels.
In particular, Deep Learning (DL) paradigm [
In this work we propose to combine diferent types of data from diferent publicly available data sources for classifying media contents and enrich them with informative labels. In Figure 1, we sketched the overall learning process. After an Information Retrieval phase in which data are gathered and wrapped in a single view, these raw data are provided as input to Machine Learning block. Our solution adopts a hierarchical Deep Learning based approach: on top, an ensemble of pre-trained models (Embedder) are fine-tuned and used to map the input (text and/or images) in a low-dimensional space. Here, the main idea is that contents with similar labels generate similar vector representations (embeddings). Then, a clustering algorithm (Clusterer) is used to group similar contents and yield sub-samples of the original dataset. Finally, each sub-sample is exploited to learn a local model focused on a limited set of labels that allows for yielding more accurate predictions for specific cases. Although our approach is totally general and capable to handle diferent type of data by adding specific models to the Embedder, the current implementation focuses on analyzing text data. An experimental evaluation conducted on a real dataset containing movie plots demonstrates the quality of our approach in providing accurate predictions in this challenging scenario.
The rest of this paper is organized as follows: in Section 2 we provide an overview of the main approaches proposed in literature to tackle the automatic content tagging problem. In Section 3, we describe the framework used to address the problem and the deep learning architecture used to learn the multi-label classification model; while in Section 4 we discuss the experimental results. Section 5 concludes the work and introduces some new research lines.
The problem of classifying movies is not new in the literature and can be considered a general
classification task on heterogeneous (video, images, audio, text) data. Wu et al. [
The work that most resembles our approach is [
Wehrmann and Barros [
Fish et al. [
Table 1 summarizes the most significant approaches among those described above. Compared
to these approaches, there are some major diferences with regards to the problem we aim to
tackle: first, the tagging task is relative to a high number of labels. Second, this large number of
labels exhibits a long-tail distribution, as illustrated also in Figure 4 and discussed later in the
paper. To the best of our knowledge, our solution is the first approach that can handle large
Approach Dataset Number of tags DL architecture Data Type XMLC Multi-Modal Metric Result
Kar et al. [
In this section we illustrate our solution and the main components of the proposed DL-based architecture. As highlighted in Section 1, we adopted a hierarchical approach composed of three main components, as shown in Figure 2: (i) an Embedder, devoted to summarizing the original input into a vector representation (embedding); (ii) a cluster module (Clusterer) that allows for identifying media with similar contents and extracting focused sub-samples of the original dataset; and (iii) the local models that perform the final predictions.
Embedder. As mentioned above, the current implementation of our technique works on text
data (i.e., the movie plots), therefore our Embedder takes the form of a widely adopted
(pretrained) neural network i.e., BERT (Bidirectional Encoder Representations from Transformers)
[
In1981,atabus stop inSavannah,Georgia,amannamed… A gangofcriminals robaGotham City mobbank;theJokermanipulates themintomurderingeachother… Amidagalacticcivil war,Rebel Al iancespies havestolenplans to theGalactic Empire's DeathStar… … e m b e d d i n g . . .
representation of the input data. The main benefit of this approach relies on the possibility of
combinatorically increasing the input size and handle the lack of labelled examples.
Clusterer. A clustering algorithm is involved in the framework to group similar movies, thus
allowing the deployment of local classification models. Each cluster includes movies that share
a minimal number of tags with respect to the whole label space. Hence, this phase further
alleviates the extreme-classification problem. New instances, to be classified, are assigned to
the closest cluster on the basis of a suitable distance metric (in our case, euclidean distance is
adopted). As shown in Figure 2, we adopted the K-Means algorithm as our clusterer.
Local Models. In our framework, local models take the form of neural networks too.
Specifically, we exploited the DNN-based architecture, shown in Figure 3, to provide more accurate
predictions also for minority classes. The base building block of our model includes three types
of layers: (i) a fully-connected dense layer equipped with Rectified Linear Unit (ReLU) activation
function [
In order to assess the quality of our approach in labelling movies with related tags, we conducted a preliminary experimentation on a real dataset extracted by fusing data from diferent data sources. In particular, first we illustrate the dataset and the challenges to address for providing accurate predictions in this scenario; then we describe the adopted evaluation protocol and DNN Blocki
D N N B l icko + 1 (a) Overview of the model.
(b) Layers composing a base DNN component metrics; finally, we show an ablation study aiming at highlight the benefits of the proposed approach.
Our approach has been evaluated on a novel media content dataset gathering diferent types of information on a movie catalogue (e.g., movie plot, trailer, poster, synopsis, tags, etc.). Specifically, we focused our analysis on text data (i.e., movie plots) and tags extracted by multiple open sources such as Wikipedia, Wikimedia, and TMDB. When available, an extended plot is associated with the movie otherwise it is replaced with its synopsis. As shown in Table 2, our dataset contains ∼ 5k movies, with about 3 tags per movie. The complete list of tags (85) is showed in Figure 4. It is important noting that a restricted number of tags (mainly the genres) occurs more frequently than others, that can be considered as keywords summarizing some aspects of the movie. In particular, we can see that the data distribution exhibits a long tail shape.
The experiments are perfomed on a DGX machine equipped with V100 GPUs. The dataset is split in training and test set respectively with 70%/30% percentages. In more detail, the dataset is partitioned in a stratified fashion so to reduce the sampling error. Adam is used as optimizer while, as mentioned above, we exploits two loss functions to handle the imbalance problem i.e., the triplet loss and the CB-loss. The first one is used during the Embedder learning phase while the last is used for training the local models. As regards the Clusterer, the number of groups has been empirically determined to 20. As a result of the clustering phase, each local model can focus the learning on a limited number of tags, in particular the average number of tags per cluster is ∼ 12.
In Table 3 we report F1 score averaged according to two diferent strategies, respectively
named macro and micro [
The results shown in table 3 highlight the poor performance of the base model, i.e. the model trained on all tags. It is unable to handle the high number of tags (i.e. classes) and provides inaccurate predictions. The comparison of the values of Micro-F1 and Macro-F1 highlight the influences of the majority classes such as Drama or Comedy on the overall performances. The low value of Macro-F1 show that the model is unable to detect the under-represented tags which are the majority in the dataset.
The adoption of the triplet loss architecture allows for improving the performances of the base model, although the low value of the Macro-F1 indicates poor performances on the minority classes. Finally, the full approach, named in table as _ℎ, allows for improving also the Macro-F1 value.
Enriching metadata with informative labels is a crucial task for the enterprises operating in the media content delivery field. However, automating this process requires to cope with diferent challenging issues. In this work we proposed a hierarchical DL-Based approach for extreme multi-label classification aiming at providing accurate predictions for movie tagging task. An experimentation conducted on a real dataset demonstrates the quality of the approach.
As a pointer of further research, we aim at boosting the overall performance of the proposed approach by integrating information coming from unlabeled data in a semi-supervised or selfsupervised way. Also, active learning schemes can be fruitfully exploited by implementing ad-hoc oracle labeling strategies. Finally, we are interested to extend the experimentation for a fully multi-modal scenario by including heterogeneous data e.g., movie posters and trailers.
This work was supported by PON I&C 2014-2020 FESR MISE, Catch 4.0.