In this paper, we present the algorithm designed for mining emotional features of movies. The algorithm dubbed Arousal-Valence Discriminant Preserving Embedding (AVDPE) is proposed to extract the intrinsic features embedded in movies that are essentially di erentiating in both arousal and valence directions. After dimensionality reduction, we use the neural network and support vector regressor to make the nal prediction. Experimental results show that the extracted features can capture most of the discriminant information in movie emotions.
A ective multimedia content analysis aims to automatically recognize and analyze the emotions evoked by multimedia data such as images, music, and videos. It has a lot of real-world applications such as image search, movie recommendation, and music classi cation [3, 7{9, 11{14].
In this 2016 Emotional Impact of Movies Task, the participants are required to design algorithms to predict the arousal and valence values of the given movies automatically. The dataset used in this task is the LIRIS-ACCEDE dataset (liris-accede.ec-lyon.fr). It contains videos from a set of 160 professionally made and amateur movies, shared under the Creative Commons licenses that allow redistribution [2]. More details of the task requirements as well as the dataset description can be found in [5, 10].
In this paper, we perform both global and continuous emotion predictions via a proposed supervised dimensionality reduction algorithm called Arousal-Valence Discriminant Preserving Embedding (AV-DPE), which learns the compact representations of the original data. After obtaining the lowdimensional features, we use the neural network and support vector regressor to predict the emotion values.
In order to derive the intrinsic factors in movies that convey or evoke emotions along the arousal and valence dimensions, we propose a supervised feature extraction algorithm dubbed Arousal-Valence Discriminant Preserving Embedding (AV-DPE) to map the original high-dimensional representations into a low-dimensional feature subspace, in which the data with similar A-V values are close to each other, while the data with di erent A-V values are faraway from each other.
Let x 2 RD be the high-dimensional feature vector
of the movie, and y = [y(
In order to describe the similarity between data samples, we de ne the following adjacency scatter matrix: n n Sa = X X Aij (xi i=1 j=1 xj)(xi xj)T ; where Aij denotes the similarity between the i-th and j-th data points. In our formulation, we use the form of inner product between the corresponding label vectors associated with xi and xj. To further normalize the similarity values into interval [0; 1], we de ne the normalized adjacency matrix A^ where ^
Aij = hy^i; y^ji = hyi=jjyijj; yj=jjyjjji:
The normalized adjacency scatter matrix is then de ned as:
(
xj)T :
xj)(xi
xj)T ;
(
Similarly, we de ne the normalized discriminant scatter matrix to characterize the dissimilarity between data points: where we simply de ne D^ ij = 1 A^ij .
In order to maximize the distance between data points with di erent labels while minimizing the distance between data points with similar labels, the objective function of AVDPE is formulated as follows:
U = arg maxftr((UT S^aU)yUT S^dU)g;
U
where tr( ) denotes the matrix trace operation and (S^a)y
denotes the Moore-Penrose pseudoinverse of S^a [6]. The
optimization problem in Eq. (
In this section, we report the experimental settings and the evaluation results.
Global emotion prediction: we construct a 34-D feature set, including alpha, asymmetry env, colorfulness, colorRawEnergy, colorStrength, compositionalBalance, cutLength, depthOfField, entropyComplexity, atness, globalActivity, hueCount, lightning, maxSaliencyCount, medianLightness, minEnergy, nbFades, nbSceneCuts, nbWhiteFrames, saliencyDisparity, spatialEdgeDistributionArea, wtf max2stdratio f1-12g and zcr. Note that all above features are provided by the task organizers.
Run #1: We use the original 34-D features as the input, and then use a function tting neural network [1] with 100 nodes in the hidden layer for prediction. The Levenberg-Marquardt backpropagation function is used in training.
Run #2: We use the original 34-D features as input, and then use the -support vector regression ( -SVR) for prediction. In -SVR, the RBF kernel is utilized with the default setting from LIBSVM [4], i.e., cost = 1, = 0:5, and is then set to be the reciprocal of the number of feature dimension.
Run #3: We rst use the proposed AV-DPE to reduce the original feature space to the 10-D subspace. Then utilize the neural network for prediction. The setting of neural network is the same as that in Run #1. Run #4: We rst use the proposed AV-DPE to reduce the original feature space to the 10-D subspace. Then we use the -SVR for prediction. The setting of -SVR is the same as that in Run #2.
Continuous emotion prediction: we downsample the size of each video to 64 36. As a result, we have a 6912-D feature vector of RGB values for each frame.
Run #1: We use the original 6912-D features as the input, and then use the neural network for prediction. The setting of neural network is the same as that in Run #1 of global emotion prediction.
Run #2: We use the original 6912-D features as the input, and then use the -SVR for prediction. The setting of -SVR is the same as that in Run #2 of global emotion prediction.
Run #3: We rst use the proposed AV-DPE to reduce the original high-dimensional feature space to the 100D subspace. Then we use the neural network for prediction. The setting of neural network is the same as that in Run #1 of global emotion prediction. Run #4: We rst use the proposed AV-DPE to reduce the original high-dimensional feature space to the 100D subspace. Then we use the -SVR for prediction. The setting of -SVR is the same as that in Run #2 of global emotion prediction.
Table 1 and Table 2 report the results of our system. From the tables we can see that after dimensionality reduction, the performance of the reduced features (Run #3 and Run #4) is generally worse than that of the original features (Run #1 and Run #2), which indicates that the emotion information in movies is relatively complex, and thus we may not be able to fully describe it using just a few dimensions. However, considering that the dimension of the reduced features is much less than that of the original features, we still can conclude that the learned subspace preserves rich discriminant information of the original feature space.
Moreover, from both tables we can observe that the neural network performs more robust than SVR after dimensionality reduction. The possible reason is that besides the discriminant ability, the neural network with the hidden layer has better representation ability of the original data than SVR, which is also of great importance in supervised learning tasks. 4.
In this working notes paper, we have proposed a dimensionality reduction method to extract the emotional features from movies. By minimizing the distance between data points with similar emotion levels and maximizing the distance between data points with di erent emotion levels simultaneously, the learned subspace keeps most of the discriminant information and gives relatively robust results in both global and continuous emotion prediction tasks.
The authors would like to thank the reviewer for the helpful comments. This work was supported in part by the National Natural Science Foundation of China under Grant 61503317.