Affective labeling of multimedia content can be useful in recommender systems. In this paper we compare the effect of implicit and explicit affective labeling in an image recommender system. The implicit affective labeling method is based on an emotion detection technique that takes as input the video sequences of the users' facial expressions. It extracts Gabor low level features from the video frames and employs a kNN machine learning technique to generate affective labels in the valence-arousal-dominance space. We performed a comparative study of the performance of a content-based recommender (CBR) system for images that uses three types of metadata to model the users and the items: (i) generic metadata, (ii) explicitly acquired affective labels and (iii) implicitly acquired affective labels with the proposed methodology. The results showed that the CBR performs best when explicit labels are used. However, implicitly acquired labels yield a significantly better performance of the CBR than generic metadata while being an unobtrusive feedback tool.
Each of the two approaches for affective labeling, explicit and implicit, has its
pros and cons. The explicit approach provides unambiguous labels but Pant
Given the advantages of implicit labeling over explicit there is a need to assess the impact of the low emotion detection accuracy on the performance of recommender systems.
In this paper we compare the performance of a CBR system using explicit affective labeling vs. the proposed implicit affective labeling. The baseline results of the CBR with explicit affective labeling are those published in Tkalˇciˇc et al. [2010a]. The comparative results of the implicit affective labeling are obtained using the same CBR procedure as in Tkalˇciˇc et al. [2010a], the same user interaction dataset [Tkalˇciˇc et al., 2010c] but with affective labels acquired implicitly. 1.2
As anticipated by Pant
The acquisition of explicit affective labels is usually performed through an
application with a graphical user interface (GUI) where users consume the
multimedia content and provide appropriate labels. An example of such an application
is the one developed by Eckhardt and P
On the other hand, the acquisition of implicit affective labels is usually
reduced to the problem of non-intrusive emotion detection. Various modalities are
used, such as video of users’ faces, voice or physiological sensors (heartbeat,
galvanic skin res
Another approach is to extract affective labels directly from the content
itself, without observin
In contrast to emotion detection techniques the usage of affective labels for
information retrieval has only recently started to gain attention. Chen et al.
[2008] developed the EmoPlayer which has a similar user interface to the tool
developed by Eckhardt and P
In a multimedia consumption scenario a user is watching multimedia content. During the consumption of multimedia content (images in our case), the emotive state of a user is continuously changing between different emotive states ǫj ∈ E, as different visual stimuli hi ∈ H induce these emotions (see Fig. 1). The facial expressions of the user are being continuously monitored by a video camera for the purpose of the automatic detection of the emotion expressions.
The detected emotion expressions of the users, along with the ratings given to the content items, can be used in two ways: (i) to model the multimedia content item (e.g. the multimedia item hi is funny - it induces laughter in most of the viewers) and (ii) to model individual users (e.g. the user u likes images that induce fear).
E ǫ4 ǫ3 ǫ2 ǫ1 ǫN
tT t(h1) t(h2) t(h3) t(h4) t Item modeling with affective metadata We use the valence-arousal-dominance (VAD) emotive space for describing the users’ emotive reactions to images. In the VAD space each emotive state is described by three parameters, namely valence, arousal and dominance. A single user u ∈ U consumes one or more content items (images) h ∈ H. As a consequence of the image h being a visual stimulus, the user u experiences an emotive response which we denote as er(u, h) = (v, a, d) where v, a and d are scalar values that represent the valence, arousal and dominance dimensions of the emotive response er. The set of users that have watched a single item h are denoted with Uh. The emotive responses of all users Uh, that have watched the item h form the set ERh = {er(u, h) : u ∈ Uh}. We model the image h with the item profile that is composed of the first two statistical moments of the VAD values from the emotive responses ERh which yields the six tuple
V = (v¯, σv, a¯, σa, d¯, σd) (1) where v¯, a¯ and d¯ represent the average VAD values and σv, σa and σd represent the standard deviations of the VAD values for the observed content item h. An example of the affective item profile is shown in Tab. 1.
User modeling with affective metadata The preferences of the user are modeled based on the explicit ratings that she/he has given to the consumed items. The observed user u rates each viewed item either as relevant or nonrelevant. A machine learning (ML) algorithm is trained to separate relevant from non-relevant items using the affective metadata in the item profiles as features and the binary ratings (relevant/non-relevant) as classes. The user profile up(u) of the observed user u is thus an ML algorithm dependent data structure. Fig. 2 shows an example of a user profile when the tree classifier C4.5 is being used.
Value
We used our implementation of an emotion detection algorithm
The emotion detection procedure used to give affective labels to the content images involved three stages: (i) pre-processing, (ii) low level feature extraction and (iii) emotion detection. We formalized the procedure with the mappings I → Ψ → E (2) where I represents the frame from the video stream, Ψ represents the low level features corresponding to the frame I and E represents the emotion corresponding to the frame I.
In the pre-processing stage we extracted and registered the faces from the
video frames to allow precise low level feature extraction. We used the eye tracker
developed by Valent
The low level features used in the proposed method were drawn from the images filtered by a Gabor filter bank. We used a bank of Gabor filters of 6 different orientation and 4 different spatial sub-bands which yielded a total of 24 Gabor filtered images per frame. The final feature vector had the total length of 240 elements.
The emotion detection was done by a k-NN algorithm after performing dimensionality reduction using the principal component analysis (PCA).
Each frame from the LDOS-PerAff-1 dataset was labeled with a six tuple of the induced emotion V . The six tuple was composed of scalar values representing the first two statistical moments in the VAD space. However, for our purposes we opted for a coarser set of emotional classes ǫ ∈ E. We divided the whole VAD space into 8 subspaces by thresholding each of the three first statistical moments v¯, a¯ and d¯. We thus gained 8 rough classes. Among these, only 6 classes actually contained at least one item so we reduced the emotion detection problem to a classification into 6 distinct classes problem as shown in Tab. 2. centroid values class E v¯ a¯ d¯ v a d ǫ1 v¯ > 0 a¯ < 0 d¯ < 0 0.5 −0.5 −0.5 ǫ2 v¯ < 0 a¯ > 0 d¯ < 0 −0.5 0.5 −0.5 ǫ3 v¯ > 0 a¯ > 0 d¯ < 0 0.5 0.5 −0.5 ǫ4 v¯ < 0 a¯ < 0 d¯ > 0 −0.5 −0.5 0.5 ǫ5 v¯ > 0 a¯ < 0 d¯ > 0 0.5 −0.5 0.5 ǫ6 v¯ > 0 a¯ > 0 d¯ > 0 0.5 0.5 0.5 Our scenario consisted in showing end users a set of still color images while observing their facial expressions with a camera. These videos were used for implicit affective labeling. The users were also asked to give explicit binary ratings to the images. They were instructed to select images for their computer wallpapers. The task of the recommender system was to select the relevant items for each user as accurate as possible. This task falls in the category find all good items for the recommender systems’ tasks taxonomy proposed by Herlocker et al. [2004].
The set of images h ∈ H that the users were consuming, had a twofold
meaning: (i) they were used as content items and (ii) they were used as emotion
induction stimuli for the affective labeling algorithm. We used a subset of 70
images from the IAPS dataset Lan
The affective labeling algorithm described in Sec. 3.1 yielded rough classes in the VAD space. In order to build the affective item profiles we used the classes’ centroid values (see Tab. 2) in the calculation of the first two statistical moments. We applied the procedure from Sec. 2.2.
We had 52 users taking part in our experiment (mean = 18.3 years, 15 males). 3.3
The results of the CBR system were the confusion matrices of the classification procedure that mapped the images H into one of the two possible classes: relevant or non-relevant class. From the confusion matrices we calculated the recall, precision and F measure as defined in Herlocker et al. [2004].
We also compared the performances of the CBR system with three types of
metadata: (i) generic metadata
We compared the performance of the classification of items into relevant or non relevant through the confusion matrices in the following way: (i) Explicitly acquired affective metadata vs Implicitly acquired metadata, (ii) explicitly acquired metadata vs. generic metadata and (iii) implicitly acquired metadata vs. generic metadata. In all three cases the p value was p < 0.01. Table 3 shows the scalar measures precision, recall and F measures for all three approaches. As we already reported in Tkalˇciˇc et al. [2010b], the application of the emotion detection algorithm on spontaneous face expression videos has a low performance. We identified three main reasons for that: (i) weak supervision in learning, (ii) non-optimal video acquisition and (iii) non-extreme facial expressions.
In supervised learning techniques there is ground truth reference data to which we compare our model. In the induced emotion experiment the ground truth data is weak because we did not verify whether the emotive response of the user equals to the predicted induced emotive response.
Second, the acquisition of video of users’ expressions in real applications takes place in less controlled environments. The users change their position during the session. This results in head orientation changes, size of the face changes and changes of camera focus. All these changes require a precise face tracker that allows for fine face registration. Further difficulties are brought by various face occlusions and changing lighting conditions (e.g. a light can be turned on or off, the position of the curtains can be changed etc.) which confuse the face tracker. It is important that the face registration is done in a precisely manner to allow the detection of changes in the same areas of the face.
The third reason why the accuracy drops is the fact that face expressions in spontaneous videos are less extreme than in posed videos. As a consequence the changes on the faces are less visible and are hidden in the overall noise of the face changes. The dynamics of face expressions depend on the emotion amplitude as well as on the subjects’ individual differences.
The comparison of the performance of the CBR with explicit vs. implicit affective labeling shows significant differences regardless of the ML technique employed to predict the ratings. The explicit labeling yields superior CBR performance than the implicit labeling. However, another comparison, that between the implicitly acquired affective labels and generic metadata (genre and watching time) shows that the CBR with implicit affective labels is significantly better than the CBR with generic metadata only. Although not as good as explicit labeling, the presented implicit labeling technique brings additional value to the CBR system used.
The usage of affective labels is not present in state-of-the-art commercial recommender systems, to the best of the authors’ knowledge. The presented approach allows to upgrade an existing CBR system by adding the unobtrusive video acquisition of users’ emotive responses. The results showed that the inclusion of affective metadata, although acquired with a not-so-perfect emotion detection algorithm, significantly improves the quality of the selection of recommended items. In other words, although there is a lot of noise in the affective labels acquired with the proposed method, these labels still describe more variance in users’ preferences than the generic metadata used in state-of-the-art recommender systems. 5.1
The usage of affective labels in recommender systems has not reached a production level yet. There are several open issues that need to be addressed in the future.
The presented work was verified on a sample of 52 users of a narrow age and
social segment and on 70 images as content items. The sample size is not big but
it is in line with sample sizes used in related work [Arapak
Another aspect of the sample size issue is the impact of the size on the
ML techniques used. The sample size in the emotion detection algorithm (the
kNN classifier) is not problematic. It is, however, questionable the sample size
used in the CBR. In the ten fold cross validation scheme we used 63 items for
training the model and seven for testing. Although it appears that this is small, a
comparison with other recommender system reveals that this is a common issue,
and is usually referred as the sparsity problem. It occurs when, even if there are
lots of users and lots of items, each user usually rated only few items and there
are few data to build the models u
The presented work also lacks a further user satisfaction study. Besides just aiming at the prediction of user ratings for unseen items research should also focus on the users’ satisfaction with the list of recommended items.
But the most important thing to do in the future is to improve the emotion detection algorithms used for implicit affective labeling. In the ideal case, the perfect emotion detection algorithm would yield CBR performance that is identical to the CBR performance with explicit labeling.
The acquisition of video of users raises also privacy issues that need to be addressed before such a system can go in production.
Last, but not least, we believe that implicit affective labeling should be
complemented with context modeling to provide better predictions of users’
preferences. In fact, emotional responses of users and their tendencies to seek one
kind of emotion over another, is tightly connected with the context where the
items are consumed. Several investigations started to explore the influence of
various contextual parameters, like being alone or being in company, on the
users’ pr
We performed a comparative study of a CBR system for images that uses three types of metadata: (i) explicit affective labels, (ii) implicit affective labels and (iii) generic metadata. Although the results showed that the explicit labels yielded better recommendations than implicit labels, the proposed approach significantly improves the CBR performance over generic metadata. Because the approach is unobtrusive it is feasible to upgrade existing CBR systems with the proposed solution. The presented implicit labeling technique takes as input video sequences of users’ facial expressions and yields affective labels in the VAD emotive space. We used Gabor filtering based low level features, PCA for dimensionality reduction and the kNN classifier for affective labeling.
This work was partially funded by the European Commission within the FP6 IST grant number FP6-27312 and partially by the Slovenian Research Agency ARRS. All statements in this work reflect the personal ideas and opinions of the authors and not necessarily the opinions of the EC or ARRS.