In this paper we present a tool that performs two tasks: given an input image, rst, (i) it detects whether the image corresponds to a male or female person and then (ii), it further recognizes which emotion the face expression of the detected person is conveying. We mapped the two tasks as multi-label classi cations. The rst one aims at identifying if the input image contains one of the following four categories: one male person, one female person, a group of both male and female persons or if the image does not contain any person. The second task is triggered whether the image has been recognized to belong to one of the rst two classes and aims at detecting whether that image is conveying one among six di erent emotions: sadness, anger, surprise, happiness, disgust, fear. For both the problems, Microsoft Cognitive Services have been leveraged to extract tags from the input image. Tags are text elements that have been adopted to form the vectorial space model, using the bag of words model, that has been fed to the machine learning classi ers for the prediction tasks. For both tasks, we manually annotated 3000 images, which have been extracted from students who agreed using our system and providing their Facebook pictures for our analysis. Our evaluation uses Naive Bayes and Random Forest classi ers and with a 10-fold cross-validation reached satisfactory accuracies both for the two tasks and for the combination of them. Finally, our system works online and has been integrated with social media. In that way, any visitors logged in to Facebook through its APIs, is allowed to quickly classify any of their photos.
useful marketing channel for business of all sizes. Users uploads statuses, posts,
images, and videos and, therefore, this contributes to the increase of available
data in Internet [
Understanding emotions [
Cognitive computing systems [
In this paper we present two multi-classi cation problems. The rst one aims at detecting whether a given image contains one male person, or one female person, or two or more people of di erent sex or no person. If an image belongs to one of the rst two classes (is a man or a woman) one more task is performed that aims at classifying the face expression of the input image according to six possible emotions: sadness, anger, surprise, happiness, disgust, fear. Microsoft Cognitive Services have been leveraged to extract textual tags from training and test set images and create a vectorial space using bag of word model (we employed term frequency and TF-IDF) that has been fed to di erent classi cation algorithms. Our system has been validated on 3000 images manually annotated and extracted from students Facebook pro les who agreed to have their Facebook pictures used for our analysis. A 10-fold cross-validation method has been performed. We obtained an accuracy of 65% for the rst classi cation task, an accuracy of 60% for the second task and an accuracy of 56% for the combined task. They system has also been developed to work online and allows any user to associate his/her Facebook account for logging in and classify any of the user's photo through the two tasks. The remainder of this paper is organized as it follows: Section 2 includes related work. A brief description of the Microsoft Cognitive Services is given in Section 3. Section 4 describes the used dataset and how we turned the input images into text representation. Section 5 includes the adopted methodology and the performance evaluation we carried out. Section 6 gives some technical details of our developed system. Finally, Section 7 draws conclusions and directions where we are headed. 2
Gender recognition is a domain that has attracted interest in both fundamental
and applied research. It has mostly been targeted using computer vision
algorithms and its resolution is still challenging. The di culties emerge from the
di erent positions of a face whose capture depends on the image acquisition
process, and the intrinsic di erences between people's faces [
We employed the Computer Vision APIs (v1.0)3. They provide state of the art algorithms to process images. They can be used to determine if an image contains mature content, or estimating dominant and accent colors and so on. When performing a request, the input consists of an image URL. Within the request, there is an optional parameter used to specify which features to return (image categories are returned by default). The response will be returned in JSON. Metadata we collected are: categories, tags, description, faces, image type, color, and adult. The subscription we used is free and, as such, we were able to perform only 20 requests per minute. For this reason, the online application might take several minutes to retrieve the metadata of all the images of the user. On the other hand, the Emotion APIs take a facial expression in a given input image, and returns the con dence across a set of emotions. The detected emotions are anger, contempt, disgust, fear, happiness, neutral, sadness and surprise. 4
Our dataset set is represented by 3000 images extracted from Facebook accounts
of students at University of Cagliari who agreed participating to our analysis. A
subset of 1000 images contained pictures of single persons and, therefore, were
used for the evaluation of the second task. These images represented face
expressions having six emotions: sadness, anger, surprise, happiness, disgust, fear.
The rst task has been evaluated on the entire collection of images. Tables 1
and 2 shows some statistics of the collected images dataset. Moreover, images
have been rst sent to Microsoft Cognitive Services to retrieve the tags and
manually classi ed according to the two classi cations: one label out of four classes
for the gender recognition task and one more label out of six for the emotion
detection class. For the emotion detection class we followed the same procedure
of the generation of the Microsoft FER dataset [
As mentioned within Section 4, we leveraged the textual tags extracted from Microsoft Cognitive Services to create our vectorial space model using bag of word and TF and TF-IDF models. Tags are text elements returned for each image based on more than 2000 recognizable objects, living beings, scenery, and actions. Tags are not organized as a taxonomy and no inheritance hierarchies exist. Naive Bayes and Random Forest classi ers have been employed for the obtained accuracy shown in Table 3 using 10-cross validation technique.
For the second task, again, we generated the vectorial space similarly as performed above for the rst task. We tested the same classi ers and in Table 4 results of the obtained accuracy are shown using 10-cross validation technique.
Finally, Table 5 shows the accuracy for the combination of the two tasks: given an image, it detects if it represents a single person and, in such a case, classi es which emotion it is conveying. For simplicity, the combined task has been represented as a multi-class classi cation problem with 8 categories: face expression of a single person conveying each of the six emotions, more than a single person, no persons and it has been tested on the entire collection of 3000 annotated images using 10-cross validation technique. where tp, tn, f p and f n are, respectively, true positives, true negatives, false positives and false negatives.
Table 6 shows the confusion table for the rst class (male) of the gender recognition task with the quantities above (true positive, etc.) properly indicated for the computation of the accuracy. Three more accuracies values have been computed using three more similar confusion tables for the remaining three classes. The overall accuracy has been calculated averaging the four accuracies thus computed. A similar process has been used to calculate the accuracy for the emotion detection task and the combined task.
Predicted class In this paragraph we will describe how our system works online. It is a web application composed by two main modules: the rst is a web application developed using CodeIgniter Web Framework4 which allows building robust solutions; the second module is a Python HTTP Apache Server.
We used the Facebook Graph API5 by obtaining rst the API KEY where we had to specify the needed security scopes for our application. We explicitly mentioned user photos to gain the URLs of the users' public photos. During this step we had to submit a short description of our project for the Facebook team to validate. Once validated, we were able to extract the list of the public images of the user navigating our web application. Figure 1 shows the architecture of the rst task6. A certain user visits our web application and he is redirected to
Facebook.com to start the authentication ow. After a successful log in, the user is redirected back to our server. The public images of the user are retrieved using a valid authentication token. The photos are sent to the Cognitive Services to fetch the related metadata. The images are shown to the user. Each photo can be clicked and a pop up appears with the related metadata and with one of the four categories related to the rst task we predict on the y with our trained classi er. In the background, a new JSON le for that user is created containing his/her images with the associated metadata and the predicted class and stored on HDFS through its REST APIs. In this paper we presented an approach that performs two multi-class classi cation tasks: detecting gender of a given image and detecting emotions of face
expression for images detected within the rst task. The system has been developed and works on-line and integrated with Facebook so that visitors can log in associating their Facebook account and deciding to perform the classi cation task on their images. For each selected image, Microsoft Cognitive Services are called and image tags collected in order to create a vectorial space representation of the input image. Our evaluation on 3000 images for the rst task and 1000 images for the second task using two di erent classi ers with 10-cross validation generated accuracy respectively of 65% using TF and 63% using TF-IDF for the rst task, 59% using TF and 55% using TF-IDF for the second task and, 56% (TF) and 55% (TF-IDF) for the combined task. As future work we are already collecting more images using crowd sourcing tools and will apply deep learning models on machines with fast NVIDIA GPUs (e.g. TitanX).
The authors gratefully acknowledge Sardinia Regional Government for the nancial support (Convenzione triennale tra la Fondazione di Sardegna e gli Atenei Sardi Regione Sardegna - L.R. 7/2007 annualita 2016 - DGR 28/21 del 17.05.201, CUP: F72F16003030002).