With this work we participate in the competition on the author pro ling task on the MEX-A3T track at IberEval 2019. Author pro ling task aims to identify gender, occupation and location from images or text of Mexican Twitter users. We propose a 3D Convolutional Neural Network for solving this task, using visual information, in this case images extracted from the user's pro le which are grouped to create a unique input of each user.
each image of the pro le using CNNs and then performs classi cation on those categories to
predict gender. Although Merler et al. [11] and Taniguchi et al. [13] work with images, they rely on
an intermediate stage that recognizes certain features in images, and thus, an additional process of
labeling and training is required. A similar work is made by Alvarez-Carmona et al. [
In this section we describe our method using visual information from Mexican Twitter users; we used 3D CNN models and images to identify the AP gender: male and female, location: north, northwest, northeast, south, southeast and center, and occupation: arts, student, social, sciences, administrative, health, sports and others.
It is worth to mention that we are working with Mexican Twitter users, so their native language is Spanish while all the previous mentioned work was focused on English language users. Another important point is that, as far as we know, there are no other works trying to predict location and occupation from both textual and visual information.
Finally, in this work we use only visual information provided by the users, that is, we will use images taken from their Twitter pro les. From the aforementioned works we take the idea that CNNs are capable of extracting features from single images and classify them in di erent classes. We propose a 3D CNN capable of classify gender, location and occupation at the same time from groups of images instead of single images. 3.1
Dataset The dataset was provided by the AP task on the MEX-A3T at IberEval 2019 [2]. It consists of images and texts extracted from Mexican Twitter users to identify gender, occupation and location. For solving this task we use images only. The dataset contains 11 images per user (in few cases the number of images is less than 11 depending on the Twitter user), the full dataset consists of 3,500 users, resulting in 38,500 images approximately. In Fig. 1 we show samples of several images of this dataset.
By analyzing the images given by the users we found that using a single image as in
AlvarezCarmona et al. [
The 3D CNN Model 1 is shown in Figure 4 and it is based on [6], where h is the height of the patch, w is the width of the patch, p is the number of images per user|in this case we decided to use 11 images per group| and n is the batch size. It consists of four convolutional layers to extract features; in this CNN we use 3D kernels of size 2x3x3 and Max-pooling with size 2x2x2; nally we add a classi cation stage.
The 3D CNN Model 2 is shown in Figure 5 and it is based on [5], where h is the height of the patch, w is the width of the patch, p is the number of images per user|in this case we decided to use 11 images per group| and n is the batch size. It consists of seven convolutional layers to extract features; in this CNN we use 3D kernels of size 2x3x3 and Max-pooling with size 2x2x2; nally we add a classi cation stage.
The main di erences between both models is the Max-pooling operation performed on the last layers of Model 2, this operation reduces the input volume just in h and w dimension to extract features at lower resolution. Also the number of layers used to extract features from the input volume is di erent in both models.
Classi cation stage. We needed to adapt the output of the models to classify each pro ling setting separately; we use Fully Connected layers to perform the classi cation, all of them with ReLU as activation function; as loss function we use the Categorical Crossentropy function [9] given by Eq. 1.
L(y; y^) =
M N X X(yij log(y^ij )) j=0 i=0 (1) where y^ is the predicted value and y the target value. Categorical crossentropy will compare the distribution of the predictions (the neurons in the output layer, one for each class) with the true distribution, where the probability of the true class is set to 1 and 0 for the other classes. This output layer consists of three stages, the rst one classi es gender (male and female) using 2 output neurons, the second one classi es location (north, northwest, northeast, south, southeast and center) using 6 output neurons and nally the third one classi es occupation (arts, student, social, sciences, administrative, health, sports and others) using 8 output neurons. In Figure 6 we can observe how each stage is divided.
The dataset provided by [2] consists of training and test sets; the training set consists of 3,500 groups of images per user and 11 images for each group and the testing set consists of 1,500 groups of images per user with the same number of images.
We trained both models using the training set and using the Backpropagation (BP) method [10] and ADAM optimizer [7] to update the weights during training, with learning rate equal to 0.05 and decay equal to 0.7; we used 100 iterations and a batch size of 13. We trained the model on two GPU NVIDIA GTX 1080Ti; it took ve hours for training and less than a second for testing a single group of images.
As mentioned before we were participants of the AP task on the MEX-A3T at IberEval 2019; we compare our method with CerpamidTeam as their results were made public for this contest as well; they use the textual information to perform their prediction. From Tables 1 to 6 we report the evaluation of our 3D CNN models, and present the comparison with CerpamidTeam. We use the metrics given by the organizers of the contest such as F measure, Accuracy (Acc), Precision (P) and Recall (R). As shown in previous tables we observe that predicting the AP of a Twitter user using only images is a di cult task due to the generality of purpose of images on this platform, and even if we look at the images is hard to classify them; nevertheless we obtain better results using the 3D CNN Model 2 on gender and using the 3D CNN Model 1 on location and occupation. As far a we know there are not other participants who solve the task using images, so we cannot compare with the methods using text. Also both models work with volumes of images instead single images as described in previous work. One disadvantage from our method is that we need the same number of images per user and if a user has less number of images we are not able to determine precisely his or her pro le. 5
AP using only visual information such as images from Twitter is a di cult task due to the ambiguity and the source of them, although we did not obtain high results, we developed a new framework in which we can obtain AP using only images. Also we created a framework in which we can use groups of images instead of single images, avoiding the problem of shared images between users; additionally, our models allow to classify simultaneously directly from the images three di erent aspects of the pro le.
As a future work we plan to create a multimodal 3D CNN model which is able to combine visual and textual information. 2. Aragon, M.E., Alvarez-Carmona, M.A., Montes-y Gomez, M., Escalante, H.J., Villasen~or-Pineda, L., Moctezuma, D.: Overview of MEX-A3T at IberLEF 2019: Authorship and aggressiveness analysis in mexican spanish tweets. In: Notebook Papers of 1st SEPLN Workshop on Iberian Languages Evaluation Forum (IberLEF), Bilbao, Spain, September (2019) 3. Argamon, S., Koppel, M., Fine, J., Shimoni, A.R.: Gender, genre, and writing style in formal written texts. Text-Interdiciplinary Journal for the Study of Discourse, 23(3), 321{346 (2006) 4. Argamon, S., Koppel, M., Pennebaker, J.W., Schler, J.: Mining the blogosphere: Age, gender and the varieties of self-expression. First Monday 12(9) (2007) 5. Hou, R., Chen, C., Shah, M.: Tube convolutional neural network (T-CNN) for action detection in videos. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 5822{5831 (2017) 6. Ji, S., Xu, W., Yang, M., Yu, K.: 3D convolutional neural networks for human action recognition.
IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 221{231 (2010) 7. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014) 8. Koppel, M., Argamon, S., Shimoni, A.R.: Automatically categorizing written texts by author gender.
Literary and linguistic computing 17(4), 401{412 (2002) 9. LeCun, Y., Boser, B.E., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W.E., Jackel, L.D.: Handwritten digit recognition with a back-propagation network. In: Advances in neural information processing systems. pp. 396{404 (1990) 10. LeCun, Y.A., Bottou, L., Orr, G.B., Muller, K.R.: E cient backprop. In: Neural networks: Tricks of the trade, pp. 9{48. Springer (2012) 11. Merler, M., Cao, L., Smith, J.R.: You are what you tweet. . . pic! gender prediction based on semantic analysis of social media images. In: 2015 IEEE International Conference on Multimedia and Expo (ICME). pp. 1{6. IEEE (2015) 12. Ortega-Mendoza, R.M., Franco-Arcega, A., Lopez-Monroy, A.P., Montes-y Gomez, M.: I, me, mine: The role of personal phrases in author pro ling. In: International Conference of the Cross-Language Evaluation Forum for European Languages. pp. 110{122. Springer (2016) 13. Taniguchi, T., Sakaki, S., Shigenaka, R., Tsuboshita, Y., Ohkuma, T.: A weighted combination of text and image classi ers for user gender inference. In: Proceedings of the Fourth Workshop on Vision and Language. pp. 87{93 (2015)