This paper describes a neural network model for the author profiling task of PAN@CLEF 2018. Traditional machine learning models have shown superior performances for the author profiling task in past PAN series. However, these models often require careful feature-engineering to improve their performance. On the other hand, neural network approaches have recently shown advanced performances in both natural language processing (NLP) and computer vision (CV) tasks. We tackle the author profiling task using neural networks for texts and images. In order to leverage the synergy of the texts and images, we propose Text Image Fusion Neural Network (TIFNN), which considers their interaction. In an in-house experiment, TIFNN achieved accuracies of 84-90% for different languages when used for gender identification.
Author profiling technologies that extract author profile traits from social media can be applied to some applications, e.g., advertisement, recommendation, and marketing. PAN 2018: Author Profiling Task [13] is identifying the user's gender from tweets that are contained texts and images in three languages (English, Spanish, and Arabic).
In PAN 2017 Author Profiling Task, various approaches based on a deep neural network (DNN) were presented [6,7,9,16,18]. However, such approaches could not outperform traditional machine learning models that were carefully modeled, such as support vector machine. In contrast, neural network approaches have shown superior performances on various NLP tasks, e.g., machine translation, summarization, and information retrieval. In addition, DNN approaches have shown advanced performances in various CV tasks.
Because PAN 2018 Author Profiling Task includes both texts and images, using both texts and images in a neural network will improve the performances. Therefore, we tackle this task using both texts and images in a DNN-based approach.
In order to leverage the synergy of the texts and images, we propose Text Image Fusion Neural Network (TIFNN), which considers their interaction. This paper makes the following contributions.
1. We propose an effective fusion strategy for a neural network to utilize texts and images for gender identification. 2. We show that TIFNN has drastically improved accuracies (3-8pt) compared with both a text-based neural network and an image-based neural network.
In the following section of this paper, we first explain the related work in Section 2. Our neural network model is described in Section 3. The details of the experiments used to confirm the model's performances are described in Section 4. Finally, we conclude the paper and outline future work in Section 5.
PAN Author Profiling Task was to identify both age and gender from social media text in the past PAN series before 2017 edition [12,15]. In the last year, the task included language variety identification instead of age identification [14]. In PAN 2017 Author Profiling Task, various models that used not only traditional machine learning but also deep neural networks were presented.
Basile et al. [1] used linear support vector machine (SVM) with character 3-to 5grams and word 1-to 2-grams features and showed that it outperforms other approaches. Martinc et al. [8] explored many approaches (e.g. linear SVM, logistic regression, random forest, XGBoost, and voting classifier combining these models) with various parameters for this task. They finally tested logistic regression because it showed the best performance. Tellez et al. [20] used a generic framework for text classification, as called MicroTC. As shown in these researches, the approaches of traditional machine learning that were carefully designed showed the superior performances in this task.
On the other hand, the approaches based on deep neural networks were also presented [6,7,9,16,18]. Miura et al. [9] used both bi-directional GRU with an attention mechanism to capture the word representations and convolutional neural network (CNN) to capture the character representations. Sierra et al. [18] applied CNN that has a set of convolutional filters of different sizes to capture n-gram features. Although the approaches using deep neural networks are strong model for many NLP tasks, the above approaches could not outperform traditional machine learning approaches in this task.
In author profiling tasks outside of PAN, researches utilizing images or multimodality also exist. The research of [17] utilized images to identify the gender of users and the object of images with a multi-task bilinear model. In addition, the research of [21] presented a state-of-the-art model that utilized both texts and images to predict users' traits such as gender, age, political orientation, and location.
As overviewed in this section, the approaches using traditional machine learning showed the superior performances in past PAN series. Although the approaches based on deep neural networks utilizing only text could not outperform traditional machine learning approaches, the researches of [17,21] indicated that utilizing images is effective in the prediction of author profile traits. Because using images is possible in PAN 2018 Author Profiling Task, utilizing both texts and images would be effective for gender identification. 3 Model
Figure 1 illustrates the proposed model. The model is constructed of a text component, an image component, and a fusion component. The proposed model processes texts and images in their respective components. The fusion component computes the relationship between the texts and images using direct-product, column-wise pooling, and row-wise pooling. Finally, the combination feature of the texts and images is fed to two fully connected layers.
In the following, we first describe each component of the model in Section 3.2 and 3.3. The details of the fusion component are also described in Section 3.4.
This section describes the text component of the model, which is the "Text Component" division in Figure 1. The component is implemented based on the previous models [9]. layers. For the RNN, we used Gated Recurrent Unit (GRU) [4] with a bi-directional setting.
First, the input words are embedded to k w dimensional word embeddings with embedding matrix E w to obtain x with x t ∈ R kw . x are then fed to RNN W with the following transition functions:
This section describes the image component of the model, which is the "Image Component" division in Figure 1. Figure 3 provides an overview of the image component. CNN I in Figure 1 represents the layers from Conv.Layers1 to FC7 in Figure 3. This architecture is implemented based on VGG16 [19]. CNN I utilizes the layers from Conv.Layers1 to FC7 to extract each image feature.
Pooling FI fuses the features extracted from images. The images posted by a single author on social media can be regarded as a kind of time series. However, we cannot know the ground truth of the images' order in time steps and the interval of time between posted images. Therefore, we simply use the average or max operation over image features as Pooling I .
The fusion component is expected to complementarily capture the relationship between the texts and images. User representation r txt ∈ R M is obtained via the text component and r img ∈ R L is obtained via the image component. The relationship between the texts and images is represented as a matrix G ∈ R M ×L with the following equation:
where ⊗ is a direct-product operation. We apply column-wise and row-wise maxpoolings over G to generate g txt ∈ R M and g img ∈ R L , respectively. Formally, the j-th elements of the vector g txt and the j-th elements of the vector g img are computed in the following operation:
We can interpret the j-th element of the vector g txt as an importance degree for the j-th text feature with regard to image features. Finally, the vectors g txt and g img are concatenated to obtain g comb as g comb = [g txt , g img ] and passed to FC1.
This section describes two datasets: PAN@CLEF 2018 Author Profiling Training Corpus and streaming tweets. PAN@CLEF 2018 Author Profiling Training Corpus was utilized to train the proposed model and comparison models. Streaming tweets were utilized to pre-train a word embedding matrix E w .
The first dataset we used to train the proposed model was the official PAN@CLEF 2018 Author Profiling Training Corpus. This dataset is constructed of users' tweets in three languages: English, Spanish, and Arabic. There are 3, 000 English language users, 3, 000 Spanish language users, and 1, 500 Arabic language users, with a gender ratio of 1:1. We used random sampling to divide this dataset into train 8 , dev 1 , and test 1 , with a ratio of 8:1:1, while maintaining the gender ratio of 1:1.
The second dataset we used to pre-train the word embeddings was composed of tweets collected by Twitter Streaming APIs1 . We used the collected tweets to pre-train the word embedding matrix E w of the proposed model and the comparison models. Table 1 lists the number of resulting tweets. The process of collecting tweets was described in [9]. We will describe the process to pre-train the word embedding matrix in Section 4.3.
We pre-trained each component for the proposed model. We used three steps to initialize the proposed model for training according to the following procedure. Initialization of text component We first pre-trained a word embedding matrix E w for the text component. The details of the pre-training of the word embeddings will be described in Section 4.3. The text component was trained using train 8 and dev 1 .
The image component was trained by fine-tuning on train 8 and dev 1 . First, the layers from Conv.Layers1 to FC7 described in Figure 3 (VGG16) were pre-trained on ImageNet [5]. We then initialized CNN I , as described in Figure 1, using the pre-trained VGG16. Finally, FC UI was then randomly initialized.
Initialization of TIFNN We described the pre-training procedure for each component using train 8 and dev 1 above. This was done because TIFNN could be successfully trained utilizing pre-trained text and image components. Thus, we used the pre-trained text and image components to train TIFNN. Therefore, all of TIFNN parameters except FC1 and FC2 were initialized with the parameters of the pre-trained components.
Text pre-processing We applied unicode normalization, user name normalization, URL normalization, and HTML normalization. We used twokenizer [10] for the English text. We used WordPunctTokenizer in NLTK [2] for the other languages for tokenization.
Image pre-processing We applied two resizing methods: direct resizing and resizingcropping.
-Direct resizing: We resized images to 224 pixels × 224 pixels.
-Resizing-cropping: We resized images to 256 pixels × 256 pixels and then cropped the center of each image to 224 pixels × 224 pixels.
Direct resizing was applied to an image-based neural network and resizing-cropping to TIFNN. After resizing, normalization was applied to all the images by subtracting the average values of the RGB channels for each language.
Initialization of word embeddings We used fastText [3] with the skip-gram algorithm to pre-train a word embedding matrix E w . The pre-training parameters were as follows: dimension = 100, learning rate = 0.025, window size = 5, negative sample = Parameters and pooling settings for proposed model Table 2 summarizes the number of parameters in the proposed model. In addition, Pooling W was applied as a max pooling layer for each language, Pooling I was applied as an average operation for each language, and Pooling T was applied as a max pooling layer for Arabic or an average pooling layer for the other languages.
Optimization strategies We used cross-entropy loss as an objective function for the models. The objective function of TIFNN was minimized over shuffled mini-batches with SGD. We also used Adam for the text component and SGD for an image component. The initial SGD learning rate for the image component was set at 1e −3 . In addition, we selected the best TIFNN learning rate for each language: 5e −3 for English and 1e −2 for the other languages.
Parameter selection The models had l 2 regularization parameter α. We selected the best parameter α of the text component from the following candidates. On the other hand, the parameter α of TIFNN was fixed at α = 1e −5 .
We explored the best parameter α for each model using dev 1 .
We next describe the details of the comparison models used for the in-house experiment. The parameter of FC1 is different from that listed in Table 2; we set it to 100.
We evaluated the proposed model and the comparison models using train 8 , dev 1 , and test 1 . With the exception of the baseline, the models were trained using Titan X GPUs. Table 3 summarizes the gender identification results. As listed in Table 3, Text NN and Image NN achieved accuracies of 80.0-82.3% for each language. TIFNN drastically improved the accuracies (3-8pt) for each language compared with Text NN and Image NN in this task. Furthermore, TIFNN has also improved the accuracies for English and Spanish compared with Text NN + Image NN. This indicated that obtaining a fusion synergy via the fusion component was an effective approach for this task.
We chose the best performing models, which were the Text NN, Image NN, and TIFNN, as described in a TIRA virtual machine [11] with CPUs. Table 4 summarizes the performances of the models in the submission run that are published as the official PAN results 2 . Although the models have lower accuracies compared with the in-house experiment, it is observed that TIFNN has better accuracies for each language compared with Text NN and Image NN. They ranked 1st in English ranking, 2nd in Spanish ranking, 7th in Arabic ranking, and 1st in Global ranking.
In this paper, we proposed Text Image Fusion Neural Network (TIFNN) for gender identification. In order to leverage the synergy of texts and images, the model computes the relationship between them using the direct-product. In-house experimental results showed that Text NN and Image NN achieved accuracies of 80.0-82.3% for each language in gender identification. TIFNN had drastically improved accuracies (+3-8pt) compared with Text NN and Image NN. Furthermore, TIFNN also had improved accuracies for English and Spanish compared with Text NN + Image NN. In addition to the results of this in-house experiment, we confirmed that TIFNN could improve the accuracy compared with individual models in a submission run.
In future work, we would like to analyze how the proposed model interacts with texts and images. We believe that understanding this interaction will make it possible to improve TIFNN.
2 https://pan.webis.de/clef18/pan18-web/author-profiling.html