This manuscript describes our solution to solve the Author Profiling task at PAN'18. In this edition, the task asks for identifying the user's gender using both their Tweets containing texts and images. We used our MicroTC ( TC) text classification framework to cope with the text problem, and a novel approach to Bag of Visual Words to solve the image classification, designed for this task to solve the image classification. Finally, we tried to improve the final prediction using a combination of both approaches.
The author profiling problem predicts specific characteristics of an author through
the analysis of his/her documents. These traits can be gender, age, personality,
native language, among others [
Our contribution solves the text and image profiling independently, in a multilingual perspective, and then merges the solutions using a convex combination. In particular, we use our text classification framework TC to profile authors based on their published texts; and for the image problem, we design a variant of Bag of Visual Words (BoVW) using the DAISY [20] feature descriptor and a transformation of these features to text, instead of using plain histograms as it is usual in traditional BoVW approaches.
The rest of the paper is organized as follows. Section 2 presents a brief review of works on gender identification. Section 3 describes our approach to solve the problem and our implemented system. Section 4 details our experimental methodology and lists our results. Finally, conclusions are given in Section 5. 2
Author profiling is a core task in the PAN contest since 2013 [
In this context, several and exciting works have been published in the research
community, most of them deal with the problem using only the written text by
the authors. Such is the case of [
Another approaches for gender classification using images typically employ
face’s images from the user (see for instance [
Recent literature shows the possibility to employ visual user content to learn
personal attributes such as gender or age. That is the case of the work presented
in [22] where the associated categories are used to acquire the posting behavior
and then to predict the user’s gender. The Bag of Visual Words model uses SIFT
features to capture the image’s content. For the experiments, 80 profiles were
used. The accuracy of this approach is 0:71, achieved using both posting behavior
and image content. Another method to extract user’s gender from images shared
in social media is presented in [
Our approach deals with both problems separately and, then, we combine predictions. The textual information is tackled with our MicroTC ( TC) framework, and the profiling based on image content was attempted using a variant of Bag of Visual Words (BoVW) that use DAISY features to feed an especially designed clustering and encoding method. Our approaches will be detailed in the following sections. 3
As in PAN’17, we tackled the text-based author profiling challenge using our text classifier MicroTC ( TC) [18], since it works regardless of both domain and language particularities.
The core idea behind TC to solve the text classification task, the problem is formulated as a model selection problem, that is, it selects a competitive configuration from a vast universe of possible ones. Each configuration is composed of a list of text transformations (normalizations and generic transformations), a combination of tokenizers, and a weighting schemes. The following steps describe the TC configuration space and its implicit work-flow: i. Preprocessing functions We use trivalent and binary parameters. The trivalent values can be set to fremove; group; noneg which means that the term matching the parameter is removed, grouped in set of predefined classes, or left untouched. In this kind of parameters, TC contains handlers for hashtags, numbers, urls, users, and emoticons. The binary parameters are boolean, and basically, indicate if the parameter is activated or not. In this parameter set, we support for diacritic removal, character duplication removal, punctuation removal, and case normalization. ii. Tokenizers After all text normalization and transformation, a list of tokens should be extracted. We allow to use n-grams of words (n = 1; 2; 3), q-grams of characters (q = 1; 3; 5; 7; 9), and skip-grams. For skip-grams we allow to select a few tokenizers like two words with gap one, (2; 1), also we allow to use (2; 2), (3; 1). Instead of selecting one or another tokenizer scheme, we allow to select any combination of the available tokenizers, and perform the union of the final multisets of tokens. iii. Weighting schemes. After we obtained a multiset (bag of tokens) from the tokenizers, we must create a vector space. MicroTC allows to use the raw frequency and the TFIDF schemes to weight the coordinates of the vector. It contains a number of frequency filters that were deactivated for this contribution, see [18] for more details.
To evaluate each configuration, we use a Support Vector Machine (SVM) with
a linear kernel to perform the final classification. It is well-known that SVM
performs excellently for large dimensional input (which is our case), and the
linear kernel also performs well under this conditions. We do not optimize the
parameters of the classifier since we are pretty interested in the rest of the
process. We use the SVM classifier from liblinear, Fan et al. [
The model selection is lead by a performance function score that is
maximized (solved) by a meta-heuristic. The only assumption is that score slowly
varies on similar configurations, such that we can assume some degree of locally
concaveness, in the sense that a local maximum can be reached using greedy
decisions at some given point. Clearly, this is not true in general, and the solver
algorithm should be robust enough to get a good approximation even when the
assumption is valid only with some degree of certainty. From a practical point
of view, a configuration is similar to another if structurally vary in a single
parameter. We name the set of all similar configurations of m as its
neighborhood. Therefore, the core idea is to start from a set of random configurations,
evaluate their neighborhoods and greedily move to the most promising set of
configurations. This procedure is repeated until some condition is achieved, like
the impossibility of improving the score function, or when a maximum number
of iterations is reached. There are several meta-heuristics to solve combinatorial
optimization problems. In particular, TC uses two types of meta-heuristics,
Random Search [
The TC framework is more detailed in [18], and it is available as open source.3 Modeling Users In the text problem, we model each user as an array of its text messages, as we did it in PAN’17 [19] following the weighting scheme of entropy+3. We introduced the entropy+b term-weighting in our previous PAN’17 participation, which consists in representing each term by the entropy of the term’s empirical distribution over the available classes, using a smoothing parameter b, in this case, b = 3. More precisely, defined as follows: entropyb(w) = log jCj
X pc(w; b) log c2C
1 pc(w; b) ; where C is the set of classes, and pc(w; b) is the probability of term w in class c parametrized with b. More detailed, pc(w; b) =
freqc(w) + b b jCj + Pc2C freqc(w) : Here, freqc denotes the frequency of the term in the class c. 3.1
Author Profiling through Posted Images Content In this problem, we model each user as the collection of all its images converted as text. In first place, all images were coverted to grayscale format and resized to 400 400 pixels. Later, for the process of image to text transformation three
main steps were followed: i) compute feature descriptors for each image (multiple dense vectors per image), ii) create a codebook using an efficient clustering algorithm, and iii) represent each image as a text using the codebook, and finally, iv) perform text classification over the generated text.
A descriptor algorithm computes many vectors for an image such that each vector describes in some sense a region of the image (sub-image). In our case, we use the DAISY [20] descriptor, which is a fast local descriptor that allows fast dense extraction, we can found it in the literature as part of a typical bag of visual words representations; in particular, we used the implementation found in scikit-image [21]. Furthermore, we explore some parameters modifications to get better results.
For the codebook computation, we use a variant of k-means found in the library SemanticWords.jl4 using approximate nearest neighbors to compute the centroids. Instead of using a Delone partition like the original algorithm, the used one selects a K nearest centroids to compute the next generation of centroids; this makes the computed clustering more robust to noisy data. The text encoding of the image is then performed using the computed codebook as follows, for each image: – The DAISY feature descriptors are computed, this computes a grid of dense vectors, with a shape that depends on the step and radius parameter of DAISY and the image geometry. – The text that represents an image is computed as follows: for each DAISY vector u, visited row-wise, left to right, we compute its k nearest centroids, then u is represented by these centroids, using a unique code for each centroid. The order induced by the distance to u is preserved. This procedure composes a visual word u^ for vector u. – In addition to visual words, we use a unique code to indicate the division among visual words.
This procedure is computed for all images, such that, the computed text will
represent each image. We use a Rocchio-like classifier [
We tried to use our TC to perform this final classification, but it exhibited a low performance, we presume this is a consequence of the large alphabet and large vocabulary setup found in our image-to-text encoding. Note that the procedure resembles a typical BoVW, but the differences produce significant
improvements in the performance. Among the differences with BoVW is the additional support for failure using k nearest centroids for encoding and the notion of syntax, captured due to our q-gram expansion. 3.2
Combining Text and Image-Based Author Profiling Results The combined prediction between text and images was computed using a convex combination between the prediction of the text and image problem in a separately way, comb(u) =
T (u) + (1 )I(u)
Where is the weight assigned to prediction of the text-classifier, T (u), and I(u) is the profiling prediction based on the image content.
More detailed, T (u) corresponds to the decision function of the underlying SVM inside TC, i.e., a number between -1 and 1. For the image problem, we normalized the absolute difference of the angles between examples and centroids, recall we use Rocchio for this subtask, so we have:
I(u) = \(u; cmale) \(u; cfemale)
D ; D where D j\(u; cmale) \(u; cfemale)j is the random variable representing the absolute differences of the distances between u and centroids, D is the mean of D and D its standard deviation. Please take into account that the order of cmale and cfemale is important, and it is dependent of the corresponding meaning of the decision function of TC, i.e., in the given definition, 1 corresponds to female and 1 to male. 4
The experiments with the training set were run in an Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz with 28 threads and 256 GiB of RAM running Ubuntu Linux 16.04. The gold-standard were evaluated in the TIRA platform using a virtual machine with 20GiB of RAM and six cores, necessary to parallelize and reduce the running time of computing image features. We use our TC using the master branch of https://github.com/INGEOTEC/microTC.5
We partitioned the full training dataset into two smaller sets, a new training set containing 70% of the users, and a validation set with the rest 30%. The core idea is then to perform cross-validation on the small training set, so we can optimize parameters, and then select best models in the validation set. Note that our participation includes both text and image tasks, and both follow the same partition scheme. The gold-standard was only accessed through the TIRA evaluation platform.
4.1
Results for the Text Classification Subtask To tackle the text classification we use our TC text classifier, already proved in PAN’17 in the author profiling task. In this sense, we tried the following four different paths: i. determine the best model for each language, ii. use the best configuration found in one language and create models all datasets in the different languages, iii. use the configuration models found in PAN’17 for each language and apply it to the new data, and iv. we use the best configuration in PAN’17 for some language and create models specific for each new dataset
Table 1 show our best results with the text classification approach using our TC system. Our best macro-F1 results in Arabic is 0:8377, in English is 0:8266, and in Spanish 0:8143. We selected the Arabic@PAN2017 parameters configuration instead of selecting the optimum for each, but it contains two of the three best performing model. Our final system is in bold, see Table 1.
We use 3-fold cross-validation for the model selection procedure. Once the model selection finished, we use the configuration found to train a TC machine with the whole (small) training set and measure the performance of that classifier on the validation set. Table 1 shows the performance in the evaluation set; the training set in the specified language was used. The set of parameters was computed in different ways, as explained above.
Surprisingly, in the evaluation set, the fourth option using our configuration of parameters found in PAN’17 for the Arabic language created superior models, in average, than other alternatives and configurations. So, we use it for all our final models. The Arabic@PAN’17 [19] configuration indicates that the entire text should be normalized to lower case, it also commands to delete diacritic symbols, consecutive duplicated characters, while punctuation should be kept. Emoticons and hashtags should be left untouched, while numbers and urls should be grouped into a common token (one per option); also, usernames should be deleted. The resulting text should be tokenized using unigrams, bigrams, and three-grams (word n-grams), and characters q-grams with q = 1 and q = 5, and also skip-grams (2; 1), i.e., three-grams without the central word. The weighting scheme indicate that low and high filtering should not be applied, and commands to use the entropy weighting with a smoothing factor of 3. It can be seen as weird that lower case is recommended on the Arabic language since the concept of upper and lower case is missing, but it is common to found messages, usernames and hashtags in other languages. Please note that the performance difference between the Arabic configuration and the resulting one of applying the parameter optimization for each dataset is of around 1 or 2 points of accuracy in the validation set, so they may perform quite similar, but it is interesting to verify the power of that configuration. The image-based profiling uses our Bag of Visual Words with 5000 centers and k = 7 (nearest centroids), with this configuration, our approach produced an accuracy of 0:5691, 0:5468, 0:5900 for Spanish, English, and Arabic languages, respectively. Figure 1 shows several configurations with different parameters and its related scores in the validation set. In this graphs, the y-axis is the value of accuracy, macro-F1, and macro-recall; the x-axis is the number of k nearest centroids used to encode the image into text. These parameters were the result of optimizing for the Spanish language and applying the same for the rest, see Figure 1. While this setup is competitive for the Spanish dataset, it has lower performance in English, and a quite bad in the Arabic dataset, as the array of figures illustrate. However, this multi-lingual analysis was performed after the deadline date, so they were not tested in the TIRA virtual machine. Based on this evidence, it is essential to optimize the parameters to each dataset.
On the other hand, the Rocchio classifier was feed with vectors produced with the text tokenized with character q-grams of length 1; 3; 5; and 7, the tokens having a frequency lower than 3 in the entire collection were removed, and finally, the TFIDF weighting scheme to encode the bag of tokens into vectors. Please note that this configuration was also selected for the Spanish language and applied for others; it is quite possible that the optimal configuration for each dataset vary; however, this analysis is beyond the scope of this document. 0.60 0.58 0.56 0.54 0.52 0.50 0.60 0.58 0.56 0.54 0.52 0.50 0.60 0.58 0.56 0.54 0.52 centers=3000, ar accuracy macro_f1 macro_recall centers=3000, en accuracy macro_f1 macro_recall centers=3000, es accuracy macro_f1 macro_recall centers=5000, ar accuracy macro_f1 macro_recall centers=7000, ar accuracy macro_f1 macro_recall centers=5000, en accuracy macro_f1 macro_recall centers=5000, es accuracy macro_f1 macro_recall centers=7000, en accuracy macro_f1 macro_recall centers=7000, es accuracy macro_f1 macro_recall The multi-modal approach uses the analysis of both text and images to predict the gender of the user. We tackled this task using a simple convex combination from the text and images author-profiling models, independently, normalize both predictions and combine them using a convex linear combination using the formulation of §3.2. Table 2 shows the performance of our approach in the evaluation set; also, the table shows the optimized that maximizes the accuracy performance in the evaluation set. In this notebook, we describe the INGEOTEC’s system used to solve the Author Profiling task in PAN’18. We used our MicroTC ( TC) framework to tackled the text classification problem, and for the images, we use a variant of Bag of Visual Words that transform images to text, and not only to histograms like typical BoVW approaches.
We observe that text-based features are dependent of the collection, and its analysis is beyond the scope of this On the other hand, we observed that women tend to share selfies and images with text-content. In case of men, they shared cartoons and humorous images, and landscape photos as well. In our approach, faces and text images are important in this specific problem. In our opinion working on this particular image classification, author profiling based on shared images, task was very difficult, mainly because of the limited amount of images per user. We will be working hard to improve our methods to improve our multimodal approach to author profiling. Finally, as part of our future work, we will be working in better combination schemes, ad-hoc to the problem particularities. 18. Tellez, E.S., Moctezuma, D., Miranda-Jiménez, S., Graff, M.: An automated text categorization framework based on hyperparameter optimization.
Knowledge-Based Systems 149, 110 – 123 (2018), https://www.sciencedirect.com/science/article/pii/S0950705118301217 19. Tellez, E.S., Miranda-Jiménez, S., Graff, M., Moctezuma, D.: Gender and language-variety identification with microtc. In: Working Notes of CLEF 2017
2017. (2017), http://ceur-ws.org/Vol-1866/paper_104.pdf 20. Tola, E., Lepetit, V., Fua, P.: Daisy: An efficient dense descriptor applied to wide-baseline stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(5), 815–830 (May 2010) 21. van der Walt, S., Schönberger, J.L., Nunez-Iglesias, J., Boulogne, F., Warner,
image processing in Python. PeerJ 2, e453 (6 2014), http://dx.doi.org/10.7717/peerj.453 22. You, Q., Bhatia, S., Sun, T., Luo, J.: The eyes of the beholder: Gender prediction using images posted in online social networks. In: 2014 IEEE International