The CLIN 29 shared task is concerned with binary gender prediction within and across di erent genres in Dutch. Our proposed approaches to this problem are a simple model, which uses character n-grams, and a more complex model which consists of two systems in a co-training setup. Both of these approaches beat the baseline scores in all in-genre and cross-genre settings. Our simple model works better in an in-genre setting. The simple model performs only slightly worse in a crossgenre setting than our complex model. This is in line with the ndings of Basile et al. (2017). We conclude that our co-training setup does not seem to work as well as expected for cross-genre gender detection. We believe that this might be caused by the fact that the di erence between instances is larger between genres than between genders, making initial predictions for the co-training setup inaccurate.
Gender prediction is a relatively common author pro ling task. The theory
behind gender prediction is that men and women use slightly di erent variations
of language, and that they write about di erent subjects. This is also why
stateof-the-art models use relatively traditional systems and features, such as n-grams
and a Support Vector Classi er
Unfortunately, these models do not seem to work well in a cross-genre setting, as the di erence in writing between genres is larger than the di erence between genders. To tackle this problem, and to determine if a cross-genre gender prediction model is currently even feasible, multiple shared tasks have been organized in the past year. For Italian, there was the EVALITA 2018 Gender x Genre Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) (GxG) task1. For Dutch, the CLIN 29 shared task2 was organized with a similar setup and datasets.
In this paper, we compare two approaches, a simple approach and a more complex approach. By participating in this shared task, we have tried not only to create a cross-genre gender prediction model, but also to answer our own research question:
Does a simple support vector machine model outperform a co-training model for gender prediction in a cross-genre setting? Task description Given a (collection of) text(s) from a speci c genre, the gender of the author has to be predicted. The task is cast as a binary classi cation task, with gender represented as F (female) or M (male). Gender prediction will be done in two ways: { using a model which has been trained on the same genre; { using a model which has been trained on anything but that genre. 2
A common approach to cross-domain prediction is transfer learning. Transfer learning is the process of training a model on a large dataset for one task and then applying that model on another dataset for a related task. This approach can be useful when there is only a small amount of training data available for the target task, while large corpora exist for the source task. This approach is often used for neural networks and for word embeddings, but can also be applied to other types of more traditional machine learning.
Co-training (also known as co-learning)
Co-training uses multiple classi ers with di erent views of a problem which, similar to transfer learning, train on one set and predict on another dataset. Unlike transfer learning, co-learning does not only attempt to build upon previous knowledge, but also on di erent views.
The classi ers add the predictions they are relatively sure about from the unlabelled set to the training set of the other classi ers. This makes it possible for the other classi ers to learn about these instances in their way, which might then improve the accuracy on the unlabelled set, which in turn adds more of the unlabelled instances to the training sets. 1 https://sites.google.com/view/gxg2018/task 2 https://www.let.rug.nl/clin29/shared_task.php
For our complex model, we have decided to use lexical normalization as a pre-processing step. Lexical normalization is the task of converting non-standard text (e.g. \Somethign liek dis") to clean text (\Something like this"). This preprocessing task tries to minimize the di erence between the di erent genres in this shared task. By converting text from all genres, we reduce the amount of spelling mistakes, phonetic substitutions, and other errors. This should increase the similarity between the genres, as news most likely has less deviation from standard spelling than the Twitter and YouTube domains. As we wanted our system to work for any genre, in a language agnostic way, we have made sure that every genre was handled in the same way.
Even though this pre-processing step most likely will not a ect news articles,
as those are more-or-less in standard form already, it could be important for the
Twitter and YouTube genres. This is especially the case for Twitter, which has
often been used as a subject in text normalization because of its character limit
Contrary to what one might expect, simple models have so far worked better than complex models for gender prediction. Basile et al. (2017) found that their model performed better when less features were used. This is why we have also created a very simple model, which we will compare to our more complex model, in order to nd out if complex models consisting of relatively simple systems also perform less well than simple models themselves. 3
The dataset consists of three genres: news excerpts, Twitter posts and YouTube comments. All genres consist of half female authored, half male authored documents. An overview of the data is given in Table 1.
Genre
Training Test News 1,832 1,000 Twitter 20,000 4,914
YouTube 14,744 10,000
Table 1. Number of instances per genre in train and test data.
As we can see, the dataset is relatively small in comparison to other NLP
datasets. The length of the instances is also shorter than it would normally be,
providing only one tweet per user for Twitter, and one reaction for YouTube
users. The news instances are highly variable in length, ranging from only one or
two sentences to entire articles. We have also used word embeddings trained on
external data. These embeddings have been trained on multiple datasets from
di erent domains, including but not limited to the SONAR 500 corpus, Twente
News Corpus
These embeddings are represented as an array containing average embedding vectors per document.
Document Lexical Normalization
P > 0.7
Balancing
SVC (embeddings)
Logistic regression (tokens) Predictions
Predictions Choose most probable class
Predictions
For the simple approach, we use no pre-processing. For the more complex approach,
lexical normalization is applied to all documents. This increases the likelihood
that a certain word would match a known n-gram combination or word embedding.
We use the MoNoise system
Approaches
In this section we introduce our simple (R1) and complex (R2) approaches to
cross-genre prediction. Both rely on scikit-learn
R1 uses 3-6 character n-grams from within word boundaries. We use TF-IDF vectors based on these n-grams to train a Support Vector Machine (SVM) with a linear kernel. This approach aims to not only be robust to slight spelling variations and mistakes, but also tries to generalize well enough.
By using character n-grams instead of the commonly used combination of word and character n-grams, our model will be less likely to accidentally model topic instead of gender. Even though such large n-grams are able to capture smaller words or topics, we did not observe this for the Twitter and YouTube genres during development. For the news genre, we found that some substrings such as `bier' (beer) occurred multiple times in the list of most signi cant features, which means that some topic modelling will still have occurred. A side e ect of this approach is that some grammatical structures might not be detected by character n-grams alone. One of the phenomena that we cannot detect with our n-grams is the relation between the words.
R2 uses two di erent systems and feature sets in a co-training setup. The rst system uses binary TF-IDF vectors that represent document tokens, which are fed to a Logistic Regression model. The second system uses external word embeddings as features in a support vector classi er with a linear kernel. A schematic overview is provided in Figure 1.
Each system is trained and predicts classes for an unlabelled dataset. In a cross-genre setting, source data is used as training set, and target data with the labels removed, to provide unlabelled data. In in-genre settings, the training data is evenly split between the training and unlabelled set.
Documents that belong to a certain class with a certainty above a set threshold according to one system, are added to the training data of the other system and vice versa. The training data is then re-balanced, to reduce the chance of over tting on one class. During development we observed that without balancing, one class would be over-represented in the new training instances. This process continues until no new items are transferred between the training sets of both systems or a maximum number of iterations is reached. If the system nds that it should train on the same sets as during a previous iteration, it will also stop early, to prevent computationally wasteful training loops.
To ensure that instances with a low con dence score will not be added to the training data, we use a threshold (P ) to lter these. Empirically, P = 0:7 showed the most stable results. For the same reason, we have chosen to limit the maximum number of iterations of the co-learning process. Using ve iterations at most gives a good balance between run time and performance on the di erent genres. Due to (run) time constraints, we could not tune this parameter in a more sophisticated manner. For the cross-genre scores, we based our choice on the performance of di erent setups on the development data. To validate our models, we split our training data evenly into a training and development set. Half of this data was used as unlabelled data for our co-training setup, while the other half was used as test data. Training was done with the full training set of another genre. Based on the scores in Table 2, we decided to use normalization for our co-training approach as it performed better on two of the three genres. We selected the best performing model for each target genre. These models are shown in bold in Table 2.
For our simple approach we did not consider using lexical normalization as we wanted to compare our co-training approach to a simple model.
The results of both systems are presented in Table 3. R1 performs better in-genre, outperforming R2 in every genre. R2 performs best on average in the cross-genre setting, although the di erence with R1 is small.
Compared to other submissions of the CLIN shared task, the results of R1 are interesting. On the in-genre task, the model ranks second. This con rms the conclusion of Basile et al. (2017) that simple traditional models still perform very well on this task.
Genre
IN R1
R2
R1 News 0.6890 0.5830 0.5260 0.5530 Twitter 0.6367 0.6241 0.5406 0.5376 YouTube 0.6156 0.5849 0.5360 0.5212
CROSS
R2
The results of the co-training setup were somewhat disappointing. We believe that the cause for this lies with the initial predictions the system uses. The di erences between instances appear to be larger between the two genres than the di erent genders. As a result, the system is fed with wrong information and is also not able to overcome the lack of new information.
Some aspects of the co-training setup that could have been improved further. The parameters regarding transfer of training instances and the maximum number of iterations were set based on balancing (run) time with the results on the di erent genres. Fine-tuning these parameters could lead to better results.
The simple model is also limited by its feature set. As we use word boundary n-grams, almost similarly spelled words result in almost similar n-grams, which results in the loss of grammatical information. Rangel and Rosso (2016) suggest that certain morphosyntactic information is reasonably indicative of gender. Extending the n-grams beyond word boundaries could help to also capture this information. 7
As our systems did not reach similar accuracy between the in-genre and crossgenre settings, we have not managed to create a domain-agnostic model. We believe that more work towards this task should be performed. One of the most important steps would be assessing how well human annotators can perform this task. This could shed some light on whether better results can reasonably be expected or not.
If human evaluators do manage to score signi cantly higher than current systems, we suggest focusing on simple approaches. These approaches seem to work just as well as a co-training approach and are often easier and faster to train.
We would like to thank the Center for Information Technology of the University of Groningen for their support and for providing access to the Peregrine high performance computing cluster. We also want to thank Rob van der Goot for both continuous feedback and his support running the MoNoise model for Dutch. The word embeddings that we used in our complex model were trained by Gertjan van Noord.