We present our entries to the Shared Task on Cross-genre Gender Detection in Dutch at CLIN 2019. We start from a simple logistic regression model with commonly used features, and consider two ways of combining training data from di erent sources. Our in-genre models do reasonably well, but the cross-genre models are a lot worse. Post-task experiments show no clear systematic advantage of one way of combining training data sources over the other, but do suggest accuracy can be gained from a better way of setting model hyperparameters.
Detection of binary author gender can be done from text alone with impressive e ectiveness. For instance, Van der Goot et al. (2018) report an accuracy of 80% on Dutch tweets for their system, and the top systems for four other languages reported in Rangel et al. (2017) also perform in the low 80% accuracy range. These results concern systems that were trained on and applied to Twitter data, with multiple documents (i.e., tweets) per author. It is to be expected that performance su ers when such systems are applied to another genre than they were initially trained on. The CLIN 2019 Shared Task on Cross-genre Gender Detection in Dutch therefore invites authors to investigate \gender prediction within and across di erent genres in Dutch."1 This paper reports on our participation this shared task.
The shared task consists of an in-genre setting and a cross-genre setting. In the former, models are trained on and applied to data from the same genre/data source. In the latter, models are trained on data from one or more sources, and applied to data from a genre that is assumed to be completely unknown during training. The genres supplied in the shared task were News, Twitter and YouTube comments.
Having access to training data from multiple sources raises the question of whether we can use that fact to construct models that generalize better and therefore perform better in a cross-genre setting. Our contribution to the shared task is a small investigation of the e ect of how the multiple sources are combined. Building upon a basic logistic regression model with features taken from existing research on author pro ling in general and gender identi cation in particular, we compare two ways of combining training data sources.
Section 2 gives a formal description of the used model and introduces an alternative objective that combines training datasets in a principled way. Section 3 describes the used features and gives some implementation details. The results for our systems in the shared task are presented and brie y discussed in Section 4. These results prompt a set of post-task experiments, whose outcome and implications are re ected upon in Section 5 2
The core of our approach is a logistic regression model. To t a logistic regression
model with L2 regularization, we need to nd an intercept b0 and feature weights
B that minimize the sum of a) the normalized negative log-likelihood of the
model given the data, and b) the squared magnitude of those weights. To be
precise, we minimize
1
jDj
log L(b0; BjD) +
2
jBj
X bi2;
i=1
(1)
where D is the training data, and a hyperparameter that lets us set the strength
of the regularization factor.
In the cross-genre setting, we are able to train on a combination of two or
more datasets from di erent genres. Ideally, we would be able to leverage this fact
to nd models that generalize better and therefore fare better when applied to a
new genre. Handling data from di erent sources is well-studied in the domain
adaptation literature. However, there one often has access to training data from
both source and target domains
d o o h li e k i -l g o l e v it a g e n d e z il a m r o N
Source D₂ (10× larger) Pooled data model Average objective LSE objective (k=1,4,16,64) (minima) 0.0 0.2 0.4 0.6
A direct way to combine training data from multiple sources is simply to pool the data. We can then proceed to train according to Equation 1, as we would with a single dataset. When we are dealing with source datasets of unequal size, we can also consider normalizing the respective negative log-likelihoods to the sizes of the datasets, so that the larger dataset does not dominate the model. Just summing/averaging normalized negative log-likelihoods for each source could still lead to one source dominating, if that source is much easier to model. We therefore combine the negative log-likelihoods by taking their maximum. The log-sum-exp function gives us the kind of smooth maximum we need in order to be able to use standard optimization algorithms. In the formulation we use, log-sum-exp also takes a scaling parameter: lse(x; y; k) = log(ekx + eky): With positive k, lse(x; y; k) is always greater than max(x; y). A higher k makes lse less smooth but closer to max.
The objective for two datasets would thus be lse
1 jD1j 1 k 1 jD2j 2 jBj X b2;
i i=1 (2) (3) log L(b0; BjD1); log L(b0; BjD2); k + where and k are hyperparameters. This is trivially extended to more datasets.
Figure 1 illustrates the three ways of combining two source datasets graphically. The hope is that by combining sources in a balanced way, we nd models that generalize better to new genres. This is on the premise that we do not know anything about the target genre. If we had information that the new genre is more like one of the source genres, we might be better o building an `unfair' model. 3
Features, data preparation, and implementation
A linear model using surface form-based n-gram features has been shown to be
very e ective in (in-domain) gender identi cation
We follow common praxis in authorship attribution
Texts were tokenized using Cutter
Fitting the logistic regression models was done with L-BFGS using the facilities supplied by SciPy2 and Autograd.3 4
For the in-genre models, we entered two groups: full models according to the speci cations above (`in-genre 1'), and models that only uses the 2500 lexical features (`in-genre 2'). We set the regularization hyperparamater by 5-fold cross-validation.
The results are in Table 1. Cross-validation gives fair estimates of the task evaluation results (except for one overestimate). In the task evaluation, the full models do better than the reduced. We speculate that the character and word length features, being less sparse, make the models more robust. Compared to the other entries to the shared task, both kinds of model perform reasonably well, 2 scipy.optimize.minimize, see scipy.org 3 See github.com/HIPS/autograd
| Average
X-val acc log10
News Twitter YouTube Avg Eval acc Eval rank with accuracies consistently in the top half. It should be noted that all entries to the shared task perform well below the 80% mentioned in the introduction. This is probably related to the training data set sizes and the fact that the task requires prediction on the basis of just one document.
For the cross-genre models, we also entered two groups of models: one group combining the datasets using the lse-based formulation of the objective (`crossgenre 1') and one pooling the data (`cross-genre 2'). The scaling hyperparameter k for lse was kept constant at 20, no attempts where made to optimize it, and was set by 5-fold cross-validation. The hyperparameter setting for the model with the highest macro-averaged accuracy between data sources was chosen.
The results are in Table 2. The cross-validated accuracies are all lower than in the in-genre case: Apparently the models su er more from having to deal with two di erent sources than they bene t from having larger training datasets. Comparing accuracies of cross-genre 1 (lse) to cross-genre 2 (pooling), we can observe that the pooled data model tends to cater better for the larger source dataset (viz., Twitter or YouTube), although this is only very pronounced in the model combing News and Twitter training data. The average cross-validation accuracies of the two combination methods are very similar, except in this last case, where lse has a slight advantage. The task evaluation accuracies are also very similar between the two model types. As is to be expected from the shift in genre, the cross-validation accuracy here is a very poor indication of evaluation accuracy: the latter is on average almost 8 percentage points lower. Compared to the other entries in the shared task, we now do a lot worse, performing in the bottom segment. The poor results on the News genre { the least similar genre { are to blame for this, as we are in the middle bracket for the other two genres. 5
As mentioned, the cross-genre models do not fare as well as the in-genre ones in the shared task. In addition, the di erences between lse and pooling is also small. Pooled data LSE model (k=1,2,4,..,128) (x-val setings)
Pooled data LSE model (k=1,2,4,..,128) (x-val setings) To see to what extent these results depend on our choice of hyperparamaters, we trained models on two genres at di erent levels of and k, and evaluated on a third genre. We only used the shared task's training data for these experiments. The results are in Figure 2. Note that the results of the shared task evaluation are not in here, since we did not use the task evaluation data. There does not seem to be a clear, systematic di erence between accuracies for the two ways of combining data. However, we can see that the hyperparameter settings from cross-validation are suboptimal for both methods in all three datasets. In addition, choosing the right hyperparameter setting for k can make a real di erence in performance, although overall it seems that a higher k is preferable. 6
We have presented our e orts in the Cross-genre Gender Detection shared task, where we aimed to compare two ways of combining data sources: simply pooling the data vs optimizing an objective that combines the respective negative loglikelihoods with log-sum-exp. The two methods perform similarly, and we have not seen evidence of a real advantage of using the more involved method. However, a set of post-task experiments does show that there is performance to be gained from a better way of picking the hyperparamaters in both methods.
In this work, we have not focussed on the feature set de nition nor studied the e ectiveness of di erent kinds of features in any depth. In theory, these issues are orthogonal to what we presented in our report. We thus reserve the investigation of model combination methods in the context of known state of the art feature sets for future work.