This paper describes the systems designed by the Fraunhofer IAIS team at the CLIN29 shared task on cross-genre gender detection in Dutch. We show two alternative classi cation approaches: a rather standard one consisting of feature engineering and a random forest classi er; and an alternative one involving a LSTM classi er. Both are enhanced by a LDA model trained on stems. We considered various features such as frequency of function words, parts-of-speech and sentiment among others. We achieved 53.77% average accuracy in the cross-genre settings.
The CLIN29 shared task is de ned as predicting the gender of the author of collections of texts in Dutch. The organizers provided a labeled dataset from three di erent genres according to their source: "Twitter", "Youtube" and "News". For each of these genres, we were asked to predict the gender of the respective author by means of two di erent models: one trained with the text collection of the same genre (in-genre setting) and one trained with anything but text from the same genre (cross-genre setting).
We decided to design a common machine learning pipeline (per presented system) that works for all the settings. Thus, all the models are learned by applying the same pipeline and they only di er by their respective training data. This pipeline is described in section 3. Due to the shared task motivation of exploring approaches for cross-genre settings, we focused on tuning the pipeline components by evaluating each con guration on the three cross-genre settings exclusively, virtually ignoring intra-genre evaluations for hyperparameter tuning.
Finally, our experiments suggested that two di erent architectures were the best to tackle this shared task: one based on term frequency of function words and topic modelling; and another based on recurrent neural network using features of di erent nature. Since we were allowed to submit two di erent runs per setting, we provided predictions from both systems for each of the de ned Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) settings. Although both approaches reached similar accuracy values in the crossvalidations that we performed on the training dataset, the evaluation on the nal test dataset shows that our rst approach can generalize better on the evaluated genres. 2
There is already a certain tradition on research on the broader area of author
pro ling, which includes detection of other author characteristics of the author
such as their age, as we can see at the PAN shared task series1. However, most of
the research has been performed for text in English. In particular, we could only
nd previous work on cross-genre author gender detection in Dutch at the PAN
2016 shared task
Syntactic features such as part-of-speech (POS) and syntactic dependency
relations can predict author gender
Chen et al. (2018) showed that emoji usage in twitter highly depends on the gender of the author. However, emojis appear mostly on social media and they hardly exist in the "News" genre. Hence, we decided to remove all emojis and emoticons in order to prevent over tting in cross-genre settings: our aim was to test the same set of features in all settings at the cost of eventually underperforming at in-genre settings. 3
Our two systems consist of three main processes: topic modeling, feature extraction and classi cation. An overview of the pipeline can be visualized in gure 1. 3.1
Data: The topic modeling module takes two external corpora as input: { NLCOW142: the Dutch web corpus from the COW initiative (Schfer and
Bildhauer, 2012; Schfer, 2015). { Wikipedia: A Dutch Wikipedia dump from 2018-10-013.
Filtering: In order to comply with the cross-genre setting rules of not using any text coming from the genre where it is evaluated, we ltered out all documents from NLCOW14 whose URL contains the string "twitter" or "youtube". 1 https://pan.webis.de 2 https://github.com/rsling/cow , https://github.com/rsling/texrex 3 https://linguatools.org/tools/corpora/wikipedia-monolingual-corpora/
Since the concept of genre is rather de ned by the data source than by the text content, we consider that this approach su ces to discard all texts belonging to the "twitter" and "youtube" genres: each of these genres comes from a single data source.
In contrast to the other genres, "News" may come from diverse data sources which we cannot reliably lter out by just checking the document URL against a blacklist in the same way that we did for the NLCOW14 corpus. Therefore, we selected the Wikipedia corpus for the topic modelling part of the two "News" settings.
Preprocessing: From each document, a set of words is ltered out, namely:
stop words, emoticons, emojis, URLs and words starting with the characters
'#'and '@' (typical for Twitter hashtags and mentions respectively). Then, each
word token is both lemmatized and stemmed consecutively. Word tokenization
and lemmatization are performed with the help of the Frog parser
Model training: We train latent Dirichlet allocation (LDA) models
The labeled data from the three genres goes through a similar preprocessing as described for topic modeling. However, instead of being lemmatized and
stemmed, non-function words are normalized into some special tokens. Namely, words which are parsed as nouns, verbs, adjectives and special words are substituted by a token related to their POS. Additionally, tokens recognized as ordinals, hours, dates and other types of numbers are normalized to respective tokens as well.
The feature vector used for classi cation is dependent on the system: System 1 We create feature vectors based on function words. In our preprocessed text, only function words and normalized tokens are left. For each document, we compute the logarithm of the term frequency vector normalized with respect to the L1-norm as proposed by Diederich et al. (2003).
Since we assume that their usage frequency vary depending on the context
where they appear, we concatenate a 50-dimensional vector inferred from the
trained LDA model in order to provide contextual information to the model.
System 2 We decided to try also a neural network architecture based on a
long short-term memory (LSTM) classi er
Each of these vectors contains: { A coarse-grained POS tag (e.g. verb). { A ne-grained POS tag (e.g. 1st person, singular, present). { Position of the word within the sentence. { Relative position of the word (position / sentence length) { Position distance of the word respect to the parent word (in the syntax dependency tree) within the sentence. { Polarity score.
{ Subjectivity score.
For each vector, we append a 50-dimensional vector inferred by the respective LDA model from the document where the respective word resides in the same fashion that we do for system 1.
The POS tags are based on the Corpus Gesproken Nederlands (CGN) tags
The polarity and subjectivity scores are obtained from the Pattern software
package
In order to have a document representation of xed size, we restrict each representation to a sequence of 100 vectors. If a document has less than 100 vectors, we prepend zero vectors until we reach the xed size. Setting Max depth Twitter in-genre 5 Twitter cross-genre 4 News in-genre 11 News cross-genre 8 Youtube in-genre 10
Youtube cross-genre 5 For both systems, the training data depends on the evaluated setting: for the in-genre settings, only the provided examples from the same genre are used to generate the feature vectors; for the cross-genre settings, the examples for the not evaluated genres are fetched. Additionally, each feature vector includes a topic vector inferred by the respective topic model. For instance, the training set the for "In-genre Twitter" setting consists of the preprocessed "Twitter" examples plus a topic vector derived from a topic model trained on the NLWAC corpus, whereas the training set for the "Cross-genre Twitter" setting is made out of the preprocessed "News" and "Youtube" examples (plus the respective topic vectors).
System 1 We train a random forest classi er
System 2 Per setting, we train the following recurrent neural network
architecture with Keras5:
{ A bidirectional LSTM layer with 50 hidden units with 0.5 dropout ratio for
the linear transformations of the inputs and the recurrent state respectively.
{ A dropout layer with 0.5 ratio
{ A softmax layer
The architecture is trained by a Nesterov Adam optimizer
Setting In-genre Twitter News Youtube Average Cross-genre Twitter News Youtube Average As we can see from table 2, system 1 performs consistently better than system 2 in all settings. This is an unexpected result considering that the latter learns from a wider range of features. It is possible that the amount of training examples might have not su ced for the second approach to work better than the rst one because LSTM classi ers require considerably more labeled examples than random forest classi ers.
Another surprising result is that the accuracy in the cross-genre setting is higher than in the in-genre one in the case of the "News" datasets. This is probably to the much larger size of the training dataset for the cross-genre setting (34,744 examples from "Twitter" and "Youtube") than for the in-genre setting (1,832 examples from "News"). Therefore, system 1 may still perform better if it had a larger training dataset.
Our models show that they can only predict author gender for only a (small) part of the evaluated text collections. Hence, we suspect that most pieces of text do not have any genre-independent characteristics that can be reliably used to predict the gender of the author, although in some of them this may be possible. Therefore, we remain uncertain about whether it is feasible (in general) to predict author gender by using features that are independent from the text genre.