However neural networks are receiving more and more attention from diferent fields of computer-aided research, application of this approach to stylometry and authorship attribution is still relatively infrequent in comparison to other domains of natural language processing. In this paper we present our attempt to analyse frequencies of diferent types of linguistic data (Part-of-speech, most frequent words, n-grams and skip-grams) with the means of simple neural networks.
The concept of artificial neural network (ANN) is quite old [
Stylometric features can be combined together in so-called n-grams being
groups of n-neighbouring elements, mostly part-of-speech tags or characters,
because they render linguistic structures of higher order (respectively,
codependencies among parts of speech or stems with afixes). There is also a category
of skipgrams being extension of the idea of n-grams [
Since we wanted to try out experimental settings of features we propose a
fashion of combing PoS-tags with most frequent words used already for
stylometry [
To produce specific text forms of text representations (like PoS-tags) we used
third-party software for tagging and lemmatisation. For all but the Polish
language we tagged and lemmatised the training corpus with tools provided by the
package SpaCy [
(1) Normal unprocessed text. “The funeral had been a nightmare. Being who he is, the security kept away those who wished to sabotage the ceremony, and allowed only a minimal number of people to enter the graveyard. It didn’t stop some of the invited people from whispering in harsh tones insults that Mycroft chose to ignore.” (2) Character trigrams. ’_-t-h’, ’t-h-e’, ’h-e-_’, ’_-f-u’, ’f-u-n’, ’u-n-e’, ’n-e-r’, ’e-r-a’, ’r-a-l’, ’a-l-_’, ’_-ha’, ’h-a-d’, ’a-d-_’, ’_-b-e’, ’b-e-e’, ’e-e-n’, ’e-n-_’, ’_-a-_’, ’_-n-i’, ’n-i-g’, ’i-g-h’, ’g-h-t’, ’h-t-m’, ’t-m-a’, ’m-a-r’, ’a-r-e’, ’r-e-_’ (only first sentence showed ) (3) PoS-tags mixed with 150 most frequent words in the lemma form. ’the’, ’DT’, ’NN’, ’have’, ’VBD’, ’be’, ’VBN’, ’a’, ’DT’, ’NN’, ’NN’, ’be’, ’VBG’, ’who’, ’WP’, ’-PRON-’, ’PRP’, ’be’, ’VBZ’, ’the’, ’DT’, ’NN’, ’keep’, ’VBD’, ’away’, ’RB’, ’DT’, ’who’, ’WP’, ’VBD’, ’to’, ’TO’, ’VB’, ’the’, ’DT’, ’NN’, ’and’, ’CC’, ’VBD’, ’only’, ’RB’, ’a’, ’DT’, ’JJ’, ’NN’, ’of’, ’IN’, ’NNS’, ’to’, ’TO’, ’VB’, ’the’, ’DT’, ’NN’, ’-PRON-’, ’PRP’, ’didn’, ’VBZ’, ’t’, ’NN’, ’stop’, ’VB’, ’some’, ’DT’, ’of’, ’IN’, ’the’, ’DT’, ’VBN’, ’NNS’, ’from’, ’IN’, ’VBG’, ’in’, ’IN’, ’JJ’, ’NNS’, ’NNS’, ’that’, ’IN’, ’NN’, ’VBD’, ’to’, ’TO’, ’VB’
(4) PoS-tags with corresponding lemma for each token. “’the’, ’DT’, ’funeral’, ’NN’, ’have’, ’VBD’, ’be’, ’VBN’, ’a’, ’DT’, ’nightmare’, ’NN’, ’be’, ’VBG’, ’who’, ’WP’, ’-PRON-’, ’PRP’, ’be’, ’VBZ’, ’the’, ’DT’, ’security’, ’NN’, ’keep’, ’VBD’, ’away’, ’RB’, ’those’, ’DT’, ’who’, ’WP’, ’wish’, ’VBD’, ’to’, ’TO’, ’sabotage’, ’VB’, ’the’, ’DT’, ’ceremony’” (only first sentence showed )
Preference to a PoS-based approach was supported by an intuition that author-specific grammar structures should vary less between diferent domains than, for instance, most frequent words containing a lot of domain-specific vocabulary dictated by the domain itself. Moreover, by mixing PoS-tags with few MWF we wanted to enhance information on sentence structure carried e.g. by some function words. The optimal solution would be, however, to construct a list of function words, or by extension functors – due to time limit and insuficient knowledge of some of the PAN corpus languages we resigned from that step.
From the corpus texts prepared in the described way we calculated frequencies of n-grams and skip-grams (both of size 2, 3 and 4) for further investigation. 3
The general idea of Artificial Neural Networks (ANN) was introduced in the
1940’s and after multiple ups and downs it is today a steadfastly growing branch
of AI research. The term “deep learning” refers to a method of stacking many
layers of artificial neurons together what improve their computational
capabilities. In our approach we use simple architecture of so-called dense layers already
proposed for stylometric analysis with n-grams of characters [
Our network consists of individual branches for each category of linguistic features (e.g trigrams of PoS-tags) with an activation function. The formula of each hidden dense layer, that data are subsequently fed in, has the following form: y = W a + b (1) where x is a vector with either raw input data (for the first layer) or output data from an earlier layer (for nth layer, n > 1) and W, b are network parameters, a matrix and a vector called respectively weights and bias, that are learnt while training the network. Since all neurons in subsequent layers are connected with each other this type of ANN-architecture is called “dense” or “fully connected” in the contrast to convolutional layers, which preselect data output from a previous layer by so-called filters.
Activation functions are vital part of any artificial neural network and
provide a non-linear transformation of the input value. It guarantees that particular
neurons of a network are activated by a particular values of data. For our
architecture we choose the exponential linear unit [
After the first activation layer we carry forward our data to a next hidden layer for each category, then the respective outputs run once again through the activation cells and are merged into one branch combing all categories together. Then this one block of data in the form of a single vector is fed in the activation cells and the final layer assigning the authors to data.
The training consists of iterative providing of input data and conditioning
the weights and biases of the hidden layers of the network. The output of the last
hidden layer is mapped into (0,1) by the softmax function and then we take the
natural logarithm of that mapping. In this way we obtain log probabilities for
each author. As a loss function (called also cost function) to measure how “bad”
the network predicts a correct class we chose the negative log likelihood loss. At
that point we compute also gradients for all parameters. The gradients can be
thought of as vectors pointing to local minima of the loss function. The intention
is to arrive at a global minimum by locally optimising particular parameters [
Fig. 3. Schematic view of the classification network. The proposed network architecture assumes a multi-branch type of input, where the sets of features described in Section 2 are simultaneously processed by each set-specific branch (e.g a branch for POS-tags, a branch for POS-tags enriched with most frequent lemma etc.). For the purpose of determining feature utility we investigated diferent combinations of those sets. Due to practical constraints k (for n types of sets we have P n sub-combinations to investigate for k being n k the size of the smallest sub-combination) we focused on the most promising ones.
The task development corpus consisted of ten apart problems. Each problem
had either five or twenty diferent authors, to which a set of anonymous texts
of varying length of a couple of hundreds words. Furthermore, the texts in the
corpus problems were written in one of five languages (English, French, Polish,
Spanish, Italian) and the texts themselves were fragments of web fan fiction
originating from diferent fandom milieus [
The baseline approach proposed by the organisers was a Linear Support
Machine Vectors classification with trigrams of letters – an appropriate Python
code was also delivered by the organisers [
To start with, we tried out our network just with singular sets with diferent frequencies:
1.) 1000, 2000, and 3000 most frequent trigrams of POS-tags enriched with 350 most frequent lemmas.
2.) 1000, 2000 and 3000 most frequent lemmas. 3.) 1000, 2000, and 3000 most frequent trigrams of POS-tags.
Since each element has a single input node assigned, frequencies directly correspond to the size of the input layer influencing the training time of the whole network. For each problem a new model is initialised with the same parameters across all the languages and the problems with the same language do not share their models or features. The input for each specified branch is a simple normalised table with frequencies of corresponding features (trigrams of PoS-tags, unigrams of lemmas etc.). We give an overview of the average F1 score of each combination of parameters in Table 2.
A superficial look at results in Table 2 reveals a simple and obvious correlation – the more features the network analyses, the better it can perform across diferent categories. Furthermore, the network fed only with lemmas seems to perform better than as fed with PoS-tags and 350 most frequent lemmas. Since an only size limitation for a neural network is the RAM of a particular machine, we constructed a multi-branch network as described in Figure 4 for diferent features categories as described in Section 2. The single categories perform, however, significantly worse than the baseline method.
All the described networks were trained until an arbitrary loss threshold of 0.01 was not achieved. Because texts in the training data sets were from diferent domains, we decided to resign from creating a validation subset of texts – this approach has of course numerous weak points, like the risk of over-fitting of the model. In future work on this approach it should be more accurately addressed.
Since our network can process diferent types of data, we extended it with 6 branches (unigrams of lemmas, trigrams of PoS-tags, trigrams of PoS-tags enriched with 350 MFLemmas, trigrams of characters, skipgrams of PoS-tags with 350 MFLemmas with the gap size of 1, 2 and 3). For each category we took 3000 most frequent features. During diferent experimental runs we noticed some significant divergence among results of the runs with similar parameters, so we decided to run our model 10-times with exactly the same parameters. As seen in Table 3, the model does not yield the same scores for subsequent runs, meaning it may be guessing for some of the unknown texts.
For the final test run we chose the above combination of features, however due to an unfortunate setting mistake the model was trained just one epoch instead of approximately tens of times. It obviously leaded to an inferior score. The test corpus consisted of 20 problems of the similar ratio of authors and languages.
In the post-evaluation phase we continued to scrutinise the performance of various parameter settings. The self-evident drawback of our method is the size of the network, since it ties one input neuron to the frequency of each element, making the whole network as wide as a whole frequency table multiplied by the number of feature categories. 5
We want to further test our approach and ameliorate the obvious disadvantages, like the size of a multi-branch network. Furthermore, we are convinced of utility of the multi-branch approach combining together diferent types of linguistic markers (PoS-tags alone, PoS-tags with MFLemmas, skipgrams of diferent size). On the other hand one should think about simplification of layers, e.g. the number of nodes.