The writing style of individuals is the basis for tasks such as authorship attribution, authorship verification or authorship profile, associated with stylometric analysis. Traditional learning methods based on neural networks use the information encoded in the last encoding layer of a model such as Transformers. In this paper, we describe our thesis project in which we propose to investigate whether a deep neural network encodes the style in any way. To do so, we explore the intermediate layers and embeddings of the initial token encoding of all layers of BERT-based Transformer models, to identify and extract style features to improve stylistic modeling systems, with emphasis on the analysis of documents written in Spanish.
Style is defined as a form of expression or way of writing, starting with the choice of words, the combination of various words, punctuation, sentence structure, grammatical patterns and all the elements that an author likes to use [1]. The analysis of authorial style, called stylometry, is based on the assumption that style is quantifiable in order to evaluate its distinctive qualities [2].
The tasks associated with stylometry include authorship attribution, authorship verification or authorship profiling, they are based on the analysis of the writing style of individuals. The problem has been extensively explored, resulting in several traditional methods and tools for extracting stylometric features from a text.
Natural Language Processing (NLP) systems were initially based mainly on learning rules from the extraction of style features from a text. Later, they were replaced by machine learning models. Current deep learning models encode the relationship between words and learn about the final embeddings of their encoding layers, with encouraging results in text classification tasks but, we do not know what information about style is contained along the encoding layers. In this sense, we are exploring what style information is collected in the embeddings throughout the coding layers of the Transformer models, in order to experiment in stylometric analysis tasks such as authorship determination applied mainly to the Spanish language. Doctoral Symposium on Natural Language Processing from the Proyecto ILENIA, 28 September 2023, Jaén, Spain. Envelope cesar.espinr@ug.edu.ec (C. Espin-Riofrio) Orcid 0000-0001-8864-756X (C. Espin-Riofrio)
In this paper, we describe our thesis project focused on the extraction of stylometric features from Spanish language documents. We highlight the importance of our research, review its origin and related works, state the hypothesis and describe our research along with the methods, experiments and specific research elements proposed.
Stylometric-based Natural Language Processing is an approach that uses style analysis techniques to study and characterize texts. Style is reflected in the words and expressions used in texts in aspects such as syntax and grammar and in other measures such as average number of words used, frequency of word usage, length of paragraphs, etc. How a computer system can represent the style of a text or a set of documents is important. Stylometry includes among its most important tasks authorship attribution, authorship verification and authorship profiling, most of them solved on the basis of the writing style of a text. Text classification is a fundamental task of NLP, where the style of a text is the basis for extracting features.
NLP applies machine learning methods to identify patterns, extract and analyze features related to the writing style of a text. Traditional models obtain features using artificial methods and then classify them with classical machine learning algorithms, the effectiveness of these methods is largely limited by feature extraction. In contrast, deep learning integrates feature engineering into model fitting by learning a set of transformations that map features directly to outputs. Since their emergence, deep learning models treat the issue of style almost blindly, the models are applied to learn about features and relationships between words within text without delving into style.
We consider it important to delve deeper into what a neural network learns in relation to style in order to apply it to new models for solving text classification tasks such as authorship detection.
On the other hand, there are about 496 million people in the world who speak Spanish natively, making it the second most spoken language in the world [3]. Therefore, it is very important to carry out studies related to machine learning methods to extract style features from Spanish language documents to solve different NLP tasks.
The beginning of stylometry dates back to Augustus de Morgan's suggestion to resolve authorship disputes by means of word length frequency, in the year 1851 [4]. His hypothesis was investigated by [5], who published the results of his work by measuring the length of several hundred thousand words from the works of Bacon, Marlowe and Shakespeare. George Zipf discovered, using logarithmic scales, that there was a relationship between the rank and frequency of words, later known as Zipf's Law [6]. [7] measured word frequency for vocabulary richness analysis, now known as the "Yule characteristic". [8] used statistical methods to investigate the authorship of the Federalist Papers. The Federalist problem has been used subsequently as stylometry's 'testing ground' for new techniques. In the late 1980s, John Burrows published a series of seminal articles in which he regenerated stylometry as a viable tool in authorship attribution [9,10,11]. The initial work involving neural networks with stylometry was presented in [12]. [13] achieved results consistent with those of Mosteller and Wallace described earlier, using just eleven of their thirty 'marker' words as input to a neural network.
The stylistic features of a text are present at various levels such as in the vocabulary, the syntax, the grammar, the semantics, and in some cases in the layout, presentation, etc. [14] carried out an exploration of 166 features used for authorship attribution including commonly used stylistic features and several others intended to capture emotional tone. [15] divided authorship attribution features into five groups, on which much work has been don: lexical [16], character [17], syntactic [18], semantic [15] and application-specific features [19].
Simple lexical features, such as word frequencies, word n-grams, function words, word or phrase length, have been widely used since early attribution work [5], functional words were useful features in [8], the usefulness of character n-grams was highlighted in [15,20]. Bag-ofwords (BoW) approaches have also been reported as being useful for authorship attribution [21]. Term Frequency-Inverse Document Frequency (TF-IDF) [22] uses the word frequency and inverses the document frequency to model the text.
Traditional methods are statistics-based models, such as Naïve Bayes (NB) [23], K-Nearest Neighbor (KNN) [24], and Support Vector Machine (SVM) [25]. In a PAN 2013 Competition [26], all participants used a machine learning algorithm for classification, including Decision Trees, Support Vector Machines and Random Forests, [27,28].
The evolution of better computer hardware like GPUs and word embeddings like Word2Vec [29] and Glove [30] increased the use of deep learning models like CNN [31] and RNN [32]. LSTM (Long short-term memory) [33] attempts to solve the short-term memory problem of RNNs by retaining selected information in long-term memory. Convolutional seq2seq [34] applies convolutional neural networks.
Transformers [35], apply self-attention, captures the weight distribution of words in sentences. The attention mechanism is often used in an encoder-decoder architecture, and there are many variants of attention implementations [36]. A Transformer encoder layer is composed of multihead self-attention following a position-wise feed-forward network (FFN) with the residual connection [37] and layer normalization [38]. Transformer architectures rely on explicit position encodings in order to preserve a notion of word order. A positional embedding should be considered together with the NLP tasks [39]. The absolute position embedding is used to model how a token at one position attends to another token at a different position [40] Pre-trained language models [41], became a trend among many NLP tasks. Pre-trained language models effectively learn global semantic representation and significantly boost NLP tasks, including text classification. It generally uses unsupervised methods to mine semantic knowledge automatically and then constructs pre-training targets so that machines can learn to understand semantics [42].
Transformed-based pre-trained language models (T-PTLM) learn universal language representations from large volumes of text data using self-supervised learning and transfer this knowledge to downstream tasks. These models provide good background knowledge to downstream tasks which avoids training of downstream models from scratch [43]. GPT [44] and BERT [45] are the first Transformer-based pretrained language models developed based on transformer decoder and encoder layers respectively.
In general, an encoder-based T-PTLM consists of an embedding layer followed by a stack of encoder layers. For example, the BERT-base model consists of 12 encoder layers while the BERT-large model consists of 24 encoder layers. The output from the last encoder layer is treated as the final contextual representation of the input sequence. In general, encoder-based models like BERT are used in Natural Language Understanding (NLU) tasks.
Transformer-based models can parallelize computation without considering the sequential information suitable for large scale datasets, making it popular for NLP tasks. Thus, some other works are used for text classification tasks and get excellent performance, such as RoBERTa [46], XLNet [47], Bart [48], deBERTa [49], ERNIE [50].
In NLP tasks related to stylometry, some lexicons have been employed as EuroWordNet [51], Spanish Emotion Lexicon (SEL) [52], [53] perform a lexicon-based sentiment analysis of short texts generated on the social network Twitter in Spanish, [54] a lexicon-based approach to extract sentiment from text, Bing Liu English Lexicon or polarity classification [55], Spanish Opinion Lexicon (SOL) [56].
About corpus, there are some publicly available corpora for Stylometrics, they are important for NLP-related research, Autextification dataset [57], Enron [58], IMDB1M reviews [59], Guardian10 corpus [60].There are several important corpora for specific tasks in the Spanish language, such as Spanish-language corpus for researching offensive language OffendES [61], SFU Spanish review corpus [62], PoliCorpus 2020 [63], eSOLHotel. For the shared task on Multi-Author Writing Style Analysis PAN@CLEF2023 [64], PAN22 Style Change Detection [65], PAN21 Profiling Hate Speech Spreaders on Twitter [66], PAN20 Profiling Fake News Spreaders on Twitter [67].
Regarding the main tasks related to stylometry, there are the shared evaluation campaigns of Natural Language Processing (NLP) systems in Spanish and other languages, such as Automatically generated texts identification: Human or Generated, Model Attribution in AuTexTification in IberLEF 2023. Spanish Author Profiling for Political Ideology in IberLEF 2022. PAN CLEF shared tasks; Multi Author Writing Style Analysis PAN23, Style Change Detection PAN22, Profiling Hate Speech Spreaders on Twitter PAN21, Profiling Fake News Spreaders in Twitter PAN20, Celebrity Profiling PAN20, Bots and Gender Profiling PAN19, among others.
Neural networks are capable of capturing stylistic information, that information combined with previously known stylistic features such as character-level characteristics, word, phrase, vocabulary richness, lexical complexity, etc., can help solve tasks such as authorship attribution, profiling users based on their writing, differentiating between synthetic text and human-written text, etc.
The question arises; what does a neural network learn that is related to style?
We propose to further investigate the topic and determine what information about style is contained throughout the layers of pre-trained Transformer-based models, and experiment with methods of extracting their embeddings to refine learning models in text classification tasks, especially in Spanish.
An exhaustive analysis of the state of the art has been carried out, determining the classical techniques for the extraction of style features from Spanish and English texts, exploring with them in different application domains and tasks.
We participated in the main international forums on NLP tasks such as PAN, IberLEF, Semeval. We use in our experiments the reference datasets proposed in those campaigns, and thus compare our results with those obtained by other researchers.
We are experimenting with current neural network models to determine what they learn about style. To this end, we are currently exploring with the extraction of initial embeddings from all layers of BERT-based Transformers models to fine-tune a learning model for various text classification tasks.
In terms of dissemination of our results, we have published several scientific papers of worldwide impact, and we are also participating in international scientific conferences such as SePLN, Laccei and SmartTech.
We explore the capacity of linguistic features of various kinds that can be extracted from the text to be considered elements of style, such as lexical diversity, lexical complexity, syntactic and semantic features, etc.
We ask whether there are style features in the parameters that a neural network learns, whether style is encoded in any way in a deep neural network such as Transformer-based models, and if so, where and how. In the deeper layers of a Tranformer encoder it is possible that there is information about the style rather than the semantics. In this sense, we are analyzing, based on a series of works, not only the final coding of BERT-based Transformer models, but also the first and intermediate layers of coding in search of style features, so we are exploring ways to analyze and extract that information to improve stylistic modeling systems.
I thank the University of Jaén for allowing me to carry out my doctoral studies there, my mentors my director PhD Arturo Montejo Ráez and tutor PhD Fernando Martínez Santiago, and PhD Luis Alfonso Ureña López coordinator of the program.