The Style Change Detection task is very important in the area of authorship profiling, having one of its main applications in plagiarism detection. Specifically, the goal of the task is to detect where (if any) stylistic changes happen in a document which can be used to estimate the number of authors of a given document. In this paper, we present a method for Style Change Detection. We use Google AI's open source BERT pretrained bidirectional models to tokenize and generate embeddings for the sentences in each document in our dataset and use those to train a random forest classifier. We achieved an F1 score of 0.86 for detecting style changes and an F1 score of 0.64 for detecting multi-author documents on the test set, placing us at the top of the competition for both tasks. The code for this project has been made open source so that it can be used for further research: https://github.com/aarish407/ Style-Change-Detection-Using-BERT Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CLEF 2020, 22-25 September 2020, Thessaloniki, Greece.
Detecting the number of authors involved in writing document by analyzing the writing
style is an important task that has been a focus of research for centuries. This area of
research has traditionally been called Stylometry and is defined by the Oxford dictionary
as, “the statistical analysis of variations in literary style between one writer or genre
and another". It is a centuries-old practice, dating back to the early Renaissance. Its
applications include plagiarism detection and forensics (e.g., Vosoughi et al. [
Unsurprisingly, style understanding has become one of the core areas of research in
natural language understanding, leading to the proposal of various computational
methods for understanding and detecting style in written text (e.g., see [
In this paper, we present a solution for this task using a Random Forest classifier in conjunction with embeddings generated by BERT, an open source large-scale pretrained language model developed by Google AI. The remaining of the paper is organized as follows. First, we introduce the dataset, next we describe our approach, including all data cleaning and pre-processing steps. Next, we describe our experiments and results. Finally, we wrap up by discussing future work and summarizing our findings. 2
There were two types of datasets [
Here is some information about the dataset: 1. Table 1 shows the number of documents in the narrow and wide train and validation sets. For each document, there was an appropriate truth file. 2. Table 2 shows the statistics of number of sentences and number of paragraphs in each document for the train narrow and wide datasets. 3. The truth files had the following data: number of authors, order of authors, source site and the results for task 1 and 2. For our solution, we did not make use of the first three keys. 4. Figures 1a and 1b show the distribution of number of style changes for the train narrow and wide datasets. 5. All datasets were balanced for task 1, i.e., detecting if a document is written by more than one author.
(a) Wide dataset.
(b) Narrow dataset. The general approach to both tasks was to generate embeddings of the words in each document at the sentence level and then use these embeddings for the classification. This is highlighted in Fig. 2. The first step was to split each document into paragraphs, since paragraphs are guaranteed to be atomic (i.e., only a single author has written a paragraph). This is important as the second task involves identifying style change between consecutive paragraphs. On first thought, the idea of splitting the paragraphs into sentences seems fairly straightforward – split on characters such as ’.’, ’?’ and ’!’. However, a lot of sentences would be generated that weren’t sentences originally. For example, the prefixes ’Dr.’ or ’Mr.’ would have their own sentences. Thus, it was important to ensure that the sentences were split in a manner that is robust to the variations in usage of the aforementioned punctuation marks. A regular expression approach was used, for which each occurrence of "." which was not meant as a sentence delimiter is identified and replaced with a special token. The following structures were identified and replaced accordingly: – Prefixes (Mr., Mrs., Dr., Ms., Prof., Capt., Cpt., Lt., Mt.) – Website domains (.com, .net, .org, .io, .gov, .me, .edu) – Acronyms (U.S.A., etc.) – Suffixes (Inc., Ltd., Jr., Sr., Co.) – Abbreviations (e.g., i.e., ...) – Any digits separated by a period
The above approach doesn’t take into consideration the different usage of ’.’, ’,’, ’?’ and ’!’ written in code, which is likely to come up in a dataset mined from the Stack family of websites. This can be added in the future to further improve this solution. 3.3
Before generating the embeddings, the sentence had to first be tokenized, which was
done by using Google AI’s BERT [
In order to generate embeddings for the tokenized sentences, Google AI’s BERT
[
Although BERT is used to capture context rather than style, the authors of this work found out that the information captured by these embeddings works well for the style change detection task as well. 3.4
Although BERT was used to generate the embeddings, they had to be combined in a specific way to fit both the tasks. The following is the method followed: 1. Each individual sentence was processed by the BERT Tokenizer, and truncated to 512 tokens if needed. 2. The tokenized sentence was then processed by BERT, which generated embeddings for each layer. This generated a tensor of dimensions 12 l 768, where l is the length of the sentence. 3. The authors of BERT found out that the best results were obtained when the embeddings of the last 4 layers were combined, either by summing them, producing a tensor of dimensions l 768, or by concatenating them and producing a tensor of dimensions l 3072. We chose to sum the embeddings of the last 4 layers in order to prevent the dimensions of the tensor from becoming too big. Thus the final dimensions of the tensor at this step are l 768. 4. The first dimension of the current tensor is the length of the sentence, and can thus change from sentence to sentence. In order to prevent this, the embeddings of the sentence are summed over the first dimension, thus producing a final vector of length 768. Summing the embeddings over the first dimension as opposed to averaging them can lead to large difference in embedding values between sentences that are long and short. However, the length of a sentence is an important factor in detecting style change, and thus it is important to capture that information.
At this stage, we change our approach of combining embeddings for the two tasks. Detecting style change at the document-level (Task 1) To produce a final tensor for the whole document, all the sentence vectors of the document were averaged. At the document-level, the following approaches were tested: 1. Generate the sentence vectors by summing the embeddings over the length dimension + summing all the sentence vectors of the document to produce a documentlevel tensor 2. Generate the sentence vectors by summing the embeddings over the length dimension + averaging all the sentence vectors of the document to produce a documentlevel tensor 3. Generate the sentence vectors by averaging the embeddings over the length dimension + summing all the sentence vectors of the document to produce a documentlevel tensor 4. Generate the sentence vectors by averaging the embeddings over the length dimension + averaging all the sentence vectors of the document to produce a documentlevel tensor
The second approach produced the best results for the style change detection task at the document-level. We have described why summing the embeddings over the length dimension as opposed to averaging them works better. While producing the documentlevel tensor, averaging all the sentence vectors seems to work better. This can be attributed to the fact that the length of the document doesn’t really factor into determining whether or not style change occurred in the document, as all style changes occur between paragraphs. Thus, it makes no difference if the document is relatively short or long, as long as it has at least two paragraphs. Thus, there is no need to capture this information. It must be noted that the difference in accuracy for all four approaches was within 2% for the validation wide dataset.
Detecting style change at the paragraph-level (Task 2) Since style change had to be determined between paragraphs, the paragraph-level data points were calculated by averaging the embeddings of two consecutive paragraphs. Thus, the data point was generated by adding the embeddings of all sentences in both paragraphs and then dividing it by the sum of both paragraph lengths (in sentences). The labels were the entries in the changes array of the truth file. It is important to note that the labels of the paragraphlevel data points are now imbalanced, as a document with no style change will have all paragraph-level labels as 0, while a document with style-change may still have some consecutive paragraphs that were written by the same author, and thus the labels for those data points would also be 0.
After this step, we essentially have two datasets - one with data points at the documentlevel and the other with data points at the paragraph-level. 3.5
Using Python’s off-the-shelf ML library Scikit-learn [
The final set of hyperparameters for each classifier are given in Table 3
Criterion Min Samples Per Leaf Min Samples Per Split
Estimators Here we show the performance of our model on the validation and test sets. The validation set was made available during the development of the model, while the test results show the performance of our model in the competition.
Table 4 shows the performance of our model on the validation set and Table 5 shows the performance of our model on the test set. Note that for the test set, we only have cumulative information of the two datasets for the two tasks
As can be observed, there is a discrepancy between the results reported on the test set and the validation set. This is because of the difference between the environments in which both the tests were carried out. During the development of this project, the BERT model was run using a GPU, which greatly increased the speed of computation. However, since the virtual machine offered by TIRA did not provide a GPU, all computations were significantly slower. The authors of this paper decide to clip the computations after a certain time in order to prevent the session from crashing and not being able to submit our solution. 5
During the course of this project, a number of different approaches were tried. For those approaches, a unique dataset was generated, where each data point was a combination of two sentences from consecutive paragraphs of a document. Thus, if the sentences were from the same paragraph, then the corresponding label would be 0, while two sentences from different paragraphs would have a label of 1 if there was a style change between the two paragraphs. This approach is also susceptible to producing an imbalanced dataset, and hence the dataset was balanced before moving on with the classification task. The dataset produced had nearly 3 million data points by just using the wide dataset. A couple of the approaches have been described below: Fine-Tuning BERT In this method, the goal was to fine tune BERT using the training data so that it could produce results that were at par or better than the submitted solution. However, it was empirically observed that accuracy plateaued after a point, and was thus not explored further.
Convolutional Neural Network This method is inspired by prior work on sentence
classification using convolutional neural networks [
Due to a lack of time, this approach could not be explored fully. However, the authors of this paper believe that there is merit to this approach, and intend to study it further in the future.
In this paper, we have shown how BERT can be used for the Style Change Detection
task. Although BERT is used to capture context, this project shows that the information
captured by its embeddings can be used for other NLP tasks as well. We intend to work
on this project further by expanding on the other methods mentioned in Section 5. The
code for the project can be found at [