The goal of style change detection is to identify text positions within a multi-author document at which the author switches. Detecting these positions is a crucial part of processing multi-author documents for purposes of authorship identification. In this year's PAN style change detection task, we asked the participants to answer the following questions for a given document: (1) Given a document, was it written by multiple authors? (2) For each pair of consecutive paragraphs in a given document, is there a style change between these paragraphs? The task is performed and evaluated on two datasets compiled from an English Q&A platform, which differ in their topical breadth (i.e., the number of different topics that are covered in the documents contained). The paper in hand introduces style change detection as a task and its underlying dataset, surveys the participants' submissions, and analyzes their performance.
The task of style change detection aims at detecting positions of author changes within
a collaboratively written text. Previous PAN editions paved the way for PAN’20 by
analyzing multi-authored documents for style changes. This includes the identification and
clustering of text segments by author in 2016 [
The remainder of this paper is structured as follows. Section 2 presents previous style change detection approaches. Section 3 introduces the style change detection task as part of PAN 2020, along with the datasets and evaluation measures employed. Section 4 summarizes the received submissions, and Section 5 analyzes and compares the achieved results and Section 6 concludes the paper. 2
Style change detection is closely related to the fields of stylometry, plagiarism
detection, and text segmentation. All of them have in common that they rely on intrinsic
stylometric analyses of documents, without referring to external documents or
corpora for comparison. Hence, stylistic profiles are created that are based on lexical
features like character n-grams (e.g., [
One of the earliest works on style change detection by analyzing stylometric
features to detect author boundaries is by Glover and Hirst [
At PAN 2017 [
Author 1 Author 1 changes, resulting from at most three different authors. 3.2
As with last year’s task [
After cleaning the data, we constructed two datasets differing in the number of topics covered to also enable an investigation into how well approaches are able to deal with topical diversity. For the first dataset, called dataset-narrow, we used only questions and answers belonging to a subset of the StackExchange sites that deal with topics related to computer technology.2 For the second dataset, called dataset-wide, we used a subset of sites covering a wide range of different topics,3 including technology, economics, literature, philosophy, and mathematics.
For every site in the subset of sites that was used for the creation of a given dataset, we grouped together all questions and answers written by the same user and split them into paragraphs, removing paragraphs with fewer than 100 characters. This yielded a list of paragraphs for every user on a particular site. In a next step, we constructed the documents making up the dataset. Each dataset contains an equal number of singleauthor and multi-author documents. For single-author documents, we selected a random user and drew paragraphs from the paragraph list of that user until the document had a sufficient length (randomly chosen to be between 1,000 and 3,000 words). For multiauthor documents, we first randomly chose whether the document should have two or three authors. Then, we randomly constructed a structure for the document (i.e., a sequence of author changes for a set of paragraphs). Based on that, we randomly chose distinct authors from our list of users and drew paragraphs from their paragraph lists until the document had the predetermined structure and the chosen length (again, randomly chosen to be between 1,000 and 3,000 words).
The resulting documents were then split into training, validation, and test sets with approximately 50% of the documents being assigned to the training set, and 25% each being assigned to the validation and test sets. The procedure we used for splitting ensured that every subset contains approximately the same number of single-author and multi-author documents. Finally, we filtered all documents based on their language. As 1See https://archive.org/details/stackexchange 2dataset-narrow contains questions and answers from the following sites: Code Review, Computer Graphics, CS Educators, CS Theory, Data Science, DBA, DevOps, Game Dev, Network Engineering, Raspberry Pi, Serverfault.com, Superuser.com. 3dataset-wide contains questions and answers from the following sites: Academia, Astronomy, Bicycles, Biology, Buddhism, Code Review, Coffee, DBA, Earth Science, Economics, Engineering, Fitness, History, Interpersonal, Linguistics, Literature, Mathoverflow.net, Outdoors, Philosophy, Serverfault.com, Skeptics, Sports, Travel, Workplace, Worldbuilding. we want our datasets to consist only of English documents, we removed all documents where at least one paragraph was identified as being written in a language other than English. For this, we used the Python library langdetect.4 A summary of the parameters for both datasets is given in Table 1. Table 2 shows an overview of the created datasets, including the number of contained documents as well as the average document lengths, partitioned by the number of authors.
For development, participants are provided with the documents and ground truth information. For each training and validation document, we provided the number of authors, the StackExchange site the texts were gathered from, the order of the authors within the document, the positions of the style changes, and whether the document was indeed multi-authored. 3.3
To evaluate and compare the submitted approaches, we report both, the achieved performance for the individual subtasks, and their combination as a staged task. Furthermore, 4https://pypi.org/project/langdetect/ we evaluate the approaches on both datasets individually. Submissions are evaluated using the F -Measure for each document, where = 1 equally weighs the harmonic mean between precision and recall. For Task 1, we compute the average F1 measure across all documents, and for Task 2, we use the micro-averaged F1 measure across all documents. The submissions for the two datasets are evaluated independently and the resulting F1 measures for the two tasks are averaged across datasets. 3.4
To ensure the reproducibility of the submitted solutions, participants were asked to
deploy their software on our TIRA platform [
For this year’s edition of the style change detection task, we received three submissions. However, only two participating teams submitted a working notes paper. In the following, we describe the approaches used in those submissions. 4.1
The approach developed by Castro-Castro et al. [
Based on this, B0-maximal clustering is performed to group the paragraphs in a
document into clusters, where every cluster is regarded to be one author. This clustering
approach assigns all paragraphs with a similarity larger than a defined threshold to the
same cluster. Since this makes it possible for a paragraph to be assigned to multiple
clusters, and hence to multiple authors, a basic approach for deciding to which cluster
a paragraph will be assigned is proposed. In such a case, from all possible candidate
clusters, the one which contains the paragraph that occurs earliest in the document is
chosen. Thus, all the paragraphs in a document are assigned to authors. From this, the
tasks posed in this year’s style change detection task are solved as follows: For the first
task, it is simply checked whether the clustering has produced more than one cluster.
For the second task, the positions in the document are identified, where consecutive
paragraphs were assigned to different clusters.
The approach of Iyer and Vosoughi [
The results for the participants submissions as well as a random baseline are given in Table 3. The table shows the F1 score for both tasks, as well as the overall average scores. Both participants’ submissions significantly outperformed the baseline with respect to individual and overall score. The best-performing submission is the one by Iyer and Vosoughi, which achieved the best scores in both tasks as well as in the overall average score. The approach developed by Castro-Castro et al. performs significantly better than the random baseline, but also significantly worse than the approach by Iyer and Vosoughi, forming a middle ground. The approach of Nath5 performs only slightly worse than that of Castro-Castro et al. 5The participant did not submit their working notes and was hence omitted from further analysis. 1.0 0.8 re0.6 o c S F10.4 0.2 0.0 (d) Topic Diversity
Iyer and Vosoughi Castro-Castro et al.
(a) Single-author Documents (b) Multi-author Documents
Participant Castro-Castro et al.
Iyer and Vosoughi
We further analyzed the performance of both approaches with regard to the specific properties of the documents in our datasets. First, compared both approaches with respect to single-author versus multi-author documents. The results for this analysis are shown in Figures 2a and b. The approach by Iyer and Vosoughi reaches an F1 score of almost 1:0 for Task 2 on single-author documents. This suggests that it may be beneficial for them to reduce their approach to one model predicting style changes between paragraphs, and then calculating predictions for Task 1 based on the output of that model (i.e., predicting a document to be multi-author if and only if there was at least one style change predicted between the paragraphs of that document). Another point to note is that the approach by Castro-Castro et al. performed best for Task 1 on multiauthor documents. This suggests that their model is especially well-suited for detecting documents that have been written by more than one author. Moreover, we analyzed how the performance for both submitted approaches changes depending on the number of authors. The results for this analysis are shown in Figure 2c, confirming that the approach of Castro-Castro et al. performs better for multi-author documents, regardless of whether the number of authors is two or three. It is interesting that Castro-Castro et al.’s approach improves for multi-author documents, whereas that of Iyer and Vosoughi performs best for single-author documents, exerting a sharp drop in performance when a document is written by multiple authors.
Finally, we analyzed the performance of the participants’ approaches dependent on topic diversity (see Figure 2d). In most cases, we found a significant difference in performance between both datasets. The exception to this is the approach by Castro-Castro et al. on Task 1, where the performance on the narrow and wide datasets are almost identical. In all other cases, the performance differs significantly, with performance on the narrow dataset being higher than on the wide dataset, implying that dealing with documents of a diverse topical variety renders the task more difficult.
In the 2020 edition of the PAN style change detection task, we asked participants to answer the following questions for a given document: (1) Given a document, was it written by multiple authors? (2) For each pair of consecutive paragraphs in a given document, is there a style change between these paragraphs? Three participants submitted their systems and two participants submitted a working notes paper. The two approaches differed fundamentally, the best-performing system relying on semantic features (i.e., BERT embeddings), while the second-best approach focused on syntactic ones. Future challenges include finding the exact position of authorship changes beyond the paragraph level, and assigning paragraphs to individual authors.