Our approach to the task of intrinsic plagiarism detection uses a vectorspace model which eschews surface features in favor of richer extrinsic features, including those based on latent semantic analysis in a larger external corpus. We posit that the popularity and success of surface n-gram features is mostly due to the topic-biased nature of current artificial evaluations, a problem which unfortunately extends to the present PAN evaluation. One interesting of aspect of our approach is our way of dealing with small, imbalanced span sizes; we improved performance considerably in our development evaluation by countering these effect using the expected difference of sums of random variables.
The task of intrinsic plagiarism detection involves distinguishing portions of a single
text which are written by different authors [24]. Key characteristics of the task are the
lack of texts written purely by one author or another, as is typically the case in
authorship attribution [23], and lack of a database of texts from which the texts are formed,
which is the focus of extrinsic plagiarism detection [
Relatively successful approaches to intrinsic plagiarism detection [
The set of features that we explore for this task falls roughly into two categories: surface and extrinsic. The distinction is not entirely clear cut, but we wish to distinguish features that use the basic properties of the words or their PoS, which have traditionally been the focus of automated stylistic analysis, from features which rely heavily on external lexical information such as word sentiment and, in particular, vector-space representations, which are more novel for this task. 2.1
Word length Word length is a common textual statistic in register and readability
studies. Readability, in turn, has been used for plagiarism detection [24], and related metrics
were consistently among the best performing features for Guthrie [
Punctuation frequency The presence or absence of punctuation such as commas, colons, semicolons, and quotes can be very good indicator of style. We also include periods, which offer a measure of sentence length.
Parts of speech Lexical categories can indicate, for instance, the degree of
nominalization, which is a key stylistic variable [
Pronouns We count the frequency of first-, second-, and third-person pronouns, which
can indicate the interactiveness and narrative character of a text [
Verb tense Past tense is often preferred in narratives, whereas present tense can give a sense of immediacy.
Type-token ratio A standard measure of lexical diversity.
Lexical density Lexical density is the ratio of the count of tokens of the four substantive parts of speech to the count of all tokens.
Contextuality measure The contextuality measure of Heylighen and Dewaele [
For those lexicons which include only lemmatized forms, the words are lemmatized before their values are retrieved.
Presence in Dale-Chall Word List A list of 3000 basic words that is used in the
DaleChall Readability metric [
Unigram count in 1T Corpus Another metric of whether a word is commonly used. We
use the unigram counts in the 1T 5-gram Corpus [
Sentiment polarity The positive or negative stance of a span could be viewed as a
stylistic variable. We use a hand-built lexicon for the SO-CAL sentiment analysis system,
which has shown superior performance in lexicon-based sentiment analysis [25].
Sentiment extremity Some sentiment lexicons provide a measure of the degree to which
a word is positive or negative. Instead of summing sentiment scores, we sum their
absolute values, to get a measure of how extreme (subjective) the span is. In our previous
work[
Formality Formality score uses a lexicon of formality that we created in previous work
[
LSA vector features In Brooke et al. [
Normalization For all the features, we also normalize at the level of the word, and
then sum across the span; for instance, to calculate a noun frequency metric, we first
assign each noun in the text a 1, and all other tokens a 0, normalize this distribution to a
mean of 0 and a standard deviation of 1, and then average the normalized values across
the span that we’re interested in. When applying unsupervised techniques with features
of vastly different ranges, normalization is a very important step [
Our general approach to both of the paragraph-clustering subtasks of the intrinsic
plagiarism detection task is to assign paragraphs into author groups that maximize the
(average) distance between authors. Following Guthrie’s work in stylistic outlier
detection [
However, there is a serious flaw in this kind of approach: all other things being equal, shorter spans have more random variation and thus are, on average, more distant (sometimes much more distant) from any given span than a longer, more homogeneous span. Fortunately, this effect can be modeled. We did this by calculating the expected distance of sums of random variables. Supposing a span of some basic length (we used 50 tokens) to have a random component of normal distribution (with a mean and standard deviation of 1), we can estimate the expected influence of this randomness on the distance measure between any pair of spans — for instance, spans of length 400 and 100 — by looking at the sum of random variables corresponding to the n basic spans that make it up — in this example, the expected difference between the sum of 8 random variables and the sum of 2 random variables with the same distribution. We ran 100 trials using a random number generator and computed a table of such expected differences, and then divided our calculated distance by the corresponding number in the table to get a new distance that takes expected difference into account.2 4
Even before we reached the final version described above, our approach had perfect performance on the two example texts provided by the PAN organizers, so we created some additional corpora for testing, collecting a few different types of texts (early modern novels, translated Russian novels, and political treatises) from Project Gutenberg, and automatically creating mixed texts of various difficulty. Here, we present results using two relatively easy corpora which consist of texts of paragraphs randomly pulled from novels by Fyodor Dostoyevsky (in English translation) and Thomas Hardy, and political texts by Thomas Paine and Jean-Jacques Rousseau. For each text in the corpus for the mixed-author task, we first choose the number of authors (between 2 and 4), then randomly selected the authors, the number of paragraphs (between 10 and 30) and then the paragraphs themselves from random locations in the text. For the two-author insertion task, we randomly choose two authors, a total number of paragraphs, and two non-equal indices within that range; for each author, a random starting location was randomly selected and consecutive paragraphs from the first author were randomly selected for paragraphs before the first index, and then after the second, and consecutive paragraphs from the second author were inserted between the two indices. Both corpora have 30 texts created in this fashion.
There are various metrics for extrinsic cluster evaluation; Amigó et al. [
We compare our algorithms with task-specific random baselines (with 50 trials) and two related alternatives: one which excludes our expected difference corrector and another that is based purely on the maximizing distances between individual paragraphs in the spans, rather than treating each cluster as a whole. The results for each of the two tasks are in Table 1.
There is little doubt that our expected difference adjustment has an overall positive effect, and, in the multi-author task, this provides the best result by a reasonably large 2 There may be a closed-form solution to this problem, but in our case it was easier to derive it empirically. margin. For the insertion task, which has a much higher random baseline, the individual span distance comparison was found to be roughly equivalent to our combined span approach with the adjustment. 5
Our linguistically motivated, vector-space clustering approach shows promise, particularly with our expected difference adjustment. There is, however, obviously more work to do in this regard; for instance, using this adjustment our multi-author method never, in practice, predicts more than two authors, probably because the differences between short spans are now being underestimated rather than overestimated, meaning that two relatively short author spans (e.g. 3rd and 4th authors) are now highly dispreferred under our distance-maximizing algorithm. This may partially explain our relatively poor performance on the multi-author intrinsic plagiarism task, but in fact there is a more obvious reason. For instance, here are two paragraphs from different authors in the multi-author task evaluation data (text1):
John did not dream about the deli. He had nightmares of Douglas falling onto swords of knights on horseback, and woke several times throughout the night sweating and breathing heavily.
But in an empty house, surrounded by evidence of Caroline’s long absence, Hillie’s words plagued him, and he was forced to accept that his mind might be capable of the cruelest of tricks. He felt desperately, hopelessly alone. Stylistically, we find the two authors nearly indistinguishable. There are small differences (the second author prefers longer sentences and hyperbole), but the easiest way, for either human or computer, to identify the two is by the names of the characters. All but two of the paragraphs contains a proper name that appears in several other paragraphs (one author talks about Geoff, Hillie, and Caroline, the other about Douglas, John, and Mrs. Cumberland). Beyond proper names, there are also recurring topics: mail in one story, a job at a deli in another.3 Any model that uses word or character 3 There are even two sets of repeated paragraphs in this particular text: paragraphs 14 and 21 are the same, as are paragraphs 24 and 30! n-grams should be able to take easy advantage of these regularities. Our model, conversely, was specifically designed not to do so; rather, it was developed to detect significant stylistic differences. In fact, in the task evaluation data there are quite clearly more obvious stylistic differences between some excerpts from the same novel than between some excerpts from different novels:
On his departure, Hillie had pressed her business card into his hand. “My number’s on there,” she told him. “Call me, all right? I want you to promise.” “I’m sorry,” Geoff said. “I don’t see the point.” “You’ve suffered a shock. You can’t be expected to cope at home on your own.” Geoff had simply smiled at her. “I won’t be on my own,” he said. “I keep telling you. Caroline will look after me.” Because of the presence of dialogue, this passage is radically different, stylistically, from the other passage from the same novel that was shown above. However, it is clear which of the two novels it comes from, since there are several proper-noun indicators. Given the range of subgenres within the novel genre, i.e. narration, description, and dialogue, this genre is a particularly bad choice for the purposes of simulating intrinsic plagiarism, since those stylistic features which exist and might be useful for distinguishing authors will be ultimately be drowned out by this confounding variation. Instead, topic-related features, which would be highly unreliable in the real world (for reasons that are obvious; what is the purpose of a student plagiarizing something that is topically distinct from the matrix text in which it is embedded?), are strongly preferred.
Thus, we argue that for the PAN evaluation to be a useful reflection of the realworld task of intrinsic plagiarism detection, it requires data that are well-controlled for topic. This might include changing proper nouns (and other highly topical elements) in the data so that they are matched across authors. Insofar as clustering by authorship (as opposed to merely detecting intrusions by one or more authors into a matrix text) is taken as an interesting research problem that mirrors aspects of intrinsic plagiarism detection, the evaluation needs to be constructed so that the focal task is not confounded by orthogonal issues such as subgenre detection. 23. Stamatatos, E.: A survey of modern authorship attribution methods. Journal of the American Society for Information Science and Technology 60(3), 538–556 (2009), http://dx.doi.org/10.1002/asi.21001 24. Stein, B., Lipka, N., Prettenhofer, P.: Intrinsic plagiarism analysis. Language Resources and
Evaluation 45(1), 63–82 (2011) 25. Taboada, M., Brooke, J., Tofiloski, M., Voll, K., Stede, M.: Lexicon-based methods for sentiment analysis. Computational Linguistics 37(2), 267–307 (2011) 26. Turney, P., Littman, M.: Measuring praise and criticism: Inference of semantic orientation from association. ACM Transactions on Information Systems 21, 315–346 (2003)