Tasks such as Authorship Attribution, Intrinsic Plagiarism detection and Sexual Predator Identification are representative of attempts to deceive. In the first two, authors try to convince others that the presented work is theirs, and in the third there is an attempt to convince readers to take actions based on false beliefs or ill-perceived risks. In this paper, we discuss our approaches to these tasks in the Author Identification track at PAN2012, which represents our first proper attempt at any of them. Our initial intention was to determine whether cues of deception, documented in the literature, might be relevant to such tasks. However, it quickly became apparent that such cues would not be readily useful, and we discuss the results achieved using some simple but relatively novel approaches: for the Traditional Authorship Attribution task, we show how a mean-variance framework using just 10 stopwords detects 42.8% and could be obtain 52.12% using fewer; for Intrinsic Plagiarism Detection, frequent words achieved 91.1% overall; and for Sexual Predator Identification, we used just a few features covering requests for personal information, with mixed results.
The PAN activity has been around since the 2007 International Workshop on Plagiarism Analysis, Authorship Identification, and Near-Duplicate Detection, and has subsequently evolved to become Uncovering Plagiarism, Authorship, and Social Software Misuse1. The first competitive PAN activity in 2009 had two parts, an external task of checking document content against a collection, which largely remains as the Plagiarism Detection task though with differences in approach between 2009 and 2012, and an intrinsic component apparently looking at writing style changes within a document. This intrinsic component is now just one small part of the Authorship Identification track (Tasks E and F), which also includes Traditional Authorship Attribution (Task A, B, C, D, I, J) and Sexual Predator identification. Authorship Attribution requires identifying, based on a sample of given texts, a likely
author – in essence, identifying the closest match to other texts, as author names need not be given. Intrinsic Plagiarism detection involves the separation of text fragments from a single document where fragments are combined from two or more authors. For Sexual Predators, the result comes from a binary classification in which the data of interest relate to those conversations where some are attempting to deceive underage children (mainly) to perform actions of an immoral and potentially illegal nature.
In this paper, we outline the approach taken at the University of Surrey to these quite varied tasks for PAN2012. In section 2, we discuss why current deception detection cues seem to be unsuited for these tasks, which leads us to develop our own approaches. Sections 3, 4 and 5 focus on each of the individual tasks and the results obtained using relatively few features in most cases. Section 6 concludes the paper with considerations for future work. 2
In Vartapetiance and Gillam [1], we discussed why current systems and approaches
for deception detection might not be effective in a variety of deceptive situations, one
reason being a lack of common data sets upon which to experiment. However, as far
as we can tell such approaches had not been explored systematically for PAN. We can
readily consider attempts at plagiarism (or copyright infringement) to be deceptive
acts, and certainly Sexual Predation would appear to be an attempt to deceive. Hence,
the treatment of deception would seem to be relevant to Authorship Identification and
vice versa. However, most of the cues for deception (e.g. in DePaulo et al. [2]) are
based on non-verbal behaviour (visual and vocal), so are immediately not fit for such
purposes. There are then different sets of verbal cues defined by various research
groups, though most are covered by three major categories: (
Consider, for example, Pennebaker’s work, which has been widely used (e.g.
[46]), it is suggested that deceptive text will have (
Following various apparently unsuccessful attempts to make use of such cues, we considered whether such kinds of deception may not be suited to detection using these cues, and looked instead at what we might obtain first from simple features across the data. 3
Much literature discusses the use of numerous NLP techniques that operate over bags of words, N-grams, and parts of speech (POS), with varying degrees of success. In many cases, stopwords are either an integral part of the analysis, without consideration for how much they drive the analysis, or are dropped from processing. Prior research in this task does not appear to have addressed whether authors’ writing styles and preferred topics lead to distinctive positional preferences for stopwords.
Church & Hanks [7] describe a mean-variance framework for detecting strong associations between co-occurring words and being able to distinguish amongst patterns using this. Their examples are of fixed phrases such as “bread and butter”, which demonstrate a clear preference over “butter and bread”. As an indicator of style, we explored grammatical preference using a mean-variance framework with just 10 stopwords.
4.1
Step 1 Select the 10 most frequent words from the Oxford English Dictionary: the, be, to, of, and, a, in, that, have, I Step 2 Generate regular expressions for all pairs of these words, e.g. the+have, have+the, and use a specific size of window N (here, 5).
Step 3 Extract concordances containing the regular expressions for all author texts Step 4 Calculate per-author frequency, mean and variance information for the pairs. Step 5 Calculate the frequency, mean and variance for the test data (per document) in the same way.
Step 6 Select the author with closest match values
An example of the values derived for three authors is shown below in Table 1, against text 12Atest01. The selected author is, in essence, decided on by the number of votes cast by matches to frequency, mean and variance as shown in Table 2.
For the open dataset, to account for data not belonging to any of the authors in the training set, we used a simple confidence measure: if the count difference between the 1st and 2nd highest values is less than 5, it is reported that there is no author. Table 3 shows an example where matches have been made to different authors but with insufficient confidence (difference = 3).
Results from PAN2012 show that this method achieves 41%, flagging 28 out of 71
documents correctly. In post-competition analysis, we investigated effects of
changing the gaps size (5, 10 and 25), changing the confidence (
Depending on the approach, Intrinsic Plagiarism Detection might be categorized as
Authorship Attribution - it can be related to identifying parts of a text least likely to
have been produced by the current author. However, there are various differences: for
Authorship Attribution, (
Step 1 Select the 50 most frequent words from the file, after removing stopwords. Step 2 Determine frequency by paragraph for these 50 words Step 3 Select (sequences of) paragraphs with fewer similarities (e.g. < 10)
If there is more than one sequence: Step 3a Select the longest sequences of paragraphs which do not share the most frequent word, and have the lowest average frequency for top 5 of these 50 words
Table 5 shows paragraphs flagged by total frequency below 10. These sequences’ average frequency for the top 5 shows that the [P4, P5, P6] sequence is least relevant to the file.
For 12Ftest02, steps 1-3 identify P01, P04 and [P06, P07, P08]. Calculating with the 5 most frequent words suggested [P06, P07, P08] to be the sequence. However, all similarly shared the most frequent words suggesting that they are all related to the topic. We allocated “no author” to that file, even though it was not suggested in the competition that the dataset could have open answers.
P04 P07 P08 P12 P13 P15 P17 P21 P29 A2 NA NA NA NA NA NA NA NA NA
3 1 1 1 1 1 1 1 1 NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA A2 A2 A2 A2 A2 A2 A2 A2 A2
The approach to Task F would not distinguish as readily. We adapted this for task E as follows: Step 1 As task F Step 2 As task F Step 3 Extract the nouns from the 50 most frequent words (excluding stopwords) Step 4 For the highest frequency noun, create a cluster and remove from consideration all other nouns enclosed by this – i.e. occurring in the same paragraphs. Repeat this step to produce new clusters from the remaining nouns.
Where paragraphs are not allocated to a cluster: If the number of consecutive unallocated paragraphs is greater than 5, these form a new cluster. For others: (a) paragraphs between two in the same cluster are allocated to that cluster; (b) paragraphs between different clusters are allocated to the subsequent cluster.
A sample of the process and the results from one of the test sets in presented in Table 6, with errors highlighted with gray.
To validate our reasoning behind not using the Task F approach, we used the 5 most frequent words (from task F) and used Step 4 of task E to cluster them into groups. Interestingly, for 12Etest01 this only mis-classified P19 as Author 2, but we have mis-classified 2 for our final submission. However, such a gain would have come at the cost of losses elsewhere.
Competition results show these approaches gave 100% (Task F) and 82.2% (Task E) accuracy, a simple average of which would make us 2nd in just this task (91.1% against 94.2%). We are now looking further at what might have improved methods to improved performance in Task E without introducing complexity.
Just like other sub-tasks in this paper, Sexual Predator actions carry a level of deception which may or may not lie in the text but with the actions followed. Since we have never attempted the analysis of such a corpus or topic previously, we have taken relatively straightforward approach, and with reference to the training corpus this appears to offer good performance (up to f1=0.66), but for which we would have concern over the rate of false negatives as we discuss later.
As it was the first time we have attempted such a task, we randomly took 10 predators IDs from the training set to set about manually discovering patterns. We found obvious similarities, and classified these as described below (examples are shown below in Table 7):
Address: asking for the address of the house or somewhere close to drive to in other to meet up. Mostly, it’s the predator who asks the question about the child’s address and it was rare for children to ask whether the predator would need/like their address. This alone detects 58 out of 142 in the training data, appearing more than once, and 28 times it appears twice or more with very high precision (85%).
Parents: another strong feature. Questions about parents are usually because of: Secrecy
To ascertain how long they would be gone for
This feature detects 84 out of 142, when appearing once or more, and 49 when appearing twice or more. Combining “address” and “parents” would detect 105 and 74 respectively.
Age: Some predators might lie about their age but most seem quite open about their age. They would usually highlight the fact that they are older, wishing the child were older, and this is mainly to retain the secrecy.
Intention: Interestingly, not many of these chats have direct references to their sexual intentions. They usually focus on the concept of meeting up and having fun time, watching TV, listen to music and have some alcohol. In some cases there may be mentions of what they would like to do – these are mostly limited to cuddles and kisses and so sexual activities are more apparent.
We tested all of the above mentioned categories individually, varying the number of occurrences, and in various combinations, with the PAN2012 training data. Our analysis of these results is shown in Table 8.
For the competition, we used the combination of all four categories that occurred twice or more, as this offered the optimal f1 score on training data (precision=0.7, recall=0.62 and F1=0.66).
However, it can easily be argued that for real detection the false negatives would be of particular concern. Let us consider an application that might filter out possible predatory conversations. What characteristics would a parent rather have: stringent filtering of suspicious behaviour but with a high false positive rate so that some genuine conversations are dropped, or conversations with real predators that may remain and appear acceptable? The competition webpage suggests that the decision on efforts put on investigation are the dominion of the system “to optimize the time of a police agent towards the "right" suspect rather than "all" the possible suspects”. This presents a disturbing view of such a system, and we would strongly contend that the system should produce results ranked according to confidence, but resourcing judgements should be left to the conscience of human beings who are fully aware of the consequences of such missed results. Recall, then, should be a higher priority – F2, not F0.5 – even if this would reduce our placing in this task.
To see if we could have improved our results, post submission but before results, we also tested the combination of best f1 scores of all categories on the training dataset. The result for “one occurrence” across all 4 classes increased from 0.42 to 0.58 because Parent was already based on two occurrences, but the two and three occurrence scores decrease respectively to 0.6 and 0.48. Currently we are looking at other ways to increase the detection rate while keeping the simplicity of the method. We have already found a few words that can improve the detection, especially in case of sexual comments related to “intentions”.
Checking the ground truth of second section raised some questions for us based on the number of lines selected and the content. For example, lines such as: 0fe0367fc3735101fbf7aa3df1cb9f4e 6bf9b33a9f4ae1df54cb89831eac1be2 94c71d9e905c390d310f3f315f9c7b19 94c71d9e905c390d310f3f315f9c7b19 37 what grade u in 5 :) 41 i promise 45 age 7
We attempted, for the first time, the Author Identification track at PAN2012, participating in all three tasks of Authorship Attribution, Intrinsic Plagiarism detection and Sexual Predator Identification. We attempted to use fairly simple approaches in each case to determine the extent to which these might be effective, and believe there is some degree of novelty presented in each approach. It was surprising, for example, that just 5 stopwords with a mean-variance framework might be able to produce reasonable performance in Authorship Attribution, given that it is essentially a stylometric approach.
Our initial intention was to determine how the supposed cues of deception might be useful against benchmark data collections, but we have seen little indication of relevance to these kinds of deception. What we have learnt from participating in these tasks can now be applied back to reported deception experiments elsewhere to see whether deception is indicated by features which are not usually in the set selected by the researchers – positive evidence being so much easier to discern.
Our best results appear to have been obtained against Intrinsic Plagiarism, which was very much an 11th hour effort. However, plenty of room for improvement remains against the other tasks, and the generalisability of our approaches can now be evaluated across previous PAN datasets also.