Wikipedia is an online encyclopedia that anyone can edit. In this open model, some people edits with the intent of harming the integrity of Wikipedia. This is known as vandalism. We extend the framework presented in (Potthast, Stein, and Gerling, 2008) for Wikipedia vandalism detection. In this approach, several vandalism indicating features are extracted from edits in a vandalism corpus and are fed to a supervised learning algorithm. The best performing classifiers were LogitBoost and Random Forest. Our classifier, a Random Forest, obtained an AUC of 0.92236, ranking in the first place of the PAN'10 Wikipedia vandalism detection task.
[...] any addition, removal, or change of content made in a deliberate attempt to compromise the integrity of Wikipedia. [...] Common types of vandalism are the addition of obscenities or crude humor, page blanking, and the insertion of nonsense into articles.
Any good-faith effort to improve the encyclopedia, even if misguided or illconsidered, is not vandalism. Even harmful edits that are not explicitly made in bad faith are not vandalism. (Wikipedia contributors, 2010a)
The Wikipedia community maintains a project for studying vandalism
Thanks to the PAN’10 organizers, to the reviewers, to Dmitry Chichkov for his contributions
to the PAN’10 group and to Sandra García Blasco for her support and ideas.
the most prominent active bot in the English Wikipedia is ClueBot, a rule-based
system.
We have used the PAN-WVC-10 corpus
The rest of the paper is structured as follows. In Section 2 of this article, we will describe the preprocessing we have applied to the data. In Section 3 we discuss all edit features that we have used. In Section 4 we present the classifiers we have used. In Section 5 we will evaluate our features and classifiers and present the classifier we have used for the PAN’10 result submission. Finally, we will present our conclusions in Section 6. 2
In our feature framework, we have defined different representations of edits to use in different features. In this section we will describe these representations and the steps to construct them.
First, we defined a tokenization function. A token can be a word, a number, a punctuation mark or a wiki-syntax element. When describing features, we will consider a word as any of these tokens. The following symbols are considered independent tokens: ., ,, :, ;, ", «, », ’, |, ?, !, =, (, ), *, [[, ]], [. ], {{, }}, { and }.
We also need to produce diffs between the old and new revisions of an edit. To that end, we used google-diff-match-patch.1
Using these funcions, we define the following representations of edit text: – Old text and new text. The old and new without any preprocess. – Case-sensitive inserted words. The set of inserted words. – Inserted words. The set of inserted words, and converted to lowercase. – Concatenated inserted words. All inserted words concatenated and separated with spaces. This is defined both case-sensitive and insensitive. – Inserted text. Inserted lines as reported by the diff algorithm.
1Available at google-diff-match-patch. Behaviour of different parameters of the diff function has not been studied yet. The parameters used were Patch_Margin set to 0, Diff_EditCost to 6 and Match_Distance to 500.
In this section, we describe our set of features. Features based on dictionaries will be
explained separately. Features marked with were already defined in
Anonymous Wether the editor is anonymous or not.
Vandals are likely to be anonymous. This feature is used in a way or another in
most antivandalism working bots such as ClueBot and AVBOT. In the
PAN-WVC10 training set
Comment length Length in characters of the edit summary.
Long comments might indicate regular editing and short or blank ones might suggest vandalism, however, this feature is quite weak, since leaving an empty comment in regular editing is a common practice.
Upper to lower ratioy Uppercase to lowercase letters ratio, i.e., 1+jupperj . 1+jlowerj Vandals often do not follow capitalization rules, writing everything in lowercase or in uppercase. 1+jupperj Upper to all ratioy Uppercase letters to all letters to ratio, i.e., 1+jlowerj+jupperj . Digit ratio Digit to all characters ratio, i.e., 11++jdjaigllijtj .
This feature helps to spot minor edits that only change numbers, which might help to find some cases of subtle vandalism where the vandal changes arbitrarily a date or a number to introduce misinformation.
Non-alphanumeric ratio Non-alphanumeric to all characters ratio, i.e., 1+jnonalphanumericj . 1+jallj An excess of non-alphanumeric characters in short texts might indicate excessive use of exclamation marks or emoticons.
Character diversity Measure of different characters compared to the length of inserted text, given by the expression length differe1nt chars .
This feature helps to spot random keyboard hits and other non-sense. It should take into account QWERTY keyboard layout in the future.
Character distributiony Kullback-Leibler divergence of the character distribution of the inserted text with respect the expectation. Useful to detect non-sense.
Compressibilityy Compression rate of inserted text using the LZW algorithm. 2
Useful to detect non-sense, repetitions of the same character or words, etc.
Size increment Absolute increment of size, i.e., jnewj joldj.
The value of this feature is already well-established. ClueBot uses various thresholds of size increment for its heuristics, e.g., a big size decrement is considered an indicator of blanking.
Size ratio Size of the new revision relative to the old revision, i.e., 11++jjnoeldwjj .
Complements size increment.
2LZW was chosen after evaluating the behaviour of LZW, gzip and bzip2, although, an
exhaustive comparison is still pending. We used the TIFF LZW algorithm
Average term frequency Average relative frequency of inserted words in the new revision.
In long and well-established articles too many words that do not appear in the rest of the article indicates that the edit might be including non-sense or non-related content.
Longest word Length of the longest word in inserted text.
Useful to detect non-sense.
Longest character sequence Longest consecutive sequence of the same character in the inserted text.
Long sequences of the same character are frequent in vandalism (e.g. aaggggghhhhhhh!!!!!, soooooo huge).
Size increment and ratio are closely related to all features, as can be seen in ClueBot and AVBOT source code, where thresholds for these measures are coded deep into most heuristics. This should be considered when building a classifier.
Upper to all, upper to lower, digit and non-alphanumeric ratios were computed using case-sensitive concatenated inserted words. These could also be computed using a diff or computing the difference of their values in the old and new text. Each method produces different results. We chose only inserted words for efficiency.
Antivandalism bots use lists of regular expressions with weights in order to
compute a vandalism score. In
We have defined features analogous to vulgarism frequency and vulgarism impact, for different categories of words: Vulgarisms Vulgar and offensive words, e.g., fuck, suck, stupid.
Pronouns First and second person pronouns, including slang spellings, e.g., I, you, ya. Biased Colloquial words with high bias, e.g., coolest, huge.
Sex Non-vulgar sex-related words, e.g., sex, penis, nipple.
Bad Hodgepodge category for colloquial contractions (e.g. wanna, gotcha), typos (e.g.
dosent), etc.
All A meta-category, containing vulgarisms, pronouns, biased, sex-related and bad words.
Good Words rarely used by vandals, mainly wiki-syntax elements (e.g. __TOC__, <ref>)
Word counters were computed using the inserted words in lowercase. In future work, it would be desirable to consider caseness and style. For example, dick is very likely to be slang for penis and DICK is a strong indicator of vandalism, but Dick is more likely to be the diminutive for Richard or a surname. Our current system does not make this distinction.
Classification has been conducted using the Weka framework
Our baseline classifier is C4.5 decision tree
We also chose Support Vector Machines (SVM) because of its resistance to class
imbalance
The following measures have been evaluated in every experiment: Precision, Recall, F-Measure and Area Under ROC Curve (AUC).
In Table 1 we can see the performance of a C4.5 decision tree built for each feature. The C4.5 algorithm, considering only one feature and with severe class imbalance, will often produce a majority class classifier. In order to prevent this, we changed the weight of vandalism edits so the cost of a misclassified vandalism edit to be 10 times higher than a misclassified regular edit. This is meant to be orientative, not an accurate ranking of features.
In Table 2 we can see the performance of all the classifiers considering all features. Every classifier has been evaluated with 10-fold cross-validation. Parameters used for classifiers were: 3 C4.5 minimum number of instances per leaf is 6, Laplace smoothing is used for predicted probabilites.
LogitBoost 100, 500 and 1000 boost iterations. The weak classifier is a Decision
Stump.
Random Forest 100, 500 and 1000 iterations.
LogitBoost and Random Forest perform similarly in this task. Random Forest improves F-Measure and AUC as we increase the number of iterations. While LogitBoost shows slightly better performance than Random Forest, its results are less stable and 3Any omitted parameter uses default parameters in Weka. For more information, check Weka API documentation at http://weka.sourceforge.net/doc.stable/. decrease while increasing number iterations, suggesting a problem of overfitting. This is, presumably, because of LogitBoost’s sensitiveness to noise and outliers.
Our final classifier was a Random Forest of 1000 trees, each one built while considering 5 random features4, obtaining an AUC of 0.92236 when evaluated with the PAN-WVC-10 testing corpus. 6
In this work, we revisited
Our final classifier, a Random Forest, obtained an AUC of 0.92236, which was ranked in the first place in PAN’10 for the vandalism detection task. Precision and recall on the training test were 0.861 and 0.568, respectively. This is far better than current antivandalism systems in terms of F-Measure, but it has neither precision near to 1, which is required for a bot in order to operate autonomously, nor a recall high enough to be used as a filtering system for human editors. In any case, our results suggest that this machine learning based approach is a suitable path for the next-generation antivandalism bots.
Future work might include a more exhaustive feature evaluation; building bigger word lists using free lexical resources such as Wiktionary and WordNet and infer their weights; introducing a feature for detecting random keyboard hits considering QWERTY keyboard layout; introducing heuristics from current antivandalism bots and experiment with measures based on ngram language models.
4That is, log2 M + 1, where M is the number of features.