In this paper, we present our research on the problem of ensuring the integrity of Wikipedia, the world's biggest free encyclopedia. As anyone can edit Wikipedia, many malicious users take advantage of this situation to make edits that compromise pages' content quality. Speci cally, we present DePP, the state-of-the-art tool that detects article pages to protect with an accuracy of 93% and we introduce our research on identifying spam users. We show that we are able to classify spammers from benign users with 80.8% of accuracy and 0.88 mean average precision.
Wikipedia is the world's biggest free encyclopedia read by many users every day. Thanks to the mechanism by which anyone can edit, its content grows and is kept constantly updated. However, malicious users can take advantage of this open editing mechanism to seriously compromise the quality of Wikipedia articles.
The main form of content damaging is vandalism, de ned by Wikipedia itself
as \any addition, removal, or change of content, in a deliberate attempt to
compromise the integrity of Wikipedia"[
In this paper, we discuss two research e orts which have the common goal of ensuring the content integrity of Wikipedia.
First, we introduce DePP, the state-of-the-art tool detecting article pages
to protect [
Second, we present our work on spam users identi cation [
Detecting damaging edits. Plenty of work has been done on detecting
damaging edits, particularly vandalism (see [
Recently, Wikimedia Foundation launched a new machine learning-based
service, called Objective Revision Evaluation Service (ORES) [
Regarding spam edits detection speci cally, previous work concentrated on
the problem of predicting whether a link contained in an edit is spam or not,
whereas, in this paper, we predict whether a user is a spammer or not by
considering her edit history. [
Page protection. When a page article is heavily vandalized, administrators may decide to protect the page by restricting its access. To the best of our knowledge, little research has been done on the topic of page protection in Wikipedia.
2 Dataset and code are available at http://www.cs.umd.edu/~vs/vews/
Hill and Shaw [
The rst problem we address consists in deciding whether or not a page should
be protected by Wikipedia administrators. Page protection consists in placing
restrictions on the type of users that can edit a Wikipedia page. Common
motivations that an administrative user may have in protecting a page include
(i) consistent vandalism or libel from one or more users, and (ii) avoiding edit
wars [
Currently, English Wikipedia contains over ve million pages. Only a small percentage of those pages are currently protected, less than 0.2 percent. However, around 17 pages become protected every day (according to the number of protected pages from May 6 through Aug 6, 2016). This ratio shows how it is di cult for administrative users to monitor over all Wikipedia pages to determine if any need to be protected. Users can request pages to be protected or unprotected but an administrative user would have to analyze the page to determine if it should be protected, what level of protection to give, and for how long the protection should last, if not inde nitely. All this work is currently manually done by administrators.
To overcome this problem, we propose DePP, the rst automated tool to
detect pages to protect in Wikipedia [
In addition to the above base features, we also include an additional set of features taking into account the page editing pattern over the time. We de ne these features by leveraging the features E1-E6 as follows. For each page, we consider the edits made in the latest 10 weeks and we split this time interval into time frames of two weeks (last two weeks, second last two weeks, etc.). Then, we compute features E1 to E6 within each time frame. The idea of these features is to monitor features E1-E6 over time to see if some anomaly starts to happen at some point. For instance, if a page is new we may observe a lot of edits of larger size in a short time after the page is created as users are building the content of the page. Later when the content is stable, we may observe fewer edits of smaller size representing small changes in the page. On the other hand, if the content of the page was stable and suddenly we observe a lot of edits from many users, it may indicate the page topic became controversial and the page may need protection.
The second group of features use information about page categories and includes:
in the training set T and a page category c, we compute the probability pr(c) that pages in category c are protected as the percentage of pages in T having category c that are protected. Then, given a page p having categories c1; : : : ; cn, we compute this feature as the probability that the page is in at least one category whose pages have a high probability to be protected as P C(p) = 1 Qin=1(1 pr(ci)).
In addition to the above two features, we de ne another group of features that shows how much features E1-E6 vary for a page p w.r.t. the average of these values among all the pages in the same categories as p. Speci cally, given the set of pages in the training set T , we computed the set C of the top-100 most frequent categories. Additionally, for each category c 2 C, we averaged the features E1-E6 among all the pages (denoted by Tc) having category c in the training set. Then, for each page p we computed 600 features (6 times 100), one for each feature Ei (1 i 6) and for each category c 2 C as follows: C(Ei; c) = (jEi(p) 0 avgp02Tc (Ei(p0))j if p is in category c otherwise where Ei(p) is the value of the feature Ei for the page p. The aim of this group of features is to understand if a page is anomalous w.r.t. other pages in the same category.
DePP 93.237%
All the features that we propose are language independent as they do not consider page content. As a consequence, DePP is general and able to work on any version of Wikipedia.
To test our DePP system we built a balanced dataset 4 containing all edit protected articles until to Apr. 7, 2016 (6,799 pages) and an almost equal number of randomly selected unprotected pages (6,824), for a total of 13.6K article pages, and up to the last 500 most recent revisions for each selected page. For protected pages, we only gathered the revisions up until the most recent protection. If there was more than one recent protection, we gathered the revision information between the two protections. This allowed us to focus on the revisions leading up to the most recent page protection. Revision information that we collected included the user who made the revision, the timestamp of the revision, the size of the revision, the categories of the page, and any comments, tags or ags associated with the revision.
The DePP accuracy in the prediction task on 10-fold cross validation is reported for random forest (the best performing algorithm as compared to Logistic Regression, SVM, and K-Nearest Neighbor) in Table 1. As we can see, DePP is able to classify pages to protect from pages that do not need protection with an accuracy of 93.237%. As no automated tool detecting which page to protect exists in Wikipedia, we de ned some baselines to compare our results. One of the main reasons for protecting a page on Wikipedia is to stop edit wars, vandalism or libel from happening, or continuing to happen on a page. Thus, we used the following baselines: [B1] Number of revisions tagged as \Possible libel or vandalism"; [B2] Number of revisions that Cluebot NG or STiki reverted as possible vandalism; [B3] Number of edit wars between two users in the page.
As we can see in Table 1, DePP signi cantly beats each individual baseline and the combination of all the three. 4
Another problem that compromises the content quality of Wikipedia articles is spamming. Wikipedia, like most forms of online social media, receives continuous spamming attempts every day. Since all non-protected pages are open for editing by any type of user, inevitably happens that malicious users have the opportunity to post spam messages into any open page. These messages remain
on the page until they are discovered and removed by another user. Speci cally,
Wikipedia recognizes three main types of spam, namely \advertisements
masquerading as articles, external link spamming, and adding references with the
aim of promoting the author or the work being referenced"[
Currently, no speci c tool is available on Wikipedia to identify neither spam edits or spam users. Tools like Cluebot NG and STiki are tailored toward vandalism detection, while ORES is designed to detect damaging edits in general. As in the case of page protection, the majority of the work to protect Wikipedia from spammers is done manually by Wikipedia users (patrollers, watchlisters, and readers) who monitor recent changes in the encyclopedia and, eventually, report suspicious spam users to administrators for de nitive account blocking.
To ght spammers on Wikipedia, we study the problem of identifying spam
users from benign ones [
We propose a machine learning-based framework using a set of features which
are based on research that has been done regarding typical behaviors exhibited
by spammers: similarity in edit size and links used in revisions, similar
timesensitive behavior in edits, social involvement of a user in the community through
contribution to Wikipedias talk page system, and chosen username. We did not
consider any feature related to edit content so that our system would be language
independent and capable of working for all Wikipedia versions. Moreover, we do
not rely on third-party services, so there is no overhead cost as in [
The list of features we considered in our system are as follows: User edit sizes based features : average size of edits, standard deviation of edit sizes, and variance signi cance (previous feature normalized by user average edit size).
Edit timing behavior based features : average and standard deviation of time between edits.
Links in edits based features : Unique link rating (the ratio of unique links posted by a user to the total number of links posted by the user) and link ratio in edits (number of edits that a user makes which contain links). Talk page edit ratio : this is the ratio of talk pages edited by a user that correspond with the main article pages that a user edits.
Username based features : Zafarani and Liu [
To test our framework, we built a new dataset 5 containing 4.2K (half spammer and half benign) users and 75.6K edits as follows. We collected all Wikipedia
Accuracy
MAP
Our Features ORES 80.8% 69.7% 0.88 0.695
Our Features + ORES 82.1% 0.886 users (up to Nov. 17, 2016) who were blocked for spamming from two lists maintained on Wikipedia: \Wikipedians who are inde nitely blocked for spamming"6 \Wikipedians who are inde nitely blocked for link spamming" 7. The rst list contains all spam users blocked before Mar 12, 2009, while the second one includes all link-spammers after Mar 12, 2009 to today. We gathered a total of 2,087 spam users (we only included users who did at least one edit) between the two lists considered.
In order to create a balanced dataset of spam/benign users, we randomly select a sample of benign Wikipedia users of roughly the same size as the spammer user set (2,119 users). To ensure these were genuine users, we cross-checked their usernames against the entire list of blocked users provided by Wikipedia 8. This list contains all users in Wikipedia who have been blocked for any reason, spammers included. For each user in our dataset, we collected up to their 500 most recent edits. For each edit we gathered the following information: edit content, time-stamp, whether or not the edit is done on a Talk page, and the damaging score provided by ORES.
We run 10-fold cross validation on several machine learning algorithms, namely SVM, Logistic Regression, K-Nearest Neighbor, Random Forest, and XGBoost, to test the performances of our features. Experimental results are shown in Table 2 for the best performing algorithm (XGBoost). Here we can see that our system is able to classify spammers from benign users with 80.8% of accuracy and it is a valuable tool in suggesting potential spammers to Wikipedia administrators for further investigation as proved by a mean average precision of 0.88.
We compared our tool with ORES only, as the tool proposed in [
In this paper, we addressed the problem of ensuring the integrity of Wikipedia pages and presented our research on detecting pages to protect and identifying spam users. Our experimental results show that we are able to classify (i) article pages to protect with an accuracy of 93% and (ii) spammers from benign users with 80.8% of accuracy and 0.88 mean average precision.
Both the methods proposed do not look at edit content and, as a consequence, they are generally applicable to all versions of Wikipedia.