The paper overviews the task “Quality Flaw Prediction in Wikipedia” of the PAN'12 competition. An evaluation corpus is introduced which comprises 1 592 226 English Wikipedia articles, of which 208 228 have been tagged to contain one of ten important quality flaws. Moreover, the performance of three quality flaw classifiers is evaluated. 1 Wikimedia, http://meta.wikimedia.org/wiki/List_of_Wikipedias. Alexa Internet, Inc., http://www.alexa.com/siteinfo/wikipedia.org.
The online encyclopedia Wikipedia is one of the largest and most popular usergenerated knowledge sources on the Web. Some facts: Wikipedia contains articles from more than 280 languages, the English Wikipedia version contains about 4 million articles, the Wikipedia community involves more than 35 million registered editors, and wikipedia.org ranks among the top ten most visited Web sites.1 Probably the biggest challenge for Wikipedia pertains to the quality of its articles, since the community of Wikipedia authors is heterogeneous and since contributions to Wikipedia are not reviewed by experts before their publication. Both the size and the dynamic nature of Wikipedia render a comprehensive manual quality assurance infeasible.
A variety of approaches to automatically assess quality in Wikipedia has been
proposed in the relevant literature, see e.g. [
The outlined facts make clear that the automated prediction of quality flaws in Wikipedia is a relevant problem, and the research on and the development of respective prediction approaches are the main goals of this PAN’12 task.
We cast quality flaw prediction in Wikipedia as a one-class classification problem, as
proposed in [
Stated formally, let D be the set of Wikipedia articles and let F be a set of quality flaws. We model the classification cf (d) of an article d 2 D with respect to a quality flaw f 2 F as the following one-class classification problem: Decide whether or not d contains f , whereas a sample of articles containing f is given. cf : D ! f1; 0g is a specific classifier for flaw f , d denotes the (vector) representation or document model of article d, and D denotes the set of document models for the Wikipedia articles D.
A key challenge of this problem is the absence of representative “negative” training data (articles that are tagged to not contain a particular flaw)—a fact which renders common discrimination-based classification techniques such as binary or multiclass classification inapplicable. The feature engineering, i.e., the development of document models that discriminate articles containing a certain flaw from all other articles is hence one of the primary challenges. 1.2
The acquisition of sensible test data to evaluate a classifier cf is intricate in the
Wikipedia setting; see [
The authors of [
Wikipedia users who encounter a flaw may tag the affected article with a so-called
cleanup tag.2 The available cleanup tags correspond to the set of quality flaws that have
been identified so far by Wikipedia users, and the tagged articles provide a source of
human-labeled data—an idea that has been proposed in [
The evaluation corpus is based on the English Wikipedia snapshot from January 4, 2012.3 The corpus contains for each of the ten quality flaws Wikipedia articles that are exclusively tagged with the respective cleanup tag. The corpus contains also untagged articles, which have not been tagged with any cleanup tag. Altogether 1 592 226 articles are provided from which 208 228 are tagged and 1 383 998 are untagged.4
For the PAN competition, the corpus is divided into a training corpus and a test corpus.5 The training corpus contains tagged articles for each of the ten quality flaws plus additional 50 000 untagged articles; in the training corpus the respective labels are given. In particular, tagged articles may be considered as “positive” training examples while untagged articles may be considered as outlier examples to evaluate and tune the classifiers. In case of a semi-supervised learning approach, the untagged articles serve as additional training examples. The test corpus contains a balanced number of tagged articles and untagged articles for each of the ten quality flaws; in the test corpus the labels are omitted. Moreover, it is ensured that 10% of the untagged articles are featured articles in order to address both the optimistic and the pessimistic setting, mentioned in Section 1.2. 3
This section briefly overviews the submitted quality flaw prediction approaches and
reports on their evaluation. From 21 registered teams three teams submitted runs for this
task, see Table 2. Feretti et al. [
Ferschke et al. regard the problem as a binary classification task, using the tagged articles as positive instances and the untagged articles as negative instances. They employ two machine learning approaches, namely a Naive Bayes classifier and C4.5 decision trees. Their document model is based on 32 feature types. In particular, a dedicated document model is used for each flaw, which is determined by a features selection approach.
Instead of using machine learning, Pistol and Iftene resort to a rule-based approach. They define a particular set of rules for each flaw and classify an article as flawed if it fulfills the formulated requirements. 3.2
The quality flaw classifiers are evaluated for each of the ten flaws individually. To determine the winning classifier, the prediction performance is judged by averaging precision, recall, and F-measure over all ten quality flaws. Table 3 shows the prediction performance of the quality flaw classifiers.
The classifier of Feretti et al. performs best in terms of the averaged F-measure and the averaged recall. The classifier of Ferschke et al. achieves a slightly higher averaged precision, but a much lower averaged recall. The third classifier of Pistol and Iftene falls far behind because of a very low averaged precision. The situation is nearly the same for the individual flaws: except for the flaw Wikify, Feretti et al. achieve in general a higher recall than Ferschke et al. For seven of the ten quality flaws Ferschke et al. achieve the highest precision. However, in terms of the F-measure the classifier of Feretti et al. performs best for seven of the ten quality flaws.
The results of the 1st International Competition on Quality Flaw Prediction in Wikipedia can be summarized as follows: three quality flaw classifiers have been developed, which employ a total of 105 features to quantify the ten most important quality flaws in the English Wikipedia. Two classifiers achieve promising performance for particular flaws. An important “by-product” of the competition is the first corpus of flawed Wikipedia articles, the PAN Wikipedia quality flaw corpus 2012 (PAN-WQF-12).
We thank the German chapter of the Wikimedia Foundation, Wikimedia Deutschland, for sponsoring the price for the winning team. [10] L. Gaio, M. den Besten, A. Rossi, and J. Dalle. Wikibugs: using template messages in open content collections. In Proceedings of the 5th symposium on wikis and open collaboration (WikiSym 2009), pages 14:1–14:7, 2009. [11] M. Hu, E. Lim, A. Sun, H. Lauw, and B. Vuong. Measuring article quality in Wikipedia: models and evaluation. In Proceedings of the 20th ACM conference on information and knowledge management (CIKM 2007), pages 243–252, 2007. [12] B. Liu, Y. Dai, X. Li, W. S. Lee and P. Yu. Building text classifiers using positive and unlabeled examples. In Proceedings of the 3rd IEEE international conference on data mining (ICDM 2003), pages 179–186, 2003. [13] N. Lipka and B. Stein. Identifying featured articles in Wikipedia: writing style matters. In Proceedings of the 20th international conference on World Wide Web (WWW 2010), pages 1147–1148, 2010. [14] B. Stvilia, M. Twidale, L. Smith, and L. Gasser. Information quality work organization in Wikipedia. Journal of the american society for information science and technology, 59(6):983–1001, 2008. [15] D. Wilkinson and B. Huberman. Cooperation and quality in Wikipedia. In Proceedings of the 3rd symposium on wikis and open collaboration (WikiSym 2007), pages 157–164, 2007.