In this article we describe a new approach to assess Quality Flaw Prediction in Wikipedia. The partially supervised method studied, called PU Learning, has been successfully applied in classi cations tasks with traditional corpora like Reuters-21578 or 20-Newsgroups. To the best of our knowledge, this is the rst time that it is applied in this domain. Throughout this paper, we describe how the original PU Learning approach was evaluated for assessing quality aws and the modi cations introduced to get a quality aws predictor which obtained the best F1 scores in the task \Quality Flaw Prediction in Wikipedia" of the PAN challenge.
Given the daily increase in the amount of data on the Web, machine-based
assessment of Information Quality (IQ) is becoming a topic of enormous interest. This
fact is rooted, among others, in the increasing popularity of user-generated Web
content and the unavoidable divergence of the delivered content's quality [
According to our literature review, there are three main research lines
related to IQ in Wikipedia, namely: (a) Featured articles identi cation [
In [
In previous editions of the PAN challenge, assessing quality issues in
Wikipedia has been addressed in the form of vandalism detection. Given the context
above, in PAN@CLEF 2012,5 the vandalism detection task has been generalized
in focussing on the prediction of quality aws in Wikipedia articles. In particular,
the quality aws to be predicted are the ten most frequent quality aws of the
English Wikipedia articles, namely: Advert, Empty Section (Empty), No footnotes
(No-foot), Notability (Notab), Original research (OR), Orphan (Orph), Primary
sources (PS), Re mprove (Ref), Unreferenced (Unref) and Wikify (Wiki).
Besides, the task is formally de ned as follows: \Given a set of Wikipedia articles
that are tagged with a particular quality aw, decide whether an untagged
article su ers from this aw". That is to say, that detection of text quality aws is
cast as a one-class classi cation, as proposed in [
In our view, the most notable proposals to quality aw predictions in
Wikipedia have been made by Anderka et al. [2{4]. In [
As mentioned above, all the work done in literature with respect to
quality aw prediction in Wikipedia, has been carried out following supervised
approaches. Despite the fact that very good results have been achieved in [
In Sect. 3, it is described in detail the research questions which guided the development of our proposal to participate in PAN@CLEF 2012. Besides, in this section it is also described a more intuitive rule-based approach to assess certain quality aws. Then, Sect. 4 reports and discusses the results obtained in the competition with our PU Learning approach and with the rule-based approach as well. Finally, in Sect. 5 some general conclusions are drawn. 2
Text classi cation is an important problem which has numerous applications. As pointed out by Liu et al.:7 \Although this classic model is important,8 in practice one also encounters another problem. That is, one has a set of documents of a particular topic or class P (positive class), and is given a large set U of mixed (unlabelled) documents that contains documents from class P and also other types of documents (negative documents). One wants to classify the documents in U into documents from P and documents not from P. The key feature of this problem is that there is no labeled negative training data, which makes the traditional text classi cation techniques inapplicable. This problem is termed, partially supervised classi cation (PSC). We also call it PU-learning (Learning from Positive and Unlabelled examples)."
In particular, given its simplicity and robust performance we decided to
implement the two-step strategy proposed in [
Step 1: Identifying a set of reliable negative documents from the unlabelled set. Step 2: Building a set of classi ers by iteratively applying a classi cation algorithm and then selecting a good classi er from the set.
8Here, \this" refers to the supervised approach.
It is well-known in Machine Learning research community that documents' representation is a key issue. However, given that the team expertise is stronger in the research eld of algorithms, we decided to use features already proposed in the literature for modelling the documents and focussing our e orts in exploiting as much as possible the characteristics of the PU learning approach described in Sect. 2. There are four research questions which guided our experiments, namely:
2. How to determine the sets of untagged documents for training the rst stage classi er to improve its performance in selecting RNs? 3. After determining the RNs set, what documents should be used for training the second stage classi er? 4. Which parameters setting of the second classi er improves its performance?
These four questions are discussed next in below subsections. Regarding the
documents' representation, we used as a guide the work performed in [
Documents Model
In [
What Classi er in Each Stage?
As mentioned above, in [
To avoid a bias in how U ( U) was determined, 40 di erent samples were selected to cover all the 50000 untagged documents in U. Figure 2 shows how these 40 di erent samples were obtained. Originally, U was split in 10 sub-sets Ui such that jUij = 5000, for i = 1 : : : 10. Then, subsets Ui:1, were built such that: Ui:1 = Ui + U(i mod 10)+1. Hence, jUi:1j = 10000 for all i = 1 : : : 10. Similarly, subsets Ui:2, were obtained as: Ui:2 = Ui:1 +U((i+1) mod 10)+1. Thus, jUi:2j = 15000 for all i = 1 : : : 10. Finally, subsets Ui:3, were built as Ui:3 = Ui:2 + U((i+2) mod 10)+1. In this way, for all i = 1 : : : 10, jUi:3j = 20000. The idea of building these untagged sets in an incremental way aims at analysing the e ect of increasing the proportion of untagged documents versus positive documents in the training sets. Larger proportions up to jU j = 50000 were tried but no improvements in the results were achieved. Moreover, increasing the size of U also increases running times, so 20000 was set as the upper amount of untagged samples to be used.
Given that the number of positive sample documents for each aw was highly unbalanced, for each aw it was also determined the minimum amount of positive documents required to get the best results. For eight of the ten aws, the number of positive documents in the training sets was set to 1000. Besides, several proportions for the respective test sets were analysed. From these experiments we decided to use for testing only 110 positive documents, since having more of them in average resulted in similar performance rates. Flaws Advert and OR contain 1109 and 507 documents, respectively. Hence, in order to have a uni ed experimental test setting, it was also set to 110 the number of positive documents to be used in the test sets of these aws. Hence, the number of positive documents for training were 999 for Advert and 397 for OR, respectively.
In this way, for each aw f it was xed a positive set Pf which was combined with each of the 40 di erent subsets of U depicted in Fig. 2, thus yielding in 40 di erent training sets for the rst stage classi er. As explained in Sect. 2, set Pf also comprise the positive sample of the training set of the second stage classi er. In the following section, the di erent approaches used to determine the negative training set of the second stage classi er, are explained. 3.4
Strategies for Selecting Reliable Negatives Four strategies were used for selecting the reliable negative documents (RNs) to compose the training set of the second stage classi er, namely:
2. Selecting jPf j documents by random from RNs set. 3. Selecting the jPf j best RNs (those assigned the highest con dence prediction values by the rst stage classi er). 4. Selecting the jPf j worst RNs (those assigned the lowest con dence prediction values by the rst stage classi er).
Strategy 1 is the original one proposed in [
A statistical analysis (a non-parametric ANOVA) showed that the existing di erences in the false negative rates of strategies 2 and 4, were signi cant when compared against strategies 1 and 3, respectively. For strategy 3, the di erences with strategy 1 were found not signi cant. Similarly, taking into account the median recall values calculated on the ten aws when trained with the 40 different training sets described in Sect. 3.3, the statistical analysis determined as signi cant the existing di erences between strategies 2 and 4, against 1 and 3. Moreover, the mean rank di erences found between strategies 2 vs. 4, and 1 vs. 3, were determined not signi cant, respectively. Table 3 presents the average recall values obtained per each aw on the 40 training sets. As it can be observed, strategy 1 for ve out of the ten aws performs very poorly. Furthermore, when considering the running times for each strategy, strategy 1 was found at least three times slower than the other ones. Based on this evidence, we decided to continue working with strategies 2 4.
When compared against strategies 3 and 4, strategy 2 is conceptually the
simplest one, since it just selects at random jPf j RNs documents to make a
balanced training set for the second stage classi er. Conversely, strategy 3 selects
those documents assigned the highest con dence prediction values by the rst
classi er, on the grounds that they are better candidates in representing the
real negative documents' features. Finally, strategy 4 aims at selecting those
documents that in spite of being predicted as negatives, are still quite similar to
the positive ones. The underlying idea of this last strategy, is that selecting these
documents could help to build a much more ne-grained borderline between both
sets of documents. As shown in Table 2, strategies 2 and 4 perform best.
In Sect. 3.2, it was mentioned that using a SVM variant as second stage
classier reported the best results in [
for all the aws, while the best values are indicated in Table 4. It is worth mentioning, that values presented in Tables 2 and 3 were obtained with the RBF kernel, accordingly set with the parameters values reported in Table 4.
During the training stage, we studied the performance of our algorithm using the default kernel, i.e., RBF. When the PAN test set was released, the only clue we had about it, was that it was balanced. That is to say, that for all the aws, there were the same amount of positives and untagged documents. In this way, approximately 50% of the documents was expected to be predicted as positives by our algorithm. When we ran the di erent variants of our algorithm, in average, for all the aws, they predicted as positive nearly 75% of the test documents.
With the aim of improving the classi ers expected performance, the same experimental setting carried out for the training set was also run for the test set. Instead of studying the classi ers performance based on recall and f n measures, they were studied with respect to their prediction rates. For each document in the test sets, statistics were gathered considering if a particular classi er predicted it or not as positive. For the aws Advert, Empty, No-foot, OR and Ref, most of the classi ers agree on their predictions, while for the remaining aws the classi ers shown di erent predicting behaviours. As this phenomenon could be caused by an over- tting in the models learned from the training sets, therefore, it was tried a more simple approach like a linear kernel instead of RBF. With this kernel, in average, the number of documents predicted as positive was 62.55%, a more balanced percentage than the one obtained for RBF.
Based on these studies on the PAN test set, the linear SVM was also studied
as second stage classi er in our PU learning approach for the PAN training set.
For this particular kernel, we used the default parameters provided by WEKA [
In this way, based on all the experiments performed with both kernels and also considering the prediction statistics gathered for each document for the PAN test set, we decided to use strategy 2 for RNs selection in nine out of the ten aws. Flaw Orph, was the only one where strategy 4 was used. Regarding the kernel selection for the second stage classi er, the linear kernel was used in eight out of the ten aws. Flaws Advert and OR were the only ones where the RBF kernel was used. In the Sect. 4 the results obtained with the PAN test set are presented. As mentioned above, for some aws like Empty, No-foot, Ref and Unref, generating a complex document model to predict them is not necessary. In this way, according to our experimental study, a document is predicted as having the Empty aw when one of the following three conditions hold: the number of empty sections is greater than zero; the number of subsections without content is greater than zero or when the pattern \< - - - : : : - - ->" is present. Similarly, the No-foot aw is predicted when at least one of the following conditions hold: there are no external links (\==External link==" = 0); there are no in-text citations (\http" = 0) or when expression \ref" is found less than 80 times. Moreover, the Ref aw is predicted when: the number of external references is less than 22; the regular expression (\ref >") is found less than 65 times or when reference section is empty. Finally, for Unref aw three conditions are used: expression \ref" is found less than 45 times; the reference section does not exist or there are no in-text citations (\http" = 0).
In [
Throughout this paper, we have mainly described the PU Learning approach
implemented to participate in the PAN challenge. Despite the fact that we
competed with this approach, as indicated in Sect. 3.6, also a rule-based approach
was developed to assess the prediction of some quality aws. As it can be
observed in Table 6, the results obtained with our rule-based approach are not very
encouraging. Nonetheless, we believe that we have failed in capturing the gist of
the wikitext patterns which characterize best these aws, and as suggested in [
Also, it has been described the exhaustive experimental setting carried out to set up as best as possible all the features of this approach, in order to participate in the PAN challenge, where our proposal obtained the best F1 scores in the task \Quality Flaw Prediction in Wikipedia". As future work, we think that exploring other di erent semi-supervised techniques is a promising direction to improve quality aw predictions in this free-access encyclopedia available to the entire world.
Edgardo Ferretti and Marcelo Errecalde thank Universidad Nacional de San Luis (PROICO 30310). The collaboration of UNSL, INAOE and UPV has been funded by the European Commission as part of the WIQ-EI project (project no. 269180) within the FP7 People Programme. Manuel Montes is partially supported by CONACYT, No. 134186. The work of Paolo Rosso was carried out also in the framework of the MICINN Text-Enterprise (TIN2009-13391-C04-03) research project and the Microcluster VLC/Campus (International Campus of Excellence) on Multimodal Intelligent Systems.