Source retrieval involves making use of a search engine to retrieve candidate sources of plagiarism for a given suspicious document so that more accurate comparisons can be made. We describe a strategy for source retrieval that uses a supervised method to classify and rank search engine results as potential sources of plagiarism without retrieving the documents themselves. Evaluation shows the performance of our approach, which achieved the highest precision (0.57) and F1 score (0.47) in the 2014 PAN Source Retrieval task.
The advent of the Web has had many benefits in terms of access to information.
However, it has also made it increasingly easy for people to plagiarize. For instance, in a
study from 2002-2005, it was found that 36% of undergraduate college students
admitted to plagiarizing [
There is an inherent assumption in most plagiarism detection systems that potential sources of plagiarism have already been identified that can be compared to a suspicious document. For small collections of documents, it may be reasonable to perform a comparison between the suspicious document and every document in the collection. However, this is infeasible for large document collections and on the Web. The source retrieval problem involves using a search engine to retrieve potential sources of plagiarism for a given suspicious document by submitting queries to the search engine and retrieving the search results.
In this paper we describe our approach to the source retrieval task at PAN 2014.
The approach builds on our previous approach in 2013 [
To generate queries, the suspicious document is partitioned into paragraphs with each
paragraph consisting of 5 sentences as tagged by the Stanford Tagger [
Queries are submitted in batches consisting of 3 queries and the results returned by each query are combined to form a single set of results. The intuition behind submitting queries in batches is that the union of the results returned by the three queries is more likely to contain a true positive than the results returned by a query individually. For each query, a maximum of 3 results is returned resulting in a set of at most 9 results. The intuition behind submitting 3 queries is that they likely capture sufficient information about the paragraph. 2.3
We assume that the order of search results as produced by the search engine does not necessarily reflect the probability of the result being a source of plagiarism. We thus infer a new ranking of the results based on a supervised method. This is a major component of our approach and will be discussed in Section 3. 2.4
Having inferred a new ordering of the results, they are then retrieved in the new ranked order. For each result retrieved, the PAN Oracle is consulted to determine if that result is a source of plagiarism for the input suspicious document. If it is, then candidate retrieval is stopped and the whole process is repeated on the next paragraph. If it is not, then the next document is retrieved. Furthermore, a list is maintained of the URLs of each document retrieved and used to prevent a URL from being retrieved more than once since this does not improve precision or recall.
The main difference between our current approach and our approach in the previous year is that now we make use of a supervised method. Specifically, we train a search result classifier that classifies each search result as either being a potential source of plagiarism or not. The classifier only makes use of features that are available at search time and without retrieving a search result unless it is classified as a potential source of plagiarism. Furthermore, an ordering of the positively classified search results is produced based on the probabilities output by the classifier. 3.1
We make use of a Linear Discriminant Analysis (LDA) classifier that tries to find a
linear combination of features for classification. We make use of the implementation of
the LDA classifier from the scikit-learn machine learning toolkit [
For each search result that is returned by the search engine, we extract the following
features, which we use for classification. All of these features are available at search
result time and do not require the search result to be retrieved, which allows for
classification to be performed as the search results become available. Many of these features
are available from the ChatNoir search engine at search result time.
1. Readability. The readability of the result document as measured by the Flesh-Kincaid
grade level formula [
Cosine(s; d) = cos( ) =
Vs Vd
jjVsjjjjVdjj
;
(2)
where V is a term vector.
13. Title-document Cosine Similarity. The cosine similarity between the result title
and the suspicious document (Eq. 2).
14. Query-snippet Cosine Similarity. The cosine similarity between the query and
the snippet (Eq. 2).
15. Query-title Cosine Similarity. The cosine similarity between the query and the
result title (Eq. 2) [
We collect training data for the classifier by running our source retrieval method (as described above) over the training data provided as part of the PAN 2014 source retrieval task. However, instead of classifying and ranking the search results returned by the queries, we instead retrieve all of the results and consult the Oracle to determine if they are sources of plagiarism. This provides labels for the set of search results.
The training data was heavily imbalanced and skewed towards negative samples,
which made up around 70% of the training data. It is well known that most classifiers
expect an even class distribution in order for them to work well [
The number of nearest neighbors considered is set to k = 3 and the minority class is increased by 200%. Since nearest neighbors are randomly selected, we train 5 classifiers. The final classification of a search result is based on the majority vote of the 5 classifiers and the probability output for ranking is based on the average probabilities produced by the 5 classifiers.
Our approach achieved a relatively high precision of 0.57 and recall of 0.48. The
F1 score, which is the harmonic mean of precision and recall was 0.47. Overall, our
approach was very competitive. Both the precision and F1 score were the highest achieved
among all participants. The number of queries submitted was relatively high compared
to the other participants; however, as shown in [
The features used for classification are also of interest to gain a better understanding
of what features are important for source retrieval. While we do not discuss it here, a
complete discussion is presented in [
We describe an approach to the source retrieval problem that makes used of supervised ranking and classification of search results. Overall, the approach was very competitive and achieved the highest precision and F1 score among all task participants. 1 The F1 score is computed by averaging the F1 score of each run rather than from the average precision and recall.