This report explains our Persian plagiarism detection system which we used to submit our run to Persian PlagDet competition at FIRE 2016. The system was constructed through four main stages. First is pre-processing and tokenization. Second is constructing a corpus of sentences from combination of source and suspicious document pair. Each sentence considered to be a document and represented as a tf-idf vector. Third step is to construct a similarity matrix between source and suspicious document. Finally the most similar documents which their similarity is higher than a specific threshold marked as plagiarized segments. Our performance measures on the training corpus were promising (precision=0.914, recall=0.848, granularity=3.85). • Information systems➝Information retrieval ➝Retrieval tasks and goals. Near-duplicate and plagiarism detection.
[1][
Plagiarism in academia is rising and multiple authors have worked
to describe these phenomena [
Two main strategies for plagiarism detection have been
considered by researchers, namely, Intrinsic and external
plagiarism detection [
For external plagiarism detection Stein, Meyer zu Eissen, and
Potthast [
The process starts with a suspicious document dq and a collection D of documents from which dq’s author may have plagiarized. Within a so-called heuristic retrieval step a small number of candidate documents, Dx, which are likely to be sources for plagiarism, are retrieved from D. Note that D can be as large as the entire web. Hence, it is impractical to compare dq with all of its members. Usually an initial inspection is made to select a rough subset of D as prospective candidates. In a so-called detailed analysis step, dq is compared section-wise with the retrieved candidates. All pairs of sections (sq, sx) with sq ∈ dq and sx ∈ dx, dx ∈ Dx, are to be retrieved such that sq and sx have a high similarity under some retrieval model. In a knowledge-based postprocessing step those sections are filtered for which certain exclusion criteria hold, such as the use of proper citation or literal speech. The remaining suspicious sections are presented to a human, who may decide whether or not a plagiarism offense is given.
Given a set of suspicious and source documents written in
Persian, the plagiarism detection task of Persian PlagDetect
competition [
Text plagiarism is a long lasting game between authors and
publishers [
In the following we describe the steps we took in order to find plagiarized segments of the document pairs.
In her MSc thesis [
تسا دارگیتناس هجرد دص بآ شوج هطقن Boiling Point of Water is one hundred Celsius degrees. تسا سویسلس هجرد دص بآ شوج هطقن تسا دارگیتناس هجرد 011 بآ شوج هطقن دیآ یم شوج هب دارگیتناس هجرد دص رد بآ
If temperature of water reaches one hundred degrees, it will boil دشوج یم ،دسرب دارگیتناس هجرد دص هب بآ یامد رگا
دشوج یم دیهد ترارح هجرد دص ات ار بآ رگا If you heat water up to one hundred degrees, it boils. A syntactic example is demonstrated in Figure 2. Key terms in the original sentence are water, boiling, and one-hundred. To communicate the same concept, plagiarized versions of the sentence had to use these terms with re-ordering and /or changing verb tenses. This fact narrows down the task of plagiarism detection to classic document matching and retrieval.
Eq 1 Eq 2 Eq 3 Given a document collection D and a query document, q, find it's most similar documents in D.
The task is usually solved by representing documents in some vector space and exploiting similarities between the query document and the collection members in this space. Bag of words representation of documents is a classic, yet powerful method for document indexing and retrieval. In this method a document is assumed to be the set of its terms, ignoring their relative positions in paragraphs and sentences. A dictionary is the set of all the distinct terms in the collection. A document, d, is represented by a one-dimensional vector, v, in which vi denotes relative importance of the i-th dictionary term in d. Using this representation, similarities between documents can be estimated by well-known techniques such as Euclidean distance or cosine of their respective vectors. Usually vi is defined as function of appearances of the word in the document and the whole collection. Term frequency (TF) of term in a document is the relative frequency of the term in the document.
In this equation Fi is the frequency of i-th term in the document and |.| denotes document length; i.e. total number of terms in the document. Assuming that there totally N documents in the collection and Ni of them contains i-th term, inverse document frequency (IDF) is defined by equation Eq 2
| | Finally, as an importance measure of the word in a document, tf-idf is defined as: tf-idf favors terms with high frequency in the document but low frequency in the collection. Common terms such as it, is, and what that appear in most of the documents will have a low tf-idf value. These so-called stop words cannot distinguish the documents efficiently; hence usually are removed in pre-processing steps of NLP tasks.
We adopt tf-idf for representation and retrieval of matching
documents. Each source and suspicious document is firstly
tokenized and then split to sentences. This step is performed using
NLTK 3.0 toolkit [
In this step, the similarity matrix between source sentences and suspicious sentences is calculated. If there are N source sentences and M suspicious sentences, the similarity matrix S, has N×M elements in which Si,j denotes the similarity between sentence i from source document and sentence j from suspicious document. There are many different similarity and distance measures. For example an option is to use Euclidian distance, which measures the distance between two vectors u and v in the encompassing Euclidian space. In order to convert the distance to a similarity measure we can use: ‖
‖ Another choice that we used is cosine similarity: ‖ ‖‖ ‖ This measure can range from -1 to 1. When the value is 1, it means that u and v are in the same direction and when it is -1, it means that u and v are in the opposite directions. We use cosine similarity in subsequent steps. Initial inspection and evaluation confirmed that it has slightly better results than Euclidean distance.
We used pairs of sentences which their similarity was greater than a pre-specified threshold as plagiarized fragments for both source and suspicious documents. The value we used as a threshold was 0.4, which is obtained through cross validation.
The training results were obtained after we run our application on
TIRA platform [
The detection granularity of detected documents collection R under source documents collection S is defined as: | ∈ | ∈ | | ∑ | | ∈ ∈ Where SR ⊆ S are cases detected by detections in R, and Rs ⊆ R are the detections of a given s: The domain of gran(S, R) is [1, |R|], with 1 indicating the desired one-to-one correspondence and |R| indicating the extreme case, where a single s ∈ S is detected multiple times.
Precision, recall, and granularity allow for a partial ordering
among plagiarism detection algorithms. To obtain an absolute
order they must be combined to an overall score:
Where F denotes the F-Measure, i.e., the weighted harmonic mean
of precision and recall. Log of granularity decreases its impact on
the overall score [
In this paper, we proposed a sentence-level algorithm based on tfidf features for plagiarism detection task, which shows competitive results on the training data.
The algorithm works is designed for near-copy and paraphrasing types of plagiarism. It relies on the fact that a plagiarist willing to publish an article in a scientific field must use popular terminology of the field. Obviously sophisticated cases such as cross language plagiarism or grabbing another ones idea and discussing it in one's own words would hard to be detected by this algorithm. In the future works we will consider improving the feature vector of the sentences by incorporating more features and also to use a method to combine overlapping fragments. Furthermore we will study language modeling and semantic text normalization for detecting harder cases of plagiarism.
[1] Alzahrani, S.M., Salim, N. and Abraham, A., 2012.
Understanding plagiarism linguistic patterns, textual features, and detection methods. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(2), pp.133-149.