As the amount of data and software applications increases, it becomes important to identify the true authors for ownership and liability of the work. Issues such as plagiarism in academic activities, open-source contributions, and identification of the creators of malware applications can be done using automatic authorship identification models. In this work, the performance of Character Count vectorization and TFIDF models are studied on the AI-SOCO data-set. We achieved a significant improvement from the baseline with 85% accuracy on the test-set and 92% accuracy on the dev-set.
The authorship identification can solve two major problems: identification of plagiarism by interpreting a program writing based on the previous works, and identification of the creators of malicious applications. Both are equally intimidating and the only solution is to device a foolproof model for identifying the author of a particular code snippet by understanding the users’ style in writing programs. This model aims to safeguards the rights of one’s work in academics, open-source contributions, projects, online coding contests, and all around the open internet.
Authorship detection on academic activity is important, for the identification of students cheating on their assignments. With online classes and online submission systems being promoted, it adds up to the need for the detection of plagiarism. In addition to this, recruitment processes for companies opting for online coding rounds escalate the need for identification of cheating. As other alternatives such as invigilation during academic exams and proctoring during recruitment processes are resource-intensive and require a large labor force, authorship identification becomes an eficient alternative.
Malware software is annoying enough to ruin a person’s or even an entire organization’s time. Even though laws against malware applications exists, it is hard to find the person involved in the program, to stop further malicious programs, and punish the person. Devising author identification models could instill fear in the first place, hence reducing the attempt for even indulging in coding malicious programs.
Data Set Number of users Number of programs per user Total samples Train-set 1000 50 50,000 Dev-set 1000 25 25,000 Test-set 1000 25 25,000
Total 1000 100 100,000
In this work feature extraction techniques and machine learning modeling techniques are analyzed. Char count vectorization with the Random Forest model is proposed for the authorship identification task. The AI-SOCO data-set containing C++ programs and corresponding user-id is used to analyze the performance of the models.
The remaining of the paper is organized in the following fashion: The data-set description and baseline analysis in Sectio2n,followed by the model architecture in Secti3o, nresults and discussion in Section4 and conclusions in the Sectio5n.
The data-set is a collection of C++ programs by 1000 users fromctohdeeforces programming platform. The programs are coded in diferent versions of C++. It contains 100 (50 for training, 25 for development, and 25 for testing) diferent code snippets of each user. It contains a mapping of user-ids (uid) to the program-ids (pid) and the set of programs. These program ifles are verified and ready to compile, working programs that were submitted to the coding platform. As the data-set contains an equal number of samples for each testing individual, the data-set is fairly distributed, without any imbalance. The distribution of the data-set among the train, test, and dev-set is explained in Tab1l.e
The baseline model considered by the AI-SOCO challenge is a char Count Logistic regression model which gives an accuracy of 29.252%. In addition to this model, a TFIDF feature extractor with KNN based baseline with 10k features and k=25 is considered. This model shows an accuracy of 62.128%1[].
The architecture included three parts - feature extraction, classification, performance evaluation. The C++ program file contains raw code, hence Count and TFIDF vectorizers are considered for feature extraction. These vectors and their corresponding uids now become a numerical classification problem, which is trained with Naive Bayes (NB) and Random Forest (RF) classifiers. As the data-set is well balanced the accuracy score is considered for comparing and evaluating the performance of the model. The scikit-lear2n] [text feature extractor is used for converting the text into numerical vectors and scikit-learn2’]sR[F and NB implementations are used for classification.
C++ Program Code
from data-set Char (Count/TFIDF)
Vectorization n-gram = 2-5
RF/NB
Classifier
User id labels
Accuracy calculation
The count vectorization provides a primitive but powerful and simple way to transform the program documents to numerical vectors, as could be seen from previous w3o]r.kT[he count vectorization generates vectors of length equal to the length of the vocabulary the vectorizer is trained on and each value presents the number of instances the particular character or word appears in a document. This is helpful in the case of author prediction as some authors would mostly use a particular set of variable names for frequent mundane tasks. For instance for looping some show preference tfoor loops, whereas some prefewrhile loops, so is the bias with if-else andswitch-case statements. Hence the set of characters or sequences of character diference is an important factor for studying a person’s individual style of programming.
The count vectorizer is trained on the AI-SOCO corpus and this transformation function is used to fit the classification model on the train-set. Character Count vectorization with diferent n-gram ranges was analysed and the range of 2-5 was the best performing. The random forest proved to be the best model for fitting the large sparse matrix generated by the count vector transformation.
Term Frequency Frequency Inverse Document Frequency (TF-IDF) method is an extension of
the Count vectorization technique. This method has shown significant results with feature
extraction from program code4s,[
Similar to the count vectorizer, the TF-IDF vectorizer also shows good performance with an n-gram range of 2-5 with a random forest classifier.
In this section the vectorization techniques are compared based on the performance on the dev-set and test-set accuracy scores. Both the techniques generated a significant improvement when compared to the baseline results. The random forest classifier is observed to perform better than the Naive Bayes classifier. With respect to the Char Logistic regression model there is a huge increase in accuracy from 29% to 92% on the dev-set. Similarly with respect to the TFIDF-KNN baseline model there is a increase in accuracy from 62% to 92% with around 48% improvement on the dev-set. The detailed scores on the dev-set and test-set are shown in Table 2 and Table3 respectively.
Authorship identification from program code is becoming important in recent times due to the rapid increase in the number of free-to-use applications, online academic and research contributions. This could help to regulate plagiarism and detect the creators of malware softwares. In this work, Char count based feature extraction technique along with a Random Forest classifier is proposed for the AI-SOCO data-set. Our system showed significant increase in performance with respect to the baseline model with an accuracy of 85% on the test-data and 92% on the dev-set.