The purpose of authorship attribution is to determine who is the author of a certain work by comparing features of the said work with other documents for which the authorship is known. In the software forensics domain, revealing who is the author behind a piece of software can prevent intellectual property infringements as well as prevent and detect cheating in academic courses. In this paper, we describe the participation of the UMUTeam from the University of Murcia in the PAN's shared task: Authorship Identification of SOurce COde (AI-SOCO'2020). Our proposal is grounded on a mixture of statistical features and heuristics that we called author's traits. From the three runs submitted, we achieve a best accuracy of 91.16% over the testing dataset reaching the sixth place in the oficial ranking. It is worth noting that our proposal, which relies in traditional machine-learning classifiers, outperforms competitive baselines based on state of the art deep-learning transformers such as RoBERTa.
In a broad sense, authorship identification deals with the problem of identifying who is the author
behind certain unidentified work by comparing and finding similarities between that work with
other works whose authorship is known. Authorship identification has many applications. For
example, it can be used to provide evidences that the person listed as the author of certain work
was not actually the person who wrote it, unveiling cases of ghostwriting, fraud or plagiarism.
Moreover, authorship identification can help to unmask who is the author of anonymous
threatening messages. Applied to the manufacturing software domain, authorship identification
(
In academia, e-learning platforms are easing for both students and teachers to do their work at home, which also promotes self-discipline and self-evaluation skills during software learning [2]. Moreover, as technology evolves quickly, these skills are also needed to employees in technological companies in order to adapt themselves to the industry and an increasingly competitive market. In this context, online judges and contest platform systems allow the participants to compile, execute, and evaluate their approaches unsuperv3is].edNe[vertheless, the feeling of not being under close surveillance in a classroom or in the ofice can encourage people to cheat4[] and, therefore, online judges and contest platform systems should incorporate mechanisms to detect evidences of cheating in order to ease the work of the reviewers.
With the aim of improving current authorship identification techniques, the Forum for Information Retrieval Evaluation (FIRE’2020) proposAeudthorship Identification of SOurce COde (AI-SOCO), a PAN shared task focused onu“ncovering the author who wrote some piece of code” [5]. This task consists of, given a pre-defined set of source codes and authors, infer who the author is for each source code. In this paper we describe our contribution with the submissions of three runs based on a machine-learning models trained with statistical features based on character n-grams and some heuristics that we called author’s traits.
The rest of the paper is organised in the following way. First, in Sec2tidoinferent approaches for solving authorship identification are discussed in detail. Then, the reader can find a description of the materials and methods employed in Sectio3n. Next, the results achieved by our runs compared with the baselines proposed by the organisers and the runs of the rest of the participants are shown and discussed in Secti4o.nFinally, the conclusions and further work are presented in Section5.
The first approaches regarding authorship identification relied in (
Author profiling task regarding the software domain has also caught the attention in scientific workshops and conferences. In 2014, the shared taDsketection of SOurce COde Re-use [14] was proposed as a PAN shared task, which consisted in the identification of source code re-use from an unbalanced dataset that consisted in code written in C and Java. A total of five teams participated in this task with a total of 17 runs. Another shared task is describe1d5]a,tin[ which the participants were required to predict the author’s personality from five big traits from source codes written in Java. At the end of the task, a total of 48 runs from 11 participants were sent.
In this work, we describe the three runs submitted by the UMUTeam to the AISOCO’2020 shared task. Accordingly, in this section we describe briefly the dataset provided by the organisers of the task (see Section3.1), and we describe our proposal pipeline (see Sect3i.o2n).
The organisers of the AI-SOCO’2020 shared task compiled the dataset from Codefo1r.cTehsis is a popular website that hosts programming contests in which the participants can propose solutions to solve the proposed challenges. Each time the participants send their programs, the online judge labels each response as accepted or attach some labels to indicate an incorrect response as well as that some constraints regarding time and memory were not fulfilled.
The dataset provided to the participants consisted in 100,000 accepted source codes written in
C++ from a total of 1000 users. The dataset is balanced so there are no authors underrepresented,
which eases the classification task. The whole corpus was released into three datasets: (
In a nutshell, our pipeline can be described as follows: First, each source code is automatically inspected in order to create two alternative versions: one that contains only the comments and another with the source code without the comments. Second, we extract the features which involves character n-grams and the author’s traits. Next, we apply feature selection techniques in order to discard highly correlated features and, finally, we train a machine-learning classifier with the training dataset that we used to predict the labels of the development and testing datasets. This pipeline is depicted in Figur1.e
Regarding the feature extraction, our contribution is grounded in the combination of character
ngrams and author’s traits features. On the one hand, we use character n-grams as the foundation
of our proposal as they can capture stylistic patterns and nuances at lexical, syntactical, and
structural level16[]. Specifically, we extracted combinations between 1 and 8 length applying
TF-IDF with Sub-linear TF scaling. We preserve the letter case both of the source code and the
comments and we set an upper-bound of 400 characters n-grams per length. That is, we select
the best character n-grams of length 1, the best character n-grams of length 2, and so on until
character n-grams of length 8. This resulted in 3200 character n-grams. On the other hand,
after a manual analysis of some instances of the source code provided, we manually handcraft
features to capture author’s traits. These features are organised in the following categories:
(
It is worth noting that some of these features were previously employed by our research
group in UMUTextStats tool to conduct diferent text classification task such as (
Once all the features were compiled, they were filtered by applying a feature selection consisting
of removing those features that have the same value in all samples in order to discard similar
character n-grams. Then, we evaluate diferent supervised machine-learning classifiers, such as
Random Forest, Multi Layer Perceptron, K-Neighbours, Support Vector Machines with linear
kernels, and multinomial Naive Bayes. Out of these, the best accuracy was achieved with
Random-Forest classifier. This algorithm was built using SciKit in Python and the
hyperparameters were: (
At the beginning of the task, the organisers provided three baselines. The first baseline consisted in predicting a random author with an expected accuracy of 0.1%. The second baseline consisted in count English printable characters and to apply a logistic regression model, which achieved an accuracy of 29.252%. The last baseline consisted in a K-nearest neighbours with the TF-IDF of 10,000 features. This baseline achieved an accuracy of 62.128%. After showing the final results of the competition, the organisers included two more baselines based on RoBE2R1T],a [ in which they vary the complexity of the neural networks layers.
The evaluation of the proposal was performed using the Accuracy metric (see Equa1)tiaosn it can defined as the relationship between items correctly identified: True Positives (TP) and True Negatives (TN) with the total of instances classified including False Positives (FP) and False Negatives (FP).
= ( + )/( + + + )
(
First, we show the results of our submissions comparing the accuracy of the development and testing datasets in Tabl2e. As it can be observed, the best accuracy with the development dataset was 91.40% and an accuracy of 91.16% for the testing dataset. In both cases, the best run is based on the combination of the statistical features and the author’s traits. However, we can observe that a similar accuracy can be achieved on both approaches by using only the character n-grams model (run-2). When looking the results of the author traits separately (run 3), we achieved an accuracy of 56.28% which beats the second baseline proposed by AI-SOCO, consisted in a Logistic Regression model on the printable characters (29.92%) as it is not far from the TF-IDF K-nearest neighbours baseline (62.78%).
In Figure4, we show the Information Gain (IG) of some features regarding the author’s traits. We decided to include features for all the categories in order to see if they contribute equality or not regarding authorship identification. In view of the results, we can conclude that those metrics regarding with stylometry are the most discriminating features. For example, it seems that the average length of the source code, comments, and the percentage of uppercase letters have more influence in authorship identification than in determining what source codes resolve the same problem. A similar guess can be done when looking the author’s traits fromcothdee category. We can observe that the number of nested loops is relevant regarding the author profiling. This fact can be related to the way in which programmers focus on how to resolve a problem eficiently in order to win the contest. Finally, we can observe that IG decreases regarding the usage of non-ASCII characters in order to forecast the country of origin of the programmer, or the usage of ASCII-art as signature in their comments.
The comparison of our proposal with the rest of the baselines and the rest of the participants are shown in Table3. It is worth noting that this table contains only the oficial runs. Some extra runs were sent after the oficial end-date, highlighting the results achieved by the Twist Bytes team achieving an accuracy of 94.4%.
As it can be observed from our results compared with the oficial participants and the baselines (see Table3), our proposal reaches the sixth place and outperforms baselines based on deeplearning transformers such as RoBERTa. We achieve an accuracy of 91.16% which indicates that our proposal is quite reliable regarding the identification of the authors. Moreover, our proposal outperforms by far the first baselines provided and the first baseline of the two generated with RoBERTa. However, the second baseline based on RoBERTa, with six layers, outperforms our proposal with an 92.88% of accuracy.
In this paper we describe the participation of the UMUTeam at the AISOCO’2020 shared task for author identification of source code. This was our first participation on one task of this kind and we are very happy with the results. It draws our attention that traditional machine-learning classifiers could outperform some models based on the state-of-the-art transformers, which suggests that the feature extraction and selection plays a crucial role in authorship attribution, as our proposal achieves competitive results reaching the sixth position among the oficial results with an accuracy of 91.16%.
The average accuracy of the participants (without the baselines) is 87.05% with a standard deviation of 0.0623, which indicates that all the participants achieved very competitive results in a dificult task regarding author identification. We consider, however, that the results of this task would be lower if the source code was provided to us without comments. Moreover, as the dataset was compiled from online sources, we suppose that the dataset does not contain any kind of cheating that could make it hard this task. We think that it is not common that participants in online contents do not know each other and that prevents direct plagiarism between the authors as could happen in academic courses.
During our review of literature we found that not all research take comments into consideration. Comments make code easier for understand and they are a valuable data source for author profiling as a comment may include copyright info, explanations about implementation decisions, or even arbitrary patterns that act as section delimi2te2r].s H[owever, they are not always available as, for example, in low-level languages which are common in malware software. As source comments can include text written in natural language, it is feasible to extract linguistic features from comments in order to create a fingerprint of the author’s writing style. For example, with the detection of slang, jokes, or the usage of figurative langua2g3e],[ it is possible to deduce which is the cultural and social background of the author. As further work, we will improve the author traits for authorship identification as we consider that these features provide more interpretability of the results that character n-grams. In addition, we will continue investigating on author identification. In this sense, we will explore programming collaborative environments such as GitHub or Stack Overflow.
This work has been supported by the Spanish National Research Agency (AEI) and the European Regional Development Fund (FEDER/ERDF) through projects KBS4FIA (TIN2016-76323-R) and LaTe4PSP (PID2019-107652RB-I00). In addition, José Antonio García-Díaz has been supported by Banco Santander and University of Murcia through the Doctorado industrial programme. [11] E. Bogomolov, V. Kovalenko, A. Bacchelli, T. Bryksin, Authorship attribution of source code: A language-agnostic approach and applicability in software engineering, 2020. arXiv:2001.11593. [12] P. Mahbub, N. Z. Oishie, S. M. Rafizul Haque, Authorship identification of source code segments written by multiple authors using stacking ensemble method, in: 2019 22nd International Conference on Computer and Information Technology (ICCIT), 2019, pp. 1–6. [13] E. Quiring, A. Maier, K. Rieck, Misleading authorship attribution of source code using adversarial learning, in: 28{tUhSENIX} Security Symposium ({USENIX} Security 19), 2019, pp. 479–496. [14] E. Flores, P. Rosso, L. Moreno, E. Villatoro-Tello, On the detection of source code re-use, in: Proceedings of the Forum for Information Retrieval Evaluation, 2014, pp. 21–30. [15] F. Rangel, F. González, F. Restrepo, M. Montes, P. Rosso, Pan@ fire: overview of the prsoco track on personality recognition in source code, in: Forum for Information Retrieval Evaluation, Springer, 2016, pp. 1–19. [16] J. Houvardas, E. Stamatatos, N-gram feature selection for authorship identification, in: International conference on artificial intelligence: Methodology, systems, and applications, Springer, 2006, pp. 77–86. [17] J. A. García-Díaz, M. Cánovas-García, R. Valencia-García, Ontology-driven aspect-based sentiment analysis classification: An infodemiological case study regarding infectious diseases in latin america, Future Generation Computer Systems 112 (2020) 614–657. doi:https://doi.org/10.1016/j.future.2020.06.019. [18] J. M. Ruiz-Martínez, R. Valencia-García, F. García-Sánchez, et al., Semantic-based sentiment analysis in financial news, in: Proceedings of the 1st International Workshop on Finance and Economics on the Semantic Web, 2012, pp. 38–51. [19] M. del Pilar Salas-Zárate, M. A. Paredes-Valverde, M. Á. Rodriguez-García, R. ValenciaGarcía, G. Alor-Hernández, Automatic detection of satire in twitter: A psycholinguisticbased approach, Knowledge-Based Systems 128 (2017) 20–33. [20] J. A. García-Díaz, M. Cánovas-García, R. Colomo-Palacios, R. Valencia-García, Detecting misogyny in spanish tweets. an approach based on linguistics features and word embeddings, Future Generation Computer Systems 114 (2020) 506–518. [21] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019). [22] D. Steidl, B. Hummel, E. Juergens, Quality analysis of source code comments, in: 2013 21st international conference on program comprehension (icpc), Ieee, 2013, pp. 83–92. [23] M. del Pilar Salas-Zárate, G. Alor-Hernández, J. L. Sánchez-Cervantes, M. A. ParedesValverde, J. L. García-Alcaraz, R. Valencia-García, Review of english literature on figurative language applied to social networks, Knowl. Inf. Syst. 62 (2020) 2105–2137. UhRtLtp:s: //doi.org/10.1007/s10115-019-01425- 3. doi:10.1007/s10115- 019- 01425- 3.