This paper describes our participation in the PAN@FIRE Personality Recognition in Source Code (PR-SOCO) 2016 shared task. We have proposed two di erent approaches to tackle this task, on the one hand, each code sample from each author was taken as an independent sample and it was vectorized using word n-grams; on the other hand, all the code from an author was taken as a unique sample, and it was vectorized using word n-grams together with handcrafted features that may determine the personality traits of an author. Regardless of the approach, a regression model was trained to classify the personality traits of the author of a sample of source code. All the systems we have submitted to be evaluated have achieved a root mean square error (RMSE) below the mean RMSE of the participants of the shared task. Moreover, one of our runs, the one that included the hand-crafted features, held the best result in the personality trait Agreeableness. This suggests that in the absence of enough independent samples to train a machine learning system, hand-crafted features are able to obtain better results.
One of the new emerging research areas in Natural
Language Processing (NLP) is Personality Recognition (PR),
which seeks to classify the personality traits of the author
of a text. In psychology, Norman et al. (1963) [
In this paper we describe our participation for addressing the PR-SOCO task. The rest of the paper is organized as follows. Next section is devoted to de ne the Personality Recognition task. In Section 4 the model proposed is described. Following, in Section 5, the results achieved are presented. Finally, in Section 6 our results are discussed, and future work is proposed. 2.
The main objective proposed by the organizers of the
PRSOCO shared task was to predict the personality traits of
developers given a collection of their source code. The
personality of a developer was determined following the Five
Factor Theory or Big Five [
The organizers have gathered 60 samples of source code from 60 di erent programmers. In order to train the partic1Free/Libre Open Source Software https://www.gnu.org/ philosophy/ oss-and-foss.en.html ipants' models, 49 samples were provided, and 21 were held to validate the results. Each sample consists of a collection of source code written in Java. In Table 1 the total number of training and test samples is shown.
We have studied the distribution of the number of samples
available for each value of each trait to classify depending
on whether we considered the number of code samples as
independent (number of pieces of source code) or not (number
of authors). Figures 1 and 2 show the number of samples
available for the trait Agreeableness. Similarly, the rest of
the traits presented an equivalent distribution of the
number of training samples available. It should be noted that
the number of authors, and therefore the number of training
samples available might be insu cient to adjust the
parameters of a machine learning system adequately. If we consider
each sample of code as an independent training sample, we
will have more training samples available, which might be
useful for ghting the curse of dimensionality [
Noteworthy, we are not exploiting any external dataset or resource to either train or ne-tune our models.
Provided that the number of data samples available for training machine learning models is crucial, two approaches were evaluated. We have proposed an Author Based (AB) approach and a Code Based (CB) approach.
Author-Based approach uses all the samples of code from an author including hand-crafted features in addition to the words n-grams. The features considered were: the number of samples of code that implemented the same class (hf1), the number of allocations (hf2), the number of loops (hf3), the appearance of pieces of code suspicious of plagiarism (hf4)2, the number of imports (hf5), the number of functions (hf6), the number of exceptions handled (hf7), the number classes developed (hf8), the number of di erent classes developed (hf9), the number of comment lines (hf10), and the number of prints (hf11).
Code-Based approach assumed independence between the samples. This nave assumption allowed us to train with 1,741 samples. The CB approach relies solely on the n-grams found in each piece of code, without considering any kind of aggregated information from each author. It generates a prediction for each sample of source code. Therefore, the nal prediction for an author is the mean of all the predictions obtained for each piece of code that this author wrote.
As text representation, several vectorizer methods were
evaluated for each approach. The vectorizers considered
were: the term frequency-inverse document frequency
(tfidf) from one to four words (t df-words), the tf-idf from
one to four n-grams of words ignoring the terms that have
a frequency strictly higher than the threshold 0.5 and
applying sub-linear scaling (sublinear-1:4), idem but
exploring n-grams from one to six words (sublinear-1:6), the tf-idf
from one to six characters (t df-chars), and a bag of words
(BOW). We carried out a preprocessing phase where code
snippets (e.g. a sequence of words that de ne a loop) were
replaced by tokens. However, the systems that included this
2We supposed that those samples of code that
instantiate classes that do not belong to the standard library are
suspicious of plagiarism, e.g. the class
SeparateChainingHashTable.
phase obtained worse results that those systems without
preprocessing. This phenomenon was previously reported in the
author pro ling literature [
Moreover, both approaches used a regression model to classify the authors automatically. The machine learning algorithms considered were: an Epsilon-Support Vector Regression (SVR) model, a Linear Regression (LR) model, a Linear Least Squares model with l2 regularization and = 0:5 (Ridge), Linear model trained with L1 prior as regularizer and = 0:5 (Lasso), a Multi-layer Perceptron classi er (MLP), a Decision Tree Regressor (DTR), and a Random Forest Regressor (RFR). The task was also evaluated as a classi cation problem using Support Vector Machines, and Random Forest. Nevertheless, the classi cation approach behaved worse than the regression approach. Therefore, this classi cation approach was discarded.
We have developed a pipeline using scikit-learn [
X trait2A;C;E;N;O
P5 fold=1(RM SEtrait&fold)=5 5 This has allowed us to obtain models with a competitive performance for all traits measured with the RMSE. Our systems were only optimized for the RMSE, which might a ect the performance using the Pearson Correlation since there is no reciprocity between the RMSE and the Pearson Correlation. Conversely, in the AB approach, the best handcrafted combination was selected applying an ablation test, and these features were concatenated to the word n-grams of the best model obtained for the CB approach.
Hereafter, we will describe the results achieved by our best models. Table 2 shows the RMSE of our best models at development time. Due to the computational complexity of performing the grid search over two metrics, we have only used the RMSE to adjust our models.
After selecting the best model for the Code-Based approach, we have selected the hand-crafted features that improved the classi cation in the Author-Based approach. The hand-crafted features selected were: the number of samples of code that implemented the same class hf1, the appearance of pieces of code suspicious of plagiarism hf4, the number of classes developed hf8, and the number of di erent classes developed hf9.
We submitted ve di erent models. Those that performed better during the development phase, which were: 1. run 1: a Code-Based approach using sublinear-1:4 and
Ridge. 2. run 2: a Code-Based approach using sublinear-1:6 and
Ridge. 3. run 3: an Author-Based approach using sublinear-1:4, the following hand-crafted features: hf1 hf4 hf8 hf9 hf10 and Ridge. 4. run 4: a Code-Based approach using sublinear-1:4 and
Logistic Regression. 5. run 5: a Code-Based approach using sublinear-1:6 and
Logistic Regression.
Two baselines were provided by the organizers: a bag of words 3-grams with frequency weight (bow), and an approach that always predicts the mean value observed in the training data (mean). The evaluation results for each personality trait over the test set can be found in Table 3.
Eleven teams have presented their respective systems. In total, 48 systems were submitted for evaluation. All the systems we have submitted have performed better than the mean of the systems proposed using the RMSE. Despite the results achieved during the development phase, our best performing system was the one that followed the Author-Based approach. This system was able to achieve the best RMSE result in the personality trait Agreeableness. Nevertheless, our systems' predictions did not nd a correlation with the gold standard following the Pearson coe cient metric. Besides, neither the baselines proposed nor the best performing participants were able to nd a signi cative correlation. The best correlation found by the participants was 0.62 for the trait Openness, which can not be considered a strong positive correlation. 6.
In this paper we have presented our participation in the PAN@FIRE Personality Recognition in Source Code 2016 shared task. Two approaches were proposed an AuthorBased approach and a Code-Based approach. The AB approach performed better for all the traits. This could be explained because the samples we used to train the systems that followed the Code-Based approach were not independent. Therefore, the results we obtained in the development phase correspond to over- tted systems.
However, provided that we did not have enough samples we still need to include proper techniques for data augmentation. If we would be able to get more labeled data, new approaches could be studied such as deep learning methods and word embeddings for text representation.
Noteworthy, the minimum error achieved by the participants' proposals in the RMSE is close to the baseline models for all the personality traits, and only for some traits a correlation with the gold standard was found. This highlights the complexity of the task. Therefore, personality recognition in source codes is an open problem and new NLP approaches could improve the performance of the systems. W. Daelemans. Overview of the 3rd author pro ling task at pan 2015. In CLEF, 2015. [18] R. S. Rubin, D. C. Munz, and W. H. Bommer.
Leading from within: The e ects of emotion recognition and personality on transformational leadership behavior. Academy of Management Journal, 48(5):845{858, 2005.