Personality recognition in source code constitutes a novel task in the eld of author pro ling on written text. In this paper we describe our proposal for the PR-SOCO shared task in FIRE 2016, which is based on a shallow recurrent LSTM neural network that tries to predict ve personality traits of the author given a source code fragment. Our preliminary results show that it should be possible to tackle the problem at hand with our approach but also that there is still room for improvement through more complex network architectures and training processes.
Written text can tell us a lot about its author.
Demographic information such as age, gender or speci c
personality traits of the author can be inferred by a human expert
by the sole observation of a written text fragment [
Source code is another form of written text, and it is becoming very accessible as software developers are now able to easily publish their work on the Web through services such as Github1 or Bitbucket.2 Although more constrained and formal than natural language, source code text may also have something to tell us about its author, as there is still room for personal preferences in its writing. For instance, 1https://github.com/ 2https://bitbucket.org/ some coders tend to use block delimiters even when they are not necessary, or to add a certain number of blank lines in order to clearly separate two function declarations. Morevoer, variable and function names are custom made by the coder, and commentaries include information in natural language. Therefore, it sounds reasonable to take advantage of this type of patterns to attempt to extract information about the author of a source code fragment, which constitutes a novel task in the author pro ling eld.
In this work, we describe our contribution to the
Personality Recognition in SOurce COde (PR-SOCO) shared
task [
The results obtained with our shallow networks are
encouraging with respect to the root mean squared error metric
(RMSE), which is aligned with the smoothed mean absolute
error criterion employed in our training process. However,
they do not perform so well for Pearson Correlation (PC),
which we have not considered at this time. We have also
found that the use of more layers in our networks can
improve their performance, agreeing with previous work [
There has been a recent surge of interest on author
proling related to personality recognition [
For written text, traditional author pro ling approaches
tend to rely on lexical and syntactical features, such as
identi cation of key words, part-of-speech tags [
However, author pro ling is not restricted to written text.
Mairesse et al [
Regarding the psychological aspects of this work, the
proposed task relies on the so-called Big Five Theory [
Written text, either natural language or source code, can be viewed as a sequence of basic elements such as sentences, words or characters, to name a few possibilities. Given a particular domain, we can choose the sequential view of the input text which best ts our needs. In our case, source code is full of reserved keywords such as if, return or while, so a word-based approach may seem appropriate at rst, as the word vocabulary seems to be relatively xed and reduced. However, the problem then comes with the custom names given by the coder to classes, variables, functions, etc. which have an unpredictable nature and do not t well in a strict vocabulary approach. Furthermore, it would be interesting that the vocabulary of sequence elements was as small as possible since this a ects the required size of the input layer of our models. In order to keep things simple, we will not follow a word-level or character-level approach but a pure byte-level approach, thus limiting the size of the vocabulary to 256 possible byte values.
To process these byte sequences we will use recurrent
neural networks, as they are a perfect t for sequential data [
It is worth noting that, in contrast with previous work (see Section 2), our approach does not require a feature engineering phase as neural networks, in their training processes, re ect the most interesting features from the input domain in the values of their parameters, sometimes referred to as weights.
Lastly, we opted for feeding the network with input
instances (sequences) which are independent from each other,
so that the important dependence relationships (patterns)
between elements (bytes) of a sequence may be observed
by our model. For this reason, we have constructed
sequences from whole source code packages. As these
sequences can be quite long (see Section 4), traditional
recurrent networks may have problems to recall important
information extracted at the beginning of the input sequence
while they are processing the last elements of it. In order
to address this limitation, we use long short term memory
(LSTM) units as the neurons in the hidden layers of the
network [
Our models were implemented using the scienti c
framework Torch [
First of all, we have preprocessed the training corpus given by the PR-SOCO organization to best t the training procedure of the neural network. In this vein, we have merged the personality information (i.e. the personality trait values) from each one of the 49 developers right after the end marker of the package of their source code les. This way our input format contains the input sequence to the network above the marker and the desired output one line below. Then we merged all resulting les into a single one and shu ed the instances, resulting in a total of 1600 instances. The validation dataset was built taking the rst 141 instances from the resulting le, leaving the rest for training.
The shortest meaningful sequence in the training corpus has length 34, the longest 27654 and the average one is approximately 4823 characters long. Regarding the personality scores in the training corpus, their values fall in the range 20{80 and their means are: 49.92 for neuroticism, 45.22 for extroversion, 49.51 for openness, 47.02 for agreeableness and 46.37 for conscientiousness.
In order to bene t from the processing power of the GPU, we gathered input sequences into batches. Since all sequences in a given batch must have the same length, we padded the shorter sequences with zeros at the beginning. Unfortunately, this was not exactly the case throughout our experiments, and until very recently the padding was being added to the end of shorter sequences instead, giving rise to a bug were these sequences were automatically discarded in the training process. This bug did not a ect experiment settings with a batch size of 1.
We show in Figure 2 how some sample input texts are represented as sequences of byte values which are then gathered into batches where they are appropriately padded with zeros at the beginning.
The training process consists of 100 full cycles (epochs) through the training corpus. The time needed to accomplish this depends on the complexity of the network and the batch size used. As an example, one epoch in a network with two hidden layers of 300 neurons each and a batch size of 1 can take up to 4.8 hours while using a batch size of 10 reduces the training time to 2.6 hours. Similarly, adjusting the batch size to 10, a network formed by a 300 neurons hidden layer needs 2 hours to train through one epoch. It is important 5.5 5 4.5 4 3.5 2.5 1.5 2 1 rrro 3 e to note that this gures would be lower if we did not run multiple training processes in parallel.
In Figure 3 we show our preliminary experiments to attest for the capacity of our models to tackle the task at hand. Although the observed behaviour in training time of these models was acceptable and invited us to use them against the test corpus (which we will describe shortly), they were a ected by the padding bug mentioned earlier and cannot be considered as clear evidence of the performance of the models. Since the batch size was established to 10, the bug caused the models to train with a tenth of the total training and validation instances. In any case, as we can see in the graph, it seems bene cial for the generalization capabilities of a neural network trained for this task to add at least one extra hidden layer to its architecture (steady training data t and lower nal validation error), while adding neurons to a sole hidden layer results in a counterproductive measure.
The test corpus supplied by the PR-SOCO organization did not undergo a preprocessing stage such as the one described above. In this case we have to evaluate 21 developers whose source code is fragmented in 750 test instances. The maximum sequence length observed is 33550, the minimum 114 and the mean 3743.
For the ve runs performed on the test data, we have used ve di erent models di ering in the number of hidden layers and batch size to be employed, which are related in Table 1.
All of them have 300 neurons per hidden layer and have been trained with the whole training corpus, including the validation part. Note that the di erence between run04 and run05-v2 is the training time, longer in the latter case. The only run a ected by the padding bug was run02.
In Tables 2 and 3 we can see our o cial results obtained for the PR-SOCO task. In general, the correlation scores are quite low while the RMSE gures are acceptable (considering that they beat the task average) except for run04, whose better results in correlation might be attributed to a mere coincidence. On the other hand, we see that RMSE scores for run02 are quite good despite being the only case a ected by the batch padding bug mentioned above. This fact seems to be related with the bene ts provided by the extra network layer that the corresponding model has with respect to the rest. We can also observe, in the di erence between run04 and run05-v2, that allowing the model to train for longer periods of time is indeed useful to attain good performance. Finally, at this time the data available do not allow us to extract any particular conclusion about the in uence of the batch size on our results.
It is worth noting that, unfortunately, we could not re-run the 2-hidden layer network without the padding bug against the test corpus because of time constraints. Nevertheless, in order to con rm the hypothesis that adding an extra layer to the network is bene cial to its performance, we have conducted some a posteriori experiments with the training corpus. In Figure 4 we can see how the 2-hidden layer network obtains, once again, better generalization capabilities than the 1-hidden layer network.
Source code is a form of written text which has been becoming very accessible in recent years. While more con50 epoch 10 20 30 40 60 70 80 90 100 strained and formal than natural language due to its very nature, it also allows for some personal preferences to pour down into its structure and content, giving rise to the possibility of author pro ling on it.
In this paper we have shown our proposal for personality recognition in source code. Viewing such text as a sequence of characters (or bytes), we have used shallow recurrent neural networks as our personality trait predictors. In order to maximize the pattern detection capabilities of our model, we have fed entire source code packages as sequence inputs to the network. The network learning criterion was a smoothed mean absolute error, less sensitive to outliers than RMSE or the mean absolute error.
Given the encouraging results obtained, we think that our approach may be a viable one to tackle this problem. On one hand, the RMSE gures obtained, which are aligned with the criterion we were optimizing for, are positive considering that we have used a shallow network, whose expressivity power is limited, with large input sequences. On the other hand, we have found some hints pointing at a better performance in the case of using deeper neural networks and training them for longer periods of time, which may constitute immediate ways of improving our results.
As future lines of work, we will try to improve our results by adding more layers to our neural network|in a one-byone fashion until we see no more signi cant improvement|, and also by introducing a new training criterion that considers the correlation between instances. Another interesting research line would be the study and visualization of the activation mechanisms which occur within the network at evaluation time in order to try to interpret the patterns, or features, that the model has previously extracted during the training phase. In other words, to analyse the behaviour of the network to try to observe human interpretable patterns and thus distil the knowledge condensed in the network.
This work has been partially funded by the Spanish Ministerio de Econom a y Competitividad through projects FFI2014-51978-C2-1-R and FFI2014-51978-C2-2-R, and by Xunta de Galicia through an Oportunius program grant.
We gratefully acknowledge NVIDIA Corporation for the donation of a GTX Titan X GPU used for this research.