This paper describes the participation of the CAPS-PRC system developed at the LMU Munich in the personality recognition shared task (PR-SOCO) organized by PAN at the FIRE16 Conference. The machine learning system uses the output of a Java code analyzer to investigate the structure of a given program, its length, its average variable length and also it takes into account the comments a given programmer wrote. The comments are analyzed by language independent stylometric features, including TF-IDF distribution, average word length, type/token ration and more. The system was evaluated using Root Mean Squared Error (RMSE) and Pearson Product-Moment Correlation (PC). The best run exhibited the following results: Neuroticism (RMSE - 10.42, PC - 0.04), Extroversion (RMSE - 8.96, PC - 0.16), Openness (RMSE - 7.54, PC - 0.1), Agreeableness (RMSE - 9.16, PC - 0.04), Conscientiousness (RMSE - 8.61, PC - 0.07).
machine learning; Big Five personality traits; source code analysis; abstract syntax tree
The main purpose of the task is to investigate whether
it is possible to predict personality traits of programmers
based on the source code written by them [
Based on the available research on the Big Five
psychological traits [
Since each programmer/author has submitted more than one program, we approach the problem from two di erent angles:
1) the feature vectors are extracted for each programmer, by rst extracting them for each program and then averaging all the underlying feature vectors into one single feature vector for the author. The classi er learns based on a single feature vector for each author, where the author represents one sample in the dataset.
2) the classi er is trained at the level of programs. Each program inherits the trait value of its author. The feature vectors are extracted for each program and then the classi er regards each program as a training instance. To get back to the level of authors (while the nal prediction should be done for the author), the predictions are averaged for each program belonging to a certain author. The nal result is a single prediction for each author based on the predictions produced for each underlying program. 2.2
2.2.1
We use a grammar speci cally designed for the analysis of a programming language , which in the context of the task was the Java programming language. The grammar combined with a parser provides a semantic representation of the source code called an abstract syntax tree (AST). Compared to normal parse trees there are some potential advantages. First, the generation of an AST can be interpreted as a normalization step of our feature generation. In contrast to the original source code, which has inconsistencies like whitespace characters or other unneeded characters, the AST represents a concise version of a given program. This also makes the generation of meta-features (compositions of di erent base features) more simple, due to the strict representation of all, to the compiler important, parts of the program. Additionally, the representing syntax tree is not necessarily bound by the original syntactic rules of the original programming language which allows for generalizations of the source code to occur.
In our approach, we use the frequency distribution of all
known entities in the grammar to build a feature list for a
given program. This shallow use of the AST provides 237
features for a given source code analysis. Some examples
would be the T ype of variables or the nature of a
statement(do, for, while, etc.) The implementation of the AST
is made possible with the help of ANTLR parser [
In addition to the AST, we used additional features for the source code and also the comments. The following is an exhaustive list of all additional features used.
Code-based features: length of the whole program (in lines of code, in characters), the average length of variable names, what indentation the programmer is using (tabs or spaces).
Comment-based features: type/token ratio, usage of punctuation marks, TF-IDF, the frequency of comments (block comments and inline comments separately), average word length.
Author-level based features: number of programs submitted (see Table 2), average length of programs in lines of code. 2.3
We experimented with a number of Regression classi ers like Linear Regression, Ridge Regression, Logistic Regression and Gradient Boosted Regression. In addition, we have tried to detect the outliers with the RANdom SAmple Consensus (RANSAC). The nal system implementation did not use RANSAC, since it delivered worse results. Although, this technique should be further investigated with a bigger dataset.
We have submitted our nal runs based on two machine learning algorithms: Gradient Boosted Regression and Multinomial Logistic Regression. Furthermore, Gradient Boosted Regression was evaluated on the level of authors and the programs level, while the Multinomial Logistic Regression
Conscientiousness
Extroversion Neuroticism
Openness was implemented on the level of authors.
The rst classi cation approach is based on Gradient Boosted
Regression with least squares regression as its loss function,
1100 estimators, 5 as the maximum depth of the
individual regression estimators, and the learning rate of 0.1. This
approach also utilized 2 test for the feature selection to
choose only the best 200 features from the AST feature
extraction pipeline. This approach was implemented using the
scikit-learn Python library [
The second approach is based on the Multinomial Logistic Regression model with the l2-regularized squared loss as its objective function. That is, each feature was multiplied with a trait-speci c weight, and the result of this linear combination was the input to a sigmoid activation. As the output of this prediction is in the range [0,1], we re-scaled the trait values in the training data to the same range for computing the squared loss.
Training was done using stochastic gradient descent with constant learning rate, and parameters were tuned on the held-out development set using random search. The search space of the parameters was: learning rate 2 0:01; 0:1; 1, number of training epochs 2 f10; 20; 50; 100; 200; 500g, regularization 2 f0; 0:001; 0:01; 0:1; 1g, (mini-)batch size 2 f1; allg. The best con guration was: learning rate: 1, training-epochs: 2, regularization: 0.6, batch-size: all. This approach was developed with theano Python library [1]. 3.
The dataset included 49 programmers in the training set (with 1790 programs in total) and 22 programmers in the test set (772 programs). Final evaluation was done with two di erent evaluation metrics: Root Mean Squared Error (RMSE) and Pearson Product-Moment Correlation (PC). In the Gradient Boosted Regression approach (GBR Approach), the system was tuned to maximize both of these metrics at the same time, while the Multinomial Logistic Regression one (MLR Approach) concentrated on RMSE. Table 3 gives a detailed overview of the results achieved using Multinomial Logistic Regression at the level of authors. Table 4 shows the results achieved using the Gradient Boosted Regression approach at the level of authors and the level of programs.
In general, the results are low using both RMSE and PC
and only slightly outperform the performance of the baseline
approaches (see Table 5). Two baselines have been provided
by the task organizers [
2) always predict the mean trait value of the training dataset.
Conscientiousness Extroversion Neuroticism
Openness
This paper describes the system that given a source code collection of a programmer, identi es their personality traits. While the RMSE and PC scores proved promising during development, further investigation suggested the dataset may be too small to create an e ective machine learning system. The compiler style feature generation process using ASTs combined with several custom features could serve as future baselines for similar tasks. 4.1
The task would bene t greatly from an expanded training corpus (more samples per programmer, more programmers). The value distribution of the training set is also an important point. The current training set exhibits normal distributed scores for each Big Five trait. A more robust system could be created when using an equal number of samples within low, mid and high value range.
Additionally, further feature engineering, additional statistical analysis of the AST output, and transferring strategies of other NLP tasks involving syntax trees onto the current task could improve the system.