We developed an approach to automatically predict the personality traits of Java developers based on their source code for the PR-SOCO challenge 2016. The challenge provides a data set consisting of source code with their associated developers' personality traits (neuroticism, extraversion, openness, agreeableness, and conscientiousness). Our approach adapts features from the authorship identi cation domain and utilizes features that were speci cally engineered for the PR-SOCO challenge. We experiment with two learning methods: linear regression and k-nearest neighbors regressor. The results are reported in terms of the Pearson product-moment correlation and root mean square error.
Computing methodologies Ñ Arti cial intelligence; Natural language processing; Computational personality recognition; ve factor model; Java source code
Author pro ling is a research eld that deals with the
prediction of user properties (e.g., age and gender prediction
of an author [
A popular personality model used in computational
personality recognition is the ve factor model [
Computational personality recognition has been applied
to various domains, such as essays [
The study of software developers' source code has many
practical applications. For instance, in the education sector
for detecting plagiarism [
In this paper, we introduce a machine learning approach
developed in the scope of the PR-SOCO [
We investigated two classes of features: structure features
dependent on the programming experience of the
programmer (architecture design, code complexity, etc.) and style
1https://github.com/
2https://bitbucket.org/
features related to the code layout that cannot be easily
changed by IDEs (comment length, variable length, etc.).
We intentionally ignored the layout features (line length,
formatting style, etc.) as these features can be easily
modi ed by IDEs using available formatting and code cleaning
functionalities [
The remainder of the paper is structured as follows: Section 2 describes the PR-SOCO challenge and our contribution to solving it. The results of our approach are described in Section 3. We conclude and outline future work in Section 4. 2.
In order to process the students' Java source code, we rst created knife3 which is an open-source wrapper for the two Java parsers QDOX 4 and JavaParser 5. Knife parses source code into classes, methods, parameters, and variables and uses the Spark micro framework to provide the parsed code as JSON. Afterwards, pisco6 consumes the parsed source code, extracts features, and uses machine learning to predict personality traits with linear regression and the k-nearest neighbors regressor. 2.1
The data for the PR-SOCO challenge comprises solutions for di erent Java programming tasks that were uploaded by students and the results of their personality tests. Each of the ve personality traits is represented by a value between 20 and 80. The students were allowed to upload more than one solution per programming task and to reuse code from previous exercises or from external resources. The training set comprises 49 data points and the test set contains 21 data points. It might be di cult to train classi ers and avoid outliers with such a low amount of data.
Figure 1 shows a boxplot for the personality traits in the training set. It can be observed that the median personality scores are between 46 and 50.
The data was not cleaned by the organizers and, therefore, its quality varied. It sometimes contained debug output, empty classes, syntax errors or even Python code. Another in uencing factor is that students occasionally used external code that was copied into the project, e.g., code from programming lectures at other universities. Since the focus of this challenge is the prediction of the students' personality types, a proper ltering step for external code seems reasonable. Otherwise, the prediction of the students' personality types can be in uenced by other coder's personality types. Unfortunately, we were not able to perform a plagiarism check via web search. 2.2
With the parsed source code from knife, we are able to implement several style and structure features for our machine learning approach. 2.2.1
While naming conventions are certainly a controversial topic of debate for software developers (who each have their own programming style), we believe that the naming of classes, methods, elds and local variables is important for the understanding of the code. For instance, overly short or overly long variable names can be di cult to understand. Therefore, the length of such names might correlate with how thoughtful a developer was while writing source code. We decided to use the following style features: F1: Length of method names F2: Length of method parameter names F3: Length of eld names F4: Length of local variables names in methods An interesting observation is that the training data contains a solution from one student who used a local variable name that is 75 characters long while the mean length of local variable names for all students is 4:02 p 3:89q. Such an outlier can be problematic for linear regression. 2.2.2
We investigated ten structure features that we consider to be related to the developer's programming experience. A more experienced developer might tend to write shorter methods with fewer lines of code or less code in general. F5: Number of classes F6: Cyclomatic complexity
The cyclomatic complexity [
F7: Number of methods F8: Number of method parameters F9: Length of methods
We included the length of methods in our feature set since long methods can be an indicator that the single responsiblity principle is violated and the methods could be refactored into multiple smaller methods. In our experiments, we tested the length of methods in terms of the number of lines and in terms of characters (without indentation).
F10: Number of elds per class F11: Number of local variables in methods F12: Duplicate code measure
We noticed that some students uploaded multiple solutions with very similar looking code. They copy pasted methods from one class to another while performing small changes to the code. This motivated us to check whether a student uploaded two methods that have a high overlap.7 The duplicate code measure was implemented as a binary feature. The code lines from all methods were tokenized and converted into bag-of-words models. Afterwards, we calculated the pairwise cosine similarity between all methods and considered two methods mi mj to be a duplicate of each other by comparing their similarity with a threshold :
#1 0 if cospmi; mjq ¡ otherwise (1) We empirically estimated 0:9. A student uploaded duplicate code if DCMpmi; mjq 1 for two of his or her methods mi mj.
F13: Usage of IDE default template text
We noticed that some students did not remove or change default IDE text content and implemented this behavior as a binary feature.
F14: Ratio of external library usage
Developers are nowadays able to share libraries via dependency managers, which allow developers to use implementations of other developers without the need to write all the code from scratch. In the case of Java, code can be grouped into packages which can be imported. This feature calculates the ratio of imports from standard Java packages to all imports. 2.2.3
F15: Number of empty classes
We noticed that the submitted solutions sometimes contain empty classes. This might be an indicator of how thoroughly a programmer works or how important cleaning up source code is for him/her.
F16: Ratio of unparsable solutions
This feature captures that students uploaded code that is not valid Java code. A student's solution might contain syntax errors that made it unparsable for QDOX. This is especially the case where students uploaded debug output or Python code. This feature is implemented as the ratio of parsable to unparsable solutions. 7This is not to be confused with a plagiarism check between the solutions of di erent students.
RM SE c °n i 1pyi n yiq2 3.2
To optimize the hyperparameters (meaning parameters that do not need to be learned as part of the model, e.g., summary statistics for features and parameters that have to be set manually for learning algorithms), we performed It re ects how careful the students were in following instructions or in testing if their code meets the speci ed requirements.
Although it might be useful to analyze code comments (e.g., the average comment length), we decided not to use features based on code comments since line and block comments may be polluted by code that was commented out. 2.3
Since most of our features are on a class or method basis, we need to aggregate their values to a vector representation of a xed length in order to deal with di erent numbers of solutions, classes, elds, methods, and parameters. In order to make our features more robust against outliers, we rst aggregate the values per solution with a summary statistic (e.g., mean, variance, range) and then calculate their mean. Given that the choice of a summary statistic is not apparent, we decided to choose it via cross-validation on the training set.
Additionally, we noticed di erent behaviors of the features depending on the personality trait. This encouraged us to estimate an optimal feature set for each personality trait individually. Since we have 16 features and the power set of all of these features contains too many combinations, it is not computationally feasible to search the entire feature space. First, we performed a cross-validation on the training set with all 16 features. Additionally, we experimented with subsets of our features and chose the subset that performed best during the 10-fold cross-validation on the training set. 3.
In total, 11 teams participated at the PR-SOCO shared task and submitted 48 runs. 3.1
Two evaluation metrics were proposed for the evaluation of the submissions. To measure the correlation between the predicted values and the gold standard values, the Pearson product-moment correlation coe cient (PC) was used. Moreover, the root mean square error (RMSE) was used to measure the average amount of prediction errors. For a vector y P Rn of truth values and its corresponding prediction vector y P Rn, the equations of the Pearson product-moment correlation and RMSE are shown in Equations 2 and 3 respectively: (2) (3) r b°n i 1 pyi °n i 1pyi yqpyi
yq yq2b°n i 1 pyi yq2 where y and y denote the average values of the vectors y and y respectively and n represents the number of data points. an exhaustive 10-fold cross-validated grid search over all hyperparameters for each personality trait individually. We used k-nearest neighbors regressor (runs 3 and 4) and linear regression (runs 5 and 6), and optimized once to minimize RMSE (runs 4 and 5) and once to maximize the Pearson correlation (runs 3 and 6). After observing the results of the cross-validation, we noticed that none of the two learning algorithms could outperform the other one. As a result, we decided to choose the learning algorithm for each personality trait individually and chose the one with the higher cross-validation score on the training data. This resulted in two more runs since we once optimized for the Pearson correlation (run 1) and once for RMSE (run 2).
The task organizers also provided two baseline
approaches: a bag of character 3-grams with frequency weight
and an approach that always predicts the mean value
observed in the training data [
The settings of the best runs, including the selected features and the applied learning algorithm, together with their corresponding RMSE values, are summarized in Table 1. Note that the numbers listed under selected features correspond to the feature indexes introduced in Section 2.2. It is observable that the k-nearest neighbors regressor has superior results over the linear regression method for all personality traits. As we discussed previously, several extracted features include outliers, which can cause large residual values by linear regression. By contrast, the k-nearest neighbors regressor is capable of coping with outliers and is preferred by the grid search.
It is also observable that the features length of eld names (F3), duplicate code measure (F12), usage of IDE default template text (F13), and number of empty classes (F15) are among the most powerful predictors of personality traits. In Figure 2, we compare our results regarding the RMSE measure to the other participants. The results not included between the whiskers are considered as outliers and are represented by empty circles. For each personality trait, the lled circle indicates the RMSE values of our best runs. For all personality traits except agreeableness, our proposed approach had RMSE values lower than the median. In particular, we achieved the lowest RMSE among all participating teams for the personality trait conscientiousness.
For comparison, we also provide the settings of the best runs regarding the Pearson correlation in Table 2. Similar to the case of RMSE, the features F3, F12, F13, and F15 were identi ed to result in higher Pearson correlations. For the personality traits extroversion and agreeableness, based on the grid search results, linear regression resulted in higher Pearson correlations in comparison to the k-nearest neighbors regressor. Nevertheless, linear regression results in negative correlation coe cients for both traits. The Pearson correlations of our best runs for the individual traits can be compared to the other submissions in Figure 3. 4.
We presented our approach to automatically predict personality types in the ve factor model from Java source code for the PR-SOCO challenge 2016. Our architecture consists of the two components knife and pisco which we made publicly available on GitHub. We used knife to parse the source code and pisco to extract features and to predict personality traits.
We achieved the best root mean squared error for the personality trait conscientiousness among all 11 participating teams. For the personality traits neuroticism and openness, our best runs ranked 3rd and 9th, respectively, based on 48 runs. Our RMSE result for the trait extroversion was better than the median. Unfortunately, the results in the dimension openness were not satisfactory. The results in terms of the Pearson correlation were mixed since we achieved positive and negative correlations.
In our future work, we want to crawl external resources in order to determine if pieces of the source code are plagiarized. We also want to evaluate non-linear machine learning approaches. During our data analysis, we identi ed that the developers sometimes used more than one natural language, for instance in comments or in variable names. We would like to investigate this behavior for possible correlations with personality types. In our work, we ignored layout features since they can easily be modi ed by an IDE. However, we could investigate if the developer is consistent in using the auto formatter of his or her IDE. k-NN k-NN k-NN k-NN k-NN Method k-NN LR k-NN LR k-NN X X X X
X X
X
X X X X
X X
This work was partially funded by the PhD program Online Participation, supported by the North RhineWestphalian funding scheme Fortschrittskollegs" by the German Federal Ministry of Economics and Technology under the ZIM program (Grant No. KF2846504), and by the ISTHochschule University of Applied Sciences. Computational support and infrastructure were provided by the \Centre for Information and Media Technology" (ZIM) at the University of Dusseldorf (Germany).