Personality Recognition in Source Code Working Note: Team BESUMich

Abstract

In this paper, we describe the results of source code personality identification from Team BESUMich. We used a set of simple, robust, scalable, and language-independent features on the PR-SOCO dataset. Using leave-one-coder-out strategy, we obtained minimum RMSE on the test data for extroversion, and competitive results for other personality traits.

-

personality; source code; regression; RMSE; Pearson correlation; extroversion; neuroticism; openness; agreeableness; conscientiousness

INTRODUCTION

Personality is an important element of human sociology and psychology. It determines and underscores our day-to-day decisions, shopping and dating behaviors, educational aptitude, and emotional intelligence – to name a few. It is therefore no coincidence that the source code a programmer writes tends to be influenced by his/her personality. While the traditional Author Profiling task consists of predicting an author’s demographics (e.g., age, gender, personality) from his/her writing, in the PR-SOCO shared task [ 15 ] the goal was to predict a programmer’s personality from his/her source code. Personality traits influence most human activities, including but not limited to the way people write [ 4, 14 ], interact with others, and make decisions. For example in the case of programmers, personality traits may influence the criteria they use to select which open-source software projects to participate [ 11 ], and the way they write and organize their code.

In PR-SOCO, given a source code collection of a programmer, the goal was to identify his/her personality. Personality was defined according to five traits using the Big Five Theory [ 6 ]: extroversion (E), neuroticism (S), agreeableness (A), conscientiousness (C), and openness to experience (O). Each programmer was rated on a numeric scale on each of the five traits. Training and test data consisted of such ratings, along with code snippets from the developers. Since the response variable was a real number rather than a class label, we used a regression framework to model the supervised learning problem. We used a set of simple, robust, scalable, and language-independent features (Section 3), and optimized the root mean squared error (RMSE) averaged across all five traits in a leave-one-out cross-validation strategy. While applied on the test data, one of our runs achieved the minimum RMSE for extroversion.

The rest of this paper is organized as follows. We discuss relevant literature in Section 2. Section 3 gives details on the PR-SOCO task, especially the data and task description. We also describe our features, regressors, and experimental methodology in this section, especially delineating why we chose these features instead of codestyle features. Section 4 provides experimental evaluation, and important insights that we gained along the way. We conclude in Section 5, outlining our contributions, limitations, and directions for future research. Relevant terminology is introduced as and when they first appear in the paper. 2.

RELATED WORK

Personality recognition usually falls under the purview of author profiling [ 2, 3, 8, 14, 16 ]. Argamon et al. [ 2 ] showed that authors of informal texts could be successfully classified according to high or low neuroticism, and high or low extroversion. Four different sets of lexical features were used: a standard function word list, conjunctive phrases, modality indicators, and appraisal adjectives and modifiers. Appraisal use was found to be the best predictor for neuroticism, and function words worked best for extroversion. An SVM SMO classifier was used on essays written by college students.

Argamon et al. [ 3 ] extended this study in 2009 to take into account gender, age, native language, and personality. Three different corpora were used, in conjunction with content-based and stylebased features. Bayesian Multinomial Regression (BMR) was used as classifier [ 9 ]. Style features were found to be very informative for personality traits. Most discriminative style features indicated that neurotics tended to refer to themselves.

Estival et al. [ 8 ] created an email dataset consisting of ten traits – five demographic (gender, age, geographic origin, level of education, native language), and five psychometric (the same ones mentioned in Section 1). They further designed a Text Attribution Tool (TAT), and subjected their data to this tool for rigorous validation, normalization, linguistic analysis, processing, and parsing. Three types of features – character-level, lexical, and structural – were extracted. It was shown that a combination of features performed best, and beat the baseline.

Rangel et al. [ 16 ] presented the Author Profiling Task at PAN 2013. The task consisted of age and gender classification in English and Spanish, and a special exercise on identifying adult-adult sexual conversations, and fake profiles for sexual predators. The task was extended by Rangel et al. in 2015 [ 14 ] to include four languages (English, Spanish, Italian, and Dutch), Big Five Personality traits, and Twitter users. The participants used content-based represents the power-law exponent of the features (bag of words, word n-grams, term vectors, tfidf n-grams, named entities, dictionary words, slang words, ironic words, sentiment words, emotional words), and style-based features (frequencies, punctuation, POS, verbosity measures, several different tweetspecific statistics such as mentions, hashtags, and URLs). The highest accuracies in gender identification were achieved in Dutch and Spanish with values over 95%.

While all the above studies are important, and ground-breaking in some cases, we found none that looked into personality recognition from source code. From that perspective, the PR-SOCO shared task breaks a unique ground [ 15 ]. 3.

TASK DESCRIPTION

The PR-SOCO task [ 15 ] released a set of text files for 70 programmers – 49 as training data, and 21 as test. Each text file consisted of several source code snippets. The number of code snippets vary significantly from programmer to programmer. We show the distribution of snippets in Table 1. It is to be noted that the distribution forms a power law with exponent = 2:86 for the training data, and 3:06 for the test data (statistically significant in both cases; cf. [ 5 ]). Furthermore, there is considerable similarity among the programmers in the way they wrote code. This stems from two factors: (a) the programmers were given standardized coding questions (prompts) to implement, and (b) they were not precluded from using the Internet and copy-pasting code thereof. This resulted in substantial similarity between programmers. Moreover, oftentimes programmers wrote comments and named variables in non-English languages (we detected Spanish in manual investigation), and also submitted run information (which should ideally remain separate from the code).

All the above observations indicate that the data contains much noise. While we could have opted for a serious filtering and preprocessing step, such procedure was considered potentially harmful, because we could end up removing useful information such as coding style and unique developer signature. Note also that much of the source code is not natural language, so standard NLP tools such as parsers, named entity recognizers, and POS taggers would have been useless in such a scenario. Explicit code style indicators such as commenting and indentation patterns could have been useful, but the possibility of copy-pasting code from the Internet renders such features useless. Since comments and run information were intermixed with code, we needed a set of simple, robust, powerful, scalable, and language-independent features.

We are of the opinion that the only type of features that can offer all five of the above desiderata comes from word and character n-grams. They kill two birds with one stone: they are robust and resistant against copy-pasting from the Internet (because of the shingling property much used in plagiarism research [ 1 ]), and they are very effective at discriminating between author styles (as evidenced in authorship attribution studies [ 7, 13, 17 ]).

We therefore experimented with the two following categories of features: (1) Bag of words, and (2) Character n-grams (n = 1, 2, 3) with and without space characters and punctuation symbols. For each category, we experimented with lowercase and original case formatting, and three representations: binary (presence/absence), term frequency (tf), and tfidf. Word n-grams (n = 2, 3), and combination of different types of features (feature fusion; cf. [ 10 ]) could not be explored due to sparsity and runtime issues, which we would like to investigate in future.

We used three different regression models (general linear models) from the scikit-learn package [ 12 ]: Linear Regression, Ridge Regression, and Lasso. For Linear and Ridge Regression, we used default parameter settings. For Lasso, we tuned the parameter as described in the next section. In the next section, we will see how the combinations of different features and regressors perform. 4.

RESULTS

As mentioned in Section 1, we performed leave-one-coder-out cross-validation on the training data to find out the optimal featureregressor combination, as well as the optimal parameter settings. We used the average across five RMSEs (for five personality traits) as our objective function. The reason we did not use Pearson Correlation Coefficient ( ) or its square (R2) is because there exists some debate as to whether we should use pure R2 or adjusted R2. RMSE avoids this debate. We would like to minimize the mean RMSE.

The main results are shown in Table 2 through Table 4. Note that overall, Linear Regression performs the worst, with high RMSEs across most feature combinations. This is expected, because the output space should be highly non-linear in terms of features. Ridge Regression and Lasso perform much better, with the best values coming out of Lasso using character unigrams (lowercased) – for binary, tf, and tfidf. This is a rather surprising finding, as it shows two things: (a) a handful of very simple character unigrams can capture very complex and highly non-linear output spaces, and (b) character unigrams beat more complex features in expressive power.

As next step, we proceeded to tune the Lasso parameter that governs the shrinkage of coefficients. Note from Table 2 to Table 4 that the lowest RMSE came from lowercased character unigrams and tfidf. Hence, we used this combination, and tweaked the parameter of Lasso. We obtained the following five top-performing combinations: 1. all characters, Lasso

= 0:05, mean RMSE = 8.38. 2. all non-space characters, Lasso 8.38. = 0:05, mean RMSE =

3. all characters, Lasso

= 0:1, mean RMSE = 8.4.

4. all non-space characters, Lasso

= 0:1, mean RMSE = 8.4.

5. all characters, Lasso

= 0:01, mean RMSE = 8.41.

We used the corresponding models on the test data as our five runs. The final results from five runs are shown in Table 5. Our Run 5 achieved the best RMSE on extroversion (8.60) and competitive results on other traits. We believe that with more parameter tuning and feature engineering (e.g., word n-grams), we can beat the performance of our existing system and be able to advance the state-of-the-art in this challenging and interesting task.

CONCLUSION

In this paper, we reported the design, feature engineering, and evaluation of the BESUMich system submitted to the PR-SOCO Shared Task [ 15 ]. One of our runs achieved the best RMSE on extroversion, and all five runs performed competitively. We could not experiment with word n-grams due to sparsity and runtime issues, but we hope to resolve them in future work. Future research directions include a more rigorous feature engineering and parameter tuning step, along with feature ranking to identify which features are the most important in this task. Another interesting idea will be to explore the learning curve to see how much training data is sufficient to obtain reasonable RMSE values. Similarly, a feature curve will be able to indicate a reasonable vocabulary size for the experiments we performed. Overall, we are hopeful that our methodology, combined with the methods presented by other participants, will significantly advance future research in this domain.

Feature Type AW AW AC SS PP SP AC SS PP SP AC SS PP SP AC SS PP SP AC SS PP SP AC SS PP SP

References

[1] S. M. Alzahrani , N. Salim , and A. Abraham. Understanding Plagiarism Linguistic Patterns , Textual Features, and Detection Methods. Trans. Sys. Man Cyber Part C , 42 ( 2 ): 133 - 149 , Mar. 2012 .

[2] S. Argamon , S. Dhawle , M. Koppel , and J. W. Pennebaker . Lexical Predictors of Personality Type . In Proceedings of the 2005 Joint Annual Meeting of the Interface and the Classification Society of North America , 2005 .

[3] S. Argamon , M. Koppel , J. W. Pennebaker , and J. Schler . Automatically Profiling the Author of an Anonymous Text . Commun. ACM , 52 ( 2 ): 119 - 123 , Feb. 2009 .

[4] F. Celli , B. Lepri , J.-I. Biel , D. Gatica-Perez , G. Riccardi , and F. Pianesi . The Workshop on Computational Personality Recognition 2014 . In Proceedings of the 22Nd ACM International Conference on Multimedia, MM '14 , pages 1245 - 1246 , New York, NY, USA, 2014 . ACM.

[5] A. Clauset , C. R. Shalizi , and M. E. J. Newman . Power-Law Distributions in Empirical Data . SIAM Rev ., 51 ( 4 ): 661 - 703 , Nov. 2009 .

[6] P. T. Costa Jr . and R. R. McCrae. The Revised NEO Personality Inventory (NEO-PI-R) . The SAGE Handbook of Personality Theory and Assessment , 2 : 179 - 198 , 2008 .

[7] H. J. Escalante , T. Solorio , and M. Montes-y Gomez . Local Histograms of Character N-grams for Authorship Attribution . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 288 - 298 , Portland, Oregon, USA, June 2011 . Association for Computational Linguistics .

[8] D. Estival , T. Gaustad , S. B. Pham , W. Radford , and B. Hutchinson . Author Profiling for English Emails . In Proceedings of the 10th Conference of the Pacific Association for Computational Linguistics (PACLING'07) , pages 263 - 272 , 2007 .

[9] A. Genkin , D. D. Lewis , and D. Madigan . Large-Scale Bayesian Logistic Regression for Text Categorization . Technometrics, 49 ( 3 ): 291 - 304 , 2007 .

[10] U. G. Mangai , S. Samanta , S. Das , and P. R. Chowdhury . A Survey of Decision Fusion and Feature Fusion Strategies for Pattern Classification . IETE Technical Review , 27 ( 4 ): 293 - 307 , 2010 .

[11] O. H. Paruma-Pabón , F. A. González , J. Aponte , J. E. Camargo , and F. Restrepo-Calle . Finding Relationships between Socio-technical Aspects and Personality Traits by Mining Developer E-mails . In Proceedings of the 9th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE '16 , pages 8 - 14 , New York, NY, USA, 2016 . ACM.

[12] F. Pedregosa , G. Varoquaux , A. Gramfort , V. Michel , B. Thirion , O. Grisel , M. Blondel , P. Prettenhofer , R. Weiss , V. Dubourg , J. Vanderplas , A. Passos , D. Cournapeau , M. Brucher , M. Perrot , and E. Duchesnay . Scikit-learn: Machine Learning in Python . Journal of Machine Learning Research , 12 : 2825 - 2830 , 2011 .

[13] F. Peng , D. Schuurmans , S. Wang , and V. Keselj . Language Independent Authorship Attribution using Character Level Language Models . In Proceedings of the Tenth Conference on European Chapter of the Association for Computational Linguistics - Volume 1, EACL '03 , pages 267 - 274 , Stroudsburg, PA, USA, 2003 . Association for Computational Linguistics .

[14] F. Rangel , F. Celli , P. Rosso , M. Potthast , B. Stein , and W. Daelemans . Overview of the 3rd Author Profiling Task at PAN 2015 . 2015 .

[15] F. Rangel , F. González , F. Restrepo , M. Montes , and P. Rosso . PAN at FIRE: Overview of the PR-SOCO Track on Personality Recognition in SOurce COde . In Working notes of FIRE 2016 - Forum for Information Retrieval Evaluation, Kolkata , India, December 7- 10 , 2016 , CEUR Workshop Proceedings . CEUR-WS.org, 2016 .

[16] F. Rangel , P. Rosso , M. Koppel , E. Stamatatos, and G. Inches . Overview of the Author Profiling Task at PAN 2013 . In CLEF Conference on Multilingual and Multimodal Information Access Evaluation , pages 352 - 365 . CELCT, 2013 .

[17] U. Sapkota , S. Bethard , M. Montes , and T. Solorio. Not All Character N-grams Are Created Equal: A Study in Authorship Attribution . In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 93 - 102 , Denver, Colorado, May-June 2015. Association for Computational Linguistics .