Determining an individual's personality traits is an important concept in Psychology. Although traits are normally assessed through self-report tests, an alternative method would be to computationally analyze an individual's linguistic markers. Studies in personality trait classification show promising results and look to continuously improve the field by either using new features or by collecting new data from social media; however, a key concept that is not always considered is the use of feature reduction techniques. This research aims to perform feature reduction techniques on linguistic features from essays and classify the author's personality traits based on the reduced feature set. The classifiers are evaluated by comparing against a baseline classifier trained with all extracted features. The feature reduction techniques used are Information Gain and Principal Component Analysis. The results show that feature reduction techniques are able to increase classification measures, but not by significant values. Reduced datasets are exceptionally beneficial in reducing the amount of data needed allowing classifiers to perform faster while still maintaining classification measures.
Personality Psychology, or simply Personality, is “the scientific study of psychological forces that make people uniquely themselves” [Friedman and Schustack, 2014, p.1]. These forces consist of organized and relatively enduring traits and mechanisms that influence one’s interactions with the intrapsychic, physical, and social environments [Larsen and Buss, 2008, p.4].
One of the most well-researched theories describing personality trait variation would be the Five Factor model, also known as the Big Five [Norman, 1963; Goldberg, 1981]. The Big Five is an organization of personality facets that are subsets of five broad traits: Extraversion, Agreeableness, Conscientiousness, Neuroticism, and Openness to Experience. John et al. [2008, p.116] mentions that these five traits “were derived from analyzing terms people use to describe themselves and others.” The Big Five have been used in studies observing individuals in environments such as at work [Richardson et al., 2009] and in academics [Komarraju et al., 2009]. Research using both natural language adjectives and theoretically based personality instruments supports the comprehensiveness of the model and its applicability across observers and cultures [McCrae and John, 1992].
Personality traits are traditionally measured through the use of questionnaires such as the Big Five Inventory (BFI) [John et al., 1991]; however, an alternative approach would be to analyze an individual‘s linguistic markers. An individual’s choice of words eventually becomes consistent over time and context and can be used as an individual difference measure [Pennebaker et al., 2003]. A study by Pennebaker and King [1999] showed multiple correlations between linguistic markers and the Big Five such as how Neuroticism is positively correlated with the use of negative emotion words and negatively with positive emotion words. Goldberg [1981, p.142] mentions that “the more important an individual difference [is] in human transactions; the more likely languages will have a term for it.”
Correlations between linguistic markers and personality traits have paved the way for research in the area of automatic personality classification. One of the earliest studies [Mairesse et al., 2007] focused on classifying personality traits based on text. They extracted linguistic features from essays using a text analysis tool and a psycholinguistic database. Their findings were modest, but still showed that computationally modeling the Big Five was possible. Subsequent studies were able to present promising methods in improving upon the findings of Mairesse et al. [2007] by introducing new linguistic features [Mohammad and Kiritchenko, 2013; Poria et al., 2013a]. Other studies [Golbeck et al., 2011; Schwartz et al., 2013; Park et al., 2014; Peng et al., 2015] focus on different data sources by taking advantage of social media, and collecting or using data from users of these growing platforms.
Although there have been advancements in the field of personality trait classification, there are still gaps in determining which linguistic features are most significant for the classification process. This research investigates the use of two feature reduction techniques in order to improve the computation involved. The data source for this research is the Pennebaker and King [1999] dataset of essays. Features are extracted using LIWC and are analyzed to see if they can be reduced to a smaller set while still being able to aid in classifying the Big Five. The classifiers are then built using the reduced set of features and are compared against classifiers using the complete set of features. This research aims to show that the use of feature reduction techniques are beneficial to future work in the field.
The remainder of this paper is organized as follows: Section 2 reviews studies on personality trait classification that work with the Pennebaker and King [1999] essay dataset. Section 3 discusses the characteristics of the data source used in this research. Section 4 explains how features are extracted from the data source. Section 5 explains the background of the feature reduction techniques used. Section 6 presents how each classifier was built. Section 7 discusses the overall performance of the classifiers and the effects of feature reduction. Finally, Section 8 concludes the paper and explains recommendations for future work. 2
Personality Trait Classification based on linguistic markers is a growing field. Although other studies [Golbeck et al., 2011; Schwartz et al., 2013; Park et al., 2014; Peng et al., 2015] use big collections of data from social media, this paper limits its review to studies that use the Pennebaker and King [1999] essay dataset for the sake of having a common ground.
One of the earliest studies regarding automatic personality classification is that of Mairesse et al. [2007]. Their methods of extracting features relied on LIWC1 [Pennebaker et al., 2001] and the MRC Psycholinguistic Database [Coltheart, 1981]. LIWC produced a total of 88 features and is further discussed in Section 4. They also used 14 psycholinguistic features2 from the MRC Psycholinguistic Database, a machine usable dictionary. They then trained classifiers based on different combinations of the set of features and had promising results. Openness to Experience was the easiest to identify among the Big Five having an accuracy of 62.5% using only LIWC features. They also showed how features from LIWC out performed those from MRC; however, both showed promising correlations to the Big Five. Their results were modest, but were significant enough to show that computationally modeling personality traits was possible.
One recent study [Mohammad and Kiritchenko, 2013] made use of fine affect or emotion category features as alternatives for personality trait classification. They were able to extract an extensive amount of emotion features with the use of the NCR Hashtag Emotion Lexicion [Mohammad and Turney, 2010]. This lexicon is able to produce either 8 basic emotions or 585 fine emotion features. They also made use of the Specificity Lexicon and Osgood Dimensions Lexicon [Turney and Littman, 2003]. The first lexicon calculated the average information content of an essay while the later was able to extract the average evaluativeness, activity, and potency scores of words. Finally, they made use of the 1It was assumed by the researchers that Mairesse et al. [2007] used the 2001 version of LIWC as to how it was cited in their paper 2However, the total number of features is listed as 26 [Wilson, 1988]
LIWC features and frequencies of unigrams of the essays. They experimented by combining different features sets and ran them through Support Vector Machine classifiers. The classifier that performed best was built using the LIWC and the 585 fine emotion features. Their results showed minimal improvement over the results of Mairesse et al. [2007], but revealed that emotion category features contain information regarding an individual‘s personality and can be considered useful for future studies.
Another study [Poria et al., 2013a] introduced a novelty approach of using of common sense knowledge. They utilize ConceptNet [Havasi et al., 2007] and EmoSenticNet [Poria et al., 2013b] to extract sentiment polarity scores and affective labels from the essays. They also extract linguistic features from LIWC and MRC. They train Support Vector Machine classifiers and compare against Mairesse et al [Mairesse et al., 2007] and Mohammad and Kiritchenko [Mohammad and Kiritchenko, 2013]. Their results show significant improvements demonstrating that the sentiment polarity and affective labels contained relevant information in classifying personality traits.
With the discovery of more and more features with information pertaining to an individual‘s personality, the issue of irrelevant features and overfitting arises. Each of the previously reviewed studies presents an opportunity to investigate the use of feature reduction due to the high volume of features presented. This research explores the application of feature reduction on LIWC features and aims to showcase the benefits of using these techniques. 3
The data used in this research was gathered and used in a study by Pennebaker and King [1999]. The actual file was retrieved from myPersonality3. It consists of a total of 2,468 essays or daily writing submissions from 34 psychology students. There are a total of 29 women and 5 men whose ages ranged from 18 to 67 with a mean of 26.4 and a standard deviation of 11.1.
The writing submissions were in the form of a course requirement or assignment but were not graded. For each assignment, students were expected to write a minimum of 20 minutes per day about a specific topic. The data was collected during a 2-week summer course between 1993 to 1996. Each student completed their daily writing for 10 consecutive days.
Students’ personality scores were assessed by answering the Big Five Inventory (BFI) [John et al., 1991]. The BFI is a 44-item self-report questionnaire that provides a score for each of the five personality traits. Each item consists of short phrases and is rated using a 5-point scale that ranges from 1 (disagree strongly) to 5 (agree strongly).
An instance in the data source consists of a filename or ID, the actual essay, and five classification labels of the Big Five personality traits. Labels were originally in the form of either yes (‘y’) or no (‘n’) to indicate scoring high or low for a given trait; however, this research changed the labels to ‘y’ to 1 and ‘n’ to 0 according to the preference of the researchers. 3www.mypersonality.org
In order to extract information from raw text, LIWC4 was utilized. LIWC stands for Linguistic Inquiry and Word Count and was developed by Pennebaker et al. [2007]. It is a text analysis tool that provides an efficient and effective method for studying the various emotional, cognitive, and structural components present in individuals’ written samples.
The tool analyses text files sequentially, one target word at a time by searching through it dictionary file. If the target word matches the dictionary word, the appropriate word category scale is incremented. Pennebaker et al. [2007] explains that there are a total of 80 output features consisting of 4 general descriptor categories (e.g., total word count, words per sentence), 22 standard linguistic dimensions (e.g., frequency of pronouns, articles), 32 word categories tapping psychological constructs (e.g., affect, cognition), 7 personal concern categories (e.g., work, home), 3 paralinguistic dimensions (assents, fillers, nonfluencies), and 12 punctuation categories (e.g., periods, commas). Values of all features except word count and words per sentence reflect percentage of total words.
Based on the methods of Mairesse et al. [2007], the essays from the data source were fed through LIWC. The output features were used to create a dataset for each of the Big Five. These datasets contain all 80 LIWC features and one of the five personality trait as the classification label. It is important to note that replication of the methods used by Mairesse et al. [2007] was chosen by the researchers as the better alternative to direct comparison of results. This was due to the difference in the number of output features from the version of LIWC reported in their paper. Replication of methods would allow for a better baseline when comparing against a reduced set of features. 5
LIWC provides a vast amount of information that it would be important to analyze whether or not classification of the Big Five can be improved by reducing the set of features. The presence of non-relevant features can influence a classifier to produce smaller error by fitting the model according to the training data. Removing such features can increase the predictive power of a classifier by focusing only on certain features. A defined model may be able to classify unseen data better and is desirable for real world scenarios. Feature reduction becomes an important concept to consider when trying to improve classification. The techniques that are performed are Information Gain and Principal Component Analysis.
The following subsections discuss how these techniques work and present their respective output. Both techniques were applied on the datasets using Waikato Environment for Knowledge Analysis or Weka, a tool of machine learning algorithms and data preprocessing [Hall et al., 2009]. 5.1
Information Gain is a measure of how effective a given feature is in classifying data [Mitchell, 1997]. It becomes essen4Developed in 2007 and is a different version than the previously mentioned LIWC in Section 2 tial to this research to evaluate each of the 80 LIWC features and determine which provide significance in classifying the Big Five.
One important concept in computing for the Information Gain is Entropy or being able to characterizes the impurity of an arbitrary collection of examples. Entropy is defined as E(F ) = n X pv log2 pv v=1 where F is a feature or classification containing a number n of different discrete values and where pv is the proportion of F belonging to value v. When values are continuous in nature, the values are discretized by splitting at a point that provides the maximum information gain. Therefore, the information gain of a feature F relative to a classification C can be defined as
IG(C, F ) = E(C)
X v2 V alues(F ) |C| |Cv| E(Cv) where V alues(F ) is the set of all possible values for feature A, and Cv is the subset of C for which feature F has values v. Basically, information gain is the entropy of class C reduced by the weighted average entropy of each subset Sv.
The Information Gain of all 80 features were computed for each of the Big Five datasets. For each of the datasets, only features with non-zero information gain were selected. Table 1 shows the remaining features per personality trait along with their respective information gain. 5.2
Principal Components Analysis (PCA) is used to identify patterns, and highlight the similarities and differences in data [Smith, 2002]. It is particularly useful when dealing with data with a high number of features as it is able to reduce the number of these features without losing much information. Concepts of covariance, matrix operations, eigenvalues, and eigenvectors are all used to compute for the principal components.
Smith [
F inalData = F V T ⇥
AdjustedV aluesT where the transposed F V is multipled with a transposed matrix AdjustedV alues containing the original datasets‘s values adjusted by each feature‘s mean.
After performing Weka’s implementation of PCA on all 80 LIWC features, a total of 56 eigenvectors were found and used to create a new dataset for each of the Big Five. 6
A 10-fold cross validation was performed on each of the 15 datasets (5 using all features, 5 using information gain, and 5 using PCA) in order to evaluate their overall effectiveness. This research recorded the accuracy, precision, and Fmeasure of all classifiers and the amount of reduction in terms of a dataset’s feature size.
Each dataset was fed through three learning algorithms in Weka and compared against a baseline classifier (ZeroR) that returned the majority class. The algorithms used are two implementations of Support Vector Machine (libSVM and SMO), and Linear Logistic Regression (SimpleLogistic). Other algorithms such as k-Nearest Neighbour (IBk, where k equaled 1 and 5), C4.5 Decision Tree (J48), Naive Bayes (NaiveBayes), and Random Forest (RandomForest) were also investigate; however, these classifiers were discarded due to poor performance. Default parameter settings were used for each of the learning algorithms. 7
An overview of the results of classification, as seen in Table 2, shows that Openness to Experience is the easiest trait to identify regardless of feature reduction techniques. The remaining traits, ranked from easiest to hardest to identify, are Neuroticism, Agreeableness, Conscientiousness, and Extraversion. The ranking corresponds to the Information Gain of each LIWC feature per Big Five as seen in Table 1. Openness to Experience had the most amount of features remaining after removing those with zero information gain. On the other hand, Extraversion had the least remaining features.
Each of the remaining features can also serve as a descriptor of how word choice of an individual is related to their personality traits. The use of Negative Emotions is relevant in determining one’s Conscientiousness, Agreeableness, and Neuroticism. Similarly, Swear Words is relevant to Conscientiousness, Openness to Experience and Agreeableness. Although, it is important to note that that higher usage rate of a linguistic feature does not equates to a high personality trait score. Features with higher Information Gain simply indicate that a feature is more effective in separating and classifying data.
The classifiers using feature reduced datasets were generally able to increase classification measures, but not by significant values. This suggests that the LIWC features do not have any more information to provide in classifying the Big Five, at least when considering only a feature set of only LIWC features. A better comparison of the classifiers built using all 80 LIWC features and the feature reduced datasets is shown in Table 3. This indicates that classifiers using feature reduced datasets were able to slightly edge out classifiers using all features in four of the five personality traits.
Agreeableness was the only trait where both best classifiers performed similarly.
This research also noted the minimal increase in classi
Agreeableness *Significantly better than the baseline (ZeroR); **Significantly better than its respective classifier trained using all features
Classifiers with A were trained using Information Gain reduced feature sets and B represents classifiers trained using PCA reduced feature sets fication measures from using all features to using reduced features when dealing with Agreeableness. This can be attributed to how the classifier SimpleLogistic works. It is important to note that SimpleLogistic includes its own implementation of feature reduction [Landwehr et al., 2005]. Interestingly, the only common attributes in comparison to those selected using Information Gain were Anger and Family. The remaining attributes selected by SimpleLogistic were Words greater than 6 letters, Common Adverbs, Negations, Anxiety, Motion, Exclamation Marks, and Dashes. The classifier also selected features for other personality traits that were not present in the set of remaining features after performing Information Gain.
Despite the results that feature reduction techniques were not able to significantly increase classification measures, it is important to note that the amount of reduction made in the size of the set of features is significant as seen in column Size Reduction of Table 2. Datasets using Information Gain had highly significant size reductions ranging from 70% to 95% while still able to perform up to par with those using all 80 LIWC feature. On the other hand, PCA was able to significantly reduce the set of features by 30% while still covering 95% of the feature set’s variance. The PCA reduced dataset was also able to significantly improve the classifier LibSVM for all personality traits which is evident when looking at the measures. Agreeableness was able to improve from an Fmeasure of 0.475 to 0.563 resulting in a 0.088 increase. On the other hand, Conscientiousness had improved from an Fmeasure of 0.52 to 0.56 resulting in the the lowest increase of 0.04. As a whole, feature reduction techniques were able to improve the classification of personality traits by both slightly increasing classification measures and heavily reducing the size of the datasets. 8
This research was able to demonstrate that feature reduction techniques like Information Gain and Principal Component Analysis are beneficial in classifying an individual’s personality traits based on text data. Applying these techniques reduced the size of the original data while slightly improving the classifiers’ level of performance. A reduced-dataset leads to a more defined model which can better handle unseen data. This research was also able to highlight LIWC features that contain the most Information Gain about an individual‘s traits. This knowledge can be useful outside of computational classification by providing additional linguistic descriptors of individual‘s with certain personality traits.
This research also recommends two areas for improvements regarding future work in the field of text-based personality trait classification. The first would concern the data source containing binomially labeled traits and thereby categorizing an individual into one or the other. This representation does not capture the dimensional nature of a trait and would be better represented as either the raw output of a certain personality inventory or its normalized form. The second area for improvement would involve studying the use of non-western personality trait theories or indigenous measures. Such works are normally gauged towards understand a particular culture and would be a good area to apply linguistic analysis. Findings would be beneficial to both culturespecific and cross-cultural psychology.