Introduction

When faced with the same situation, different humans behave differently. This is, of course, due to different backgrounds, education paths, and life experiences, but according to psychologists there is another important aspect: personality (Snyder, 1983;Parks and Guay, 2009).

Human Personality is a psychological construct aimed at explaining the wide variety of human behaviours in terms of a few, stable and measurable individual characteristics (Vinciarelli and Mohammadi, 2014).

Such characteristics are formalised in Trait Models, and there are currently two of these models that are widely adopted: Big Five (John and Srivastava, 1999) and Myers-Briggs Type Indicator (MBTI) (Myers and Myers, 1995). The first examines five dimensions (OPENNESS TO EX-PERIENCE, CONSCIENTIOUSNESS, EXTROVER-SION, AGREEABLENESS and NEUROTICISM) and for each of them assigns a score in a range. The second one, instead, considers 16 fixed personality types, coming from the combination of the opposite poles of 4 main dimensions (EXTRAVERT-INTROVERT, INTUITIVE-SENSING, FEELING-THINKING, PERCEIVING-JUDGING). Examples of full personality types are therefore four letter labels such as ENTJ or ISFP.

The tests used to detect prevalence of traits include human judgements regarding semantic similarity and relations between adjectives that people use to describe themselves and others. This is because language is believed to be a prime carrier of personality traits (Schwartz et al., 2013). This aspect, together with the progressive increase of available user-generated data on social media, has prompted the task of Personality Detection, i.e., the automatic prediction of personality from written texts (Youyou et al., 2015;Argamon et al., 2009;Litvinova et al., 2016;Whelan and Davies, 2006).

Personality detection can be useful in predicting life outcomes such as substance use, political attitudes and physical health. Other fields of application are marketing, politics and psychological and social assessment.

As a contribution to personality detection in Italian, we present Personal-ITY, a new corpus of YouTube comments annotated with MBTI personality traits, and some preliminary experiments to highlight its characteristics and test its potential. The corpus is made available to the community1 .

Related Work

There exist a few datasets annotated for personality traits. For the shared tasks organised within the Workshop on Computational Personality Recognition (Celli et al., 2013) (Essays (Pennebaker and King, 2000) and myPersonality 2 ) and two in 2014 (YouTube Personality Dataset (Biel and Gatica-Perez, 2013) and Mobile Phones interactions (Staiano et al., 2012)).

For the 2015 PAN Author Profiling Shared Task (Pardo et al., 2015), personality was added to gender and age in the profiling task, with tweets in English, Spanish, Italian and Dutch. These are also annotated according to the Big Five model.

Still in the Big Five landscape, Schwartz et al. (2013) collected a dataset of FaceBook comments (700 millions words) written by 136.000 users who shared their status updates. Interesting correlations were observed between word usage and personality traits.

If looking at data labelled with the MBTI traits, we find a corpus of 1.2M English tweets annotated with personality and gender (Plank and Hovy, 2015), and the multilingual TWISTY (Verhoeven et al., 2016). The latter is a corpus of data collected from Twitter annotated with MBTI personality labels and gender for six languages (Dutch, German, French, Italian, Portuguese and Spanish) and a total of 18,168 authors. We are interested in the Italian portion of TWISTY.

Table 1 contains an overview of the available Italian corpora labelled with personality traits. We include our own, which is described in Section 3.

Regarding detection approaches, Mairesse et al. ( 2007) tested the usefulness of different sets of textual features making use of mostly SVMs.

At the PAN 2015 challenge (see above) a variety of algorithms were tested (such as Random Forests, decision trees, logistic regression for classification, and also various regression models), but overall most successful participants used SVMs. Regarding features, participants approached the task with combinations of style-based and contentbased features, as well as their combination in ngram models (Pardo et al., 2015).

Experiments on TWISTY were performed by 2 http://mypersonality.org the corpus creators themselves using a Lin-earSVM with word (1-2) and character (3-4) ngrams. Their results (reported in Note that all results are reported as micro-average F-score.

3 Personal-ITY First, we explain two major choices that we made in creating Personal-ITY, namely the source of the data and the trait model. Second, we describe in detail the procedure we followed to construct the corpus. Lastly, we provide a description of the resulting dataset.

Data YouTube is the source of data for our corpus. The decision is grounded on the fact that compared to the more commonly collected tweets, YouTube comments can be longer, so that users are freer to express themselves without limitations. Additionally, there is a substantial amount of available data on the YouTube platform, which is easy to access thanks to the free YouTube APIs.

Trait Model Our model of choice is the MBTI. The first benefit of this decision is that this model is easy to use in association with a Distant Supervision approach (just checking if a message contains one of the 16 personality types; see Section 3.1). Another benefit is related to the existence of TWISTY. Since both TWISTY and Personal-ITY implement the MBTI model, analyses and experiments over personality detection can be carried out also in a cross-domain setting.

Ethics Statement

Personality profiling must be carefully evaluated from an ethical point of view. In particular, often, personality detection involves ethical dilem-mas regarding appropriate utilization and interpretations of the prediction outcomes (Weiner and Greene, 2017). Concerns have been raised regarding the inappropriate use of these tests with respect to invasion of privacy, cultural bias and confidentiality (Mehta et al., 2019). The data included in the Personal-ITY dataset were publicly available on the YouTube platform at the time of the collection. As we will explain in detail in this Section, the information collected are comments published under public videos on the YouTube platform by authors themselves. For a major protection of user identities, in the released corpus only the YouTube usernames of the authors are mentioned which are not unique identifiers. The YouTube IDs of the corresponding channels, which are the real identifiers in the platform, allowing to trace the identity of the authors, are not released. Note also that the corpus was created for academic purposes and is not intended to be used for commercial deployment or applications.

Corpus Creation

The fact that users often self-disclose information about themselves on social media makes it possible to adopt Distant Supervision (DS) for the acquisition of training data. DS is a semi-supervised method that has been abundantly and successfully used in affective computing and profiling to assign silver labels to data on the basis of indicative proxies (Go et al., 2009;Pool and Nissim, 2016;Emmery et al., 2017).

Users left comments to some videos on the MBTI theory in which they were stating their own personality type (e.g. Sono ENTJ...chi altro? [en: "I'm ENTJ...anyone else?"]). We exploited such comments to create Personal-ITY with the following procedure.

First, we searched for as many Italian YouTube videos about MBTI as possible, ending up with a selection of ten with a conspicuous number of comments as the ones above 3 .

Comment

User -MBTI label

Io sono ENFJ!!! User1 -ENFJ

Ho sempre saputo di essere connessa con Lady Gaga! ISFP!

User2 -ISFP

Table 3: Examples of automatic associations user -MBTI personality type.

was associated to a user if they included an MBTI combination in one of their comments. Table 3 shows some examples of such associations. The association process is an approximation typical of DS approaches. To assess its validity, we manually checked 300 random comments to see whether the mention of an MBTI label was indeed referred to the author's own personality. We found that in 19 cases (6.3%) our method led to a wrong or unsure classification of the user's personality (e.g. O tutti gli INTJ del mondo stanno commentando questo video oppure le statistiche sono sbagliate :-)). We can assume that our dataset might therefore contain about 6-7% of noisy labels.

Using the acquired list of authors, we meant to obtain as many comments as possible written by them. The YouTube API, however, does not allow to retrieve all comments by one user on the platform. In order to get around this problem we relied on video similarities, and tried to expand as much as possible our video collection. Therefore, as a third step, we retrieved the list of channels that feature our initial 10 videos, and then all of the videos within those channels.

Fourth, through a second AJAX request, we downloaded all comments appearing below all videos retrieved through the previous step.

Lastly, we filtered all comments retaining those written by authors included in our original list. This does not obviously cover all comments by a relevant user, but it provided us with additional data per author.

Final Corpus Statistics

For the final dataset, we decided to keep only the authors with a sufficient amount of data. More specifically, we retained only users with at least five comments, each at least five token long.

Personal-ITY includes 96, 815 comments by 1048 users, each annotated with an MBTI label. The average number of comments per user is 92 and each message has on average 115 tokens.

The amount of the 16 personality types in the corpus is not uniform. Figure 1 shows such distribution and also compares it with the one in TWISTY. The unbalanced distribution can be due to personality types not being uniformly distributed in the population, and to the fact that different personality types can make different choices about their online presence. Goby (2006) for example, observed that there is a significant correlation between online-offline choices and the MBTI dimension of EXTRAVERT-INTROVERT: extroverts are more likely to opt for offline modes of communication, while online communication is presumably easier for introverts. In Figure 1, we also see that the four most frequent types are introverts in both datasets. The conclusion is that, despite the different biases, collecting linguistic data in this way has the advantage that it reflects actual language use and allows large-scale analysis (Plank and Hovy, 2015).

Figure 2 shows more in detail, trait by trait, the distribution of the opposite poles through the users in Personal-ITY and in TWISTY. As we might have expected, in line with what is observed in Figure 1, the two datasets present very similar trends. Such similarities between Personal-ITY and TWISTY are these similarities are a further confirmation of the reliability of the data we collected.

Preliminary Experiments

We ran a series of preliminary experiments on Personali-ITY which can also serve as a baseline for future work on this dataset. We pre-processed texts by replacing hashtags, urls, usernames and emojis with four corresponding placeholders. We adopted the sklearn (Pedregosa et al., 2011) implementation of a linear SVM (LinearSVM), with standard parameters. We tested three types of features. At the lexical level, we experimented with word (1-2) and character (3-4) n-grams, both as raw counts as well as tf-idf weighted. Character n-grams were tested also with a word-boundary option. At a more stylistically level, we considered the use of emojis, hashtags, pronouns, punctuation and capitalisation. Lastly, we also experimented with embeddings-based representations, by using, on the one hand, YouTube-specific (Nieuwenhuis and Nissim, 2019) pre-trained models, on the other hand, more generic embeddings, such as the Italian version of GloVe (Pennington et al., 2014), which is trained on the Italian Wikipedia4 . We looked for all the available embeddings of the words written by each author, and used the average as feature. If a word appeared more than once in the string of comments, we considered it multiple times in the final average.

We used 10-fold cross-validation, and assessed the models using macro f-score. Note that the original TWISTY paper uses micro f-score. Thus, for the sake of comparison, we include also micro-F in Table 5 for the MAJ baseline and our lexical n-gram model. Table 4 shows the results of our experiments with different feature types. 5 Overall, lexical features (n-grams) perform best. Combining different feature types did not lead to any improvement. Classification was performed with four separate binary classifiers (one per dimension), and with one single classifier predicting four classes, i.e, the whole MBTI labels at once. In the latter case, we observe that the results are quite high considering the increased difficulty of the task. Table 5 reports the scores of our models on TWISTY. As for Personal-ITY, best results were achieved using lexical features (tf-idf n-grams); stylistic features and embeddings are just above the baseline. Our model outperforms the one in (Verhoeven et al., 2016) for all traits (micro-F).

To test compatibility of resources and to assess model portability, we also ran cross-domain experiments on Personal-ITY and TWISTY. In the first setting, we tested the effect of merging the two datasets on the performance of models for personality detection, maintaining the 10-fold crossvalidation setting and by using the model performing better on average for YouTube and Twitter data (a character n-grams model). In the second setting, instead, we divided both corpora in fixed training and test sets with a proportion of 80/20 and ran the models using lexical features, in order to run a cross-domain experiment. For direct comparison, we run the model indomain again using this split.

Conclusions

The experiments show that there is no single best model for personality prediction, as the feature contribution depends on the dimension considered, and on the dataset. Lexical features perform best, but they tend to be strictly related to the context in which the model is trained and so to overfit. The inherent difficulty of the task itself is confirmed and deserves further investigations, as assigning a definite personality is an extremely subjective and complex task, even for humans.

Personal-ITY is made available to further investigate the above and other issues related to personality detection in Italian. The corpus can lend itself to a psychological analysis of the linguistic cues for the MBTI personality traits. On this line, it is interesting to investigate the presence of evidences linking linguistic features with psychological theories about the four considered dimensions (EXTRAVERT-INTROVERT, INTUITIVE-SENSING, FEELING-THINKING, PERCEIVING-JUDGING). First results in this direction are presented in (Bassignana et al., 2020).