We present a novel corpus for personality prediction in Italian, containing a larger number of authors and a different genre compared to previously available resources. The corpus is built exploiting Distant Supervision, assigning MyersBriggs Type Indicator (MBTI) labels to YouTube comments, and can lend itself to a variety of experiments. We report on preliminary experiments on Personal-ITY, which can serve as a baseline for future work, showing that some types are easier to predict than others, and discussing the perks of cross-dataset prediction.
When faced with the same situation, different
humans behave differently. This is, of course, due
to different backgrounds, education paths, and life
experiences, but according to psychologists there
is another important aspect: personality
Human Personality is a psychological construct
aimed at explaining the wide variety of human
behaviours in terms of a few, stable and measurable
individual characteristics
Such characteristics are formalised in Trait
Models, and there are currently two of these
models that are widely adopted: Big Five
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). second one, instead, considers 16 fixed personality types, coming from the combination of the opposite poles of 4 main dimensions (EXTRAVERTINTROVERT, INTUITIVE-SENSING, FEELINGTHINKING, PERCEIVING-JUDGING). Examples of full personality types are therefore four letter labels such as ENTJ or ISFP.
The tests used to detect prevalence of traits
include human judgements regarding semantic
similarity and relations between adjectives that
people use to describe themselves and others. This
is because language is believed to be a prime
carrier of personality traits
Personality detection can be useful in predicting life outcomes such as substance use, political attitudes and physical health. Other fields of application are marketing, politics and psychological and social assessment.
As a contribution to personality detection in Italian, we present Personal-ITY, a new corpus of YouTube comments annotated with MBTI personality traits, and some preliminary experiments to highlight its characteristics and test its potential. The corpus is made available to the community1. 2
There exist a few datasets annotated for
personality traits. For the shared tasks organised within the
Workshop on Computational Personality
Recognition
For the 2015 PAN Author Profiling Shared Task
Still in the Big Five landscape, Schwartz et al. (2013) collected a dataset of FaceBook comments (700 millions words) written by 136.000 users who shared their status updates. Interesting correlations were observed between word usage and personality traits.
If looking at data labelled with the MBTI traits,
we find a corpus of 1.2M English tweets annotated
with personality and gender
Table 1 contains an overview of the available Italian corpora labelled with personality traits. We include our own, which is described in Section 3.
Regarding detection approaches, Mairesse et al. (2007) tested the usefulness of different sets of textual features making use of mostly SVMs.
At the PAN 2015 challenge (see above) a
variety of algorithms were tested (such as Random
Forests, decision trees, logistic regression for
classification, and also various regression models), but
overall most successful participants used SVMs.
Regarding features, participants approached the
task with combinations of style-based and
contentbased features, as well as their combination in
ngram models
Experiments on TWISTY were performed by
the corpus creators themselves using a LinearSVM with word (1-2) and character (3-4) ngrams. Their results (reported in Table 2 for the Italian portion of the dataset) are obtained through 10-fold cross-validation; the model is compared to a weighted random baseline (WRB) and a majority baseline (MAJ).
PJ Avg
WRB First, we explain two major choices that we made in creating Personal-ITY, namely the source of the data and the trait model. Second, we describe in detail the procedure we followed to construct the corpus. Lastly, we provide a description of the resulting dataset.
Data YouTube is the source of data for our
corpus. The decision is grounded on the fact that
compared to the more commonly collected tweets,
YouTube comments can be longer, so that users
are freer to express themselves without
limitations. Additionally, there is a substantial amount
of available data on the YouTube platform, which
is easy to access thanks to the free YouTube APIs.
Trait Model Our model of choice is the MBTI.
The first benefit of this decision is that this model
is easy to use in association with a Distant
Supervision approach (just checking if a message
contains one of the 16 personality types; see
Section 3.1). Another benefit is related to the
existence of TWISTY. Since both TWISTY and
Personal-ITY implement the MBTI model,
analyses and experiments over personality detection can
be carried out also in a cross-domain setting.
Ethics Statement
Personality profiling must be carefully evaluated
from an ethical point of view. In particular,
often, personality detection involves ethical
dilemmas regarding appropriate utilization and
interpretations of the prediction outcomes
The data included in the Personal-ITY dataset were publicly available on the YouTube platform at the time of the collection. As we will explain in detail in this Section, the information collected are comments published under public videos on the YouTube platform by authors themselves. For a major protection of user identities, in the released corpus only the YouTube usernames of the authors are mentioned which are not unique identifiers. The YouTube IDs of the corresponding channels, which are the real identifiers in the platform, allowing to trace the identity of the authors, are not released. Note also that the corpus was created for academic purposes and is not intended to be used for commercial deployment or applications. 3.1
Corpus Creation
The fact that users often self-disclose information
about themselves on social media makes it
possible to adopt Distant Supervision (DS) for the
acquisition of training data. DS is a semi-supervised
method that has been abundantly and successfully
used in affective computing and profiling to assign
silver labels to data on the basis of indicative
proxies
Users left comments to some videos on the MBTI theory in which they were stating their own personality type (e.g. Sono ENTJ...chi altro? [en: ”I’m ENTJ...anyone else?”]). We exploited such comments to create Personal-ITY with the following procedure.
First, we searched for as many Italian YouTube videos about MBTI as possible, ending up with a selection of ten with a conspicuous number of comments as the ones above3.
Second, we retrieved all the comments to these videos using an AJAX request, and built a list of authors and their associated MBTI label. A label 3Links to the 10 YouTube videos: https://www.youtube.com/watch?v=VCo9RlDRpz0 https://www.youtube.com/watch?v=N4kC8iqUNyk https://www.youtube.com/watch?v=Z8S8PgW8t2U https://www.youtube.com/watch?v=wHZOG8k7nSw https://www.youtube.com/watch?v=lO2z3_DINqs https://www.youtube.com/watch?v=NaKPl_y1JXg https://www.youtube.com/watch?v=8l4o4VBXlGY https://www.youtube.com/watch?v=GK5J6PLj218 https://www.youtube.com/watch?v=9P95dkVLmps https://www.youtube.com/watch?v=g0ZIFNgUmoE
Io sono ENFJ!!! Ho sempre saputo di essere connessa con Lady Gaga! ISFP!
was associated to a user if they included an MBTI combination in one of their comments. Table 3 shows some examples of such associations. The association process is an approximation typical of DS approaches. To assess its validity, we manually checked 300 random comments to see whether the mention of an MBTI label was indeed referred to the author’s own personality. We found that in 19 cases (6.3%) our method led to a wrong or unsure classification of the user’s personality (e.g. O tutti gli INTJ del mondo stanno commentando questo video oppure le statistiche sono sbagliate :-)). We can assume that our dataset might therefore contain about 6-7% of noisy labels.
Using the acquired list of authors, we meant to obtain as many comments as possible written by them. The YouTube API, however, does not allow to retrieve all comments by one user on the platform. In order to get around this problem we relied on video similarities, and tried to expand as much as possible our video collection. Therefore, as a third step, we retrieved the list of channels that feature our initial 10 videos, and then all of the videos within those channels.
Fourth, through a second AJAX request, we downloaded all comments appearing below all videos retrieved through the previous step.
Lastly, we filtered all comments retaining those written by authors included in our original list. This does not obviously cover all comments by a relevant user, but it provided us with additional data per author. 3.2
Final Corpus Statistics For the final dataset, we decided to keep only the authors with a sufficient amount of data. More specifically, we retained only users with at least five comments, each at least five token long.
Personal-ITY includes 96; 815 comments by 1048 users, each annotated with an MBTI label. The average number of comments per user is 92 and each message has on average 115 tokens.
The amount of the 16 personality types in the
corpus is not uniform. Figure 1 shows such
distribution and also compares it with the one in
TWISTY. The unbalanced distribution can be
due to personality types not being uniformly
distributed in the population, and to the fact that
different personality types can make different choices
about their online presence. Goby (2006) for
example, observed that there is a significant
correlation between online–offline choices and the MBTI
dimension of EXTRAVERT-INTROVERT:
extroverts are more likely to opt for offline modes of
communication, while online communication is
presumably easier for introverts. In Figure 1, we
also see that the four most frequent types are
introverts in both datasets. The conclusion is that,
despite the different biases, collecting linguistic data
in this way has the advantage that it reflects
actual language use and allows large-scale analysis
Figure 2 shows more in detail, trait by trait, the distribution of the opposite poles through the users in Personal-ITY and in TWISTY. As we might have expected, in line with what is observed in Figure 1, the two datasets present very similar trends. Such similarities between Personal-ITY and TWISTY are these similarities are a further confirmation of the reliability of the data we collected. 4
We ran a series of preliminary experiments on
Personali-ITY which can also serve as a baseline
for future work on this dataset. We pre-processed
texts by replacing hashtags, urls, usernames and
(a) Extravert - Introvert
(b) Sensing - Intuitive
(c) Thinking - Feeling
(d) Judging - Perceiving
emojis with four corresponding placeholders. We
adopted the sklearn
We used 10-fold cross-validation, and assessed
the models using macro f-score. Note that the
original TWISTY paper uses micro f-score. Thus,
for the sake of comparison, we include also
microF in Table 5 for the MAJ baseline and our lexical
n-gram model. Table 4 shows the results of our
experiments with different feature types.5
Overall, lexical features (n-grams) perform best.
Combining different feature types did not lead to any
improvement. Classification was performed with
four separate binary classifiers (one per
dimension), and with one single classifier predicting four
classes, i.e, the whole MBTI labels at once. In the
latter case, we observe that the results are quite
high considering the increased difficulty of the
task. Table 5 reports the scores of our models on
TWISTY. As for Personal-ITY, best results were
achieved using lexical features (tf-idf n-grams);
stylistic features and embeddings are just above
the baseline. Our model outperforms the one in
To test compatibility of resources and to assess model portability, we also ran cross-domain experiments on Personal-ITY and TWISTY. In the first setting, we tested the effect of merging the
wordembeddings
5In Tables 4–5, we report the highest scores based on averages of the four traits. Considering the dimensions individually, better results can be obtained by using specific models.
PJ Avg 40.55 44.34 35.01 29.49
Lex 51.65 49.04 50.86 51.03 37.35 51.24 43.03 41.74 50.65 two datasets on the performance of models for personality detection, maintaining the 10-fold crossvalidation setting and by using the model performing better on average for YouTube and Twitter data (a character n-grams model). Table 6 contains the result of such experiments6. Scores are almost always lower compared to the in-domain experiments (excepts for NS as regards Twitter scores reported in Table 5: 46.15 ! 48.31), but quite increased compared to the majority baseline.
PJ Avg
MAJ 41.64 44.93 35.04 30.66 38.07
Lex 50.57 48.31 51.31 48.24 49.61
In the second setting, instead, we divided both corpora in fixed training and test sets with a proportion of 80/20 and ran the models using lexical features, in order to run a cross-domain experiment. For direct comparison, we run the model indomain again using this split. Results are shown
IN Pers
CROSS MAJ
TWI IN TWI
CROSS MAJ Pers in Table 7. Cross-domain scores are obtained with the best in-domain model.7 They drop substantially compared to in-domain, but are always above the baseline. 5
The experiments show that there is no single best model for personality prediction, as the feature contribution depends on the dimension considered, and on the dataset. Lexical features perform best, but they tend to be strictly related to the context in which the model is trained and so to overfit.
The inherent difficulty of the task itself is confirmed and deserves further investigations, as assigning a definite personality is an extremely subjective and complex task, even for humans.
Personal-ITY is made available to further
investigate the above and other issues related
to personality detection in Italian. The
corpus can lend itself to a psychological analysis
of the linguistic cues for the MBTI
personality traits. On this line, it is interesting to
investigate the presence of evidences linking
linguistic features with psychological theories about
the four considered dimensions
(EXTRAVERTINTROVERT, INTUITIVE-SENSING,
FEELINGTHINKING, PERCEIVING-JUDGING). First
results in this direction are presented in
The work of Elisa Bassignana was partially carried out at the University of Groningen within the framework of the Erasmus+ program 2019/20. 7Better results can be obtained with other specific models.