=Paper=
{{Paper
|id=Vol-2006/paper042
|storemode=property
|title=Monitoring Adolescents' Distress using Social Web data as a Source: the InsideOut Project
|pdfUrl=https://ceur-ws.org/Vol-2006/paper042.pdf
|volume=Vol-2006
|authors=Roberto Basili,Valentina Bellomaria,Niels Jonas Bugge,Danilo Croce,Francesco De Michele,Federico Fiori Nastro,Paolo Fiori Nastro,Chantal Michel,Stefanie Schmidt,Frauke Schultze-Lutter
|dblpUrl=https://dblp.org/rec/conf/clic-it/0001BBCMNNMSS17
}}
==Monitoring Adolescents' Distress using Social Web data as a Source: the InsideOut Project==
https://ceur-ws.org/Vol-2006/paper042.pdf
Monitoring Adolescents’ Distress using Social Web data as a Source:
the InsideOut Project
Basili Roberto†‡ , Bellomaria Valentina‡ , Bugge Niels J.? , Croce Danilo†‡ ,
De Michele Francesco• , Fiori Nastro Federico• , Fiori Nastro Paolo• ,
Michel Chantal?∗ , Schmidt Stefanie J.? , Schultze-Lutter Frauke?◦
†
University of Roma, Tor Vergata ‡ Reveal srl ? University of Bern ∗ University of Geneva
•
Sapienza University of Rome ◦ Heinrich-Heine University, Düsseldorf
{basili|croce}@info.uniroma2.it, bellomaria@revealsrl.it, niels.bugge@gmail.com
{chantal.michel|stefanie.schmidt|frauke.schultze-lutter}@kjp.unibe.ch
paolo.fiorinastro@uniroma1.it, {francescodemichele1981|federico.fiori.nastro }@gmail.com
Abstract 1 Introduction
English. The role of Social Media in the Among adolescents, the use of Social Media, such
psychological and social development of as Twitter, Facebook or Instagram, has grown ex-
adolescents and young adults is increas- ponentially in the past years. This makes them a
ingly important as it impacts on the quality valuable source of information on the well-being
of their interpersonal communication dy- of adolescents, but also concerning on their men-
namics. The InsideOut project explores tal health. Mental disorders are the main cause of
the possibility to use Social Web mining disability in adolescents and young adults (Gore
methodologies and technologies to col- et al., 2011), affecting an average of 10 to 20%
lect information about adolescents’ dis- of youth worldwide (Kieling et al., 2011). Thus,
tress from their micro-blogging activities. for the emerging complex relationship between the
The project is promoting a complex lan- use of Social Media, mental health and well-being
guage processing workflow to approach (Best et al., 2014), Social Media are a valuable
the collection, enrichment and summariza- source of information on the mental health and
tion of user generated contents over Twit- well-being of adolescents.
ter. This paper presents the general archi-
Social Media thus play an increasingly impor-
tecture of the InsideOut Web Platform and
tant role in the psychological and social devel-
the resources produced by an integrated
opment of adolescents as it impacts on the qual-
effort among computer science and men-
ity of their social interactions and networks. Any
tal health professionals.
attempt to study and govern mental health in
Italiano. Il ruolo dei Social Media nella young communities (adolescents, students, inter-
crescita psicologica e sociale risulta es- est groups) must take into account an effective and
sere sempre più importante poiché in- large scale methodology to monitor all the behav-
fluisce sulla qualità e sulle dinamiche iors on the Web that exhibit and impact on men-
di comunicazioni interpersonali, special- tal habits, trends and social practices. The pos-
mente riguardo le ultime generazioni. Il sibility of predicting writers demographics from
progetto InsideOut esplora la applica- their writings is an important research topic in the
bilità di metodologie e tecnologie che con- Computational Linguistic Community. In fact, the
sentono l’individuazione nel Web di evi- idea that a writer’s style may reveal age, gender
denze riferibili a sorgenti di stress negli or other sociodemographic information has been
adolescenti. Il progetto propone un work- also targeted in the “Plagiarism analysis, Author-
flow di elaborazione linguistica in grado ship identification, and Near-duplicate detection”
di gestire la raccolta, l’arricchimento e la (PAN) (e.g., (Rangel et al., 2014; Rangel et al.,
sintesi dei contenuti generati dagli utenti 2015; Rangel et al., 2016)) or other experiences
su Twitter. Nel paper verrà presentata (Sulis et al., 2016) whose aim was to infer a user’s
l’architettura generale della piattaforma gender, age, native language or personality traits,
Web InsideOut e le risorse che derivano by analyzing the respective texts.
dal lavoro congiunto di ricercatori prove- In this paper, the InsideOut project is presented.
nienti dall’ambito informatico e medico. It explores the possibility to use Social Web min-
ing methodologies and technologies to collect in- fa schifo...” (”This school sucks...”) or ”Devo stu-
formation about adolescents’ distress from their diare.” (”I have to study.”).
micro-blogging activities. The project is promot- In order to enable such queries the following
ing a complex language processing workflow to services have been implemented:
approach the collection, enrichment of user gen- Data collection services: services dealing with
erated contents on Twitter: messages written by the extraction of data (messages/user information)
a set of targeted community of users (e.g. from from targeted social networks. These services are
a school) are enriched with semantic metadata re- designed both to collect messages referring to a
flecting the expressed topics (e.g. social vs inti- specific topic or hashtag, such as ”#maturità” or
mate relationships) and the attitude of the writ- messages exchanged between users belonging to
ers. The goal is to use this large scale evidence specific communities, such as a members of a tar-
to support a comprehensive psychological charac- geted school class. Among such services, we also
terization of adolescent communities and to pave implemented Author Profiling services that auto-
the way towards effective applications of preven- matically determine the age of the writers (e.g. to
tive and intervention efforts. The general archi- filter adolescent’s messages) but these specific ser-
tecture of the InsideOut Web Platform and the re- vices are out of the scope of this work.
sources produced by an integrated effort of com- Semantic annotation services: services dealing
puter science specialists and mental health profes- with the semantic annotation of gathered mes-
sionals will be presented. These data supported sages; once downloaded, they are automatically
the exploratory evaluation where inter-annotation annotated with the semantic metadata described in
agreement scores and the performance over real the next section.
data in the task of psychologically enriching user Storage services: services to store (possibly
writings have been obtained. large-scale) collections of messages, communities
In the rest of the paper, Section 2 describes the and semantic metadata in NoSQL databases, im-
overall workflow underlying the InsideOut Plat- plemented in MongoDB.
form. Section 3 describes the semantic models at Reporting services GUI: services that aggregate
the base of the semantic annotation process whose messages, metadata and users to enable advanced
first result is the annotated corpus and the ex- report, such as shown in Figure 1.
ploratory evaluation presented in Sections 4 and
5, respectively. Section 6 derives the conclusion. 3 Distress Characterization: The
semantic modeling
2 The InsideOut Web Platfrom
In order to synthesize the amount of information
The InsideOut Web Platform aims at supporting made available on Social Media, we need to look
mental health studies concerning the causes of dis- at different semantic dimensions that can be as-
tress in adolescents. To this aim, a comprehensive sociated with the writer’s emotion, sentiment and
service-oriented architecture has been designed mental status. Given that no direct diagnosis about
and implemented to collect messages from So- mental health of an individual can be traced from
cial Networks (such as Twitter) written by targeted or over one single message (but it is rather in-
communities of adolescents and enrich them with spired by the observation of behaviors across tem-
semantic information reflecting discussed topics poral and social dimensions) we need to frame the
and corresponding attitudes of the writers. mental state related information observable in So-
This enables specific kinds of queries and data cial Media within a comprehensive description of
aggregations, such as the pie chart shown in Fig- a subject.
ure 1, which summarizes the topics discussed by So we decided to focus on the experiential di-
a community of users, e.g. concerning S CHOOL, mension and start from the so-called Life Event
FAMILY, or A LCHOOL AND D RUGS. By select- dimension that expresses topics of interest and
ing a specific topic, such as S CHOOL, the system daily events in a young person’s life. At the mo-
shows only those messages where the writer ex- ment of writing, these have been discretized in
presses a specific attitude, such as a D ISTRESS. eighteen different classes, as listed in Table 1.
In the same Figure, the distressful messages con- Each message can be assigned to one or more
cerning school are shown, such as ”Questa scuola classes characterizing the possibly multiple topics
Figure 1: The InsideOut Interface
that can be mentioned in a message. For exam- experienced as a negative event and as distressing
ple, in the message ”Odio la scuola ma adoro i when teacher’s or parent’s judgment is negative.
miei compagni” (”I hate school but I love my class- It is worth noting that the Subjective and Expe-
mates”) the writer refers to the S CHOOL and S O - rience dimension are nevertheless correlated, but
CIAL R ELATIONSHIP life events. they target different kinds of perception: the fol-
Moreover, a Subjective emotional dimension lowing message ”Mi sono rotto una gamba.” (”I
is targeted to capture the way the subject relates broke my leg.”) can be considered D ISTRESSFUL
to the event in the micro-blog he writes, i.e., for the writer even if no agreement or rejection is
whether it is related to as a clearly positive or neg- made w.r.t. the event.
ative event, as a rather neutral statement, or in an The information observable in a tweet is thus
ironic way. We referred to the traditional mod- mapped into a set of three independent dimen-
eling for subjectivity analysis (Rosenthal et al., sions: (i) the type of Life Events le the message re-
2017; Barbieri et al., 2016), adopting P OSITIVE, lates to (ii) the sentiment s of the event (P OSITIVE,
N EGATIVE and N EUTRAL classes; as an example N EGATIVE, N EUTRAL) and (iii) experience-level
”Odio la scuola” (”I hate school”) is N EGATIVE, e related to the event (among H ELPFUL, D IS -
while ”Domani la scuola è chiusa” (”Tomorrow TRESSFUL or N EUTRAL ). For example, the tweet
my school is closed.”) is N EUTRAL. ”Quanto odio la mia classe... per fortuna mia
sorella mi aiuta!” (”I hate my class so much...
Finally, a further dimension called Experience thankfully, my sister helps me!”) is assigned to
tried to capture the writer’s personal affect towards the (le,s,e) triples: (S CHOOL, N EGATIVE, D IS -
an event, e.g., whether it (i) is causing distress or TRESSFUL ) and (FAMILY , P OSITIVE , H ELPFUL).
other negative feelings such as anger or sadness,
(ii) is regarded as helpful or causing positive feel- 4 The InsideOut Annotated Corpus
ings such as happiness or affection or (iii) is not
associated with any perceivable emotional reac- In the annotation process, annotators selected
tion (neutral). As an example, a school perfor- tweets written by adolescents (that have been pre-
mance can be a positive experience if satisfactory viously manually validated) both in English and
for the teacher or the parents, thus being experi- Italian and enriched them with triples (le, s, e), as
enced as helpful by the writer, while it might be discussed in the previous section. In the annota-
Table 1: Life Events description
Life Events Definition
A LCHOOL , D RUGS All actions and ideas involving the misuse of medications or the use of illegal drugs or alcohol.
A PPEARANCE All messages related to the physical appearance of the writer or of other people.
Thoughts, references and considerations directly connected to the world of crime or that express an attitude
C RIME , ABUSE AND MOBBING
or an opinion of the adolescent towards that sphere.
Events involving or statements related to the family members, such as parental habits, relationships, gener-
FAMILY
ational clashes.
Event related to the financial status of the young person or his own family; needs of money for important
F INANCIAL AND P OSSESSION needs or expectations; strongly perceived needs that strictly depend on the economic status and capability of
the subject or his family.
F OOD AND DRINK All actions and ideas involving food and drink (not alcohol).
F UTURE Events or thoughts related to the perception the adolescent has about his own future.
G IRLFRIEND / BOYFRIEND (Usually strongly emotional) relationships based on sentimental and sexual attraction, involving gender
PERSONAL RELATIONSHIP aspects.
All events, expectations or preferences evoked by entertainment related activities or personal interests (e.g.
H OBBIES AND INTERESTS hobbies, fun, VIP) usually producing fun or connected with time-consuming helpful activities (e.g. games,
TV, Social media, Celebrities).
Expressions related to mental well-being and to the health dimension but not related to physical aspects; this
M ENTAL (W ELL - BEING ) H EALTH
class includes sleep problems.
P ERSONAL , I NTERNAL STRESSORS , General opinions, convictions or beliefs of the subject related to his own feelings and his personal sphere;
BELIEFS general considerations regarding emotions, spirituality, stressors but not politics or social issues.
Thoughts, complains, considerations related to the physical health dimension, including conditions, nutri-
P HYSICAL (W ELL - BEING ) H EALTH
tion, diseases, remedies and treatments.
All thoughts, considerations, reports regarding social, political and anthropological aspects of the close or
P OLITICS , SOCIAL ISSUES , ETC
general environment, as perceived by the young person.
Every perception about the locations where the subject lives or spends most of his time, including environ-
R ESIDENCE
mental aspects or weather.
All events dominated by the school experience (comprehends social interactions IF only limited to school
S CHOOL
environment).
S EX AND ROMANCE Events or experience specifically grounded at the sexual level, not including boyfriend-hood.
All thoughts and events related to the relational dimension of the young people, but not involving the family,
S OCIAL RELATIONSHIPS
the criminal, the working/school and the boyfriend-hood dimension.
All events related to the relational dimension of the young people, caused or maintained alive by activities
W ORK
or dependencies based on the working condition of the subject of a member of his family.
Table 2: Corpus Statistics Table 3: Inter-annotation agreement
Language Italian English Annotators Agreement - IT
Number of tweets 2,037 1,072 Precision Recall F1
- at least two annotators 1,074 1,072 Life Event 85.76% 60.24% 70.76%
- only one annotator 963 - Sentiment 72.43% 50.88% 59.77%
# annotators 4 4 Experience 74.28% 52.17% 61.29%
# of (le, s, e) triples 2,517 2,811 Annotators Agreement - EN
Avg (le, s, e) for tweet 1.2 2.6 Precision Recall F1
Avg token per tweet 16 15 Life Event 80.69% 56.17% 66.09%
Sentiment 63.77% 44.05% 51.99%
Experience 64.16% 44.35% 52.35%
tion process, each annotator starts by associating
one or more le to a message1 and, for each of
them, the corresponding s and e must be provided.
Each message was initially annotated by two an- initial annotations as measured annotations. Re-
notators. After this first stage, the annotators in sults are shown in Table 3. For the Sentiment and
disagreement were asked to converge, in order to Experience dimension, we only focused on those
acquire a gold standard dataset. Only for Italian, messages sharing the same le. These agreement
we extended the dataset with a set of 963 messages scores are quite low, confirming the difficulty of
that were annotated by only one annotator, with- these kinds of analyses in Social Networks. The
out further refinements. The overall statistics of lowest score is Recall: it means that annotators
the dataset are shown in table 2. generally assign a reasonable class, but it is very
In order to measure the complexity of the anno- difficult to be exhaustive: as an example in the
tation process, we measured the inter-annotation tweet ”Odio la gente che mastica rumorosamente.
agreement2 . Given the possibility to associate Mi innervosisce troppo!!!” (”I hate the people who
more than one le to a message, we decided to mea- chew loudly. It makes me very upset!!!”) has been
sure the agreement in terms of Precision, Recall assigned to S OCIAL R ELATIONSHIPS by an an-
and F1, by considering the annotations confirmed notator while to P ERSONAL , INTERNAL STRES -
SORS , BELIEFS by the other one. At the end,
after the agreement step as gold-standard and the
both were accepted and added to the gold stan-
1
Each annotator can associate zero, one or more les to a dard. Agreement measured over the Italian mes-
message.
2
The inter-annotation agreement considered only mes- sages is higher if compared with the English coun-
sages annotated by at least two annotators. terpart: one of the main reasons for this is due to
Table 4: Results concerning the quantitative analysis of messages.
Life Event Sentiment Experience
Lang. Tweets Prec. Rec. F1 Accuracy Accuracy
En 1062 76.0% 31.3% 44.0% 62.8% 62.0%
It 1992 72.2% 47.2% 57.1% 67.8% 67.6%
the fact that Italian messages were annotated by class while no LE is assigned, otherwise. Per-
native speakers, while English messages were an- formance is thus measured in terms of Precision
notated by German native speakers. (the percentage of le correctly introduced by the
system), Recall (the percentage of le from the or-
5 Exploratory Evaluation acle that have been correctly recovered) and F1
(the harmonic mean between Precision and Re-
In order to assess the applicability of the an-
call)5 . Regarding the Subjective and Experience
notation process, we measured the quality of
dimensions, once a le is known, the classifier is
the system in the automatic recognition of Life
always requested to associate a message to the s
Event (LE), Sentiment and Experience classes.
and e labels, in order to generate consistent triples
We modeled this problem as a classification task
in the form (le, s, e). Being a multi-classification
and adopted the Support Vector Machine learn-
schema were the classifier always outputs a class,
ing algorithm (Vapnik, 1995) in a One-VS-ALL
Precision is always equal to Recall6 , as well to the
schema, implemented within the Kernel-based
F1. In order to avoid redundancy, only one mea-
Learning Platform (KeLP), presented in (Filice et
sure is reported and it is referred as Accuracy as
al., 2015)3 . We evaluated the three targeted di-
it also corresponds to the percentage of messages
mensions of LE, Subjectivity and Experience sep-
correctly associated to the gold-standard label.
arately4 in a 10-Fold cross-validation schema: at
Preliminary results are shown in Table 4, both
each time a fold is selected as test set, while an-
for English and Italian. Regarding the LE dimen-
other set is the validation set used to estimated the
sion, the adopted strategy results in a Precision
SVM parameters. Each tweet is modeled by using
higher than 70%, but at in a lower Recall. We be-
the following feature representations: a Bag-of-
lieve this is mainly due to the reduced size of the
words representation, Bag-of-n-grams (with n = 2
dataset: it is even more relevant for English where
and n = 3) and a distributional representation
only a 31% of Recall was detected. This number
based on Word Embedding (Mikolov et al., 2013)
is consistently higher for the Italian dataset, where
so that a message is the linear combination of
almost the double of examples is in fact provided
its nouns, verbs, adjective and adverbs. For the
and almost half of the tweets were only annotated
LE classifier, we built a similar distributional rep-
by one person, thus reducing the odds for differ-
resentation of the eighteen LE definitions shown
ences in annotations. Anyway, these results are
in Table 1: we introduced additional features in
consistently higher with respect to a baseline: the
terms of the 18-dimensional vector containing the
correct LE classification given the random selec-
cosine similarity between the distributional repre-
tion from 18 classes would achieve a F1 no higher
sentation of a tweet and the LE definitions. For
than 3%; if we require two correct classifications,
Subjectivity and Experience, we added some spe-
in line with the average le per tweet shown in Ta-
cific features, modeling the presence of emoticons,
ble 2, this baseline drops to 0.3%. Moreover, it is
punctuation marks (such as exclamation points),
worth noting that the adopted conservative strat-
upper case words and elongated words. Moreover,
egy has been adopted to have a higher precision:
we added features such as the length of the mes-
since we are able to collect a huge amount of mes-
sage (in terms of words and characters).
sages from social network, we can afford to lose
Regarding the LE dimension, we adopted a con-
5
servative strategy so that the system assigns a new Since a message could be associated to multiple le the
evaluation is not message-based but annotation-based.
LE to a message whereas the SVM classifier pro- 6
It may be the case that the LE classifier produces a num-
vides a positive confidence for the corresponding ber of les different from the number of the ones provided in
the gold-standard. As a consequence, when evaluating this
3
Available at www.kelp-ml.org. specific classifier, each message potentially introduces a dif-
4
When considering Subjectivity and Experience, a gold ferent number of false positives and false negatives, so Preci-
standard Life event is assumed. sion and Recall will diverge.
some messages (often characterize by too little in- Simone Filice, Giuseppe Castellucci, Danilo Croce,
formation in very short messages) instead of in- and Roberto Basili. 2015. Kelp: a kernel-based
learning platform for natural language processing.
troducing too many noisy meta-data in the over-
In Proceedings of ACL: System Demonstrations,
all workflow. Results concerning sentiment are Beijing, China, July.
generally consistent with respect to international
Fiona M Gore, Paul JN Bloem, George C Patton, Jane
benchmark in English (Rosenthal et al., 2017) or
Ferguson, Vronique Joseph, Carolyn Coffey, Su-
in Italian (Barbieri et al., 2016) where almost all san M Sawyer, and Colin D Mathers. 2011. Global
systems achieved an Accuracy between 60% and burden of disease in young people aged 1024 years:
65% (even using larger datasets). Overall, this re- a systematic analysis. The Lancet, 377(9783):2093
sult seems to be significant, as in line with the first – 2102.
outcome of the inter-annotation agreement. How- Christian Kieling, Helen Baker-Henningham, My-
ever, a further analysis is required to adopt more ron Belfer, Gabriella Conti, Ilgi Ertem, Olayinka
complex models for classification of such short Omigbodun, Luis Augusto Rohde, Shoba Srinath,
Nurper Ulkuer, and Atif Rahman. 2011. Child and
messages, such as more complex kernels (Agar- adolescent mental health worldwide: evidence for
wal et al., 2011) or deep methods (Kim, 2014). action. The Lancet, 378(9801):1515–1525.
6 Conclusions Yoon Kim. 2014. Convolutional neural networks for
sentence classification. In Proceedings of the 2014
This paper summarizes the InsideOut project Conference on Empirical Methods in Natural Lan-
guage Processing, EMNLP 2014, October 25-29,
where the possibility to use Social Web min- 2014, Doha, Qatar, pages 1746–1751.
ing methodologies and technologies to gather ev-
idence about the adolescents’ mental distress.The Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey
Dean. 2013. Efficient estimation of word represen-
semantic model defined here and the annotated re- tations in vector space. CoRR, abs/1301.3781.
source pave the way to a long-term joint research
between computer science specialists and mental Francisco Rangel, Paolo Rosso, Irina Chugur, Martin
Potthast, Martin Trenkmann, Benno Stein, Ben Ver-
health professionals. The outcomes suggest the hoeven, and Walter Daelemans. 2014. Overview of
applicability of the devised methodology to larger the 2nd author profiling task at pan 2014. In CLEF
communities and different languages. Since the evaluation labs and workshop, pages 898–927.
system is currently active over Twitter, the final Francisco Rangel, Fabio Celli, Paolo Rosso, Martin
version of the paper will discuss about 5 months Potthast, Benno Stein, and Walter Daelemans. 2015.
of continuous monitoring outcomes towards Ital- Overview of the 3rd author profiling task at pan
ian and English speaking communities, with inter- 2015. In CLEF 2015 Evaluation Labs and Work-
shop Working Notes Papers, pages 1–8.
esting evidences about the future of our project as
a novel and ambitious Social Computational Sci- Francisco Rangel, Paolo Rosso, Ben Verhoeven, Wal-
ence application. ter Daelemans, Martin Potthast, and Benno Stein.
2016. Overview of the 4th author profiling task at
pan 2016: cross-genre evaluations. Working Notes
Papers of the CLEF.
References
Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017.
Apoorv Agarwal, Boyi Xie, Ilia Vovsha, Owen Ram-
SemEval-2017 task 4: Sentiment analysis in Twit-
bow, and Rebecca Passonneau. 2011. Sentiment
ter. In Proceedings of the 11th International Work-
analysis of twitter data. In Proceedings of the Work-
shop on Semantic Evaluation, SemEval ’17, Van-
shop on Languages in Social Media, LSM ’11, pages
couver, Canada, August. Association for Computa-
30–38, Stroudsburg, PA, USA.
tional Linguistics.
Francesco Barbieri, Valerio Basile, Danilo Croce, Emilio Sulis, Cristina Bosco, Viviana Patti, Mirko Lai,
Malvina Nissim, Nicole Novielli, and Viviana Patti. Delia Irazú Hernández Farı́as, Letizia Mencarini,
2016. Overview of the evalita 2016 sentiment polar- Michele Mozzachiodi, and Daniele Vignoli. 2016.
ity classification task. In Proceedings Fifth Evalua- Subjective well-being and social media. A semanti-
tion Campaign of Natural Language Processing and cally annotated twitter corpus on fertility and parent-
Speech Tools for Italian. Final Workshop (EVALITA hood. In Proceedings of Third Italian Conference on
2016), Napoli, Italy, December 5-7, 2016. Computational Linguistics (CLiC-it 2016), Napoli,
Italy, December 5-7, 2016.
Paul Best, Roger Manktelow, and Brian Taylor. 2014.
Online communication, social media and adolescent V Vapnik. 1995. The nature of statistical learning the-
wellbeing: A systematic narrative review. Children ory.
and Youth Services Review, 41:27 – 36.