A report of the CL-A O MyChest Shared Task: Modeling Supportiveness and Disclosure

Abstract

This overview describes the o cial results of the CL-A Shared Task 2020 { #O MyChest. The dataset comprised a semi-supervised classi cation task, and an open-ended knowledge modeling task on a dataset of Reddit comments with annotations crowdsourced from Amazon Mechanical Turk. The Shared Task was organized as a part of the 3rd Workshop on A ective Content Analysis @ AAAAI-20, held in New York, USA, on February 7, 2020. This paper compares the participating systems in terms of their accuracy and F-1 scores at predicting di erent facets of self-disclosure. Feedback from the system runs was used to weed out labeling errors in the test set. The annotated test and training datasets, instructions, and the scripts used for evaluation are available at the GitHub repository.

-

There is a growing interest in understanding how humans initiate and hold conversations online. A plethora of social media platforms has emerged and been adopted by internet communities worldwide. Di erent cultures and communities have emerged around di erent social media platforms [ 3 ], where some social networking sites are intended more for discussions among professional contacts, e.g., LinkedIn; others are often appropriate for pursing topical interests, e.g., Twitter; for having reasoned debates, e.g., Reddit; still others were developed to provide technical support, e.g., StackOver ow. A de ning feature of these platforms is how their social norms di er. On di erent platforms, people choose to respond di erently to each other and share di erent kinds of information about themselves [ 6 ]. An interesting research problem that arises is to quantify the levels of disclosure and to apply them for cross-sectional or longitudinal analysis of social norms and platforms. In this Shared Task, we take the rst step towards approaching these problems, by examining the a ective aspect of online conversations among strangers. Our aim is to build a new resource to model how social media users reciprocate in conversations, with emotional and informational behavior that either o ers self-revelation or moral support. In this paper, we introduce the O MyChest conversation dataset and present the results of the concluded 2nd Computational Linguistics A ect Understanding (CL-A ) Shared Task on modeling interactive a ective responses. It was held in February 2020 as a part of the AAAI Annual Meeting in New York. 2

Background

Previous work exploring disclosure and support has usually examined its evidence in health forums [ 14,12 ]. In studies on general social media posts [ 11 ], women were found to self-disclose more than men, and people with a stronger desire for impression management are less likely to disclose about themselves online. Cross-platform di erences in language can enable greater or lesser predictive accuracy at identifying users' demographic information [ 6 ]. Anonymity is one of the many technological a ordances which is expected to make it easier for individuals to express negative feelings online [ 8 ]. Previous ndings o er a way to understand how platform behavior can di er, but they do not di erentiate between the information and emotional aspects of disclosure and support. Our Shared Task is motivated to address this research gap and to o er a way to distinguish emotional expressions from emotional support and informational disclosure from informational support. The ability to distinguish between these aspects would allow targeted interventions where mental health issues may be evident or where users' personal information may be at risk when they share too many personal details about themselves.

The work closest to our interest has provided annotation schemes to codify the type of disclosure [ 2 ] and support [ 12 ] in online help forums. Their work reports that support forums o er a higher degree of self-disclosure than discussion forums [ 2 ]. Furthermore, they reported that self-disclosure was often reciprocal, and reciprocity was more likely among female than male respondents. Other ndings suggest that it is emotional support [ 12 ], rather than information support, that predicts users' longevity in a health support group. On the other hand, informational support satis ed members' short-term information needs.

We were inspired to explore how easily these notions of disclosure and support can generalize into understanding casual conversations between users. To denoise the data, we decided to focus on discussions of relationships and opted to focus on Reddit sub-communities, which are likely to o er better training data thanks to the enforced community rules and strict moderation.

First, we provide the de nitional scope of disclosure and support for the CLA Shared Task: Emotional Disclosure: Comments that mention the author's feelings. Examples: { "My only concern was for my son." { "Fuck me that is beautiful." { "Thanks for sharing the story." { "My heart melted reading this xx"; { "I'm literally too jealous"; { "My heart is breaking for you."

Informational disclosure: Comments that contain at least some personal information about the author. Examples: { "I'm now 65 years old"; { "I've worked with kids with ODD and autism." { "I live in West Philly." { "Sounds like our bipolar kid."; { "She posted a screenshot of his porn history (gross)"; { "My mum told me that she was sexually abused as a kid."

Emotional Support: The comment is o ering sympathy, caring, or encouragement. Examples: { "Good luck, this shit is tough"; { "Good luck! but I'm afraid I have no advice"; { "You sound like a great person"; { "I'm so sorry."; { "That's a great story."

Informational support: This comment is o ering speci c information, practical advice, or suggesting a course of action. Examples: { "I wouldnt.."; { "You shouldn't.."; { "You can't..".; { "Why didn't you try this?"; { "Please talk to a professional." 3

Corpus

On Reddit, discussions of relationships typically happen on the r/relationships community. However, a preliminary examination suggested that the discussions are not the kind of `casual' conversations we were aiming for, and are instead more similar to a support forum. Responses to posts in this community would be skewed towards greater support and disclosure. We wanted a neutral, easy-togeneralize situation, where the pressure to reciprocate is substantively reduced. After further exploration, we decided to mix data from two subreddits. The rst one we selected was r/CasualConversations, a `friendlier' sub-community where people are encouraged to share what's on their mind about any topic. In essence, this is similar to the posting behavior encouraged on a typical social media platform. The second one we selected was r/O myChest, intended as `a mutually supportive community where deeply emotional things you can't tell people you know can be told.' We anticipated that a mixture of labeled data from both these platforms would give us a degree of heterogeneity in the confessional and emotional behavior while preserving the high topicality and post quality that is typical of Reddit posts. We provide further details of the dataset in the following subsections. 3.1

Dataset description

The CL-A

corpus comprises the following: { Unlabeled training set of posts (N=17,392): The top posts in 2018 in /r/CasualConversations and /r/O MyChest mentioning any of the terms boyfriend, girlfriend, husband, wife, gf, bf. Posts that are parents of comments in the training and test sets are separately identi ed. { Unlabeled training set of comments (N = 420,000): Over 420k sentences extracted from 130k comments posted to the unlabeled set of posts mentioned above. { Labeled training set (N = 12,860): 12,860 labeled sentences, extracted from the top comments posted to the top posts of the Reddit communities mentioned above. { Test set: (N = 5,000) Labeled sentences, extracted from the top comments made to the posts mentioned above.

A detailed breakdown of the labeled training and test sets is provided in Table 1. Data was collected by rst subsetting on the posts discussing relationships that were posted to either r/O myChest or r/CasualConversation. Posts about relationships were identi ed based on the presence of the seed words relating to romantic partners. Posts were then deduplicated, and all their underlying comments were collected. A sentence splitter was applied to obtain sentences, and a random sample of sentences which were at least 10 characters in length was then used for the pilot and con rmatory annotation tasks.

Annotation

Annotators were required to annotate each moment according to the inset questionnaire. The Disclosure and Support characteristics of each sentence were nally transformed into a binary (yes/no) coding and the labels were assigned based on a simple majority agreement between ve independent annotators. Only labels with 60% - 100% agreement were retained. The pairwise percentage agreement on the nal dataset was 71.2% each for emotional and informational disclosure, and 84.5% and 83.9% for emotional and informational support.

Instructions In this job, you will be presented with a comment made on Reddit, a popular discussion forum worldwide. The topic of the discussion is a casual conversation or a confession. Review the text of the comment and help us by answering a few yes/no questions about it. Each HIT takes about 30 seconds: <Comment appears here> Is this comment SHARING PERSONAL FEELINGS? NO/A LITTLE/A LOT { NO: This comment does not mention the author's feelings about anything.

("It's a book by Hemingway"; "Are you ok?"; "She was really mad at me.") { A LITTLE: This comment mentions the author's mild positive or negative feelings. ("My only concern was for my son."; "Fuck me that is beautiful."; "Thanks for sharing the story.") { A LOT: This comment contains deep positive or negative feelings or tears. ("My heart melted reading this xx"; "I'm not crying, you're crying!"; "I'm literally too jealous"; "My heart is breaking for you.") <Comment appears here> Is this comment SHARING PERSONAL INFORMATION? NO/A LITTLE/A LOT { NO: This comment does not mention the author's feelings about anything.

("It's a book by Hemingway"; "Are you ok?"; "She was really mad at me.") { A LITTLE: This comment mentions the author's mild positive or negative feelings. ("My only concern was for my son."; "Fuck me that is beautiful."; "Thanks for sharing the story.") { A LOT: This comment contains deep positive or negative feelings or tears. ("My heart melted reading this xx"; "I'm not crying, you're crying!"; "I'm literally too jealous"; "My heart is breaking for you.") <Comment appears here> Is this comment SUPPORTIVE? YES/NO { YES: This comment is o ering support to someone, either through sympathy, encouragement, or advice. ("Good luck, this shit is tough"; "Good luck! but I'm afraid I have no advice"; "Hey you tried your best"; "Have you tried family therapy?") { NO: This comment does not o er any support.. ("Thank you for your time."; "This is so sweet."; "Badass grandpa."; I'm now 65 years old"; "I've worked with kids with ODD and autism"; "I live in West Philly.") { GENERAL SUPPORT: The comment is o ering general support through quotes and catchphrases. ("What's the worst that could happen?"; "You only die once."; "All's well that ends well." ) (YES/NO) { INFORMATIONAL SUPPORT: The sentence is o ering information, advice, or suggesting a course of action. ("I wouldnt.."; "You shouldn't.."; "You can't..". "Why didn't you try this?"; "Please talk to a professional.") { EMOTIONAL SUPPORT: The sentence is o ering sympathy, caring, or encouragement. ("Good luck, this shit is tough"; "Good luck! but I'm afraid I have no advice"; "You sound like a great person"; "I'm so sorry."; "That's a great story.") 5

Overview of Approaches

Twelve teams signed up, and six teams nally submitted their results by the Shared Task deadline. The following paragraphs discuss the approaches followed by the participating systems, sorted in alphabetical order: { GATech USA[ 4 ]: The team from GATech followed a semi-supervised approach comprising transformer-based models. Their regularization was predicated on the assumption that the class distribution in the test set would be similar to that of the training set. { Gyrfalcon[ 10 ]: The team from Gyrfalcon Technology, California, proposed an algorithm to map English words into squared glyphs images, which they call Super Characters. These were implemented on a CNN Domain-Speci c Accelerator in order to capture properties of disclosure and support. { International Institute of Information Technology India [ 9 ]: The IIIT-H team employed a predictive ensemble model that combined predictions from multiple models based on ne-tuned contextualized word embeddings, RoBERTa and ALBERT. { Pennsylvania State University USA (PennState)[ 1 ]: The PennState team also followed an ensemble approach, but with BERT, LSTM, and CNN neural networks. In their rst model, they performed classi cation using BERT, ne-tuned their word representations, and obtained the hidden attention and sentence representation features in the CNN model, where they replaced the typical embedding layer with the pre-trained BERT model. { Sungkyunkwan team (SKKU)[ 5 ]: The SKKU team used a semi-supervised approach, with the original posts as contextual information, and applied BERT, GLoVe, and Emotional GLoVe embedding models, to represent the text for label prediction. { University of Ottawa (UOttawa) Canada[ 13 ]: The University of Ottawa team applied a deep multi-task learning approach that employed the logical relationship among the di erent labels to create `fragment layers,' that were used to build a multi-task deep neural network.

Results

Task 1: Predicting Disclosure and Support

This section compares the participating systems in terms of their performance. The results with the best-performing system runs from each of the participating teams are provided in Figure 1. The performance of individual system runs is provided in Table 2 and Table 3. For the detailed implementation of the individual runs, please refer to the system papers which are included in this proceedings volume.

Figure 1a shows that predicting disclosure was evidently a harder problem than predicting support. The best performance at predicting both emotional and informational disclosure was obtained from the team from UOttawa [ 13 ](Accuracy = .69). The second and third spots for predicting emotional disclosure went to IIIT [ 9 ] and GATech [ 4 ], with an accuracy of .62 and .61, respectively. Predictive performances for informational disclosure were rather close to one another, with Gyrfalcon [ 10 ] and GATech [ 4 ] coming in a close secondand third-places with accuracies of .64 and .63 respectively.

Figure 1b shows that IIIT [ 9 ], UOttawa [ 13 ], and GATech [ 4 ] were neckand-neck at predicting emotional and informational support, with IIIT getting a slight edge thanks to its performance on emotional support.

The most successful runs can be identi ed by referring to Table 4. 1 0.95 0.9 0.85 0.8 00..067.755 .069 .560 0.6 0.5 0.5 UOTTAWA CANADA

Four of the six systems that did Task 1 also did the bonus Task 2 to share insights based on the hidden attention or fragment layers in their deep learning models. The visualizations provided by UOttawa [ 13 ] are helpful in understanding how exactly the logical relationships between di erent labels are computed. Interestingly, their approach did not use any of the unlabeled data. Instead, their fragment layers appeared to infer the hierarchical relationship underlying the categories of disclosure and support. . 7

Error Analysis

We conducted a meta-analysis of system performances for Task 1 over all the sentences in the test set. When we ltered the sentences for which all or most of the approaches reported a false negative, we noted that the errors could be attributed to mislabeling, especially in the case of emotional disclosure, which had an unexpectedly high error rate. We expect that this may have happened because we transformed a 3-level annotation into a binary form; however, lowdisclosure sentences may be vastly di erent from high-disclosure sentences. In Table 5, we provide a count of the labeling errors identi ed (and corrected) through this process. In the true spirit of a Shared Task, we have applied this feedback to identify and correct these labels. The data with corrected labels has been released. We encourage future researchers to test their approaches with the new labels.

As is expected in such tasks, other errors appeared to be because of knowledge that was implicit in a sentence and formed the basis of annotators' labels but was not directly present in the sentence. For example, \Clearly, that's disturbing for anyone to experience." was marked positive for emotional disclosure by annotators, but was predicted to be negative by most participating systems. 8

Conclusion and Future Work

The 2nd CL-A Shared Task AAAI-20 is the rst of its kind of annotated datasets about disclosure and support in social media discussions. We have published the complete dataset to GitHub. We plan to release other labels complementary to this dataset in future tasks.

We conclude this overview with some of the main takeaways shared by our participating teams: { UOttawa suggests that when training a model on a task using noisy datasets, it is recommended to identify and separate the data-dependent noise from the signal, and to rely on patterns and relationships based on other features. Their exemplary approach does suggest new paradigms for conceptualizing deep multi-task learning problems. However, we wonder whether the presumptions could break, for instance, when the logical relationships are accidental. In the case of GATech [ 4 ], they relied on the label distribution information to regularize their models. However, we had consciously made the decision to have a larger proportion of positive cases in the test set, which may have ultimately hurt their model performance. Perhaps the takeaway would be to look for the semantic relationships in the data and not rely solely on numerical trends. { GATech rea rms our belief in the power of semi-supervised learning for model training and prediction at scale, showing respectable performance with an entropy-minimization approach for generated more labeled data from the unlabeled sample provided. However, they rely on the data distribution to introduce another error term to minimize the entropy of the output, and to minimize the divergence in output and input label distributions. For future modeling, we would recommend this approach only if the data generation and sampling processes are the same for both the training and the test set. { Gyrfalcon's Super Characters approach did not appear to wholly satisfy its authors, who recommend possibly upsampling, data augmentation, or word replacement, especially when ne-tuning on small datasets. { While it would logically be expected that adding context to models would improve model accuracy, SKKU observed no such performance gain. They recommend that rather than concatenation, adding suitable representations of context could be the right approach to enhance model performance.

Like our Shared Task last year [ 7 ], the ndings do support the emerging notion about the English language as a contextualized emotional vector space, with the best performances reported by approaches that incorporated task-speci c embeddings from other language models. Relying on emotional signals and the hierarchical structure of labels alone appears to have provided su cient predictive performance. We note that in this version of the Shared Task, we did not observe any of our teams to have used syntactic information, or in building domain-speci c embeddings, which were some of the more successful approaches last year.

It remains an open problem whether the models trained on this data will generalize to measure disclosure and support other platforms and conversations, and one for which we welcome future work and feedback.

Acknowledgement. Support for this research was provided by a Nanyang Presidential Postdoctoral Award and an Adobe Research Award.

References

1. Akiti , C. , Rajtmajer , S. , Squicciarini , A. : Contextual representation of selfdisclosure and supportiveness in short text . In: Proceedings of the 3rd Workshop on A ective Content Analysis @ AAAI (A Con2020) . New York, New York ( February 2020 )

2. Barak , A. , Gluck-Ofri , O. : Degree and reciprocity of self-disclosure in online forums . CyberPsychology & Behavior 10 ( 3 ), 407 { 417 ( 2007 )

3. Boyd , D.M. , Ellison , N.B. : Social network sites: De nition, history, and scholarship . Journal of computer-mediated Communication 13 ( 1 ), 210 { 230 ( 2007 )

4. Chen , J. , Wu , Y. , Yang , D. : Semi-supervised models via data augmentation for classifying interactive a ective responses . In: Proceedings of the 3rd Workshop on A ective Content Analysis @ AAAI (A Con2020) . New York, New York ( February 2020 )

5. Hyun , J. , Bae , B.C. , Cheong , Y.G. : [CL-A Shared Task] Multi-label text classi cation using an emotion embedding model . In: Proceedings of the 3rd Workshop on A ective Content Analysis @ AAAI (A Con2020) . New York, New York ( February 2020 )

6. Jaidka , K. , Guntuku , S.C. , Ungar , L.H. : Facebook versus twitter: Di erences inselfdisclosure and trait prediction . In: Twelfth International AAAI Conference on Web and Social Media ( 2018 )

7. Jaidka , K. , Mumick , S. , Chhaya , N. , Ungar , L. : The CL-A happiness shared task: Results and key insights ( 2019 )

8. Ma , X. , Hancock , J. , Naaman , M. : Anonymity, intimacy and self-disclosure in social media . In: Proceedings of the 2016 CHI conference on human factors in computing systems . pp. 3857 { 3869 ( 2016 )

9. Pant , K. , Dadu , T. , Mamidi , R.: Bert-based ensembles for modeling disclosure and support in conversational social media text . In: Proceedings of the 3rd Workshop on A ective Content Analysis @ AAAI (A Con2020) . New York, New York ( February 2020 )

10. Sun , B. , Yang , L. , Sha , H. , Lin , M.: Multi-modal sentiment analysis using super characters method on low-power cnn accelerator device . In: Proceedings of the 3rd Workshop on A ective Content Analysis @ AAAI(A Con2020) . New York, New York ( February 2020 )

11. Wang , Y.C. , Burke , M. , Kraut , R.: Modeling self-disclosure in social networking sites . In: Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing . pp. 74 { 85 ( 2016 )

12. Wang , Y.C. , Kraut , R. , Levine , J.M.: To stay or leave? the relationship of emotional and informational support to commitment in online health support groups . In: Proceedings of the ACM 2012 conference on computer supported cooperative work . pp. 833 { 842 ( 2012 )

13. Xin , W. , Inkpen , D. : [CL-A Shared Task] Detecting disclosure and support via deep multi-task learning . In: Proceedings of the 3rd Workshop on A ective Content Analysis @ AAAI (A Con2020) . New York, New York ( February 2020 )

14. Yang , D. , Yao , Z. , Kraut , R.: Self-disclosure and channel di erence in online health support groups . In: Eleventh International AAAI Conference on Web and Social Media ( 2017 )