Introduction

The purpose of the CL-Aff Shared Task is to challenge the current understanding of emotion through a task that models the experiential, contextual and agentic attributes of happy moments. It has long been known that human affect is context-driven, and that labeled datasets should account for these factors in generating predictive models of affect. The Shared Task is organized in collaboration with researchers at Megagon Labs and builds upon the HappyDB dataset [1], comprising human accounts of 'happy moments'. The Shared Task comprised of two sub-tasks for analyzing happiness and well-being in written language, on a corpus of over 80,000 descriptions of happy moments, as described here: Given: An account of a happy moment, marked with individual's demographics, recollection time and relevant labels.

-Task 1: Semi-Supervised classification task -Predict thematic labels (Agency/Sociality) on unseen data, based on a small labeled and large unlabeled training data. 5-Task 2: Suggest interesting ways to automatically characterize the happy moments in terms of affect, emotion, participants and content.

The task, given its predictive and open-ended interpretive aspects is relevant for the computational linguistics, natural language processing, artificial intelligence and the psycholinguistics communities. The aim is to engage scholarly interest and crowdsource new ideas and linguistic approaches to define happiness. Details on the psycholinguistic underpinnings of the annotation task are provided in a different, forthcoming paper [5].

Evaluation: The performance of Systems was compared based on their Accuracy and F-1 measure at predicting the Agency and Sociality labels on the unseen test dataset. This was done using an automatic evaluation script, available on Github6 .

Dataset description

The CL-Aff corpus comprises the following:

-Labeled training set (N = 10,560): Single-sentence happy moments from the available HappyDB corpus, annotated with demographic labels of the author, as well as labels that identify the 'agency' of the author and the 'social' characteristic of the moment, as well as concept labels describing its theme. Authors' demographic labels were available to the Shared Task participants but not the 'agency' or 'social' characteristics.

The Agency and Sociality characteristics of each happy moment were decided by a simple majority agreement between three independent annotators using a binary (yes/no) coding.

Corpus development

Collecting the happy moments

We followed the format of the original HappyDB AMT task [1] to collect a second dataset of 20,000 happy moments, which was to be the unseen test data in the CL-Aff Shared Task. The following instructions were provided to the workers.

Instructions

What made you happy? Reflect on the past <duration>, and recall three actual events that happened to you that made you happy. Describe your happy moments with a complete sentence. Write three such moments. You will also be asked to note for how long each event made you happy. This task also has post-task questions. Please be sure to answer the questions. Examples of happy moments we are NOT looking for (e.g., events in distant past, incomplete sentence): The day I married my spouse; My dog. < Enter moment here > For how long did that event make you happy? Select the answer that is most appropriate.

Each AMT worker was required to enter three happy moments experienced within a specific time period. Half of the questionnaires specified a time period of 24 hours, while the other half with a <time period> of 3 months. The options provided for the follow-up question about the duration (i.e., the length) of happiness were 'All day, I'm still feeling it,' 'Half a day,' 'At least one hour,' 'A few minutes' or 'Not Applicable.' After the participant answered these questions, demographic information was collected about their country, age, gender ('Male','Female','Other','Not Applicable'), marital status ('single', 'married', 'divorced', 'separated', 'widowed' or 'Not Applicable'), and whether or not they have children ('yes','no').

Annotation

Annotators were required to annotate each moment along two binary dimensions -Agency and Sociality. We draw from Paulhas' conceptualization of selfpresentation according to the two factors of Agency and Communion [7]. Previous work exploring the evidence of agency in writing has adapted it to mean their locus of control, or the degree to which an author in control of their surroundings [9]. Sociality conceptualizes interpersonal engagement, evinced in writing as the description of any activity performed with or in the company of others [6].

Instructions Read the following happy moment. Choose any of the following that applies:

Agency: Is the author in control? YES/NO Examples of sentences where the author is in control (Answer is YES):

-"I ran on the treadmill for 20 minutes straight when I could barely do 5 minutes 3 months ago." -"Going out to a special birthday lunch for my great-grandmother in law's birthday."

Examples of sentences where the author is not in control (Answer is NO):

-"My youngest daughter got accepted to many prestigious universities and accepted an offer to attend college in San Diego."

-"A small business deal change over for small profit."

Social: Does this moment involve other people other than the author? YES/NO Please note that objects (e.g., bus, work) should not be counted as social.

Examples of sentences which involve other people (Answer is YES):

-"Going out to a special birthday lunch for my great-grandmother in law's birthday." -"My youngest daughter got accepted to many prestigious universities and accepted an offer to attend college in San Diego."

Note that sometimes a person is implicitly involved although not explicitly mentioned. In this case, we still wish to label the happy moment as social. E.g., "I received compliments on my tattoo."

Examples of sentences which are not social (Answer is NO):

-"I ran on the treadmill for 20 minutes straight when I could barely do 5 minutes 3 months ago." -"The bus came on time, so I reached work early." <Happy moment appears here> Agency: Is the author in control? YES/NO Social: Does this moment involve other people other than the author? YES/NO

Topic labeling

Annotators were presented with a happy moment, and a set of four potential topics which it was likely describing. Annotators were asked to mark all the tags which referred to what it was about. Each moment could score a maximum of four tags if at least two annotators agreed on them.

Instructions Read the following text. Select all categories that are relevant to the text from among those provided. If none of the categories is a great fit, select "none of the above" <Topic 1> <Topic 2> <Topic 3> <Topic 4>

Overview of Approaches

Eleven teams participated in the Shared Task. The following paragraphs discuss the approaches followed by the participating systems, sorted in the order in which they signed up to participate in the task.

-Arizona State University (ASU) [10]: The team from ASU proposed a Word Pair Convolutional Model (WoPCoM) to accomplish Task 1. The proposed model is motivated by the hypothesis that a small set of word-pair features are important to capturing the agency/social nature of happy moments. They trained a convolutional neural network (CNN) to predict on the unlabeled data. -University of California Santa Cruz (UCSC) [15]: The UCSC team participated in both tasks. For Task 1, they explored the use of syntactic, emotional, and survey features with semi-supervised learning, specifically experimenting with XGBoosted Forest and CNN models. For Task 2, the team trained similar models to predict concepts, and based on the difficulty of doing so, hypothesized about the nature of the themes in the happy moments. -International Institute of Information Technology Hyderabad (IIIT-H) [12]:

The IIIT-H team employed an inductive transfer learning technique (ITL). They pre-trained a AWD-LSTM neural net on the WikiText-103 corpus, and then introduced an extra step to adapt the model to Happy moments. -Gyrfalcon [11]: The team from Gyrfalcon Technology, California, proposed an algorithm to map English words into squared glyphs images. Then, they applied a 2D-CNN model over these images in order to capture the sentiment. -A*STAR [4]: The IHPC-A*STAR team participated in both tasks. For Task 1, they used emotion intensity in happy moments to predict agency and sociality labels. They defined a set of five emotions (valence, joy, anger, fear, sadness) and use a previously developed tool, CrystalFeed, to label each moment with the corresponding five emotion intensities. Combining these features with additional word-embedding features, they trained a logistic regression model. For Task 2, the team explored how these different emotions are manifested across the different concept labels. -University of British Columbia (UBC) [8]: The UBC team primarily experimented with different embedding methods, such as CoVe and ELMo, on deep neural networks. They modeled their neural networks as long shortterm memory networks and BiLSTM, with and without attention. -University of Ottawa (UOttawa) [16]: The University of Ottawa team also proposed a deep learning CNN solution. They experimented using different kind of word embeddings, and also experimented with training a multi-task classifier to see whether performance could be enhanced by shared knowledge between agency and sociality. -Escuela Superior Politecnica del Litoral (ESPOL) [14]: The ESPOL team proposed a semi-supervised adaptation to traditional k-means clustering using neural networks. -Sungkyunkwan team (SKKU) [2]: The SKKU team used a semi-supervised approach. They built four one-class autoencoder models, one for social, nonsocial, agentic, and non-agentic moments. Each autoencoder model had a deep learning architecture consisting of two neural networks, one for encoding the the input, and the other for reconstructing the compressed vector. -Jordan University of Science and Technology (JUST) [13]: The JUST team proposed used a Recurrent Convolutional Neural Network, and combined words with their context in order to get a more precise word embedding.

-Fraunhofer (FKIE) [3]: The team from Fraunhofer FKIE trained a three-layer CNN. They experimented with using different embeddings including Fast-Text and GloVe. Additionally, they experimented with splitting the dataset by demographic location of the author, and showed that training separate classifiers on the splits enhanced performance.

Results

Task 1: Predicting Agency and Sociality

This section compares the participating systems in terms of their performance. Four of the eleven systems that did Task 1 also did the bonus Task 2. The results are provided in Table 1. The detailed implementation of the individual runs are described in the system papers included in this proceedings volume.

Fig. 1: Accuracy scores for the best performing system runs on CL-Aff Task 1 for each of the participating teams

Task 2: Happiness Insights

Some of the systems used their neural models of happiness for Task 1 to produce visual knowledge representations [11], and general insights about happiness [10,8,15,3]. Most notably, Gyrfalcon [11] transformed textual moments into visualizations to explore whether they could encode more multi-dimensional information in this manner. UBC [8] provided a visualization for "attention" in their bi-directional long short-term memory networks which highlights the patterns that were considered important by the neural network while predicting Agency and Sociality when a sequence of words was input into the model. ASU [10] showed the codependence of the individual Agency and Sociality labels across the dataset through a t-SNE visualization. Team 33 [3] and UCSC [15] both attempted to capture the linguistic patterns in the construction of happiness and their potential cultural underpinnings.

Error Analysis

In this section, we present a meta-analysis of system performances for Task 1 over all the (a) topics and (b) moments in the test set. Furthermore, in their Table 1: Systems' performance in Task 1, ordered by their accuracy on predicting Agency.

Agency Sociality System

Accuracy F1 Accuracy F1 UBC [8] .85 .9

.92 .93 ASU [10] .85 .89 .91 .92 IIIT-H [12] .84 .89 .92 .93 JUST [13] Run 1 Supervised CNN (2,3,4,5) on GloVe features UCSC [15] Run 2 Semi-Supervised CNN (2,3,4) on GloVe features UCSC [15] Run 3 Semi-Supervised CNN (2,3,4,5) on GloVe features UCSC [15] Run 4 Supervised CNN (2,3,4) on GloVe features UCSC [15] Run 5 Supervised CNN (2,3,4) on GloVe, syntactic and emotionl features UCSC [15] Run 6 Semi-Supervised CNN (2,3,4,5) on GloVe features UCSC [15] Run 7 Supervised XGBoosted Forest on syntactic and emotion features UCSC [15] Run 8 Semi-Supervised SGBoosted Forest on syntactic and emotion features UCSC [15] Run 9 Semi-Supervised CNN (2,3,4) on GloVe, syntactic and emotion features UOttawa [16] CNN Multi-task learning on GloVe UOttawa [16] Run 1 50 dimensions, 10 epochs UOttawa [16] Run 2 100 dimensions, 10 epochs UOttawa [16] Run 3 25 dimensions, 10 epochs UOttawa [16] Run 4 200 dimensions, 10 epochs UOttawa [16] Run 5 25 dimensions, 50 epochs UOttawa [16] Run 6 50 dimensions, 50 epochs UOttawa [16] Run 7 100 dimensions, 50 epochs UOttawa [16] Run 8 200 dimensions, 50 epochs UOttawa [16] Run 9 50 dimensions, 200 epochs UOttawa [16] Run 10 200 dimensions, 100 epochs data pre-processing step, the team from Fraunhofer [3] identified that in the subset of happy moments contributed by authors from India alone, there were duplicate or near-duplicate happy moments in the data, which reduced the total number of training samples by 25%. We will include data cleaning as an extra preprocessing step in future data releases.

Topic-level analysis: We expect that happiness in different situations would be experienced and expressed differently. Table 3 aggregates the failures produced by each of the approaches (out of the set of best approaches submitted by each of the teams).

Moment-level meta-analysis: We suspect that some of the errors in our data may occur due to mislabeling or the coding scheme not being applicable to the moment. In Table 4 we provide the happy moments for which 100% of the best approaches submitted by each of the teams reported failure. We observe that in some of the cases, (e.g., "Topanga running away to Cory"), the happy moment was actually mislabeled and thus the systems actually did have the correct prediction. Overall, many of the happy moments in this Table describe a single moment in the author's life, which seem ordinary when considered in the context of regular living. In some cases, the authors have attempted to explain why this moment was special to them (e.g., the second part of the moment "I finally got a hold of my auto mechanic, and that enabled me to schedule a time to bring in my car to get my custom exhaust installed" only serves to explain the significance of the moment to the author.) Agency (10% noise) I was given off phone work for the day. i was promoted in my job and i felt so appreciated and happy. Children of different races playing together at pool. I spoke to a friend on the phone that I hadn't heard from in two months. I realized earlier today that last month I made twice as much as my income for the same month the previous year. I won a new lawnmower, which I desperately needed, at a local event. My son singing songs with me. seeing a co worker I was able to talk to my boyfriend on the phone for an hour even though he is on vacation. i slept well last night -no nightmares I got a raise at work My father gifted me a car on my birthday that left me surprised and extremely happy. I won a lucky draw two weeks back to a five star resort for two days, which made me really exiting and joyfull.

Social (5% noise) I was applying for jobs for many months and finally got an interview and a offer later. I learned that we're moving into a bigger, better apartment for less money. I won the first prize in cricket match. I was happy when I was offered to work on a new television show Having my hair cut and it turning out just the way I wanted it to look. I had a job interview this morning and I think it went well. Topanga running away to Cory. Being brought McDonald's for lunch. I was gifted a very nice bottle of wine. I got an iphone 7 as gift I forgot my credit card and was given a free chicken biscuit and drink anyway. A package I ordered that I was anticipating was delivered to me. I received approval to take a month off of work to go on a backpacking trip. Got a small raise at work and even it doesn't amount to much per check, it's still something. When i return home my house was very clean and nice,and the yard was mowed.

Eleven teams participated in the inaugural CL-Aff Shared Task AAAI-19. We have published the complete dataset to Harvard Dataverse8 . Furthermore, we expect to release other resources complementary to the challenges of modeling affect and emotion language from language.

In summary, our meta-analysis of system performance identifies the following key takeaways and recommendations:

-Predictive modeling approaches are greatly improved when modeled as a semi-supervised task, enriched with unlabeled data or by knowledge or feature vectors trained from a different domain. This also highlights the generalizability of the Shared Task to other domains. -Syntactic knowledge is important for modeling Agency and Sociality (and hence, for modeling happiness). Participants incorporated the importance of the head noun and subject-verb-object word order in their language models either through interacting layers in convolution neural networks, or by mining it using lexical pattern analyses methods. -The CL-Aff dataset offers replicability of more traditional emotion modeling approaches. It was feasible to apply the models developed on other annotated emotion datasets to improve the predictive modeling performance on the Shared Task [4]. We anticipate that language models from the CL-Aff dataset will also generalize well to other problems and datasets for emotion and affect analysis. -In future work, scholars could consider training their classifiers based on domain-specific word embeddings derived from the Shared Task dataset itself. -Findings support the emerging notion about the English language as a contextualized emotional vector space, with the best performances reported by approaches that incorporated task-specific embeddings from other language models, such as ELMo and CoVe.