We present a three-layer convolutional neural network for the classification of two binary target variables 'Social' and 'Agency' in the HappyDB corpus exploiting lexical density of a closed domain and a high degree of regularity in linguistic patterns. Incorporating demographic information is demonstrated to improve classification accuracy. Custom embeddings learned from additional unlabeled data perform competitive to established pre-trained models based on much more comprehensive general training corpora. The top-performing model achieves accuracies of 0.90 for the 'Social' and 0.875 for the 'Agency' variable.
The CL-Aff Shared Task [1], held as a part of the Affective Content Analysis workshop at AAAI 2019, invited participants to analyze and classify the contents of HappyDB [2], a corpus of 100,000 'Happy Moments'. Subtask 1 consisted of classifying contents with respect to two binary variables, 'Agency' and 'Social', with 'Agency' indicating whether the author of a happy moment was in control of events and 'Social' indicating whether additional people were explicitly or implicitly involved. In addition, an open-ended second subtask invited participants to share insights from the corpus with respect to 'ingredients of happiness'.
To the best of the author's knowledge, no similar shared task or challenge has previously been proposed, and while there has been extensive research on sentiment and affect analysis, the task at hand is very specific and its scope is limited to pre-classified data describing 'happy moments'. The task at hand could therefore not be approached with established techniques for sentiment or polarity analysis. It was rather considered a classification task aiming for the detection of semantic ('Social' variable) and syntactic ('Agency' variable) patterns, with both implicit and explicit concepts present in the data.
In recent years, embedding-based deep learning techniques have gained momentum superseding conventional machine learning techniques in a broad range of linguistic tasks, currently constituting the absolute majority of publications at the four major venues of computational linguistics [5]. The use of neural networks employing the technique of vector embeddings seemed a natural choice given the need to extend the language model to abstract concepts beyond the lexical surface structure.
A comprehensive description of the dataset provided along with informative basic statistics can be found in the original HappyDB paper ( [2]). The following section describes some additional insights into the data structure that proved to be relevant for classification approach and performance.
It was quickly noted that 95.2% of the provided happy moments were tagged as submitted from just two countries, United States (8378 or 79.3%) and India (1674 or 15.9%), while the remainder of the corpus of just 508 happy moments was distributed among 69 other countries. In light of this uneven distribution and the resulting challenges for claiming statistically significant insights on this data, only the subsets from the aforementioned two countries were considered for further evaluation and additional classification experiments.
While the authors of HappyDB took basic cleaning and quality assurance measures with respect to misspellings and removal of non-informative entries, the corpus contains a considerable proportion of duplicates.
While the corpus contains 1,674 entries with the country tag 'IND', a manual inspection of those moments revealed the presence of a high number of duplicates. After removing exact literal duplicates, the subset was 391 entries lighter, leaving 1,283 entries. Removing punctuation to further catch small variations in otherwise identical utterances, like the college example in table The seven most common duplicate entries alone make up 391 (23.3%) of all moments with country tag 'IND'. Note that the entry "i went to college" occurs with and without full stop 15 times each. Additionally, the majority of these duplicates were submitted along with contradicting demographic information. While sentences like "i went to college" might indeed have been submitted by multiple participants, more distinct duplicates like the irregular pattern second to the bottom or complex utterances like the example at the bottom (shortened from originally "when i am getting ready to go to my office my parents send off with cute smile and say have a nice day and take care") were almost certainly submitted multiple times by the same worker. Even the cleaned-up subset still contains several very similar complex utterances. Undeniably, the presence of such a high proportion of duplicates in one category has a considerable distorting effect on training and evaluation of a classifier.
The situation was far less critical for the 'USA' subset of the corpus with 208 duplicates amounting to less than 2.5% of entries in the corpus. The overall duplicate ratio over the entire corpus was 6.2%
Only the cleaned up versions of the 'USA' and 'IND' subsets were considered for further analysis and training the classifiers.
The material provided by participants from the US and India differed from each other in several linguistic dimensions. The exact linguistic background of individual authors remained unclear as both countries are polyglot, however, it seems reasonable to assume the majority of US participants to be native speakers of English or highly fluent in the language. The vast majority of authors submitting from India is in contrast assumed to use English as a second language, with a more diverse linguistic background than US participants. Assuming a descriptive rather than prescriptive point of view, it is not of particular interest whether particular patterns in the Indian subset might be considered correct or appropriate by native and proficient speakers of English as long as they are distinct and reproducible enough for a classifier to learn. The intuition that patterns in this subset might be distinct enough for the classifier to benefit from learning them separately was proven correct experimentally.
American and Indian submissions differed considerably with respect to syntactic patterns to start with: While statements from US authors contained 13.52 tokens on average per sentence with a standard deviation of 6.78, Indian statements contained 12.71 tokens on average with a considerably higher standard deviation of 10.59 caused e.g. by a larger proportion of particularly long statements. While the authors were originally instructed to state complete sentences, the level of compliance varied between the two groups, with e.g. US authors starting 8.4% of sentences by a gerund form compared to 5.7% of Indian authors. Tables 2 and 3 show the most common trigrams starting sentences from the two different groups, demonstrating US authors use a considerably higher share of idiomatic expressions such as "i got to" and framing expressions such as "an event that [made me happy]" and "i was happy", marked in bold. The Indian statements might in that light tentatively be characterized as being more straightforward. Additional differences involve Indians using simple and progressive present substituting simple past more often than US authors and a higher rate of omission of particles such as prepositions. Indian statements were lexically more dense with a types to tokens ratio of 9.67 compared to 8.00 in US statements.
Participants in the crowdfunding process creating the HappyDB corpus were explicitly asked to state moments that made them happy in single full sentences.
While not all participants submitted strictly complied to those instructions, the overwhelming majority of statements are in the form of full declarative sentences. Syntax in the corpus can thus be regarded fixed and discarded as distinct piece of information in the classification process. 3 Experiments and results
Given the almost uniform syntactic structure of the corpus with respect to declarative sentences, a convolutional neural network was determined to be an appropriate architecture rather than a time-step based approach: Considering syntax more or less fixed relieves the classifier of the effort to interpret the complete input as sequences and allows to focus on detecting the presence or absence of features relating to agency or social participation in the utterance. Two binary classifiers were trained to address each variable separately. A large search space of configurations was explored, yielding the following configuration with the best performance in terms of accuracy: Two convolutional layers with 128 filters each with a step size of 5 and a dense layer with 128 units. Applying dropout of 10 and 20 % yielded slight but statistically insignificant improvements. Batch sizes were iterated in steps of 8, 16, 32, 64 and multiples of 64 up to 1024, with medium batch-sizes of around 384 performing best in the vast majority of configurations. Table 4 shows overall results in the best-performing configurations with the architecture described above.
As higher dimensional embeddings consistently outperformed low-dimensional models, only the 300 dimensional models were considered for further experiments.
Three major groups of pre-trained embeddings were used for the initialization layer of the neural network: FastText by Facebook AI [4], GloVe by Stanford University [3] and custom FastText embeddings trained on the joint set of labeled and unlabeled HappyDB data provided by the task's authors.
To assess the degree to which the supplied labeled and unlabeled HappyDB data were able to reflect syntactic and semantic relations of the domain in comparison to broader knowledge of predefined embeddings as distributed by the authors of FastText and GloVe, FastText embeddings of different dimensionality and with both available approaches, CBOW and SkipGram, were trained and evaluated as displayed in Table 4.
Based on aforementioned considerations, one binary classifier was constructed for each dependent variable, 'Agency' and 'Social', each with the target values 'yes' or 'no' as labeled in the training data.
Table 5 shows the uneven distribution of the two variables and their co-occurrences in the corpus, illuminating some basic connections in agreement with the psychological findings quoted by the authors of HappyDB: A majority of 73.8% of happy moments involves active participation or control by the author. Within these moments, an absolute majority of 54.4% involves no other people than the acting authors themselves. In turn, within the 26.2% of moments with no active participation of the author, the probability is 74.9% that other people are involved, reflecting the intuition that in most instances, something, or somebody, needs to cause the happiness after all. This connection raised interest in the performance of a classifier considering each combination of the two variables a distinct class, thus forming four classes "Agency no, social no", "Agency yes, social no", "Agency no, social yes" and "Agency yes, social yes". While there is apparently a strong conditional probability of "Social: yes" given "Agency: no", the significantly lowered number of samples was expected to cause a drop in performance, especially with only 693 samples for the "Agency no, social no" class, an assumption that was confirmed by the experimental results as displayed below. The results of the three top-performing high-dimensional configurations instantly affirmed those expectations and ceased interest in further experiments: Combining the two variables into four categories decreased the performance even when evaluating only for one variable per category well below the achievable results in the binary setting.
The presence of aforementioned distinct syntactic and lexical characteristics in the two largest groups by country inspired the question whether classification performance would benefit from training separate classifiers for each group. Since only the USA and India subsets contained more than 1000 samples, the exploration was limited to those subsets. The results show a modest but statistically significant (confidence level 0.95) improvement for both language groups with the moments submitted under country code IND benefitting considerably stronger. We suggest this might be an effect of more compact syntax patterns (see above). The picture is even clearer for the Social variable. For both variables, the separated classifiers achieve better performance than their combined average. However, the degree of convergence for this phenomenon towards larger training sets has not been investigated.
The authors of HappyDB report on successful efforts to categorize the corpus by a set of crowd-sourced category labels. Additionally, they identified a set of concepts or topics of happy moments in a seemingly rather intuitive and subjective way. To apply a limited test of replicability to this set of topics, a classifier with the aforementioned architecture was trained on a subset of the corpus consisting of happy moments labeled with exactly one concept, limited to concepts with more than 1000 labeled examples, which were careeer (1280), entertainment (1135), family (1259) and food (1007).
We introduced a rather simplistic architecture to classify the HappyDB contents with respect to the two binary variables 'Agency' and Social. HappyDB prove to be a high-quality linguistic resource with a high degree of replicability in terms of machine learning and classification as proven by experimental results for both the target variables defined by the Shared Task and the ability to reproduce the concepts introduced by the HappyDB authors. We observe that while embeddings trained only on HappyDB without any external world knowledge supplied cannot statistically significantly outperform established general purpose embeddings such as FastText and Glove trained on Wikipedia and crawled web content, they appear to be almost competitive utilizing a database of not even 20,000 types as opposed to up to 2 million types in the pre-trained embeddings. We observe no particular social media benefit for embeddings in accordance to the assumption that most statements were given in a rather formal register as intended by the corpus' authors. Classification appears to benefit from taking linguistic backgrounds of different groups of authors into account, and we recommend cleaning the corpus from remaining duplicates to avoid distortions.
I would like to express my gratitude to Fahrettin Gökgöz and Albert Pritzkau of Fraunhofer FKIE and Maria Jabari of University of Bonn for their expertise and insights supporting the system design and dataset analysis.