In this paper, we propose a deep learning model-based approach that combines a language embedding model and an emotion embedding model in the classi cation of text for the CL-AFF Shared Task 2020. The task aims to predict the disclosure and supportiveness labels of the comments (to the posts) in the O MyChest dataset which consists of a small labeled dataset and a large unlabeled dataset. We investigate the e ectiveness of the BERT, Glove, and Emotional Glove embedding models, to represent the text for label prediction. We also propose to use the original posts in the dataset as contextual information. We evaluated our approach and report the results.
This paper describes our approach to solve the CL-AFF (Computational
Linguistics - A ect Understanding) Shared Task 2020. In the CL-AFF 2020 task,
the O MyChest conversation dataset is introduced to help understand the role
of emotion in conversations (see [
Text classi cation in natural language processing has traditionally employed classi cation algorithms in machine learning such as support vector machines, Bayesian classi ers, decision trees, etc. A variety of features in text are given as the input of the classi ers, which are crucial to traditional text classi cation algorithms. For the representation of text, word frequency-based approaches (e.g., bag-of-words feature) or sequence-based approaches (e.g., N-grams feature) have been commonly used.
Text classi cation using neural models are comprised of embedding models
and classi cation models. Along with the success of word embedding models
such as Word2vec [
In this paper, focusing on emotional words in the data, we propose a deep
learning model-based text classi cation approach using an embedding model,
which has the combination of a language model and an emotion embedding
model. We particularly focus on emotional words from O MyChest dataset to
improve learning emotional labels (emotional disclosure, emotional support). We
combine labeled and unlabeled comments data for our semi-supervised method
to help supervised learning. And then the sentence features extracted from the
embedding model are given as TextCNN[
Furthermore, to improve the classi cation performance, we apply EDA (Easy
Data Augmentation)[
The O
O MyChest conversation dataset is comprised of three sets - labeled training set, unlabeled training set, and test set: unlabeled training data set includes unlabeled posts and comments; labeled training set and unlabeled test set include about 10,000 labeled sentences and 3,000 unlabeled sentences respectively from the top commented posts. Table 1 lists several excerpts sampled from the labeled data.
Text id ED ID S GS IS ES Hope you have a nice day 91px39 0 0 1 1 0 0 My wife came in when I was around half way through 91px39 1 1 0 0 0 0 this and asked why I was all choked up and watery eyed, so we read it together and now we're both crying.
I am crying a lot of happy tears right now. 91px39 1 0 0 0 0 0 He's my father in every sense of the word but name, I 946qw9 1 0 0 0 0 0 still call him by his rst name but only because we are both used to it and he doesn't mind a bit.
Stepdad will be the one walking me down the aisle when 946qw9 1 1 0 0 0 0 I get married. dThat's wonderful. :) My step-dad has been around for 946qw9 1 1 1 1 0 1 30 yrs now. Category Emotional Disclosure Informational Disclosure Support General Support Informational Support Emotional Support None (all label is 0) All
Number of label 1 Percentage in the category 3,948 31% 4,891 38% 3,226 25% 680 5% 1,250 10% 1,006 8% 4,157 32% 12,860
To understand the data, we examine the number of labeled data in each category (Table 1). As seen in the table, the number of the data labeled as class 1 in the `Support' category groups (e.g., support, general support, informational support, emotional support) are far less than the number of data labeled as class 0. For instance, the data labeled as 1 in the `General Support' category occupy only 5% of the a the data. This means that the data exhibits class imbalance problems, severely in the labels of `general support', `informational support', `emotional support'. Although these categories seem to be the sub-types of the `Support' category, we treat them independently for the classi cation because the sum of all their instances with label 1 does not match with the number of instances in the `Support' category (N =3,226).
Furthermore, we investigate the word usage in the emotional categories. First, we extract the top 100 most frequently used words in each emotional category. Then, we remove the words that also appear frequently in the non-emotional categories (e.g., informational disclosure, support, general support, informational support). Finally, we normalize the raw count by the number of labels of the (a) Emotional disclosure (b) Informational disclosure (c) Emotional support (d) Informational support (e) general support (f) no label corresponding category and sum up the two numbers. Table 4 shows the top 10 words ranked by the normalized frequencies. Therefore, these words can be used to characterize the emotional categories.
Finally, Figure 1 shows the word cloud for each category, visualizing the frequently used words in the categories. While only nouns are generally used to construct word clouds, we also use adjectives as they can represent emotion. The word clouds show that many words (e.g., life, time, good) overlap across di erent categories. There are some words unique to each group: for example, `way' in the informational disclosure category, and `sorry' in the emotional support category. 3
This section details our approach which employs pre-trained embedding models
to generate the vectors representing the text. These vectors serve as the input
for the textCNN [
As our word embedding model, we utilize the pre-trained BERT [
Additionally, we use an emotional embedding model that incorporates
emotional information into a word embedding model such as Word2vec [
Our textCNN model uses one-dimension convolutions with the lter size of 256 and the kernel size of 3, 4, 5. After max pooling on convolutions result, the features are concatenated and atten to be connected with a fully-connected layer as the nal layer for the classi cation. The output dimension is 6, and the sigmoid function is used as the activation function. The output represents prediction probability of each class; the probability of greater than or equal to 0.5 is labeled as 1, and otherwise labeled as 0. We adopt the Adam optimizer with learning rate 1e-4 and epsilon 1e-8 in this study. The binary cross entropy loss function is used to train our model. 3.3
Via informal experimentations, we discovered that the prediction performance
of `General support' is very low. We attribute this to the small number of the
label 1 data. On the other hand, the numbers of the `Info support' and the
`Emo support' labels are 1250 and 1006 respectively. Those labels show half
the performance of the rest of labels. To address this class imbalance issue, we
apply EDA (Easy Data Augmentation) [
When training, we want the model to learn from the labeled data more than the pseudo-labels as the pseudo-labels can be incorrect. Thus, we build the initial model trained with the augmented labeled data on 10 epochs. Then, we re-train the model with the pseudo-labeled data on 3 epochs. While we applied the semi-supervised learning for the system run submission, the experiment results presented below are obtained without the semi-supervised learning scheme because our limited computing facility cannot allow the 10-fold cross validation.
To evaluate the proposed approach in this paper, we use 10-fold cross validation on the labeled training data. In this experimentation, EDA and semi-supervised learning were not used due to limited computing resources for 10-fold cross validation. The di erent conditions examined in our experiments are described as below.
Glove [
Emotional Glove [
BERT [
Context: utilizing the posts as a contextual information to test if it can enhance the prediction performance. 4.1
Table 4 reports the accuracy of label prediction. Overall, the BERT embedding model without using the post shows the best performance in accuracy for three categories. The combination of BERT with Emotional Glove results in the best performance in the emotional disclosure category (without context) and the general support category (with context). The combination of BERT with Glove results outperforms the other models in the information support category. The performance in accuracy seems promising in the support group categories, ranging from 0.814(support) to 0.938(general support). Yet, their F1 scores show di erent ndings.
Table 5 shows the F1 scores of our model in each category. Overall, the `BERT + Emotional Glove' model and the `BERT + Glove' model show the best performance. The performance of the support subgroups (i.e., General Support, Informational Support, and Emotional Support) are poor, as low as 0.05 (for the general support label prediction), which are not su cient for its practical use. Meanwhile, its corresponding accuracy is 0.934. This means that accuracy is not a good metric when the class is imbalanced. Precision and recall performances of the selected model are described in the following Table 6.
Methods Glove (Baseline) Emotional Glove BERT BERT + context BERT + Glove BERT + Emotional Glove BERT + Emotional Glove + context { As for the evaluation measures, accuracy is a poor metric to evaluate the proposed approach due to class imbalance problems. Therefore, we propose to use an F1 score, a harmonic mean of precision and recall, instead. { For the word embedding model, the BERT models perform better than the Glove models. { The combination of BERT and Glove enhances the classi cation performance. { The use of emotional Glove improves the classi cation performance for the categories where classes are imbalanced (i.e., support, information support, emotional support). { Opposed to our initial assumption, the use of the original post as a context did not contribute to enhancing the prediction performance. We postulate that it is because our method of concatenating one comment to one context (i.e., a post associated with the comment) fails to give proper weight to contexts, as the comments data are presumably labeled without considering the contexts.
For the CL-A Shared task 2020 competition, we used `BERT + Emotional Glove' models as the system run model, as it reports the best F1 scores without considering context. In the system run we applied EDA (Easy Data Augmentation) and semi-supervised learning as described in Section 3.3 and 3.4. Our submission contains the labels generated using 4 di erent settings from the combination of [`with context' and `without context'] and [`with pseudo-label' and `without pseudo-label']. 5
This paper describes our approach that combines the word and emotion embedding models to predict `Disclosure' and `Supportiveness' in O MyChest dataset. Three language embedding models - BERT, Glove, and Emotional Glove - were compared, and BERT showed a better performance (about 10%) for the label prediction than Glove. Our evaluation results also indicate that the combination of embedding models can improve the performance. It is particularly noted that Emotional Glove better represents the text than Glove when the class is imbalanced. We adopted the original posts along with their associated comments to increase the prediction performance. However, the result shows that using the post makes no contribution to increasing the model's classi cation performance. In the future, we plan to investigate how to convey the context of a post in an e cient manner rather than just concatenating to increase prediction performance.
This research was partially supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(2016R1D1A1B03933002). This work was partially supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MEST) (No. 2019R1A2C1006316). This work was also partially supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (2017R1A2B4010499).