We propose a novel way of deploying deep multi-task learning models for the task of detecting disclosure and support. We calculate all possible logical relations among six labels, represented in a Venn diagram. Based on it, the six labels are distributed to multiple fragment clusters. Then, a multi-task deep neural network is built on the groups.
Deep Learning (DL) has achieved great success in many elds, including, but not limited to, natural language processing, computer vision, and speech recognition. But there are still many limitations and challenges related to training DL models, such as over tting, hyperparameter optimization, long training times, high memory usage, etc.
Even if we do not consider the high demands for computing power, there
is still some interesting techniques in the classical neural network models to
improve the performance, like, deep multi-task learning structures. Multi-task
learning (MTL), particularly with deep neural networks, can not only reduce
the risk of over tting, but also improve the results for each task, compared with
single-task learning. [
Switching the topic to the 2020 CL-A Shared Task [
For task 1, the O MyChest conversation dataset is provided. Twelve thousand samples are included in this dataset, and each entry contains a sentence and six binary labels, including Information disclosure, Emotion disclosure, Support, General support, Info support, and Emo support. The distribution of the six labels in training data is shown in table 1. We can see that in all the labels, the negative data accounts for a high proportion, especially for the label General support, in which the negative data reaches a proportion of 94.7%. This tells us to pay attention to the class weights during training. Nonetheless, it is likely that the result on the label General support will be among the lowest since it has the highest class imbalance. Label True False Emotional disclosure 3948 8912 Information disclosure 4891 7969 Support 3226 9634 General support 680 12180 Info support 1250 11610
Emo support 1006 11854
Table 2 shows the token analysis of the training dataset. As shown in the table, although the maximum token length reaches 171, 95% percent of sentences have a length of no more than 34. So the objective length in sentences preprocessing is 34. After retrieving the word embedding vectors, all sentences will be transformed into vectors with shape (34 embedding dimension).
Value Max token length 171 Min token length 1 Mean token length 15.07 Median token length 13 Number of unique tokens 11460 Total number of tokens 193837
Length that covers 95% of sentences 34 3
Preprocessing steps included standard operations like splitting the sentences
into words, omitting all punctuation marks, transforming the sentences into
sequences and padding them to the same length, 34. The word embedding method
we chose is BERT [
In the model, we want to fully utilize the power of the neural networks on
multitasking with hard parameter sharing [
In general, when training a model on a task using noisy datasets, we need
to ignore the data-dependent noise and to learn good patterns based on other
features. Because di erent tasks have di erent noise patterns, a model trained
for multiple tasks can generate a more general representation and can average
the noise patterns on the di erent tasks [
Furthermore, as similar tasks have similar patterns, we want their
taskspeci c layers to be at a closer position compared with other tasks in the
model. For example, among the six labels of our shared task, it is easy to image
that the label Support has a strong relationship with the label General support,
Info support and Emo support, as they all refer to something about support.
Considering each label as a set, which contains entries in the training data where the
corresponding label is 1, the relationship among four labels can be described by
the Venn diagram in Fig. 1. The numbers on the graph show the size of
intersections between or among the sets. Venn diagram [
From the Fig. 1, we can see that the label Support almost covers all the cases in the other three sets, except for some trivial cases. Then how to re ect this relationship in the neural network? Because the label Support covers a more general concept, it should be treated at a lower layer. The other three labels re ne information from the Support layer, using part of its information, while sharing some neurons between themselves, as shown in Fig. 2. The bottom of the Fig. 2 is a large dense layer, which is split into several parts; we call it a fragment layer. Above the fragment layer in Fig. 2 are territories of four labels; this means that the label-corresponding task-speci c layers will only connect to their speci c territories (neuron s).
Each label's territory has some overlap with other labels' territories, and the label Support occupies the whole layer, from the leftmost node to the rightmost node.
The example above is only for four labels. Nevertheless, Support can contain all the other three, which means that the intersections only appears among three layers. What about the six labels in our task? The Venn diagram is far less clear, as shown in Fig. 3. Discarding the pieces in the gure whose size is too small (intersections comprised of less than 10 instances), there are in total 31 major intersections in the Venn diagram. And for six-labels, each label is comprised of 15, 16, 28, 12, 12, and 15 intersections separately.
Roughly, from the bottom to the top, the network contains the input layer, shared hidden layers, task-speci c layers and task-speci c outputs. The connection between shared hidden layers and task-speci c layers is based on the fragments and the Venn graphs, as shown above.
As we have two types of embeddings, of shape (12860 1024) and shape (12860 34 1024), we tried two types of models in the experiment.
For the data with shape (12860 1024), from the bottom to the top, we have an input layer, fragment dense layers, concatenate layers and output layers. As we mentioned above, the embedding layer is not included in the model because the word embeddings are already generated in preprocessing, using bert-as-service.
For the data with shape (12860 34 1024), from the bottom to the top, the
model is composed of an input layer, bidirectional LSTM layer, fragment dense
layers, concatenate layer, attention layer [
During training, we use mini-batch gradient descent with size at least 512 and
the Adam optimizer [
We evaluate the performance of models on the training dataset. The split ratio we used is 0.6:0.2:0.2, which means 60% of data is used for training, 20% for validation and 20% for testing.
Table 3 shows the result of two models. The parameters used in Model 1 are: learning rate is 2e 5, epochs is 20 and batch size in mini-batch SGD is 1024. The corresponding le name in submission is system runs uottawa1. The parameters used in Model 2 are: learning rate is 2e 5, epochs is 20 and batch size in mini-batch SGD is 512. The corresponding le name in submission is system runs uottawa2.
From the table, we can see that General support always has the worst result. This is resasonable, considering the imbalance of this label, in which the positive cases:negative cases is 680:12,180.
In this paper, we presented a multi-task deep learning model. Our model has a reasonable result on some of the labels, but not all, especially not for General support. The reason is that General support classi es quotes and catchphrases, which have less distinctive features than Emotional XXX or Information XXX, while having less positive cases appearing in the dataset.
During the experiments, we tried several other methods for training models,
for instance, using LIWC [
One possible direction of further work for this task is to make use of large unlabeled data provided. An idea here is to use it to nd the di erent patterns in the texts in order to split them into several groups, and then to train models on each group. Sentences can have di erent patterns and structures, which requires di erent mapping functions in the network. If we can separate them into clusters in which sentences have similar patterns, there might be an improvement in the classi cation results.