We propose the Word Pair Convolutional Model (WoPCoM) for the CL-A 19 shared task at the AAAI-19 Workshop on A ective Content Analysis. The challenge is the classi cation of speaker-described happy moments as social events and/or activities in which they have agency. WoPCoM leverages the regular structure of language on an architectural level in a way that recurrent models cannot easily answer, by learning convolutional word pair features that capture the important intra- and inter-phrasal relationships in a sentence. It performs with an average accuracy of 91.45% predicting the social label and 86.49% predicting the agency label assessed on a 10-fold cross validation. This represents a performance improvement of 0.92% for predicting the social label, but only 0.04% on predicting the agency label, over a simpler deep LSTM baseline. In spite of similar performance on these metrics, WoPCoM demonstrates desirable results when other metrics such as training time, model-intermediate class separation, and over tting propensity are considered, warranting further study.
1.1
CL-A
This paper will address the challenge put forth in the CL-A shared task [
HappyDB is a corpus of 100,000 crowd-sourced happy moments. Workers
on Amazon Mechanical Turk were tasked with recalling and writing sentences
? Equal contribution
describing happy moments from three \recollection periods" since they have
occurred, the past 24 hours, the past week, and the past month. [
As this is a new shared task, previous approaches to this speci c problem do not exist. Because we have selected a deep learning-based approach to the task, we consider neural networks for sentence-level semantic classi cation, and the word embedding approaches that underpin most methods in that domain related work.
Word embeddings are a powerful tool for NLP tasks due to their capacity to
capture both semantic and syntactic data without a need for feature engineering
[
Embeddings from Language Models (ELMo), 1024-dimensional word vectors
assessed using a bidirectional LSTM applied across the characters in a sentence,
are the current state of the art in latent word vector representation [
Our neural network design arose by rst considering how we would implement a feature engineering approach, and then determining how a neural network could learn similar features to the ones we would have hand-engineered.
The \agency-ness" and \social-ness" of the vast majority of sentences we considered at this stage hinged on a few critical features. For example, the \I walked" in \I walked my dog to the park" is critical for understanding the agency of the speaker. Perhaps this pair of words could be captured by a twoword lter evaluating personal pronouns to the left of active verbs. Through the consideration of these toy examples our hypothetical feature engineering approach began to take shape.
By approximating distributions of certain semantic-syntactic concepts such as personal nouns, social verbs, and action verbs in the embedding space the probability a word belongs to such classes can be estimated. The word count distance between two words in a sentence can be roughly correlated with their syntactic relationship. Coupled with semantic-syntactic information about the individual words themselves features correlated with sentence meaning would arise.
Rather than hand engineer these word probability distributions we would
design the initial layers of the neural network to learn them. Rather than hand
engineer the two-word lters we would use convolutional layers to learn them.
Inspired by the dilated convolution approach employed by WaveNet [
English is a strongly head-initial language with a subject-verb-object word
order. The word determining the syntactic category of an English phrase
generally precedes its complements, leading to the formation of right-branching
grammatical structures. Nouns generally form constituent noun phrases of the
verbs they follow [
These observations justi ed the design of the Word Pair Convolutional Model (WoPCoM), hinging on the importance of pairs of words and the exibility of neural networks to form a prediction. 2
Almost all of the classes we considered boiled down to a prototypical feature f (x(n); k) = P (x(n) 2 S1)P (x(n + k) 2 S2) for input sentence x of words x(n), word count separation k and word classes S1 and S2. For example, one possible feature for testing speaker agency in a sentence might consider S1 to be the set of personal nouns and S2 to be the set of active verbs for k = 1, which would capture subclauses common to active sentences such as \I took" in the example sentence above, or \we went," \I made," etc. Similar features for labelling social interactions can be envisioned. Through the convolution of these features across the sentences, the labels could be ascertained. While modelling the probability distributions of the various word sets and considering all the important features by hand is not feasible, we hypothesized that a convolutional neural network (CNN) could solve both of these problems at once, implicitly learning the distributions of word classes and choosing which classes to select for simultaneously during the learning process. What follows are the insights from our model design process through the models we considered.
We chose to use ELMo vectors as the input feature for all models. All
models were implemented using PyTorch [
We rst considered a nave model employing a long short-term memory (LSTM) layer with variable hidden size taking the sentence set of ELMo embeddings as input. The nal hidden state of the LSTM was processed through a single fully-connected layer with output size 2, and then a sigmoid function to map all possible outputs to the (0; 1) probability range, representing the probability of membership to the two classes. 2.2
In our prototypical feature engineering formulation of the task we considered a feature extractor convolving across a vector of class membership probabilities for each word in the sentence to extract various intermediate word pair class attributes that could be used in higher-level classi cation. However, due to the feasibility issues discussed above and the lack of a ground truth objective to train word class probabilities with, we opted for a looser type of pre-CNN processing, a set of linear layers separated by ReLU operations with no strict class probability constraint at the output. We do this by taking the ELMo-evaluated sentence set of word vectors through a 1024 N linear layer, with N corresponding to the number of word attributes we want the WoPCoM to learn.
Two-vector convolutional lters of varying stride are then applied across the sentence set of size N word attribute vectors to evaluate word pair semanticsyntactic features. Through the varying dilation of the lters di erent types of syntactic relationships can be captured, allowing for intra- and inter-phrasal relationships to be assessed. We implemented our feature extractors as a set of ve 1-dimensional 2 N N C convolutional layers with dilation factors varying from 1 to 5. The output dimension N C (number of classes) is con gurable to allow for evaluating di erent quantities of word pair relationships, and can be considered analogous to the hidden state size in the LSTM implementation.
To \pool" the ve sentence-length convolution outputs we feed them through an LSTM with hidden size N C and take the nal N C 1 hidden state as the \pooled" output of the ve lter sets. Those vectors are then concatenated into one vector, and fed through a linear 5N C 2 layer to map these feature outputs Sentence length 1024
N
N
D = 1
D = 2 Sentence in
ELMo Embedder
Some word NC 5
NC
Sigmoid Linear (NC x 5) x 2
P(Agency) P(Social) (More N x N and ReLU) Linear 1024 x N
ReLU
5x 1d CNN
Dilation D=[
LSTMs for Pooling All 5 layers concat to the two classes. Finally, these feature outputs are evaluated as probabilities through the sigmoid function. Figure 1 depicts the WoPCoM model by showing selected layer outputs in the processing of a single sentence batch. 2.3
Baseline For the baseline recurrent model a hidden representation size of 25 was used.
WoPCoM For the nal implementation the word class count N was set to 100, and the convolutional lter set output size N C was set to 50. 2.4
The Adam optimizer [
The models were trained with random batches of 50 same-length sentences to minimize necessary input padding. A patience factor of 10 was used to allow variable-length training, once 10 epochs pass without a drop in validation loss, training is halted. Mean-squared error was used as the loss metric.
Class labels were tokenized as 0 for \yes" and 1 for \no." This means that the model is really learning negative probabilities (probability that the sentence is not social/agency) but this has no bearing on nal accuracy. To compute accuracy, F1, and AUC the output probabilities are rounded to 0 or 1.
We evaluated WoPCoM and our baseline model using 10-fold cross validation, achieving the results in Table 1. Results that have over a 0.5% improvement from the baseline are in bold.
To illustrate the general separation of classes performed by WoPCoM t-SNE
projection [
75 50 25 0 −25 −50 −75
Social & Agency Social Only Agency Only
None
WoPCoM shows a modest improvement over the LSTM baseline on the social classi cation task, but the di erence between its performance and the LSTM performance on the agency classi cation task is meager. This disparity could have meaningful implications about the underlying data, warranting further study.
Through the process of training and testing these models many times we have come up with some informal observations in addition to the concrete. Some conrm basic concepts in machine learning, such as the e cacy of deeper architectures at tting to the complex underlying distributions at the cost of over tting. We began paying attention to which models quickly t to high accuracy across relatively few epochs, and which models maintained a relatively narrow gulf between the training and validation loss across many epochs.
One potential advantage we found to WoPCoM as opposed to the LSTM is that it tends to resist over tting. The training and test loss both saturate around the same epoch, and there does not come a point where the over tting pattern of simultaneously decreasing training loss and increasing validation loss takes place. This might be a result of the constraints on the solution space structure described brie y above. We are interested in applying a more detailed analysis to this phenomenon.
We plan to complete future work to look more rigorously to validate parameters of our approach that were chosen almost arbitrarily, such as the choice of two-word convolution lters, and dilation factors up to ve, against similar approaches leveraging three- or four-word lters and larger dilation factors, as well as pit WoPCoM or models designed with a similar philosophy against more radically di erent architectures on more established tasks.
Currently we are using pretrained ELMo embeddings, we suspect that for task/dataset-speci c problems such as this shared task word embeddings learned from the unlabeled data directly could improve performance. An unsupervised approach employing an autoencoder that shares the sentence-feature compression architecture of WoPCoM could potentially be adapted to improve performance as well.
We would also like to investigate the utility of the WoPCoM architecture we've developed to applications with scarcity of training data and build on this architecture to t on such problems better. 5
We would like to thank Dr. Hemanth Venkateswara for his guidance, particularly in shooting down our worst ideas early, so we never had to discover how bad they were ourselves.