Causality can be understood as the relationship between two events such that the occurrence of one event results in the occurrence of the other event either directly or indirectly. This paper aims to identify whether the given sentences have a causality efect present in them and to classify the cause and efect words/phrases if present. The approach used for classification uses deep learning algorithms and the annotation task uses machine translation. These models are applied to the dataset provided by CEREX@FIRE2020. The best result for the causality identification part was obtained from BiLSTM with an F1 score of 0.60 and for the second task of annotation as cause and efect, NMT with Bahdanau attention mechanism with an F1 score of 0.44.
Causality is defined as the relation between a cause and its efect. A cause is why an event happens. An efect is an event that happens because of the cause. Any sentence with a causal expression has the following three components: cause, efect, and causal connective.For example, in the sentence : “Due to inflation, the dollar is worth less than before”, - the event “inflation” is the cause and the event “the dollar is worth less than before” is the efect. “Due to” is the causal connective here. In recent times, automatic extraction of semantic relations, in particular, automatic extraction of causal relations, has become essential for many natural language processing (NLP) applications like question answering, document-summarization, opinion mining, event analysis. One of the easiest ways to express cause-efect relations is through the form “A causes B” or “B is caused by A”. But causality can be expressed through a wide variety of syntactic expressions, and these variations are hard to represent using a single model. Therefore, due to the presence of complex grammatical structures in sentences, the automatic extraction of causal relations becomes a hard NLP problem to solve. The task here is two-fold.1 The first task is to identify whether a given sentence contains a causal event (cause/efect). Two models were employed for this task - Logistic Regression and Bidirectional Long short term memory (Bi-LSTM). The second task is to annotate each word in a sentence in terms of four labels ( C - cause, E- efect, CC- Causal Connective, and None). This task was implemented using the NMT model.
Several research work has been reported in the field of automatic extraction of cause and efect from
natural language text. The authors of [
Cause-efect relations from documents in Metallurgy and Materials Science are extracted in this
paper[
The plausible cause-efect pairs are identified through a set of logical rules based on dependencies
between words. Then Bayesian inference is used to reduce the number of pairs produced by ambiguous
patterns[
The author[
The approach used by Blanco for the detection and extraction of causations is based on the use of
syntactic patterns that may encode causation. They[
The training data for both the tasks released by CEREX 2020 included 6000 rows containing 4 columns, the sno, sentence, cause, and efect. The sno is unique for every row, sentence consists of the entire sentence, cause is the causal part of the sentence and efect is the efect part of the sentence. For Task A, the test dataset consisted of 764 sentences having 2 columns i.e. the sno and sentence. And the test dataset for Task B contained 178 rows and two columns which were Sent_id i.e the sentence and id and sentence. The sno and Sent_id are distinct for every sentence.
The pre-processing done for Task A included removing emojis and adding an additional label column which consists of two values i.e. 0 and 1. A sentence is classified as not causal if it doesn’t contain both cause and efect. These kinds of sentences are labeled as 0. Otherwise, if a sentence has both cause and efect or just has either one of them, it is causal and labeled as 1. The cause and efect columns are then dropped. Finally, the data is split in the ratio of 80%-20% and given to the Bi-LSTM model. For the Logistic Regression classifier, the data is split in the ratio of 70%-30% and given as the input.
For Task B, the data was preprocessed by removing emojis and punctuations with the help of regular expressions. Extra spaces in each sentence were removed. We created a list of common causal connectives, so along with the cause and efect words we can identify the causal connectives in the sentence too. The list contained both words and phrases. Using wordnet2, synonyms of each causal connective were found and their past, present, future participle forms were added to the list. Each word in the dataset was annotated by its specific label. If a word did not belong in either cause, efect or was not a causal connective it was labeled as O. For the training, the dataset was split into multiple files. ’train.in’ consists of the first 4200 sentences from the dataset and ’train.out’ consists of the respective labels of each word present in the ’train.in’. The labels were C(cause), E(efect), CC(causal connective), O(None). ’dev.in’ consists of the remaining 1800 sentences from the dataset and ’dev.out’ consists of the labels generated by the model for each word in ’dev.in’. ’vocab.in’ contains all the distinct set of words present in the dataset and ’vocab.out’ contains the distinct set of labels to be generated by the model i.e CC, C, E and O.
The general methodology which we have followed contains two steps i.e. model training and postprocessing. For the first task, the input to the model consisted of the sentences with their respective labels, i.e. 0 or 1. For the second task, the preprocessed data was stored in the form of an input file i.e. train.in. The output file, train.out, consisted of the corresponding labels for each word in train.in. After the result is obtained from the model, postprocessing is performed. For the first task, the predicted labels along with their sentences are returned in a file. For the second task, the resultant data obtained is post-processed by the removal of extra spaces. This is followed by annotating each word with its respective label i.e C, E, and CC in the format word/label. If a word contains the label O then it is not annotated. The result is stored in a new CSV file. 4.1. TASK A : In this task, our goal is to identify whether or not a sentence contains a causal event and classify the sentences as 1 (if the sentence contains a causal event) or 0 (if the sentence does not contain a causal event). Two approaches have been proposed for this task, which are Logistic Regression and Bi-LSTM.
0.94 0.72 0.95 0.77 0.95 0.74 We have employed the Logistic Regression Classifier as it works well with small datasets. Since the data isn’t evenly distributed across the two classes, we have performed oversampling for the minority class 0. This is done using SMOTE (Synthetic Minority Oversampling Technique)3 such that the ratio of the two classes is 1:1. Then, the Logistic Regression Classifier was used to classify sentences as having causal events or not. Our second approach was based on Bi-LSTM as it efectively increases the amount of information available to the network, improving the context available to the algorithm. It outperformed LR based on the training dataset.
2https://wordnet.princeton.edu/ 3https://imbalanced-learn.readthedocs.io/en/stable/generated/imblearn.over_sampling.SMOTE.html 4.2. TASK B: For this task, our aim is to annotate each word with its respective label which is CC, C, E or None. We have used three NMT4 models to do the same, i.e. NMT with Scaled Luong and Normed Bahdanau attention mechanisms, and NMT without an attention mechanism. In all three models, the recurrent unit is LSTM.
We started with the NMT model without an attention mechanism. The input is encoded into a fixed dimensional vector and then is decoded into the target labels from the vector. For the Scaled Luong attention mechanism, the input file is encoded and reduced to attention scores using simple matrix multiplication which makes it faster and more space-eficient and is then decoded into the target labels present in train.out. Finally, we used Normed Bahdanau which performs a linear combination of encoder states and the decoder states. The model predicts a target word based on the context vectors associated with the initial position and the previously generated target words. We chose Normed Bahdanau as it works better with smaller datasets and achieved a higher accuracy on the train dataset compared to the other two models.
We have evaluated our models using the test data of CEREX@FIRE2020 . The performance was analyzed using the metrics namely precision, recall and F1-Score. We secured the second place for the Binary Classification task. For the Tagging task, we outperformed the other teams and came first.
0.36 0.32 0.91 0.87 0.57 0.51 0.65 0.60 0.44 0.39
The submission made by ssn_nlp for Task A i.e classification of sentences as causal or not was the deep learning approach using BiLSTM. The model obtained F1-Score of 0.60 against the test set. A precision score of 0.46 and a recall score of 0.87 was encountered.
Our team ssn_nlp oficially submitted an NMT model using Normed Bahdanau attention mechanism for Tak B i.e tagging each word with its respective label. The model performed on the test set with F1-Score of 0.44, a precision score of 0.36 and, a recall score of 0.57.
We used a Bi-LSTM model for the classification of sentences as containing causal events or not. This performed better than the Logistic Regression Classifier. The model achieved an accuracy of 91% and an F1 score of 0.95. For annotating the sentences with their respective labels, NMT with Bahdanau attention mechanism was employed as it works better with smaller datasets compared to the ScaledLuong attention mechanism that is generally used for larger datasets. It produced an accuracy of 0.44. Future developments would include constructing an enriched corpus of causal connectives and incorporating more linguistic features.