The Natural Language Processing (NLP) community has recently experienced a growing interest in Semantic Role Labeling (SRL). The increased availability of annotated resources enables the development of statistical approaches specifically for SRL. This holds potential impact in NLP applications. We examine and reproduce the Marcheggiani's system and its individual components, including its annotated resources, parser, classification system, the features used and the results obtained by the system. Then, we explore different solutions in order to achieve better results by approaching to Verb-Sense Disambiguation (VSD). VSD is a sub-problem of the Word Sense Disambiguation (WSD) problem, that tries to identify in which sense a polysemic word is used in a given sentence. Thus a sense inventory for each word (or lemma) must be used. Finally, we also assess the challenges in SRL and identify the opportunities for useful further research in future.
One of the fields where AI is gaining great importance is the NLP. Nowadays, NLP has many applications: search engines (semantic/topic search rather than word matching), automated speech translation, automatic summarization, etc. Therefore, there are many sub-tasks for natural language applications that have already been studied. An example is the syntactic analysis of the words of a sentence. The object of this research study is the realization of a system able to perform SRL.
A SRL system does nothing more than take a set of input phrases and, for each of them, it starts to determine the various components that could play a semantic role. A component of a proposition that plays a semantic role is defined as constituent. Once the possible candidates are determined, Machine Learning techniques are used to label them with the right role.
This task becomes important for advanced applications where it is also necessary to process the semantic meaning of a sentence. Moreover, all this applications have to deal with ambiguity. Ambiguity is the term used to describe the fact that a certain expression can be interpreted in more than one way.
In NLP, ambiguity is present at several stages in the processing of a text or a sentence, such as: tokenization, sentence-splitting, part-of-speech (POS) tagging, syntactic parsing and semantic processing. Semantic ambiguity is usually the last to be addressed by NLP systems, and it tends to be one of the hardest to solve among all types of ambiguities mentioned.
For this type of ambiguity, the sentence has already been parsed and, even if its syntactic analysis (parse tree) is unique and correct, some words may feature more than one meaning for the grammatical category they were tagged with.
Usually this difference in meaning is associated to
syntactic properties. In order to overcome these
issues, this research study approaches to the VSD
task. The majority of the systems used in the VSD
task are based on Machine Learning techniques
We approach both the tasks by following two different solutions. 2.1
Until recently, state-of-the-art Semantic Role
Labeling (SRL) systems relied on complex sets of
lexico-syntactic features
Recent studies
An overview of the most used techniques and
features for WSD was also conducted, based on the
systems evaluated at the SensEval3. The most
common learning algorithms
The Naive Bayes algorithm, which estimates the most probable sense for a given word w based on the prior probability of each sense and the conditional probability for each of the features in that context.
The Decision List algorithm
The Vector Space Model algorithm, which considers the features of the context as binary values in a vector. In the training phase, a centroid is calculated for each possible sense of the word. These centroids are then compared with vectors of features from testing examples using the cosine function.
Support Vector Machines, the most widely
used classification technique in WSD at
SensEval3
The most commonly used features used by the systems proposed and presented at SensEval3 can be divided as follows:
Collocations: n-grams (usually bi-grams or tri-grams) around the target word are collected. The information stored for thengrams is composed by the lemma, word-from and part-of-speech tag of each word.
Syntactic dependencies: syntactic dependencies are extracted among words around the target word. The relations most commonly used are subject, object, modifier. However, depending on the system, other dependencies might also be extracted.
Surrounding context: single words in a defined window size are extracted and used in a bag-of-words approach.
Knowledge-Based information: Some systems also make use of information suchas WordNet’s domains, FrameNet’s syntactic patterns or annotated examples, among others. 3
The dataset used is the CoNLL 2009 Shared Task built on the CoNLL 2008 task which has been extended to multiple languages. The core of the task was to predict syntactic and semantic dependencies and their labeling.
Data was provided for both statistical training and evaluation, in order to extract these labelled dependencies from manually annotated Treebanks such as the Penn Treebank for English, the Prague Dependency Treebank for Czech and similar Treebanks for Catalan, Chinese, German, Japanese and Spanish languages, enriched with semantic relations. Great effort has been dedicated in providing the participants with a common and relatively simple data representation for all the languages, similar to the 2008 English data. Role-annotated data makes it available for many research opportunities in SRL including a broad spectrum of probabilistic and machine learning approaches.
We have introduced the dataset associated with SRL; we are now prepared to discuss the approaches to automatic SRL and VBS. 4
For many of these subtasks there are standard
evaluations techniques and corpora. Standard
evaluation metrics from information retrieval include
precision, recall and a combined metric called F1
measure
Precision is a measure of how much of the information that the system returned is correct, also known as accuracy. Recall is a measure of how much relevant information the system has extracted from text, thus a measure of the system’s coverage. The F1 measure balances recall and precision.
A corpus is often divided into three sets: training set, development set and testing set. Training set is used for training systems, whereas the development set is used to tune parameters of the learning systems, and sselecting the best model. Testing set is used for evaluation. Cross-corpora evaluation is used in some tasks, for which a fresh test set different from the training corpora is used for evaluation.
In this case, F1 measure is computed as the harmonic mean of Precision and Recall. 5
The model architecture for SRL is inspired from
the one ideated by Marcheggiani et al., 2017
Then, a table with all the hyperparameter values will be shown. 5.1
The word representation component builds from a word wi in a sentence w a word representation xi. Each word w is represented as the concatenation of four vectors: A randomly initialized word embedding xre 2 Rdw .
A pre-trained word embedding xpe 2 Rdw . A randomly initialized part-of-speech tag embedding xpos 2 Rdp .
A randomly initialized lemma embedding xle 2 Rdl that is only active if the word is one of the predicates.
Then, it has been used the Predicate-Specific Encoding. Specifically, when identifying arguments of a given predicate, the authors added a predicatespecific feature to the representation of each word in the sentence by concatenating a binary flag to the word representation. The flag is set as 1 for the word corresponding to the currently considered predicate, it is set as 0 otherwise. In this way, sentences with more than one predicate will be reencoded by Bidirectional LSTMs multiple times. 5.2
Recurrent neural networks (RNN)
Bidirectional LSTMs make use oftwo LSTMs: one for the forward pass, and another for the backward pass. In this way the concatenation of forward and backward LSTM states encodes both left and right contexts of a word.
In this case, the Bidirectional Long-Short Term Memory (BiLSTM) Encoder takes as input the word representation xi and provides a dynamic representation of the word and its context in a sentence. 5.3
The goal of the classifier is to predict and label arguments for a given predicate.
The basic role classifier takes the hidden state of the top-layer bidirectional LSTM corresponding to the considered word at position i and uses it to estimate the probability of the role r.
However, since the context of a predicate in the sentence is highly informative for deciding if a word is its argument and for choosing its semantic role, the authors provides the predicate’s hidden state as another input to the classifier.
Finally, it has been proven advantageous to jointly
embed the role r and predicate lemma l using
a non-linear transformation: ReLU
In the following table the hyperparameter values.
English word embeddings POS embeddings Lemma embeddings LSTM hidden states Role representation Output lemma representation BiLSTM depth Learning rate
Multi-Task Learning: by sharing representations between related tasks (VBS), we can enable our model to generalize better on our primary task (SRL).
Babelfy: usage of a pre-trained model that helps to disambiguate sentences and verbs. In the first solution the two models run in parallel. In the second solution, since the SRL model uses as input the Babelfy’s output, the two models run sequentially. 6.1
In Machine Learning we typically care about optimizing a particular metric. In order to do this, we generally train a single model to perform our desired task, then fine-tune and tweak this model until its performance no longer increases. Even if it is possible to achieve generally acceptable performance, in this way we could miss information that might help us to optimize the relevant metric. Specifically, information deriving from the training signals of related tasks.
We can consider multi-task learning as a form of inductive transfer. Inductive transfer can help to improve a model by introducing an inductive bias, defining a model as preferable with respect to other hypotheses.
Furthermore, the Verb Sense Disambiguation model has been created; following, a brief explanation of the model. We use the same Word Representation and Encoder of the Marcheggiani’s system explained in sections 5.1, 5.2.
The output of the Encoder is used to predict the sense of the verb by applying the Softmax activation function.
Lei (2015) FitzGerald (2015) Roth and Lapata (2016) Marcheggiani (2017) SRL+VSD Model
As Table 2 shows, performance worsens in terms of Precision and Recall.
Therefore, we have a lower value in term of F1 score, which, as already mentioned above, is the harmonic mean of Precision and Recall.
For this reason another solution was developed in
order to improve the results on both Precision and
Recall and then of F1.
B
Specifically, Babelfy performs the tasks of multilingual Word Sense Disambiguation and Entity Linking.
Extracted senses have been used as input of the SRL Model, by replacing the randomly initialized lemma embedding xle 2 Rdl of the word representation of 5.1
In this case we can observe improvements in all
fields. This improvement is not so significant
Moreover, this results shows that improving the VSD task determines improvements in SRL task. 7
The realized work represents the development of a complete system for the Semantic Role Labeling, an important tool to be used in advanced Natural Language Processing applications.
A system of SRL alone is not very useful and it necessarily must be included in a wider application, for example a Question&Answering system or a Neural Machine Translation system. In conclusion, as all the new applications of natural language processing must be able to handle semantic information if they want to have good performances, this type of system can be considered a valuable solution to achieve such performances. The statistical analysis of the errors registered by the system, developing from this analysis new algorithms in order to correct such errors, is another aspect to be considered in the evaluation of this system. 8
As for future works we could certainly try to develop a new Semantic Role Labeling model architecture trying to discover approaches related to models based on Attention.
Attention
An interesting effect of attention
Another future research activity could be the examination of the abovementioned models under different languages, such as Italian.