The identification and classification of some circumstance semantic roles like Location, Time, Manner and Direction, a task of Semantic Role Labeling (SRL), plays a very important role in building text understanding applications. However, the performance of the current SRL systems on those roles is often very poor, especially when the systems are applied on domains other than the ones they are trained on. We present a method to build open domain SRL system, in which the training data is expanded by replacing its predicates by words in the testing domain. A language model, which is considered as a text mining technique, and some linguistic resources are used to select from the vocabulary of the testing domain the best words for the replacement. We apply our method on the case study of transferring a semantic role labeler trained on the news domain to the children story domain. It gives us valuable improvements over the four circumstance semantic roles Location, Time, Manner and Direction.
Playing an essential role in text understanding, Semantic Role Labeling is the
task of natural language processing that specifies “Who did What to Whom,
and How, When and Where?” in text [
For example, the processing of the sentence “Mary gave Peter a book at school yesterday” should result in the identification of a “giving” event with “Mary” as the Agent of the event, “Peter” as the Recipient and “a book” as the Item being given. The Location of the “giving” event, or where it took place, is “at school” and the Time of the event is “yesterday”.
In this paper, we call an event (“giving” event) in a sentence the semantic frame, the verb or noun that evokes the frame (“gave”) the predicate, the words (“Mary”, “Peter”, “a book”, “at school”, “yesterday”) that play a role in the event the arguments and their roles (“Agent”, “Recipient”, “Thing being given”, “Location”, “Time”) the semantic roles.
The task of semantic role labeling is to detect the event, to identify its arguments and assign the correct semantic roles to them. Thanks to the availability of semantic annotated resources (e.g. PropBank3, FrameNet4 ), supervised machine learning approaches have been very successful in constructing automatic semantic role labellers. Assuming the predicates are already given, those systems can reach an F15 score of 85% when the training and testing data are in the same domain. But, when testing on other domains, the scores often drop significantly6.
Text mining is the task of automatic discovery of new, previously unknown
information from unstructured document collections. Meanwhile, a language model
tries to capture the properties of a language, and predicts the next word in a
word sequence. It is trained on a collection of unlabeled texts, and therefore is
considered as a text mining technique. Recently, we some attempts to use such
language models in a semi-supervised setting for semantic recognition [
In this paper, we develop a methodology to generate additional training data for SRL by replacing selected verbal predicate words in training examples using a language model. For each selected predicate in the training examples, from the vocabulary of the domain that the SRL is applied on, a list of replacement words which we believe can occur at the same position as the selected word, are generated. We introduce and explore a variety of features for identifying how words should be replaced, including predicate vs. argument status, POS, WordNet related words, and a replacement score based on a language model. As for experiment, we present a case study of improving the performance of a SRL system trained on the news domain when applying to the children story domain. The case study is based on our ongoing European project “Machine Understanding for interactive StorytElling” (MUSE)7. One of the fundamental goals of MUSE is to detect actors, actions, plots in children stories, and render them as 3D worlds. SRL with its function of identifying the events in texts plays an essential role in solving our problem. Among the set of semantic roles, some circumstance semantic roles like Location, Time etc. are very important to understand the full meaning of an event, while the performance of the current SRL systems on them is often very poor, especially when testing on a domain other than the one they are trained on. Thus, in our case study, we target to improve SRL on the four PropBank circumstance roles: AM-LOC (Location), AM-TMP (Time), AM-MNR (Manner) and AM-DIR (Direction).
In the next sections, we present related work (Section 2), linguistic resources
4 https://framenet.icsi.berkeley.edu 5 Harmonic mean of recall and precision. 6 http://ufal.m↵.cuni.cz/conll2009-st/ 7 http://www.muse-project.eu/ (Section 3), underlying assumptions, objectives and task definition (Section 4), methodology (Section 5), case study (Section 6), and conclusion (Section 7). 2
Semi-supervised approaches to semantic role labeling recently have received the
attention of the computational linguistics community. Information from language
models have been used as extra features to improve the performance of SRL. [
Besides the semi-supervised approaches that extend the feature set of SRL, there
are other attempts to generate new training examples automatically by using
unlabeled data. [
None of the above works consider both structural similarity and language models as a source of evidence for generating training examples, nor do they evaluate di↵erent approaches to similarity depending on the roles sought. In contrast to most of the above works, we evaluate the proposed methods when porting the learned model to texts from a domain that is di↵erent from the one the semantic role labeler was trained on. 3
The Penn Proposition Bank (PropBank) [
In VerbNet [
The most frequently encoded relation among synsets is the super-subordinate relation (also called hyperonymy, hyponymy or IS-A relation). It links more general synsets like furniture, piece of furniture to increasingly specific ones like bed and bunkbed. 4
Underlying assumptions, objectives and task definition
Semi-supervised learning is very dicult to accomplish in natural language
processing tasks. In general, it is successful when the labeled training data contain
seed examples that are representative for the whole data set and when the
cluster hypothesis holds, that is, when a suitable similarity metric can correctly
cluster the unlabeled examples with the labeled seed examples [
We assume that a language model (e.g., [
The notation of the symbols used in this paper is given in Table 4. The task of the semi-supervised semantic role labeler discussed in this paper is to learn from a set of manually annotated sentences, a set of unannotated sentences, a language model and some linguistic resources, a model that assigns semantic roles to the set of semantic frames of sentences in a test set. A sentence may contain more than one frame. Each semantic frame consists of one predicate that evokes the frame and several arguments that play a role in the frame. Predicates and arguments may be composed of more than one word. In this paper, we work with verbal predicates, and use head word labeling, which means if an argument consists of more than one word then the semantic role is assigned to only the head word. For instance, if “in the park” is the argument playing the role AMLOC, then only the head word of the phrase “in the park”, “in”, is labeled with the label AM-LOC. Given a sentence s composed of n words w1, w2, ..., wn, for each semantic frame f in s, each word wi (i 2 { 1, 2, ...n}) has received a label ri 2 R [ { NULL} during the manual annotation for training, or will receive a label ri 2 R [ { NULL} during testing or evaluation, where R is a set of predefined semantic roles and NULL means empty label. If ri 6= NULL, then wi is the head of an argument of f with ri as the semantic role. In the approach that we describe in this paper, R is the set of PropBank semantic roles (see Table 1).
Instead of training a SRL system on the given manually annotated sentences Sl, we generate a set of new training examples by using a language model trained on a set of unannotated sentences, then use them together with Sl as the training data for the SRL. 5
The steps of our methodology to generate new training examples and train a SRL system are shown in Figure 1.
Given a manually annotated sentence set Sl which can be used as training data, a set of unannotated sentences Sul, a vocabulary V including words in the domain of the testing data St, a language model L, and some linguistic resources, we create new training examples and train the SRL system as follows: First, L is trained on Sul. Then, L, Sl, V and the linguistic resources are used to generate new training examples Snl. The algorithm to generate new training examples is presented below. Finally, the SRL system is trained on Sl [ Snl. 5.1
In what follows, we present our general model to generate new training instances for the semantic role labeling task. The detail of our algorithm is given in Figure 1 (in dashed rectangle) and Algorithm 1. It consists of five main steps: Step 1. Selecting data for replacement. We select a set of semantic frames from Sl used for the replacement. The selection can be performed in di↵erent ways depending on specific case studies. For each of the selected semantic frame, we choose its predicate as the word to be replaced. For example, given a sentence “Mary gave a book to Peter at school”, the semantic frame “giving” has “gave” as predicate. “gave” is the word selected for the replacement.
Unannotated examples
Sul Vocabulary of test set
V Annotated examples
Sl
Train language model Generate replacement words Select data for replacement
Step 2. Generating replacement words for each word selected to be replaced. A statistical language model assigns a probability to a sequence of m words by means of a probability distribution. For each word selected to be replaced, we use the language model L trained on Sul, and the vocabulary V of the testing domain, to generate a list of replacement words. Given a sentence composed of w1, w2, ..., wn, with wi is the word to be replaced, for each nwj 2 V, the score of replacing wi by nwj is calculated by the probability of the sequence of words wi z, wi z+1, ..., wi 1, nwj , wi+1, wi+2, ..., wi+z obtained by putting nwj in the context of wi where z is size of the context window taken into account: ReplacementScore(wi, nwj) = P (wi z, wi z+1, ..., wi 1, nwj, wi+1, wi+2, ..., wi+z) This probability score is calculated by the language model. It is used to rank the replacement words in our algorithm. Since the size of V may be very large and the words at the end of the list may have a very low score which often represents noise, only N words that have highest scores are chosen. After this step, we receive a ranked list of the top N replacement words for each candidate of the replacement. Step 3. Applying filters to reduce noise in the list of replacement words. There may be a great deal of noise in the replacement words returned by the language model since it does not take into account enough information (syntactic, semantic etc.) to generate a replacement word that can be replaced perfectly for a word in a given sentence assuring the same semantic role. Thus, some linguistics filters are needed to improve the correctness and meaningfulness of the replacement.
Step 4. Replacing words in each sentence selected to be replaced by their replacement words that passed the filters, then we form a new unannotated set of sentences Su. Step 5. Guessing semantic frames and their semantic labels for each sentence in Su to have an annotated semantic frame set Snl.
In the following sections, we will present in more detail the language model used, some proposed filters, how to perform replacement and guess semantic role labels for the new sentences obtained by the replacement. Algorithm 1 Generate novel training examples. 1: procedure GenerateNewExample(L, Sl, V, z, N ) 2: Su = ; , Snl = ; , Stemp = ; ; 3: Select semantic frames that are used for the replacement from Sl: Ssl = selected semantic frames 4: for each sentence s 2 Sl do 5: for each word wi in s do 6: if wi is the predicate of a semantic frame 2 Ssl then 7: for each nwj 2 V do 8: ReplacementScore(wi, nwj) =
P (wi z, wi z+1, ..., wi 1, nwj, wi+1, wi+2, ..., wi+z) obtained by using L; 9: end for 10: Sort nwj according to ReplacementScore, then choose top N words that have highest scores forming the ranked list Listi; 11: Stemp = Stemp [ (s, wi, Listi); 12: end if 13: end for 14: end for 15: for each (s, wi, Listi) in Stemp do 16: for each replacement word nwj in Listi do 17: if nwj passes filters then 18: for each semantic frame f of s that is in Ssl and receives wi as the predicate do 5.2
In this paper, we use the Recurrent Neural Network Language Model9 (RNNLM) [
A language model tries to capture the properties of a language and is trained on a collection of unlabeled texts, so it can be considered as a text mining technique. 5.3
Because the list of top N replaceable words returned by the language model may contain a great deal of noise, we propose specific filters to improve the performance of the system.
Part-Of-Speech filter (POS filter): We keep replacement word nwj for wi if nwj has the same POS tag as wi, when replacing wi in sentence s.
WordNet filter : We keep replacement word nwj for wi if nwj and wi are synonyms or have the same hypernym in WordNet. Here, we ignore the problem of word disambiguation. We only use the first word sense when finding the synonyms and the words that have the same hypernym.
For example, “January” has “Jan” as synonym, “February”, “March”, etc. are the words that have the same hypernym “Gregorian calendar month”.
Predicate filter : A suitable replacement word of a predicate should also evoke a frame with correct roles when it is placed in the sentence of the target. Our idea is to assign role labels to the new frame based on the role labels of the current frame, but it raises the problem of how to find a mapping between the role sets of the two frames, and detect the correct sense of the new frame. Based on this idea, one possibility is to select only replacement words for which the mappings between role sets are available. By using SemLink (see Section 3), we define a filter specifically for predicates: for each predicate wp evoking a frame f , we keep replacement predicate word nwj for wp if f and one frame evoked by nwj are mapped to the same VerbNet class and the mappings from those two frames to the VerbNet class are defined in SemLink. 5.4
Replacement words that have passed filters are used to generate new training examples. Given a semantic frame f of a sentence s composed of n words w1, w2, ..., wn, wp (p 2 {1, 2, ..., n}) is the predicate of f , and wa1, wa2, ..., wam (a1, a2, ..., am 2 { 1, 2, ..., n}) are the heads of the m arguments of f with r1, r2, ..., rm as semantic role labels, respectively. After the filtering step, the list of replacement words of wp, Listp, includes j words {nw1, nw2, ..., nwj}. For each nwt 2 Listp, we replace wp by nwt in sentence s and obtain sentence s0 composed of n words w1, w2, ...wp 1, nwt, wp+1, ..., wn. If nwt has passed the Predicate filter - which we use as a default filter -, it can be a semantic predicate and the argument structure of the frame evoked by nwt is similar to the argument structure of the frame evoked by wp. Thus, we guess that nwt also invokes a semantic frame in s0 with wa1, wa2, ..., wam as arguments (the new semantic frame and f - the frame evoked by wp - have the same argument words). In order to predict the sense and role labels of the new semantic frame, we use the mappings between PropBank semantic frames and VerbNet classes that can be found in SemLink. We first find a frame f 0 of nwt so that both f 0 and f are mapped to a same VerbNet class c. The mappings from f and f 0 to c are denoted by m1 and m2, respectively. If the Predicate filter has been applied before, f 0 exists and can be found in this step. Then, we can guess that f 0 is the new frame evoked by nwt in s0. As for semantic role labels of f 0, if in f , wai (i 2 { 1, 2, ..., m}) with semantic role label ri, is a circumstance role AM-s, then its role does not change in f 0. That means ri is also the role label of wai in f 0. Otherwise, if the role label of wai in f is ri, then the role label of wai in f 0 is m2 1(m1(ri)). For example, the sentence “Rachel wore a hat in her room” has the frame “wear.01” (wore) with “Rachel” as A0, “hat” as A1, “in” (the head of the prepostion phrase “in her room”) as AM-LOC, and the predicate “wore” has “donned” as a replacement word. By replacing “wore” by “donned” in the sentence, we have a new sentence “Rachel donned a hat in her room” and “donned” evokes a new frame. In SemLink, we can find the VerbNet class “simple dressing-41.3.1” linked to the Propbank frame “wear.01” and one PropBank frame of the predicate “don”, “don.01”. The role mapping between the VerbNet class and the two frames can be found in Table5. By applying our method, we have a new frame “don.01” with “Rachel” as A0 (mapped to the “Agent” VerbNet role), “hat” as A1 (mapped to the “Theme” VerbNet role), and “in” as AM-LOC (circumstance role) (See Figure 3). In the EU-EP7 MUSE project10, in which KU Leuven is involved, we instantiate a virtual world with information extracted from children stories. Our fundamental goal is to introduce a new way of exploring and understanding information by “bringing text to life” through 3D interactive storytelling. Children stories will be taken initially as input, due to the relative simplicity of such stories and the relative ease of results evaluation, 10 http://www.muse-project.eu/ Original sentence s
Original frame f m1 m2
A0 A0
Rachel wore a hat in her room w1 w2 w3w4 w5 w6 w7 wa1 wp wa2wa3 predicate
A1
AM-LOC
Listp={donned (nw1)} f = wear.01 A1
AM-LOC predicate
f' = don.01 Verbnet class c Agent
Theme
c= simple_dressing-41.3.1 New frame f' New sentence s' wa1 wa2wa3
Rawc1hel dnown1nedwa3hwa4t win5hwe6r rowo7m and will then be translated into formal knowledge that represents the actions, actors, plots and surrounding world. In a next step this formal knowledge is rendered as virtual 3D worlds in which the user can explore the text through interaction, re- enactment and guided game play. Finally, we will evaluate to which degree this procedure enhances the understanding by children of simple fantasy stories. In our project, NLP techniques are necessary to bring natural language closer to 3D immersive environments. Among the needed NLP techniques, SRL is one of the most important ones, since it labels the textual content of the story by semantic roles identifying actors and activities, that are then visualized in a 3D world. In this paper, we present our work on building a SRL system for the children story domain.
To create virtual 3D worlds, the location where the activities happen, the order of the activities, the tools used by the actors in the activities and the direction where the actors move toward, are very important information. Such kinds of information can be represented by the semantic roles AM-LOC, AM-TMP, AM-MNR and AM-DIR respectively in SRL. However, the performance of the current SRL systems on those roles is often very poor, especially when testing on a domain other than the one they are trained on. Therefore, in our work, we target to improve SRL system on those four roles: AM-LOC, AM-TMP, AM-MNR and AM-DIR. 6.1
We select “Tuk the Hunter” story as the data for our project demonstration. The story (with some slight changes in the content) is the testing data in our case study. To evaluate the performance of SRL on our domain, we annotate semantic roles for the story following the PropBank annotation guideline and dependency head word labeling. The total number of annotated semantic frames in our testing data is 154. The detailed number of instances per role is given in Table 6.
Most of the annotated data available for semantic role labeling are in news domain. To Data AM-LOC AM-TMP AM-MNR AM-DIR Testing 19 28 32 10
Training 10387 23347 11837 1146 be used as our training data, we select CoNLL 200911 training dataset which contains parts of the Wall Street Journal corpus12. The detail number of instances per role in the training data is given in Table 6. 6.2
In our case study, we collect 252 children stories (mostly are fairy tales) to create the
domain of children story. They are used together with the first 80 million words of the
Reuters corpus to train the Recurrent Neutral Network Language Model13, and the
vocabulary of those stories is used to generate new training examples.
We realize that most of the instances of the four targeted roles are prepositions. It
suggests us to choose semantic frames that contain at least one preposition argument
in the CoNLL 2009 training data as the base for our training data expansion.
The SRL system is used in our experiment is the Lund university’s semantic parsing [
In this paper, we present a methodology of building an open-domain semantic role labeling. In our case study, we transfer the SRL model trained on the news domain to the children story domain, by collecting children stories to create the new domain, then replacing verbal predicates in the training data by the words of the new domain given a language model. We keep the precision score from dropping by using the occurrence probabilities and some linguistic filters to verify linguistic patterns obtained by the replacements. The valuable enhanced results over the four circumstance roles AM-LOC, AM-TMP, AM-MNR and AM-DIR show clearly the e↵ectiveness of our methodology on this case study14.
This work is funded by the EU ICT FP7 FET project “Machine Understanding for interactive StorytElling” (MUSE) http://www.muse-project.eu/.