English. We investigate head-noun identification in complex noun-compounds (e.g. table is the head-noun in three legs table with white marble top). The task is of high relevancy in several application scenarios, including utterance interpretation for dialogue systems, particularly in the context of e-commerce applications, where dozens of thousand of product descriptions for several domains and different languages have to be analyzed. We define guidelines for data annotation and propose a supervised neural model that is able to achieve 0.79 F1 on Italian food noun-compounds, which we consider an excellent result given both the minimal supervision required and the high linguistic complexity of the domain.
Italiano. Affrontiamo il problema di identificare head-noun in nomi composti complessi (ad esempio "tavolo" is the headnoun in "tavolo con tre gambe e piano in marmo bianco"). Il compito é di alta rilevanza in numerosi contesti applicativi, inclusa l’interpretazione di enunciati nei sistemi di dialogo, in particolare nelle applicazioni di e-commerce, dove decine di migliaia di descrizioni di prodotti per vari domini e lingue differenti devono essere analizzate. Proponiamo un modello neurale supervisionato che riesce a raggiungere lo 0.79 di F-measure, che consideriamo un risultato eccellente data la minima quantitá di supervisione richiesta e la alta complessitá linguistica del dominio.
Noun-compounds are nominal descriptions that
hold implicit semantic relations between their
constituents
There are a number of tasks and application
domains where identifying noun-compound
headnouns is relevant. A rather general context is
ontology population
In this paper we propose a supervised
approach based on a neural sequence-to-sequence
model
The paper is structured as follow: we first define noun-compound head-noun identification, with specific reference to complex noun-compound (Section 2). Then we introduce the neural model we have implemented (Section 3), and finally the experimental setting and the results we have obtained (Section 4). 2
In this Section we focus on Italian compoundnouns referring to food, the domain on which we run our experiments. Similar considerations and same methodology can be applied to compoundnouns in different domains and languages.
There is a very high variety of food compoundnouns, describing various aspects of food, including: simple food names, like mortadella di fegato, pesce, gin and tonic, aglio fresco; recipes mentioning their ingredients, like scaloppine al limone, spaghetti al nero, passato di pollo, decotto di carciofo; recipes focusing on preparation style, like mandorle delle tre dame, cavolfiore alla napoletana; food names focusing on visual or shape properties, like filetto di vitellone senza grasso visibile, palline di formaggio fritte; food descriptions containing a course name, like antipasto di capesante, dessert di mascarpone; food using fantasy names, like frappé capriccioso, or insalata arlecchino; food including proper names or brands, like saint-honoré, tagliatelle Matilde, formaggio bel paese; food names focusing on cooking modalities, like pane fatto in casa, or peperoni fritti; and focusing on alimentary properties, like ragù di carne dietetico, or sangria analcolica.
We assume that the head-noun of a food description is the more informative noun in the nouncompound, i.e. the noun that better allows to answer questions about properties of the food being described by the noun-compound. We consider the following four property related questions, in order of relevance: 1. What food category (e.g. meat, vegetable, cake, soup, pasta, fish, liquid, salad, etc.) is described by the noun-compound? 2. What course (e.g. main, appetizer, side dish, dessert, etc.) is described by the nouncompound? 3. Which is the main ingredient (in term of quantity) described by the noun-compound? 4. Which could be the overall quantity (expressed in grams) of food described by the noun-compound?
Although our approach does not require any domain knowledge, for the purpose of human annotation and evaluation it is useful to assume a simple ontology for food, where we define the properties used for judging head-nouns and the set of possible values for each property. Table 1 reports the food ontology at the base of our work.
quantity
Values meat, vegetable, cake, soup,
pasta, fish, liquid, salad... main, first, second, appetizer, side , dessert... <simple food>
<grams>
A good head-noun should be as much informative as possible about the noun-compound properties, or, in other terms, it should allow to infer as much as possible answers to questions 1-4. Answers to such questions are in most of the cases graduated and probabilistic, as a noun-compound contains just a fraction of the knowledge needed to answer them. For instance, given question 1) for the food noun-compound insalata noci e formaggio should be posed in the following way: knowing that formaggio is part of a food description, which is the probability that the overall description correctly refers to a food of category salad? When the probability is very low, we assume a "no guess" value for the answer.
The core procedure for human annotations considers each content word in a food description, fills in the values of the four attributes, and then select the noun with the best guesses. Below some examples (in black the selected head of the food description): insalata noci e formaggio: because insalata is a better predictor of the food category than formaggio or noci. involtini di peperoni: because peperoni is a better predictor of food category (i.e. vegetable) and of the main ingredient than involtini. budino al cioccolato fondente: because budino is a good predictor of food category (i.e. dessert) and a better predictor than cioccolato of the main ingredient (i.e. milk) of the noun-compound. 2.1
Given a food noun-compound, the task we address is to predict its head-noun, labelling one or more consecutive tokens in the food description. We assume that a head is always present, even in case it is poorly informative.
Two annotators were selected to annotate a data set of 436 food names, collected from recipe books, with their head-noun. The inter annotator agreement, computed at the token level, is Cohen’s kappa: 0.91, which is considered very high.
In table 2 we give an overview of the data set of food-description head (FDH) we created focusing on two main orthogonal characteristics: whether the head-noun is comprised of a single token or of a multi-token, and whether the head-noun corresponds to the beginning of the food description or not. As can be seen, the vast majority of headnouns is either made of a single token (almost 90% of cases), or starts at the beginning of the entity name (almost 80% of cases). The combination of Position 1st token Not 1st token Total the two accounts for roughly 70% of the cases. From the point of view of predicting the headnoun of a food name, easier cases are given by single token in first position, while harder cases are given by multi-token head inside the food name. 3
The architecture we use to recognize head-nouns
is based on a bidirectional LSTM (Long Short
Term Memory) network
Recurrent Neural Network (RNN) is a class of
artificial neural network that resemble a chain of
repeating modules to efficiently model sequential
data
The hidden state ht of the network at time t
captures only the left context of the sequence for the
input at time t. The right context for the input at
time t can be captured by performing the same
operation in the negative time direction. The input
can be represented by both its left context !ht and
!
right context ht as ht = [ht ; ht ]. Similarly, the
representation of the completed sentence is given
by hT = [h!T ; h0]. Such processing of the input in
both forward and backward time-step is known as
bidirectional RNN. Though a vanilla RNN is good
at modelling sequential data, it struggles to
capture the long-term dependencies in the sequence.
Long Short Term Memory (LSTM)
Ct = ft C(t 1) + it C~t ot = (Wo [ht 1; xt] + bo
ht = ot tanh(Ct) where xt is the embedding for input at time t; it, ft, ot are the input, forget and output gates, respectively.
The task of head-noun identification aims to predict a sequence of tags y = fy1; y2; ::; yng given an input sequence X = fx1; x2; ::xng. The system is modeled as a sequence labelling task and consists of three main steps: i) word embedding: each word in the sequence is embedded to a higher dimension; ii) Input encoder: encoding the sequence of embeddings; iii) Classification: labelling the sequence.
Word embeddings. Each word in the input
sequence is represented by a vector of d-dimensions
that captures the syntactic and semantic
information of the word. The representation is carried by
a word embedding matrix E 2 Rd jvj where jvj
is the input vocabulary size. In addition to this,
the model combines a character embedding that is
learned during training using a Bi-LSTM network
to deal with out of vocabulary terms and possible
misspellings
To represent the core structure of a complex
noun-compound, we also use the following
handcrafted features of a head-noun candidate token
The concatenation of word embedding, final states of bidirectional character embeddings network, and hand crafted features is used as the word representation.
Input encoder. LSTM nodes are used to encode the input sequence of word embeddings. We employ a bidirectional LSTM (Bi-LSTM) to capture the context in both forward and backward timesteps. The hidden representation of a word at time t is given as,
ht = [!h t; h t]
Classification. The output layer receives the
hidden representation from the Bi-LSTM and
outputs a probability distribution over the possible
tag sequences. Then, a conditional random field
(CRF) layer
n n X Ayi;yi+1 + X Pi;yi i=0 i=1 where A is the transition matrix representing the transition scores from tag i to tag j. The probability of the tag sequence is then computed using a softmax operation as follows, p(y^jX) = Py~2Y^ exp(Score(y~)
exp(Score(y^)) The training is done by maximizing the log probability of the correct output tag sequence. 4
The dimension of character embedding is set to 30
and embeddings are learned using 50 hidden units
in each direction. For the word embeddings, as
learning this level of representation with a small
dataset is highly inefficient, we decided to use
pre-trained embeddings trained using skip-gram
To compare the performance of the proposed approach, we provide two baselines: i) 1st token, where the 1st token of a noun-compound is chosen as its head-noun; ii) Spacy1, where the root token of the dependency tree for the noun-compound is chosen as its head-noun. 1st token. This baseline implicitly accounts for a number of linguistic behaviours of head-nouns in Italian language: (a) avoids stop words as headnouns, as they do not occur at the first position of a noun-compound; (b) avoids adjectives as headnouns, as they usually occur after the noun they modify; (c) captures the syntactic head of the noun-compound, which, in Italian is likely to be the first noun in a Noun Phrase; as already seen in Table 2. Summing up, the first-token baseline captures relevant linguistic behaviours, and is a strong competitor of our neural model, as in more than 80% of the entries in our dataset the first token belongs to head-noun of the noun-compound. Spacy. This is a widely known open-source library for natural language processing and include a syntactic dependency parser. Given an input sequence, based on the result returned by the dependency parser, the root of the sequence is chosen to be the head-noun. We used the statistical model it_core_news_sm2 released by Spacy for Italian language. 4.3
The performance of the models are evaluated
using F1 score as in CoNLL-2003 NER evaluation
The results for the FDH dataset are shown in Table 3. The baselines 1st token and Spacy achieve 1https://spacy.io/ 2https://spacy.io/models/it Baselines 1st token
Spacy Bi-LSTM a) word_emb b) a + hc_feat c) a + char_emb d) b + CRF d) d + char_emb a performance of 70.27 of 62.67 respectively. In particular, the performance of syntactic dependency parser from Spacy reiterates the difference between the semantic and syntactic head. The results are shown by incremental features, for the proposed approach. The models reported without CRF, are trained using a softmax function as output layer to predict the tag. We can notice from the results that using only the pre-trained embeddings, the network suffers from a poor recall and fails to achieve even the baseline performance. However, using either character embedding or the hand-crafted features, improves the performance of the model on par with the baseline. Since the single token head-noun in FDH dataset is very high (as shown in table 2), learning the multi token head-nouns and the dependency of tags is a challenge. However, introducing the CRF layer to jointly predict the sequence of tags in combination with the hand crafted features, enables us to predict multi-token heads and improve the performance of the model to 78.09. Finally, the character embeddings learned during training helps to improves the recall further, reaching a F1 score of 79.58. 5
We have addressed head-noun identification in complex noun-compounds, a task of high relevancy in utterance interpretation for dialogue systems. We proposed a neural model, and experiments on Italian food noun-compounds show that the model is able to outperform strong baselines even with a small amount of data. For the future we plan to extend our investigation to other domain and languages.