English. This paper describes a collection of modules for Italian language processing based on CoreNLP and Universal Dependencies (UD). The software will be freely available for download under the GNU General Public License (GNU GPL). Given the flexibility of the framework, it is easily adaptable to new languages provided with an UD Treebank.
The fast-growing research field of Text
Mining and Natural Language Processing (NLP) has
shown important advancements in recent years.
NLP tools that provide basic linguistic annotation
of raw texts are a crucial building block for further
research and applications. Most of these tools, like
NLTK
In this paper, we present CoreNLP-it, a set of customizable classes for CoreNLP designed for Italian. Our system, despite being simpler than any of the above mentioned toolkits, both in scope and number of features, has the advantage of being easily integrated with the CoreNLP suite, since its development has been grounded on the principle that all data structures be natively supported by CoreNLP.
The key properties of CoreNLP-it are:
UD based and compliant: The toolkit and models are based on UD and follow its guidelines for token and parsing representation. It can provide all annotation required in the UD framework, and produces a CoNLL-U formatted output at any level of annotation, as well as any other type of annotation provided in CoreNLP.
word tokens (e.g., enclitic constructions) are handled by providing separate tokens. Moreover, the CoNLL-U output can represent such information following the UD guidelines.
hybrid tokenization and sentence splitting module replaces the original rule-based annotators for this task.
it is built (including the exclusive usage of 1http://universaldependencies.org/ CoreNLP classifiers and data structures), the add-on can be seamlessly integrated with the latest available version (3.9.1) of CoreNLP, and is expected to work with upcoming versions as well.
out-of-the-box new capabilities of supporting basic annotations for other languages provided with a UD Treebank.
This paper is organized as follows: in Section 2, we present the architecture of the toolkit, whereas its core components (annotators) are described in Section 3. The results on Italian are discussed in Section 3.5. Section 4 shows preliminary experiments for the adaptation of the software to two additional languages provided with a UD treebank, namely Spanish and French. 2
CoreNLP-it has been built as an add-on to the
Stanford CoreNLP toolkit
The main architecture of CoreNLP consists of an annotation object as well as a sequence of annotators aimed at annotating texts at different levels of analysis. Starting from a raw text, each module adds a new annotation layer such as tokenizaton, PoS tagging, parsing etc. The behavior of the single annotators can be controlled via standard Java properties. Annotators can analyze text with both rule-based or statistical-based models. While rule-based models are typically language dependent, statistical based ones can be trained directly within the CoreNLP toolkit in order to improve the performance of the default models or to deal with different languages and domains. 2.2
The main goal we pursued in developing CoreNLP-it was to keep the original CoreNLP structure and usage intact, while enabling it to deal with Italian texts in order to produce a UDcompliant and UD-complete output. More specifically, we aimed at building a system capable of providing all textual annotations required by the UD guidelines. Moreover, our system is also compatible with standard CoreNLP functions (e.g., Named Entity Recognition (NER) and Sentiment annotation). For these reasons,we implemented a series of custom annotators and statistical models for Italian. The custom annotators replace the corresponding CoreNLP annotators leaving intact the annotation structure and output of the annotators they are replacing.
For simplicity, we used only one of the UD
treebanks available for Italian, namely the UD
adaptation of the ISDT Italian Treebank
The current version of CoreNLP-it can be easily integrated and configured into CoreNLP by adding the custom annotator classes and their respective models into the pipeline. Such classes and their properties can be added in a configuration file or called via the API interface. This procedure follows the standard CoreNLP documentation and guidelines for custom annotator classes. In addition, we provide a new class (resembling a CoreNLP one) for the training of the hybrid tokenization and sentence splitting. The configuration of the classifier and the required dictionaries (cf. Section 3.1) can be specified in a separate property file. 3
The annotators described in the following sections are aimed at producing a UD compliant and complete output. The following information is extracted from text: Sentences, Tokens, Universal PoS Tags, language specific PoS Tags, Lemmas, Morphological Features, and Dependency Parse Tree for each sentence.
In this section, we briefly describe each module of our linguistic pipeline, focusing on the annotators and models it implements.
Sentence Splitting and Tokenization are handled by a single classifier, namely the annotator it tok sent. The process splits raw text into sentences, and each sentence into tokens. Crucially, the tokenization process can deal with both single and multi-word tokens as specified by the CoNLLU format.
Multi word tokens such as verbs with clitic pronouns (e.g., portar-vi “carry to you”) and articulated prepositions (prep + determiner) (e.g., della, di+la “of the”), are split into their respective components. The information about the original word and its position in the sentence is however retained within each token by exploiting the token span and original word annotations.
Tokenization is usually solved with rule-based systems able to identify word and sentence boundaries, for example by identifying white spaces and full stops. However, in order to avoid encoding such set of rules, we implemented a model inspired by Evang et al. (2013). At its core, the process is driven by a hybrid model. First, it uses a character-based statistical model to recognize sentences, tokens, and clitic prepositions. Then, a rule based dictionary is used to optimize the multiword tokens detection and splitting.
The classifier tags each character with respect to one of the following classes: i. S: start of a new sentence; ii. T: start of a new token; iii. I: inside of a token; iv. O: outside of a token; v. C: start of a clitic preposition inside a token (e.g. mandarvi).
The classifier is a simple implementation of the maximum entropy Column Data Classifier available in the Stanford CoreNLP. To train the model, we used the following feature set: i. window: a window of n characters before and after the target character; ii. the case of the character; iii. the class of the previous character.
In order to deal with multi-tokens, the system allows for a full rule-based tagging of a parametric list of multi-tokens typically belonging to a strictly language dependent closed class words. In the Italian implementation, such words are articulated prepositions (prep + determiner). The word list to be ignored is fed to the classifier during training.
Moreover, an additional set of rules can be applied after the classification step in order to deal with possibly misclassified items. In particular, the system simply checks each token against a dictionary of multi-words and split them accordingly. In the case of Italian, we built a dictionary of clitic verbs (which are instead an open class) by bootstrapping the verbs in the treebank with all possible combinations of clitic pronouns. A final tagging phase was used to merge the rule-based and statistical predictions. 3.2
The Maximum Entropy implementation of the
Part-of-Speech Tagger
In order to annotate Universal PoS (uPoS) tags, a separate annotator class, namely upos, has been implemented.
For what concerns the xPoS Tagger, the Maximum Entropy model was trained on the UD-ISDT Treebank. uPoS tags are instead approached with a rule based strategy. In particular, we built a mapping between xPoS and uPoS based on the UDISTD Treebank. The mapping is used within the annotator to assign the uPoS tag based on the predicted xPoS tag. 3.3
In order to annotate each token with its corresponding lemma and morphological features, we developed a rule-based custom annotator. The annotator exploits a parametric dictionary, to assign lemmas based on the word form and PoS. In particular, the dictionary contains the lemma and UD morphological features for n (f orm; P oS) pairs. The form is used as the main access key to the dictionary, while PoS is used to solve ambiguity, e.g., between amo as ”I love” or as ”fishing hook”. Finally, in cases of PoS ambiguity, corpus frequency is used to select the target lemma.
The dictionary can be manually built or extracted from a UD treebank. In the latter case, the provided Vocabulary class has methods to extract and build a serialized model of the vocabulary. 3.4
The Neural Network Dependency Parser
implemented in Stanford CoreNLP
As for Italian, we used FastText
We must point out that one of the main
shortcomings of implementing a more statistically
oriented model for tokenization with respect to a rule
based one is that it may underperform in the case
of badly formatted or error-filled texts, which we
cannot find in most Treebanks. However, we
believe that such an approach could be nonetheless
very useful in that it can be automatically scaled
to different linguistic registers and text genres.
Moreover, most typical errors could be avoided by
means of data augmentation strategies and the use
of more heterogeneous data for training, such as
for example the PoSTWITA-UD Treebank
It is important to stress that the main focus of this work was to build a framework allowing for a fast and easy implementation of UD models based on Stanford CoreNLP from a software engineering point of view. The basic pre-trained models are intended as a proof of concept, and will require further parameter tuning to increase their performance. 4
One of the key goals that has driven the development of CoreNLP-it is keeping the core code implementation as language independent as possible. To obtain the required linguistic knowledge, the framework exploits statistical models or external resources. On the one hand, the use of big linguistic resources to perform some of the tasks can affect the computational performances, but the system enables the construction of basic resources from the treebank used for training. On the other hand, this framework is very flexible, especially by considering tasks like tokenization and lemmatization. In particular, the system is able to produce a full UD-compliant Stanford Pipeline for languages for which an UD Treebank is available.
In order to validate this claim, we focused on
two languages closely related to Italian, namely
Spanish and French. We trained the respective
models on the UD-adapted corpora ES-ANCORA
Since the core of the adaptation of the Stanford Pipeline to Universal Dependencies relies on the Tokenization phase, we report here the results obtained for this task. It is clear that the rest of the models (i.e., PoS tags and Parsing) can be trained simply by following the Stanford CoreNLP guidelines. Results obtained for the tokenization modules for French and Spanish are shown in Table 2.
Tok., S.Split. Spanish French Lemma Spanish French
99,9 99,7 66 69
All statistical models have similar performances with respect to Italian ones. The main differences, as expected, concern the tasks most dependent on external resources (e.g., Lemmatization). For example, we noticed a much lower recall for multiword token identification, given the exclusive use of the examples found in the training set. The approach shows very promising results especially for tokenization and sentence splitting modules which are central for all the subsequent levels of analysis based on UD. It is clear that for PoS Tagging and Parsing further developments based on Stanford CoreNLP and language-specific resources are required to account for the specific features of each language. 5
In this paper, we presented CoreNLP-it, a set of
add-on modules for the Stanford CoreNLP
language toolkit. Our system provides basic language
annotations such as sentence splitting,
tokenization, PoS tagging, lemmatization and dependency
parsing, and can provide a UD-compliant output.
Our rule based and statistical models achieve good
performances for all tasks. In addition, since the
framework has been implemented as an add-on
to Stanford CoreNLP, it offers the possibility of
adding other new annotators, including for
example the Stanford NER
In the near future, we plan to refine the core code by performing extensive tests to better deal with additional UD-supported languages and optimize their performances. We also plan to release the tool as well as the basic trained models for Italian. Moreover, we intend to perform data augmentation strategies to refine our models and make them able to work properly also with ill-formed or substandard text input.