In this paper we present a deep neural network model capable of inducing the rules that identify the syntax complexity of an Italian sentence. Our system, beyond the ability of choosing if a sentence needs of simpli cation, gives a score that represent the con dence of the model during the process of decision making which could be representative of the sentence complexity. Experiments have been carried out on one public corpus created speci cally for the problem of text-simpli cation.
Text Simpli cation (TS) is Natural Language process that aims at making a text more easily understandable for a determined target of people by changing the lexical and syntactic content of the original text.
The usefulness of TS can be appreciated by di erent kind of people, such as those
who are not mother tongue or have language disabilities. For example, people
a ected by aphasia during the reading process have di culties to understand
syntactic structure [
For what concerns the Italian language, TS is an underdeveloped research area
and this is evident from the availability of few resources and the number of
developed methodologies. A cause for this is probably that the English Language
is more widespread. Nonetheless, works have been done trying to face di erent
NLP problems in Italian Language.[
The problem of evaluating the complexity of a document has already been
tackled in the past using indexes like GulpEase [
In the domain of TS, words like complex and simple should be used keeping in mind that the complexity of a sentence is strictly related to a determined kind of people that could have di erent needs.Since the corpus we have used contains examples that represent the simpli cation process for di erent classes of readers, our simpli cation system is not specialized for any speci c target reader. Nonetheless, the corpus is suited for the the goal of this work that is to understand the potentiality of a model based on Neural Network (NN) to classify Italian sentences using only the part-of-speech(PoS) tags which represent the syntactical aspects of the text.
In this paper, we give a contribution to the TS eld using NN for developing a system capable of inducing the patterns which characterize the syntactic complexity of a sentence. Our system classi es the sentence in 2 classes di cultto-read and simple-to-read and produces a score which represent the con dence of the network during the decision making process that could be interpreted as a measure of complexity of the given sentence.
The paper is organized as follow: in section 2 we will describe the system and our approach of facing the problem, in section 3 we will explain the methodology of carrying out the tests and results, in section 5 we will give conclusion. 2
Our method is based on NN algorithms and it is able to discriminate if an Italian sentence needs to be simpli ed in order to be more easily understandable by di erent classed of target readers. Furthermore, the network gives a score that could be interpreted as a score of the sentence complexity and that represents the con dence of the network during the decision making.
To manage the task of understanding the sentence complexity we have chosen to use Recurrent Neural Networks (RNNs) that are a class of NN useful for analyzing sequences. In the recent past RNNs have shown their e ectiveness in many di erent linguistic elds since it is well known that a sentence can be structured as a sequence of tokens such as words, punctuation symbols or part-of-speech. 2.1
We have evaluated a sentence as a sequence of part of speech tags calculated
using a pre-trained version of TreeTagger3 [
Both the tag-sets contain tags that identify linguistic elements such as adverbs, adjective, verb, noun but they have di erent way to represent these linguistic categories. For instance, in the description of verbs one tag-set (Baroni) contains 17 di erent verb categories while the other one (Stein) contains 12 di erent verb categories. In total, the Baroni tag-set contains 52 di erent categories of part-of-speech tags while the Stein tag-set contains 38 di erent categories of part-of-speech tags.
Each part-of-speech tag obtained by TreeTagger is then coded as a vector using the one-hot encoding in which a part-of-speech tag becomes a vector full of 0s except for a unique one position in which the value is 1. Every sentence is evaluated as sequence of one-hot encoded vectors that are passed as input to the network that analyzes them. The complete process is shown in gure 1 and 2. The network that we have used to tackle the problem of evaluating the complex
Salve, avrei bisogno di una informazione piuttosto urgente
TreeTagger
TK1 TK2 TK3 TK4 TK10
NOM PON VER:cond NOM ... SENT
ity of a Italian sentence is an RNN based on Long Short Term Memory (LSTM)
arti cial neurons [
The rst layer of the network is made up of 512 LSTM arti cial neurons. The
outcome of this layer is then handled by fully connected layer composed by two
neurons which use the softmax activation function. Finally, we have applied L2
regularization [
We have used the well known cross-entropy as loss function which has been
minimized using the RMSPROP [
PoS1 PoS2 PoS3 ... PoSn
ONE
HOT ENCODING
LSTM LAYER
FULLY CONNECTED
LAYER
SOFTMAX
COMPLEX CLASS
SIMPLE CLASS
There is a lack of corpora useful to tackle the text simpli cation problem for the
Italian language by means of machine learning algorithms. Thus we have chosen,
to the best of our knowledge, the biggest available dataset created for the italian
text simpli cation nowadays [
The corpus [
The corpus has been entirely tagged with the Treetagger parser, both training and tests are based only on the tags without taking into account neither lemmas or punctuation symbols. The experiments suggest that a NN is capable of discovering the syntactics rules which characterize both approaches by understanding how to associate each sentence to the correct class. 3.2
The evaluation of the model has proceeded using the K-FOLD cross-validation (K-FOLD) method. K-FOLD is a validation method useful for assessing the abilities of a statistical model especially in presence of few data, which is our case. In fact, 63:000 pairs of sentences are not enough to evaluate this kind of model and the use of K-FOLD evaluation methodology is necessary for clearly understand how well the classi er is capable of generalizing his knowledge to an independent dataset. The method partitions randomly the dataset into K equal sized subsets (in our case K=10): the method selects all possible K-1 subsets that are used to train the model and use the last one to validate it. The K models have been trained to classify two classes of sentences that are present into the corpus: di cult-to-read (positive class), simple-to-read (negative class).
The quantization of the results has been done using the Precision, Recall, True Positive Ratio (TPR) and True Negative Ratio (TNR) measures for each iteration of K-FOLD. The Recall and Precision measures, respectively, the percentage of positive class elements that the model is able to correctly classify and the percentage of mistakes that it has done during the classi cation of the positive class elements. TPR6 and TNR measure respectively the proportion of elements correctly identi ed as positive and the proportion of elements correctly identi ed as negative. Finally, the results have been averaged on the K executed iterations. We have decided to use as baseline model a support vector machine (SVM) model trained using two di erent kernel methods: RBF and polynomial. This choice is justi ed by the fact that, to our knowlege, does not exist another classi cation system that take as input only the part-of-speech tags. READ-IT can measure the syntactical complexity of a sentence but it makes available an online interface that is not handy to make a huge amount of tests. The SVM model takes as input the part-of-speech tags of the input sentence as a vector in which each position represents a di erent part-of-speech tag whose value is the number of the corresponding part-of-speech in the source text. Table 1 shows the results obtained by both models.
Model Kernel TAG-SET Recall Precision True Positive Ratio True Negative Ratio RNN-S - STEIN 0.819 0.834 0.819 0.837 RNN-B - BARONI 0.764 0.845 0.764 0.859 SVM-SP polynomial STEIN 0.589 0.832 0.589 0.881 SVM-SR RBF STEIN 0.750 0.798 0.750 0.810 SVM-BP polynomial BARONI 0.506 0.839 0.506 0.903 SVM-BR RBF BARONI 0.731 0.793 0.731 0.809
The results show the performance of the NN model compared to those obtained by the SVM using di erent kernel methods. The RNN reaches the best result of Recall and, obviously, on the True Positive Ratio with the STEIN tag-set and the best result of Precision using the BARONI tag-sets. The True Negative Ratio is better using the SVM model with the polinomial kernel for both the tag-set. Although the good performance of the SVM-BP measured by the True Negative Ratio, the relative Recall measure reaches only a value of 0.506. In our opinion, the best model is the RNN-B one that uses the BARONI tag-set, because it shows a good value of Recall that is better than the ones obtained by the SVM and the best value of Precision. Furthermore, both Recall and True Negative Ratio measures are not much di erent from the best ones obtained respectively by RNN-S and SVM-BP (approximately 0.05 points of di erence). The results suggest the e ectiveness of our model to evaluate the syntactical complexity aspects of an Italian Sentence. The SVM model reaches a high value of True Negative Ratio that will be studied in our future works trying to understand what is the key of this outcome and if it can be embedded in the RNN-B model.
Looking into how the tag-sets in uence the results we observe that both of them allow the models to obtain good value of Precision and True Negative Ratio, in fact the maximum di erence, carried out as the best value minus the worst value, among the precision results is 0.052 and the maximum di erence among the True Negative Results is 0.094. Conversely, their usage a ect more the Recall measure in which the maximum di erence is 0.313. The problem is speci cally related to the polynomial kernel that seems to have more di cult to infer a considerable number of rules that identify the elements of the class di cult-to-read. The good performance achieved by the models, except for the Recall of the SVM model with polynomial kernel, suggests that both tag-sets express well the syntactic features of the text and they are suited to address this kind of problem coupled with a neural model. 5
We have presented a system for measuring the syntactic complexity of a sentence written in Italian language. Our system takes as input a sentence and it expresses the syntax of the sentence as a sequence of part-of-speech tags. The RNN at the base of our system, after learning the patterns that determine the syntactic complexity through a speci c corpus created for TS, is capable of classifying a sentence as being di cult-to-read or simple-to-read. We have tested the system using two di erent tag-sets and we have compared the RNN with a SVM model using di erent kernel methods. Results show the e ectiveness of the Neural Network model to address the task of classifying Italian sentences based on their readability complexity. The system can be used either as a stand-alone system or as a support tool for the creation of a complex system to address di erent problems such as the generation of simpli ed text.