English. In this paper we present a work aimed at testing the most advanced, stateof-the-art syntactic parsers based on deep neural networks (DNN) on Italian. We made a set of experiments by using the Universal Dependencies benchmarks and propose a new solution based on ensemble systems obtaining very good performances.
Syntactic parsing of morphologically rich languages like Italian often poses a number of hard challenges. Various works applied different kinds of freely available parsers on Italian training them using different resources and different methods for comparing their results (Lavelli, 2014; Alicante et al., 2015; Lavelli, 2016) and gather a clear picture of the syntactic parsing task performances for the Italian language. In this direction seems relevant to cite the EVALITA1 periodic campaigns for the evaluation of constituency and dependency parsers devoted to the syntactic analysis of Italian (Bosco and Mazzei, 2011; Bosco et al., 2014).
Other studies regarding the syntactic parsing of Italian tried to enhance the parsing performances by building some kind of ensemble systems (Lavelli, 2013; Mazzei, 2015).
By looking at the cited papers we can observe that they evaluated the state-of-the-art parsers before the “neural net revolution” not including the last improvements proposed by new research studies.
The goal of this paper is twofold: first, we would like to test the effectiveness of parsers based on the newly-proposed technologies, mainly deep neural networks, on Italian, and, second, we would like to propose an ensemble system able to further improve the neural parsers performances when parsing Italian texts. 2
We considered nine state of the art parsers
representing a wide range of contemporary approaches
to dependency parsing whose architectures are
based on neural network models (see Table 1). We
set-up each parser using the data from the Italian
Universal Dependencies
In Chen and Manning (2014) dense features are used to learn representations of words, tags and labels using a neural network classifier in order to take parsing decisions within a transition-based greedy model. To address some limitations, in Andor et al. (2016) the authors augmented the parser model with a beam search and a conditional random field loss objective. The work of Ballesteros et al. (2015) extends the parser defined in Dyer et al. (2015) introducing character-level representation of words using bidirectional LSTMs to improve the performance of stack-LSTM model which learn representations of the parser state. In Kiperwasser and Goldberg (2016) the bidirectional LSTMs recurrent output vector for each word is concatenated with any possible heads recurrent vector, and the result is used as input to a multi-layer perceptron (MLP) network that scores each resulting edge. Cheng et al. (2016) propose a bidirectional attention model which uses two additional unidirectional RNN, called leftright and right-left query component. Based on Kiperwasser and Goldberg (2016) and Cheng et al. (2016) model, in Dozat and Manning (2017) a biaffine attention mechanism is used, instead of traditional MLP-based attention. The model proposed in Nguyen et al. (2017) train a neural network model that learn jointly POS tagging and graph-based dependency parsing. The model uses a bidirectional LSTM to learn POS tagging and the Kiperwasser and Goldberg (2016) approach for dependency parsing. Shi et al. (2017a,b) described a parser that combines three parsing paradigms using a dynamic programming approach.
Method Parsing Tb: a-s Greedy Parser Ref.-Abbreviation (Chen and Manning, 2014)
CM14 (Ballesteros et al., 2015) - Tb: a-s Be-se
BA15 (Kiperwasser and Goldberg, 2016)- Tb: a-h Greedy
KG16:T (Kiperwasser and Goldberg, 2016)- Gb: a-f Eisner
KG16:G (Andor et al., 2016)
AN16 (Cheng et al., 2016) - Gb: a-f cle
CH16
DM17 (Shi et al., 2017a,b)
SH17 Tb: a-s Beam-S Tb: a-h./ Greedy -eager Gb: a-f Eisner Gb: a-f Eisner (Nguyen et al., 2017)
NG17
We trained, validated and tested the nine considered parsers, as well as all the proposed extensions, by considering three different setups: setup0: only the UD Italian 2.1 dataset; setup1: only the UD Italian PoSTWITA 2.2 dataset; setup2: UD Italian 2.1 dataset joined with the UD Italian PoSTWITA 2.2 dataset (train and validation sets) keeping the test set of PoSTWITA 2.2;
After the influential paper from Reimers and Gurevych (2017) it is clear to the community that reporting a single score for each DNN training session could be heavily affected by the system initialisation point and we should instead report the mean and standard deviation of various runs with the same setting in order to get a more accurate picture of the real systems performances and make more reliable comparisons between them.
Table 2 shows the parsers performances on the test set for the three setups described above executing the training/validation/test cycle for 5 times. In any setup the DM17 parser exhibits the best performances, notably very high for general Italian. As we can expect, the performances on setup1 were much lower than that for setup0 due to the intrinsic difficulties of parsing tweets and to the scarcity of annotated tweets for training. Joining the two datasets in the setup2 allowed to get a relevant gain in parsing tweets even if we added out-of-domain data. For these reasons, for all the following experiments, we abandoned the setup1 because it seemed more relevant to use the joined data (setup2) and compare them to setup0. 3
The DEPENDABLE tool in Choi et al. (2015) reports ensemble upper bound performance assuming that, given the parsers outputs, the best tree can be identified by an oracle “MACRO” (M A), or that the best arc can be identified by another oracle “MICRO” (mi). Table 3 shows that, by applying these oracles, we have plenty of space for improving the performances by building some kind of ensemble system able to cleverly choose the correct information from the different parsers outputs and combine them improving the final solution. This observation motivates our proposal.
To combine the parser outputs we used the following ensemble schemas:
Voting: Each parser contributes by assigning a vote on every dependency edge as described in Zeman and Zˇabokrtsky´ (2005). With the majority approach the dependency tree could be ill-formed, in this case using the switching approach the tree is replaced with the output of the first parser.
Reparsing: As described in Sagae and Lavie (2006) together with Hall et al. (2007) a MST algorithm is used to reparse a graph where
each word in the sentence is a node. The MSTs algorithms used are Chu-Liu/Edmons (cle) and Eisner as reported in McDonald et al. (2005). Three weighting strategies for Chu-Liu/Edmons are used: equally weighted (w2); weighted according to the total labeled accuracy on the validation set (w3); weighted according to labeled accuracy per coarse grained PoS tag on the validation set (w4).
Distilling: In Kuncoro et al. (2016) the authors train a distillation parser using a loss objective with a cost that incorporates ensemble uncertainty estimates for each possible attachment. 4
Tables 4, 7 and 9 show the performances of the ensembles built on the best results on validation set obtained in the 5 training/test cycles considering both setup0 and setup2. Table 6 reports the number of malformed trees for the majority strategy.
Table 5 and 8 report the number of cases when the ensemble combination output differs from the baseline, including both labeled (L) and unlabeled (U) outputs. On the average the percentage of different unlabeled output varies from 2% to 15% with respect to baseline. For the best result (DM17+ALL) the difference on setup0 and setup2 is about 4%.
The results of the voting approach reported in Table 4 shows that the majority strategy is slightly better than the switching strategy, although it must be taken into account that there might be illformed dependency trees for the former strategy.
The percentage of ill-formed trees on valid./test set vary from a minimum of 2% to a maximum of 8%. For this reasons the majority strategy should be used when it is followed by a manual correction phase. The switching strategy performs well if the first parser of voters is one of the best parsers, in fact the combinations AN16+ALL and AN16+CM14+SH17 have worst performance than the counterparts which using the best parser (DM17) as the first voter. Overall, the highest performance is achieved using all parsers together with DM17 as the first voter. For setup0 the increases are +0.19% in UAS e +0.38% in LAS, while in setup2 are +0.92% in UAS e +2.47% in LAS with respect to the best single parser (again DM17).
The results of the reparsing approach reported in Table 7 shows that the Chu-Liu/Edmonds algorithm is slightly better than the Eisner algorithm. In this case, the choice of which strategy
Validation Test Voters/Strategy UAS LAS UAS LAS DM17+CH16+BA15/maj. 94.20% 92.27% 93.77% 92.13% DM17+CH16+BA15/swi. 94.11% 92.16% 93.79% 92.14% AN16+CM14+SH17/maj. 90.43% 87.96% 91.03% 88.47% AN16+CM14+SH17/swi. 89.44% 86.77% 90.17% 87.43% DM17+CM14+SH17/maj. 93.84% 92.03% 93.82% 92.27% DM17+CM14+SH17/swi. 93.76% 91.94% 93.82% 92.25% AN16+ALL/maj. 94.37% 92.65% 93.83% 92.27% AN16+ALL/swi. 93.99% 92.15% 93.43% 91.73% DM17+ALL/maj. 94.42% 92.67% 93.94% 92.41% DM17+ALL/swi. 94.38% 92.60% 93.91% 92.37% DM17 (baseline) 93.74% 91.66% 93.75% 92.03% therefore the choice of which strategy to use is due, in part, to the properties that we want to obin-domain data (social media), giving an improvetain on the combined dependency tree. ment of 1% in UAS and 2:5% in LAS.
Our work is inspired by the work of Mazzei
Validation Test Voters/Strategy UAS LAS UAS LAS DM17+CH16+BA15/cle-w2 93.82% 91.85% 93.54% 91.83% DM17+CH16+BA15/cle-w3 93.89% 91.82% 93.78% 92.06% DM17+CH16+BA15/cle-w4 94.20% 92.28% 93.72% 92.04% DM17+CH16+BA15/eisner 94.05% 92.05% 93.46% 91.78% ALL/cle-w2 94.31% 92.53% 93.85% 92.23% ALL/cle-w3 94.16% 92.41% 94.00% 92.48% ALL/cle-w4 94.29% 92.58% 93.95% 92.38% ALL/eisner 94.31% 92.53% 93.95% 92.35% DM17 (baseline) 93.74% 91.66% 93.75% 92.03% setup2
Validation Test Voters/Strategy UAS LAS UAS LAS DM17+CH16+BA15/cle-w2 90.33% 86.95% 87.69% 83.31% DM17+CH16+BA15/cle-w3 89.82% 85.96% 87.59% 81.95% DM17+CH16+BA15/cle-w4 90.41% 86.99% 87.94% 83.32% social media texts. DM17+CH16+BA15/eisner 90.50% 87.05% 88.04% 83.51% ALL/cle-w2 90.52% 87.53% 88.36% 84.25% ALL/cle-w3 89.90% 86.75% 87.79% 83.54% ALL/cle-w4 90.42% 87.46% 88.19% 84.11% ALL/eisner 90.45% 87.41% 88.31% 84.08% We gratefully DM17 (baseline) 89.82% 85.96% 87.59% 81.95%
proaches on the best models in setup0 and setup2. The baseline is again defined by the best results of DM17. networks, in order to gain more diversity among models in setup0 and setup2. In brackets are reported the differences between the distilled models and the best results of DM17, as baseline. the models used in the ensembles; furthermore we have experimented the distilling strategy and eisner reparsing algorithm. Moreover, we built ensembles on larger datasets using both generic and acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research. Anita Alicante, Cristina Bosco, Anna Corazza, and Alberto Lavelli. 2015. Evaluating italian parsing across syntactic formalisms and annotation schemes.
Bosco, Rodolfo Delmonte, Alessandro Moschitti, and Maria Simi, editors, Harmonization and Development of Resources and Tools for Italian Natural Language Processing within the PARLI Project, Springer International Publishing, Cham, pages 135–159.
Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta,
neural networks. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
ACL, Berlin, Germany, pages 2442–2452. Miguel Ballesteros, Chris Dyer, and Noah A.
Smith. 2015. Improved transition-based parsing by modeling characters instead of words with lstms. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. ACL, Lisbon, Portugal, pages 349– 359.
Cristina Bosco, Felice DellOrletta, Simonetta Montemagni, Manuela Sanguinetti, and Maria Simi. 2014. The evalita 2014 dependency parsing task. In Proceedings of the Fourth International Workshop EVALITA 2014. Pisa, Italy, pages 1–8.
Cristina Bosco and Alessandro Mazzei. 2011. The evalita 2011 parsing task. In Working Notes of EVALITA 2011, CELCT, Povo, Trento.
Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, Doha, Qatar, pages 740–750.
Hao Cheng, Hao Fang, Xiaodong He, Jianfeng Gao, and Li Deng. 2016. Bi-directional attention with agreement for dependency parsing. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
ACL, Austin, Texas, pages 2204–2214.
Jinho D. Choi, Joel Tetreault, and Amanda Stent. 2015. It depends: Dependency parser comparison using a web-based evaluation tool. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
ACL, Beijing, China, pages 387–396.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In Proceedings of the 2017 International Conference on Learning Representations.
Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). ACL, Beijing, China, pages 334–343.
Johan Hall, Jens Nilsson, Joakim Nivre, Gu¨lsen Eryigit, Bea´ta Megyesi, Mattias Nilsson, and Markus Saers. 2007. Single malt or blended? a study in multilingual parser optimization. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007. ACL, Prague, Czech Republic, pages 933–939.
Eliyahu Kiperwasser and Yoav Goldberg. 2016.
Simple and accurate dependency parsing using bidirectional lstm feature representations. Transactions of the Association for Computational Linguistics 4:313–327.
Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, and Noah A. Smith. 2016. Distilling an ensemble of greedy dependency parsers into one mst parser. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. ACL, Austin, Texas, pages 1744–1753.
Alberto Lavelli. 2013. An ensemble model for the evalita 2011 dependency parsing task. In Bernardo Magnini, Francesco Cutugno, Mauro Falcone, and Emanuele Pianta, editors, Evaluation of Natural Language and Speech Tools for Italian. Springer Berlin Heidelberg, Berlin, Heidelberg, pages 30–36.
Alberto Lavelli. 2014. Comparing state-of-theart dependency parsers for the evalita 2014 dependency parsing task. In Proceedings of the Fourth International Workshop EVALITA 2014.
Pisa, Italy, pages 15–20.
Alberto Lavelli. 2016. Comparing state-of-the-art dependency parsers on the italian stanford dependency treebank. In Proceedings of the Third Italian Conference on Computational Linguistics (CLiC-it 2016). Napoli, Italy, pages 173– 178.
Alessandro Mazzei. 2015. Simple voting algorithms for italian parsing. In Roberto Basili, Cristina Bosco, Rodolfo Delmonte, Alessandro Moschitti, and Maria Simi, editors, Harmonization and Development of Resources and Tools for Italian Natural Language Processing within the PARLI Project, Springer International Publishing, Cham, pages 161–171.
Ryan McDonald, Fernando Pereira, Kiril Ribarov, and Jan Hajic. 2005. Non-projective dependency parsing using spanning tree algorithms. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing. ACL, Vancouver, British Columbia, Canada, pages 523–530.
Dat Quoc Nguyen, Mark Dras, and Mark Johnson. 2017. A novel neural network model for joint pos tagging and graph-based dependency parsing. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. ACL, Vancouver, Canada, pages 134–142.