We present an application in the sphere of Natural Language Processing to tackle the problem of inference relation (NLI) between pairs of sentences in Italian without an Italian NLI training dataset. The model has been developed making use of the Knowledge Distillation technique. The model may be exploited in industrial scenarios to analyse any kind of user-generated content. Specifically, applications may include product or service review analysis, brand reputation analysis, or social media sentiment analysis. To support this claim, we evaluate the proposed architecture over the native Italian ABSITA dataset on diferent tasks, such as Aspect-Based Sentiment Analysis, Sentiment Analysis, and Topic Recognition. We balanced the dataset used for each task to obtain a 50/50 division, and we respectively achieve the performances in terms of accuracy of 94.03%, 88.12%, and 71.19%. By applying the model over diferent scenarios, we empirically demonstrate the generality and exploitability of the NLI technique. While the developed model has still room for improvement, it is suitable to operate unsupervised free-text information extraction - mainly over reviews - in a production environment.
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License
The lack of a comprehensive NLI dataset in Italian has been the key driver to adopt the architecture described
Natural Language Inference (NLI) is the task of determining the inference relation between two short and ordered texts, usually defined “premise” and “hypothesis”. It is a challenging task that requires understanding the nuances of language and context, as well as the ability to reason and make logical implications.
One common approach to tackle NLI is to use
Neural Networks, such as Recurrent Neural Networks or
Transformer models [
Examples from the benchmark NLI dataset are shown
in Table 1 — namely, Stanford NLI [
If the hypothesis may be inferred from the premise, the NLI task may be reinterpreted as a task of information extrapolation. We could query a dataset of reviews to extract any custom information — e.g. topic, or sentiment. 1.2. Goal Let us have a large corpus of sentences in Italian, with the objective to extract information from each sentence.
Premise “A soccer game with multiple males playing” “An older and younger
man smiling” “A man inspects the uniform of a figure in some East Asian country” between sentences.
Hypothesis “Some men are playing a sport” “Two men are smiling and laughing at the cats playing on the floor”
Label Entailment
Neutral “The man is sleeping”
Contradiction
The goal of this research is to build a model with the following traits: • Ability to understand the inference relation between sentence pairs in a specific language. Informally, it means to train a model on a NLI dataset. • Ability to understand Italian. The majority of the published models are in English. This reflects the fact that research is published in English, and relevant NLI datasets are all in English. To the best of our knowledge, we did not find any comprehensive NLI datasets in the Italian language to train a Deep Learning model.
The industrial applications include, among others: • Product or service review analysis. We focused
on this application to run our tests. • Brand reputation analysis. • Social media sentiment analysis. in 2. This procedure can be generalised to any task involving a language-specific dataset. Another plus of this methodology is that we require no dataset labeling costs. This architecture has been demonstrated to have NLI capability in Italian without being exposed to a NLI training dataset in Italian. 2.
Method 2.1. NLI training (, ,=( , )) = 1 || ∈|| ∑ ( ( )−( ))2 (1)
Two instances of the encoder described in 2.1 have been taken for the experiment. One acts as teacher encoder model , the other as a student encoder model .
The application of Knowledge Distillation has the objective to share the domain knowledge of the teacher encoder model to the student encoder model, and at the same time learn a new vectorial representation for the target language. A schematic representation is provided
The obtained NLI classifier, able to understand Italian, accepts a sentence pair to output a NLI label.
Execution-wise, the Knowledge Distillation task on a
Tesla P100–PCIE–16GB GPU was completed in
approximately five (5) hours on the TED2020 (English–Italian)
dataset [
After a training session over a NLI English dataset, the result is a model, based on Transformers, that can proifciently address the NLI task — only in English though, despite being originally trained on multiple languages.
More information about this work available at Ref. [
In our work, we want to enable a multilingual model,
based on Transformers, previously fine-tuned for a
specific task only in one specific language, to proficiently
address that specific task in Italian.
2.2. Knowledge Distillation
_ = 24
_
_
• ℎ
•
•
• ℎ
•
• ℎ = 6
•
• = 1 − 6
_ = 2 − 5
_ = 1 − 2
_ = 4
_ℎ = 256
_ℎ = 128
The second step of our methodology is to employ a train- either by trial-and-error, or suggested online by informal
ing without language-specific NLI training data. The
selected approach is Knowledge Distillation [
We employed Knowledge Distillation to perform NLI in Italian, with the objective of forcing a translated sen- can be made. tence to have the same location mapping in the vector space as the original sentence. Compared to other approaches, this has the following advantages: languages.
space. • Easy to extend models with few samples to new • Easy to ensure desired properties for the vector • Low hardware requirements.
We require a teacher model (encoder) , that maps sentences in the source language to a vectorial representation. Further, we need parallel (translated) sentences = (( 1, 1), ..., ( ,
)) with being a sentence in the source language and a sentence in the target language. We train a student encoder model such that ( given mini-batch , we minimise the Mean Squared Error ) ≈ ( being
). For a loss function:
The parameters were selected in an unstructured way; sources. No cross-validation or grid-search analyses have been performed for computational constraints. Therefore, no guarantees on the optimality of the parameters
The discussed architecture has been tested over the standard NLI task in Italian. We made use of small Italian NLI datasets, available online, such as RTE3-ITA and RTE-2009. We also exploited an open-source machine translation model, developed by Facebook, named No Language Left Behind [ 7], to obtain a comprehensive Italian NLI dataset, suitable for testing.
The discussed architecture, based on Knowledge Distillation, demonstrated to perform better than another tested architecture that was directly trained over machine translated NLI datasets, despite having an objective disadvantage. We stress the fact that the proposed architecture was never directly trained over any kind of Italian NLI data.
Aspect-Based Sentiment Analysis at EVALITA (ABSITA) [8] is an Aspect-Based Sentiment Analysis dataset. Contains Italian hotel reviews that may touch diferent topics (such as price, location, cleanliness, etc.) and a sentiment associated to each topic (knowing that sentiments for Table 3 diferent topics may be contrasting). By choosing arbi- ABSITA results, over the Topic Recognition task. trary NLI hypotheses, this dataset may emulate a total of three (3) diferent tasks, namely Sentiment Analysis, Topic Recognition, and Aspect-Based Sentiment Analysis. 3.2.3. Aspect-Based Sentiment Analysis The core idea behind this setting comes from the desire Aspect-Based Sentiment Analysis (ABSA) is the task to to query a text — in NLI, a set of premises (e.g. a set of recognise the sentiment about each sub-topic in a senreviews), in an unsupervised way, to receive specific an- tence. As detailed above, we would like to exploit the swers from a predefined list of answers (e.g. the presence models to apply ABSA in an unsupervised manner — to of a topic from a list of topics). In the case of open an- do this, we fix a hypothesis arbitrarily. We assume that swers, a question-answer architecture would have been the hypothesis we have chosen captures the logical immore suitable. plication that is the core of NLI. Follow results for the ABSITA dataset, detailed in Table 4. Note that the hy3.2.1. Sentiment Analysis pothesis has been arbitrarily set to “La camera é pulita” (“The room is clean”).
Metric
Accuracy
Min F1-Score Macro-Avg F1-Score
Accuracy
Min F1-Score Macro-Avg F1-Score
Balancing 1:1 1:1 1:1 1:15 1:15 1:15
Where the fiteen (15) in the “Balancing” column stands for the number of diferent tuples (topic, sentiment) in the dataset. The 1:1 balancing has been obtained by randomly sampling sentences from the fiteen (15) classes that do not compose the target. The two scenarios have been proposed to extensively test the generalisation capability of the models. 3.2.4. Summary In this work we tested diferent architectures showing that it is possible to obtain reasonable accuracies over different Natural Language Processing tasks by fine-tuning a single architecture based on sentence embeddings over the NLI task.
We showed that various Natural Language Processing problems may be mapped into a NLI task — in this way, we empirically proved the generality of the NLI task. We would like to stress over the lack of need to retrain any models to obtain the results over each specific task.
The full, updated model is available at this link1, along with all instructions for usage. We named it I-SPIn 1https://huggingface.co/Lutech-AI/I-SPIn Sentiment Analysis is the task to recognise the overall sentiment of a sentence. As detailed above, we would like to exploit the models to apply Sentiment Analysis in an unsupervised manner — to do this, we fix a hypothesis arbitrarily. We assume that the hypothesis we have chosen captures the logical implication that is the core of NLI. Follow results for the ABSITA dataset, detailed in Table 2. Note that the hypothesis has been arbitrarily set to “Sono soddisfatto” (“I feel satisfied”).
Dataset ABSITA ABSITA ABSITA
Balancing 1:1 1:1 1:1
Task Sentiment Analysis Sentiment Analysis Sentiment Analysis
Metric
Accuracy
Min F1-Score Macro-Avg F1-Score
Result 88.12% 86.89% 88.02% 3.2.2. Topic Recognition Topic Recognition is the task to recognise whether or not a sentence is about a topic. As detailed above, we would like to exploit the models to apply Topic Recognition in an unsupervised manner — to do this, we fix a hypothesis arbitrarily. We assume that the hypothesis we have chosen captures the logical implication that is the core of NLI. Follow results for the ABSITA dataset, detailed in Table 3. Note that the hypothesis has been arbitrarily set to “Parlo di pulizia” (“I’m talking about cleanliness”).
Where the seven (7) in the “Balancing” column stands for the number of diferent topics in the dataset. The 1:1 balancing has been obtained by randomly sampling sentences from the seven (7) classes that do not compose the target. The two scenarios have been proposed to extensively test the generalisation capability of the models. (Italian-Sentence Pair Inference). dings useful for neural machine translation? (2018).
To test our model perfomances over Sentiment Anal- doi:1 0 . 1 8 6 5 3 / v 1 / N 1 8 - 2 0 8 4 . ysis, Topic Recognition, and Aspect-Based Sentiment [7] N. Team, M. R. Costa-jussà, J. Cross, O. Çelebi, M. ElAnalysis, we employed arbitrary hypotheses. We tried bayad, K. Heafield, K. Hefernan, E. Kalbassi, J. Lam, our best to avoid any biases (e.g. hypotheses were chosen D. Licht, J. Maillard, A. Sun, S. Wang, G. Wenzek, by colleagues that had never taken a look at the dataset), A. Youngblood, B. Akula, L. Barrault, G. M. Gonzalez, but we acknowledge that some bias may have been in- P. Hansanti, J. Hofman, S. Jarrett, K. R. Sadagopan, troduced. This is currently considered an open problem. D. Rowe, S. Spruit, C. Tran, P. Andrews, N. F. Ayan, S. Bhosale, S. Edunov, A. Fan, C. Gao, V. Goswami, F. Guzmán, P. Koehn, A. Mourachko, C. Ropers, 4. Figures S. Saleem, H. Schwenk, J. Wang, No language left behind: Scaling human-centered machine translation (2022). doi:h t t p s : / / d o i . o r g / 1 0 . 4 8 5 5 0 / a r X i v .
2 2 0 7 . 0 4 6 7 2 . [8] P. Basile, D. Croce, V. Basile, M. Polignano, Overview of the evalita 2018 aspect-based sentiment analysis task (absita) (2018). URL: https://ceur-ws.org/ Vol-2263/paper003.pdf.