1 English. With the arrival of BERT 2 in 2018, NLP research has taken a significant step forward. However, the necessary computing power has grown accordingly. Various distillation and optimization systems have been adopted but are costly in terms of cost-benefit ratio. The most important improvements are obtained by creating increasingly complex models with more layers and parameters. In this research, we will see how, by mixing transfer learning, zero-shot learning, and ONNX runtime3, we can access the power of BERT right now, optimizing time and resources, achieving noticeable results on day one.
Italiano. Con l'arrivo di BERT nel 2018, la ricerca nel campo dell'NLP ha fatto un notevole passo in avanti. La potenza di calcolo necessaria però è cresciuta di conseguenza. Diversi sistemi di distillazione e di ottimizzazione sono stati adottati ma risultano onerosi in termini di rapporto costo benefici. I vantaggi di maggior rilievo si ottengono creando modelli sempre più complessi con un maggior numero di layers e di parametri. In questa ricerca vedremo come mixando transfer learning, zero-shot learning e ONNX runtime si può accedere alla potenza di BERT da subito, ottimizzando tempo e risorse, raggiungendo risultati apprezzabili al day one. 1 Copyright ©️ 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1
In a process with data that change very quickly and the need to resort to complete training in the shortest possible time, transfer learning techniques have made possible a fast fine-tuning of BERT models. The distillation of a model made it possible to decrease the load and the times without significantly losing accuracy. These models, therefore, require, at least, constant fine-tuning training. In addition, a BERT model specially designed for the Italian language and with a vocabulary containing technical terms increases its effectiveness.
Constant and multi-disciplinary training requires specific skills and tailor-made services. In this research, we will see an effective way to make both things possible. The idea is to use a way to exchange AI models between library and frameworks, the ONNX project, and a runtime, the ONNX runtime project, to optimize inference for many platforms, languages and hardware. The ONNX runtime is still working to optimize the training directly in the ONNX format.
The second goal is to find a viable alternative with
acceptable performance at the start of a new
project while waiting for a trained BERT model.
The research was carried out for the ATE
ABSITA
To start using a sentiment analysis system, we need several elements. Certainly, a starting dataset 2 https://github.com/google-research/bert 3 https://microsoft.github.io/onnxruntime/ with the related labels. In the tasks of the challenge, we have the reviews of 23 different products. Each review has a corresponding rating assigned by the end-user. For each review, it was required to extract the aspects contained in it. By aspect, we mean every opinion word that expresses a sentiment polarity. Finally, each aspect was classified as a pair of values: positive or negative, for 4 possible states.
Imagine a system that receives an unspecified number of reviews in real-time with new products and different categories. We find ourselves in the situation of always having to fine-tune our models.
The complexity of BERT makes training time difficult for constant alignment. Being able to reduce the training time, or being able to put in place an alternative in the meantime, new perspectives open up, such as: Made inference calls before a full trained model is completed.
(optional) reclassify recent product reviews after the model update.
In this perspective, in order to validate my
hypotheses, I used the AlBERTo
The results for the three tasks for the second
submission, called ghostwriter19_b, were obtained in
two different way:
for the first two tasks, I used the model of
the first submission but exported on ONNX
and ran with the ONNX runtime.
for the third task, I trained the model with
TensorFlow using a Zero-Shot learner
[ZSL]
To test the models, I used two different machines with Ubuntu 20.04 LTS: 6 vCPU on Intel Xeon E5-2690 v4 - 112GB with P100 (GPU) 14 cores on Intel Xeon E5-2690 v4 - 32GB (CPU) 2.1
To identify an aspect, the dataset contains a label for every single word with three possible values: - B for Begin of an aspect. - I for Inside an aspect. - Or for Outside, not in an aspect.
For example, the review “La borraccia termica svolge egregiamente il proprio compito di mantenere la temperatura, calda o fretta che sia. La costruzione è ottimale e ben rifinita. Acquisto straconsigliato!” is labeled as: The model will be evaluated with the F1-score. The score results from the full matched aspects, the partial matched ones, and the missed ones. The preliminary results with the Ktrain model were encouraging (table 1).
ghostwriter19_a
At this point, the model has been exported with ONNX in maximum compatibility mode. The model ran with the ONNX runtime optimized for CPU.
The performances have remained unchanged, but the speed of inference has significantly improved (table 2). 4 https://github.com/amaiya/ktrain 5 https://www.tensorflow.org/ ghostwriter19_a CPU ghostwriter19_b CPU with ONNX runtime ghostwriter19_a GPU ghostwriter19_b GPU with ONNX runtime 4 68 124 217
The improvement is 17x for the CPU version and 1.75x for the GPU version. For this task, the aspects identified in Task 1 have been used. This implies that an error in Task 1 will have a decisive impact on Task number 2.
positive (POS:true,NEG:false) negative (POS:false,NEG:true) mixed polarity(POS:true, NEG:true) neutral polarity (POS:false, NEG:false) As showed to the image from the challenge website6: The results on the DEV test outperform the baseline (table 3). Also, for this task, the performance is improved with the use of ONNX runtime (table 4). 6 http://www.di.uniba.it/~swap/ate_absita/task.html
Task 3 is a classification problem. However, fully understanding the score is not easy. The evaluation operation is carried out by different people and with different styles. A product with a similar review is rated according to the expectations and judgment of other users differently.
Furthermore, in order to obviate the long training
time that a constant updating requires, compared
with systems used by the previous version of
EVALITA, such as an ensemble system with Tree
Random Forest and Bi-LSTM
The proposed labels were the possible numbers for evaluation, then the numbers from 1 to 5. The proposed prediction value is a weighted average of the two values with the highest probability, if and only if the gap between the two values is less than 10−3. Otherwise, only the value with the highest probability will be considered valid. For this task, I omitted the ONNX runtime test because a stable converter for the ZSL version is not available.
The score for this task is the Root Mean Squared Error between the polarity predicted and the polarity assigned by the user.
The loss in performance is 18%, but the entire previous training phase is skipped (table 5). 3
The results obtained with the DEV dataset are very positive both in terms of accuracy and performance. ZSL has proven to be an incredible technology to invest in. The Ktrain seems to suffer a heavy overfit.
The research aims not to have a relevant model but to prove that a model could be productionready with fewer resources and time.
However, in all three tasks, the models outperformed the baseline with a significant gap in terms of accuracy/RMSE. 3.1
The final results with the TEST dataset are:
ghostwriter19_a_D ghostwriter19_a_T
The results are about 12% lower than those obtained in the research phase (table 6).
It will be interesting to continue experimenting with different ONNX options to find a better combination of compatibility and performance. 3.2
The final results with the TEST dataset are: ghostwriter19_a_D ghostwriter19_b_T
0.6019 0.4994 0.2 The loss from DEV to TEST is about 17% (table 7). However, the percentage of the difference between the results of Tasks 1 and 2 have been maintained with the DEV and TEST datasets. This is in line with expectations, worse model performance in Task 1 impacted Task 2 proportionally. In return, working on a better model will improve both tasks. 3.3
ghostwriter19_a_D ghostwriter19_b_D ghostwriter19_a_T ghostwriter19_b_T
0.6997 0.8526 0.81394 0.83479 1.0806 The difference between the DEV and TEST datasets is marked here only for trained model, 14% (table 8). The untrained one performed slightly better, 2%, with the TEST dataset.
This result confirms that an underperforming model has the same performance of a model that use ZSL, as assumed.
The price to pay, however, is that the average inference time for the ZSL is 157x higher than the pure TensorFlow model obtained with Ktrain. The results demonstrated that it is possible to create hybrid systems for training and inference to make the power of BERT more accessible. In the time it takes to train a new and optimized model, an untrained ZSL model can make up for it in the meantime.
Optimizing, and in future training, our models to be intrinsically optimized for the platform and framework we have chosen to use does not affect performance and future use.
The improvements obtained in the use of ONNX
runtime for these Italian tasks are in line with what
Microsoft demonstrated, for the English language,
at the beginning of 2020