Given the number of Arabic speakers worldwide and the notably large amount of content in the web today in some fields such as law, medicine, or even news, documents of considerable length are produced regularly. Classifying those documents using traditional learning models is often impractical since extended length of the documents increases computational requirements to an unsustainable level. Thus, it is necessary to customize these models specifically for long textual documents. In this paper we propose two simple but efective models to classify long length Arabic documents. We also fine-tune two diferent models-namely, Longformer and RoBERT, for the same task and compare their results to our models. Both of our models outperform the Longformer and RoBERT in this task over two diferent datasets.
the proposed solutions are based on the sliding window
paradigm [
Transformers while ofsetting their major shortcomings In this paper we present two BERT-based language are needed at this stage of development, as they could models and fine-tune two others for Arabic long doculead to full maturation of a concept that has been demon- ment classification. The first language model consists of strated to be impressively successful with semantic tasks. four main layers: sentence segmentation layer, BERT
There have been numerous attempts to improve the layer, a linear classification layer, then the sentence
performance and eficiency of BERT with long docu- grouping layer with respect to each document, and
fiments, using a wide variety of approaches. Some of nally the softmax layer. In this model, we segmented
the document into meaningful sentences and then fed
Woodstock’22: Symposium on the irreproducible science, June 07–11, these sentences into BERT model along with their
docu⋆20Y2o2u, Wcaonodusstoeckth,iNsYdocument as the template for preparing your ment ID. The second model has the same idea of dividing
publication. We recommend using the latest version of the ceurart the document into sentences, but instead we
hypothestyle. size that a majority of semantically important
informa* Corresponding author. tion is concentrated within specific sentences inside of a
† These authors contributed equally. longer text, making it unnecessary to check for
connec$ mualqurishi@elm.sa (M. AL-Qurishi); rsouissi@elm.sa tions between all words in a document. Instead, we used
(R. Souissi) BERT-based similarity match algorithm that can
recog(R. 0S0o0u0i-s0s0i)02-7594-7325 (M. AL-Qurishi); 0000-0002-7594-7325 nize high-relevance sentences and pass them as input
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License to the BERT-base model that can complete the desired
CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org)
classification task. Both of those models are based on attention mechanism only in a limited role, thus avoiding
BERT architecture, and require supervised training for the exponential growth of complexity [
In addition, we have fine-tuned two well-known lan- velop specialized model for long document classification.
guage models for long documents classification task, Works with a narrower scope tend to stay closer to the
which are the Longformer [
In terms of methodological choices, practically all Evaluation of the proposed changes to established
alworks from this group acknowledge the unmatched gorithms is crucially important, and all of the reviewed
power of the attention mechanism for analyzing seman- works include some form of empirical confirmation of
tic relationships, and incorporate it in some way into their premises. While the numbers seemingly validate
the proposed architecture. There is a division between that the proposed solutions achieve state-of-the-art
reworks that mostly (or completely) embrace an existing sults under the best possible conditions, those findings
architecture and perform only minor operations such as are self-reported and may often be too optimistic. All
ifne-tuning or knowledge transfer in order to reduce the of the papers are interested in document classification
computational demands [
verge from self-assessed results [
When it comes to practical use of the proposed so- on the domain of research which is long length Arabic lutions, there is a general lack of field data and even documents. The datasets are vastly diferent in terms of discussions of use cases are rare. This is understandable size and diversity of classes. considering the main focus is on discovering more eficient methods, but without real world testing it’s dificult 3.1. Mawdoo3 Dataset to predict whether any of the solutions can deliver similar results to their reported findings. Some works may be The first dataset was scraped from Mawdoo3 which is the directed as specific niches such as legal or medical, but largest Arabic content website3. The number of classes even in this case little attention is paid to practicalities as- from mawdoo3 is 22 class and each category contains sociated with real world application. This weakness may between 700 to 12K articles. We have selected almost one reflect the current state of the field, which is highly exper- thousand of long articles from each category as presented imental and mostly built on data collected in a controlled in Figure 2. environment.
loaded them from diferent sources [
4. Models ument classification task. Our proposed model consist of
multiple layers as shown in figure 1; namely, sentence
segIn this section we introduce two BERT-based language mentation layer, BERT layer, a linear classification layer,
models. Both of those models are based on BERT archi- then the sentence grouping layer with respect to each
tecture, and require supervised training for best perfor- document, and finally the softmax layer. The first layer
is to make a segmentation of sentences from the long
long texts, which are explained in [
We train the model on all the sentences and each sen- attention pattern with specified locations that need to its meaning or break. The second layer is the BERT tokenizer followed by the embedding representation layer.
tence is considered as document. The training outputs are the classification probability for each class as well as the sentence ID and orginal document ID. We make a grouping of text sentences with the probabilities of each category in each sentence, and in the end we aggregate all sentence in the category with the highest probability with respect to the document ID.
plexity of the self-attention matrix. This can be done by making the matrix sparser through the introduction of be prioritized. By using a sliding window with a fixed length, the model doesn’t enter exponential progression and instead scales linearly with input sequence length.
model’s ability to be trained for specific tasks, which
required the addition of global attention to the model.
LinThis model has the same idea of dividing the document
ear projections are used to calculate the attention scores,
into sentences, but instead we hypothesize that a
majorand in this work an extra set of projections related to
ity of semantically important information is concentrated
global attention are used to make training more reliable.
within specific sentences inside of a longer text, making it
The resulting linguistic model has an impressive capacity
unnecessary to check for connections between all words
for contextual analysis, but expends far less
computain a document. Instead, we used BERT-based similarity
tional resources when used with long-form documents
match algorithm that can recognize high-relevance sen- than traditional BERT and other Transformer
architectences and pass them as input to the BERT model that
tures. Nonetheless, Longformer was trained for
autorecan complete the desired classification task. The
highgressive modeling with left-to-right word sequence and
relevance sentences were selected by applying a maximal
train it with Arabic needed some preprocessing. We
conmarginal relevance (MMR) [
document classification task. = ∈ [ 1(, ) − (1 − )
∈
max 2(, )] (1)
Where is the sentence vector and is the document
vector related to . is a subset of documents in our
dataset we already selected and is a constant in range
of [
important research works in the literature for processing
by cosine, euclidean, Jacard and any other distance simi- recurrent layer before a classification decision is made in
ways to extend the usefulness of the BERT linguistic model to text samples longer than a few hundred words.
procedure and separate the input into smaller chunks. After those chunks are processed by the base BERT model, they are passed through another Transformer or a single the softmax layer. Those variations were named RoBERT (Recurrence over BERT) and ToBERT (Transformer over BERT), collectively described as Hierarchical Transformers because they maintain the hierarchical structure of representations both on the level of extracted segments and the whole document. Those models were found to converge very quickly when trained on a narrowly focused dataset and to perform better than the original BERT with long text sequences. Suitability of those derivative models was examined for diferent tasks, including topic identification and satisfaction prediction during a customer call, which are possible real world applications. Unlike the Longformer, with RoBERT the ifne-tuning process was straightforward because it was a BERT-Based model.
accuracy on classification task performed over long text sequences and computational simplicity. Therefor we tried to utilize the base version of BERT which have less memory size of 500MB and faster prediction process where the length of the embedding is 768. We used Google Colab pro to train and fine-tune our models. In terms of accuracy, we use standard metrics to track all of those qualities for the tested models. Macro F1 score is used as a general measure of accurate prediction on all comparisons , as it provides a basis for comparison of results between studies.
Several hyperparameters have been setup to fine-tune the experimented models. Our proposed classification solutions were tested using two collection of documents mentioned in Sec. 3 where 80% of the dataset was used to train the model and 10% as a validation set and 10% utilized for conducting the tests. Table 1 shows the general parameters used in the training and fine-tuning processes. All models were empirically evaluated on long document classification task. We compared our proposed models with Longformer as well as with the RoBERT on Mawdoo3 dataset. The results were very close between the two proposed solutions and the Longformer, with a very slight superiority to the language model based on extracting key sentences using MMR method with macro F1 score equal to 83%. While Robert performed very poorly on Mawdoo3 dataset with macro F1 score of 21%. The overall results of all models in the long document classification task are explained in Table 2. We can say that this results support our hypothesis of identify the most relevant parts of the text. The resulting solution retains the ability to capture relationships between distant tokens, but doesn’t have to actively back-propagate all of them and instead focuses only on key sentences. Because of this, the model avoids geometric progression of complexity and continues to be eficient with much longer texts than the original BERT is able to. It is worth noting that we have pre-processed and removed the information at the beginning of each article in this dataset because that the parts of the document containing easily identifiable indicators of the class. 6.2. Arabic News Dataset number of epochs 5 maximum sequence length aggregation and similarity models 128 maximum sequence length longFormer model 1024 maximum sequence length truncation , RoBERT models 1024 mawdoo3 data number of training steps: 107466 adam epsilon 1e-8 train batch size 64 valid batch size 128 epochs 20 learning rate 5e-5 warmup ratio 0.1 max grad norm 1.0 accumulation steps 1
Table 1 with the Arabic news dataset. All models performed very Hyperparameters used in the training and fine-tuning pro- well, and in this experiment, the first model outperformed cesses the rest with macro F1 score of 98.4% which revealed that Parameter Name Value additional modification can have a positive impact on model performance, but it’s important which dataset is used. It was discovered that classifying each sentence is better than classifying the whole sequence, which could even increase performance when working with short sentences. However, both Longformer and our second model with MMR are still performing very well with macro F1 score of 96% and 96.2%, respectfully. Whereas RoBERT model has macro F1 score of 74.4%. The overall results of all models in the long document classification task are described in Table 3.
The evaluation was conducted using standardized hyper parameters such as batch size and sequence length and others as shown in Table 1, with two diferent datasets suitable for Arabic long document classification task as described in Sec 3. We will try to report the results and analyze them according to each data set separately.
Unmatched flexibility of BERT is one of the main reasons for its rapid acceptance as state-of-the-art language model. With additional algorithm and some modifications and fine-tuning, the model can be adjusted for certain topics or tasks and its accuracy pushed to even higher level. This work explores this possibility in detail, taking long text classification as the target task and searching for the best parameters for this type of usage. In particular, diferent possibilities for supervised pre-training and fine-tuning were examined on two diferent datasets. Through detailed experimentation, we were able to identify the most optimal procedures that enable BERT to be more accurate with our particular downstream task. While the value of the proposed training and tuning actions was confirmed only for text classification, it stands to reason that analogous procedures could prove to be useful for other linguistic tasks as well. Finally, we want to denote that we did not explore all hyperparameters which can be a future work to have along with trying another language models such as Roberta and Electra.