The existing patent text classification models perform well in many commonly-used languages but show suboptimal performance with corpora from the Association of Southeast Asian Nations (ASEAN). This paper proposes a multi-task modeling approach to train a multilingual patent classification model, which applies to patent texts of multiple languages, including commonly-used and low-resource languages. The model learns joint text embeddings by training a multilingual patent classifier and a cross-lingual text-pair similarity discriminator using monolingual classification labeled data and aligned bilingual corpora at the sentence or paragraph level. The method shows competitive performance, achieving up to 5% improvement in accuracy over mBERT on cross-lingual patent classification tasks in Thai and English.
Automatic classification of patent documents is crucial for intellectual property protection, patent management, and information retrieval. Designing an accurate and automated patent classification model can provide significant support for patent inventors and examiners. While existing text classification models perform well with patents in commonly-used languages such as English and Chinese, they struggle with patents written in low-resource languages from the ASEAN region. This challenge is often due to the lack of labeled data, a common issue across various NLP tasks, including text classification, named entity recognition (NER), and sentiment classification.
Recently, multilingual pre-trained models have been
widely used to fine-tune and train text classification
models. Experiments have proven that fine-tuned multilingual
pre-trained models perform well in situations with limited
labeled data. One of these models is multilingual BERT, also
called mBERT, proposed by Devlin et al.[
MultiFit by Eisenschlos et al.[
In addition to selecting the most advanced model at the
pre-training stage, some scholars have also enhanced model
performance by using bilingual dictionaries or multilingual
parallel corpora [
In this work, we first adopt a multilingual pre-trained model to enhance text representation. Then, we jointly train a multilingual text classifier and a cross-lingual text similarity discriminator using a multi-task learning approach. This allows us to learn multilingual patent text embeddings effectively. During this process, the patent classification task and the text similarity discrimination task interact through shared embedding layers, continuously adjusting their respective parameters. The text similarity discrimination task, as a sub-task within the multi-task framework, influences the main patent classification task through its gradient adjustments.
The motivation behind this method is to learn highquality multilingual text embeddings, including sentencelevel and paragraph-level, in low-resource scenarios. The contributions of this research are twofold. Firstly, we construct a patent corpus for low-resource languages of ASEAN countries and a parallel patent corpus combining common and low-resource languages. Secondly, we combine stateof-the-art multilingual pre-trained models with multi-task frameworks for training. Specifically, the shared word embedding layer enables both the text classification task and the text similarity task to update parameters simultaneously during training. By jointly training these models, the framework can leverage both text classification data and parallel text pairs to learn multilingual text embeddings and achieve better patent classification results.
In this work, multi-task learning is used to jointly train patent text representations. The main task is patent classification, while the secondary task is cross-lingual text similarity, which transfers additional linguistic semantic information to the main task.
As the main task, patent classification involves dividing patents into multiple categories. During training, the objective is to minimize classification error. We use features extracted from a multilingual pre-trained model as the text representation, and a two-layer feed-forward network with a softmax layer as the classifier.
Given a multi-class classification scenario, the softmax layer normalizes output values to ensure that each class’s probability is non-negative and their sum is one. The calculation formula for one label’s probability is shown in Eq.(1). Let denote the probability that the patent text belongs to the -th class, and denote the number of labels. Here, is a natural constant, and represents the input vector’s power value.
=
∑︀=− 01 , ∈ {0, 1, ..., − 1} (1) The following equation Eq.(2) is about the loss function, in 1–7 (2) (3) which denotes the loss of patent text classification and represents the real label of the example. If the model predicts correctly, is 1; otherwise, it is 0.
= − − 1 ∑︁ · () =0
The cross-lingual text similarity task actually learns more language-related semantic information by increasing the correspondence between diferent languages as auxiliary signals. The aim of this task is to map diferent languages into the same semantic space, which means finding the mapping function for two languages. Assuming that one language is Language and the other is Language , our purpose is to determine the parameters that satisfy Eq.(3).
() =
We regard function as a series of operations, such as transforming and rotating the semantic space, as shown in Fig.1. However, consists of a complex neural network, which is dificult to express as a specific mathematical formula.
Text similarity discrimination is essentially a binary classification task. The final prediction results of the model are either similar or dissimilar. The calculation method of the loss is shown in Eq.(4), where represents the loss of the discriminator and represents the sample label. If two sentences or paragraphs are similar, the label is 1; otherwise, the label is 0. Similarly, refers to the probability that belongs to a positive sample (labeled as 1).
sim = − [ · log() + (1 − ) · log(1 − )] (4) Finally, the total loss of multi-task cross-lingual patent classification is the sum of two loss functions, as shown in Eq.(5).
total = clsf + sim (5)
The model we proposed consists of an embedding layer, a joint feature extractor, a patent label classifier, and a text similarity discriminator, as shown in Fig.2.
The purpose of the embedding layer is to convert text into vectors, which are numerical representations that can be processed by machine learning models. In this work, we chose English, a widely-used language, and Thai, one of the ASEAN languages, to demonstrate the model’s capability to handle multilingual data.Despite the diferent inputs, the text representation model mBERT shares a series of parameters.
To optimize for two distinct tasks—patent label classification and cross-lingual text similarity discrimination—we modify the form of inputs according to their specific requirements. For patent label classification, the input consists of the entire sentence or paragraph in either English or Thai. This task aims to classify the patent text into predefined categories based on its content.
In contrast, the input for the similarity discrimination task involves paired sentences or paragraphs in English and Thai, separated by the delimiter [SEP]. This setup enables the model to learn the relationship and semantic similarity between texts in diferent languages. Despite the diferent input structures, the text representation model mBERT shares a series of parameters, ensuring that the multilingual embeddings are consistent and comparable across both tasks.
Furthermore, by leveraging mBERT’s pre-trained capabilities, we significantly reduce the amount of data required for efective training. This is particularly advantageous in scenarios involving low-resource languages, where labeled data is scarce. The embedding layer thus plays a critical role in bridging linguistic gaps and enhancing the model’s applicability to diverse multilingual tasks.
In our conception, we aim to learn relationships between diferent languages and incorporate them into text classification as complementary knowledge. If the feature extractor can extract features that satisfy both text classification and text similarity tasks, the model may efectively map text in diferent languages to the same semantic space. This unified semantic space allows for improved cross-lingual understanding and classification accuracy.
To achieve this, we designed a joint feature extractor with the following combinations: mBERT+FC, mBERT+CNN, mBERT+BiLSTM, and mBERT+Attention.
(1) mBERT+FC: This combination uses a fully connected layer to reduce the dimensionality of the embeddings from mBERT. It provides a straightforward approach to integrating and processing the extracted features.
(2) mBERT+CNN: TextCNN captures local patterns and n-grams within the text. By applying convolutional filters, it detects key phrases and local dependencies, which are crucial for understanding the contextual meaning in sentences and paragraphs.
(3) mBERT+BiLSTM: BiLSTM captures long-range dependencies and sequential patterns by processing the text in both forward and backward directions. This approach is essential for understanding the nuances and intricate relationships within complex patent texts.
(4) mBERT+Attention: Attention mechanisms allow the model to focus on the most relevant parts of the text, dynamically weighing the importance of diferent words and phrases. This selective focus enhances the model’s ability to highlight critical information, improving the accuracy of both classification and similarity tasks.
This joint training process ensures that the embeddings are optimized for multiple tasks simultaneously, improving the model’s overall performance. The joint feature extractor enables the model to understand and classify multilingual patent texts more efectively, particularly in low-resource scenarios where annotated data is limited.
Generally, the similarity between two sentences or
paragraphs is a score representing the relevance and likelihood
between them. Despite using the same words with identical
writing, text similarity should encompass both syntactic
and semantic information. Syntactic similarity is based on
string matching, while semantic similarity refers to the
similar meaning between words, sentences, paragraphs, and
even documents. Since the calculation of text similarity
in this paper involves diferent languages, more attention
should be given to the degree of semantic similarity across
languages. When calculating semantic similarity, it is
natural to consider a common semantic vector space [
By leveraging the shared semantic space provided by the multilingual pre-trained model, the cross-lingual text similarity discriminator efectively measures the semantic relevance between texts in diferent languages. This capability is essential for enhancing the performance of multilingual applications, ensuring that the model can accurately identify and utilize semantic similarities across languages. The output of the discriminator is the probability of text similarity, represented by a binary classification: 1 for similar and 0 for dissimilar.
The patent label classifier corresponds to the patent classification task, which is the main goal of our work. This classifier is designed to categorize patent texts into predeifned classes based on their content, helping in the eficient organization and retrieval of patent information.
The classifier operates by first receiving the vector representations from the joint feature extractor. These vectors encapsulate the semantic information derived from the original patent texts. To make these high-dimensional vectors 1–7 manageable and suitable for classification, the classifier reduces their dimensionality through a fully connected layer. This reduction helps in simplifying the complexity of the data, making the subsequent classification process more eficient and efective.
An important feature of our patent label classifier is its applicability to multilingual data. Since both English and Thai texts are converted into vectors using the same embedding process, the classifier can seamlessly handle patents in both languages.
In this study, we constructed two main datasets: a parallel corpus for training the similarity computation model and a classification corpus for the classification task.
The cross-lingual parallel corpus is a multilingual dataset composed of original texts and their corresponding target texts by translation. Given the scarcity of available corpora in the field of patents, we independently built a patent parallel corpus for ASEAN languages. Specifically, we selected 10,000 English patents from the Google BigQuery dataset and translated their titles and abstracts of claims into Thai using the Google Translate API. This translation process created a bilingual parallel corpus, enabling the model to learn mapping relationships between texts from diferent languages. The parallel corpus includes both sentence-level and paragraph-level data, with labels indicating whether the text pairs have translation correspondence (1) or not (0), with the latter obtained through shufling.
For the classification task, we accessed patent data in
both English and Thai from the Google BigQuery dataset by
writing SQL queries to randomly download patents.
Considering that Lee and Hsiang [
To ensure the dataset was balanced, we used random sampling to adjust the number of patents in each language.
After processing, the total amount of data for each class is very close, with approximately 8,000 samples for each section, although slight variations occur due to the random sampling process. The dataset is then divided into a training set, validation set, and test set in the ratio of 8:1:1, resulting in 64,000 training samples, 8,000 validation samples, and 8,000 test samples. See details in Table 1.
By constructing these two datasets, we aim to facilitate both the cross-lingual text similarity task and the patent classification task, ensuring that our multi-task learning framework can leverage the rich semantic information from multilingual texts. *Multi-Task(Par-level): Cross-lingual text similarity task with paragraph-level parallel corpus *Multi-Task(Sen-level): Cross-lingual text similarity task with sentence-level parallel corpus
In the experiments, we set the following parameters. The number of iterations is 20 rounds, and the learning rate is uniformly set to 1e-4. Since the two tasks need to use shared text embeddings, the parameters of the two pretrained models are shared, which is equivalent to using the same mBERT for the vectorized representation of text. Additionally, since BERT is a 12-layer transformer structure, we freeze the parameters of the first 9 layers of mBERT and only allow the parameters in the last three layers to be updated, with the aim of saving time.
For training, we alternated between text classification and text similarity tasks in each batch. Specifically, one batch would be used to update the model based on the text classification task, and the next batch would be used to update the model based on the text similarity task. This alternating training approach helps in jointly optimizing the model for both tasks.
The input dimension for mBERT is 768, which corresponds to the size of the embeddings produced by the model. After passing through the feature extractor (which can be a fully connected layer, CNN, BiLSTM, or Attention mechanism), the representations are reduced to 128 dimensions before being fed into the classifier or similarity discriminator.
We adopt the commonly used classification accuracy (Acc) and macro-F1 value as evaluation metrics to measure our proposed model. To comprehensively analyze the performance of our multi-task model, we choose fully connected (FC) layers, CNN, BiLSTM, and Attention as feature extraction models for comparison. In addition, we set corresponding single-task learning models as baseline methods. The evaluation was carried out on both the patent classification task and the text similarity task to ensure a holistic assessment of the model’s performance.
With the settings above, the results of the models are shown in Table 2. Compared to the baseline experiments, our proposed multi-task learning model shows better performance in patent classification overall. The results indicate that multi-task learning can achieve performance improvements through related auxiliary tasks during the training process. After multi-task learning, the performance gap among models is reduced to less than 3.47% in accuracy and less than 3.48% in macro-F1. This suggests that learning sentence pair similarity classification is the most important factor afecting the performance of multi-task learning models. The mBERT+Attention combination achieves the best results in both single-task and multi-task learning. Attention uses an auto-focusing mechanism to capture the most important semantic information in sentences, enhancing text representation and classification performance. Results on diferent datasets in multi-task learning show that sentencelevel learning performs better than paragraph-level learning, except in the mBERT+BiLSTM model. We suppose that sentence-level learning can focus more on local information, which is representative of the text topic. Meanwhile, CNN and Attention have advantages in capturing local information to an extent, resulting in good outcomes in our multi-task learning. BiLSTM is more efective at obtaining semantic information from long texts and performs better in paragraph-level learning, but the introduction of BiLSTM has not brought significant improvement in multi-task learning overall.
The results show that joint multi-task learning of patent text classification and cross-lingual text similarity based on a multilingual pre-trained model is efective, improving accuracy by 5.39% and macro-F1 by 5.50% over the baseline, with the highest result reaching 76.54% in accuracy and 76.03% in macro-F1. This demonstrates that joint multitasking can fully utilize the mapping relationship between two languages in parallel text pairs and incorporate this information into the main task of patent classification through the sub-task of discriminating text similarity. Theoretically, the overall scheme designed in this study can be transferred to any low-resource language, and its application can be extended to multiple tasks such as sentiment classification and natural language reasoning, not only text classification. In addition to the portability of tasks, the domain is also transferable; that is, in addition to patents, it can also be applied to fields such as medicine and social media.
In this paper, we demonstrate that cross-lingual text discrimination can supplement semantic information for multilingual text classification tasks. However, this approach still requires the use of parallel corpora or translation text pairs. Considering the convenience of data acquisition, future research needs to focus on reducing dependence on large data volumes and enhancing the model’s ability to learn from limited data. Currently, technologies such as adversarial transfer and domain adaptation can address this expectation. These technologies eliminate the reliance on parallel data, enabling the training of models suitable for multiple languages in an unsupervised manner. However, these technologies are still in development, particularly in addressing the issue of distributional shift across languages. Our future work will also explore more advanced methods for intelligent multilingual information processing.
The authors warmly thank reviewers for their valuable suggestions. This research was partly supported by KeyArea Research and Development Program of Guangdong Province (NO.2021B0101420004). 1–7