Pretrained transformers for Chinese show remarkable performances on various natural language processing tasks. However, these models are purely data-driven and fail to incorporate explicit semantic knowledge, like HowNet. In this paper, we propose H-BERT, which enhances the semantic representations of Chinese BERT by incorporating sememe knowledge from HowNet in the pretraining stage via multi-head attention. Our experiments demonstrate that HBERT can significantly outperform the vanilla BERT on the downstream tasks. Ablation study compares different settings of H-BERT and shows that and case study also shows that knowledge injection is required at both the pretraining and fine-tuning stage.3
Since the rise of BERT, pretrained language models (PLMs) have dominated state of
the art (SOTA) for a comprehensive list of natural language tasks [
However, the literature falls short on three aspects. First, many PLMs that inject
knowledge from pretraining suffer from catastrophic forgetting and thus can only perform
well on entity-related tasks such as named entity recognition and relation classification,
but performs poorly on sentence-level tasks like GLUE [
Attribution 4.0 International (CC BY 4.0).
In this work, we adopt the core data of HowNet [
This work proposes HowNet BERT (H-BERT), a transformer-based model that uses a simple multi-head attention module to incorporate How-Net knowledge. Our H-BERT model is depicted in Figure 1. A sentence is encoded via two modules. First, it is tokenized and embedded via the token embedding layer. Second, we recognize the words which are included in How-Net and obtain their sememes. Tokens in the same word will have the same sememes. We can treat the sememes of a word as subword features. Sememes are treated as the minimum units, and the sememes of the tokens will be embedded via the sememe embedding layer. Knowledge from How-Net is injected via a multi-head attention layer from the token representation to the tokens’ sememe representation (denoted as attn-to-sememes). Here, this attention module can be implemented right after the token embedding layer or after obtaining the sequential output of the token encoder.
We conduct experiments on sentence classification (CLS) and sentence pair classification (NLI), which are whole-sentence level tasks, and named entity recognition (NER), which is an entity-level task. Tasks from different domains are selected. Experimental results show that our model consistently outperforms the vanilla ALBERT and K-BERT on a series of tasks, indicating that our model can handle both whole sentence-level and entity-level tasks equally well. Moreover, we find that pretraining with attn-to-sememe but exclude this module during fine-tuning also improves the performance of the vanilla PLM.
Now we discuss how to incorporate the HowNet knowledge. First, in a sentence, we
match all the words (not overlapping) included in the sememe via FlashText [
For pretrained H-BERT, we can fine-tune it in two approaches: (a) discarding the attn-to-sememe module and fine-tuning H-BERT in the same way as BERT; (b) taking advantage of HowNet during fine-tuning, that is, extracting the sememe informaitons from the sentence, and using the attn-to-sememe module to inject sememe information to H-BERT’s encoder.
3.1
Pretraining is done on the Chinese Wikipedia corpus. We use the vocabulary of Google
Chinese Bert [
During fine-tuning, we mainly follow the hyper-params of [
We experiment across a diverse set of 6 benchmark NLP tasks and demonstrate the
effectiveness of our model. For text classification, we select ChnSentiCorp (chn)4. For
NLI, we select LCQMC [
The experimental results are reported in Table 1. The main takeaways are: – H-BERT v0 consistently outperforms ALBERT base and K-BERT with comparable parameters, demonstrating H-BERT’s effectiveness. – H-BERT v1 outperforms ALBERT large, demonstrating that our method can also work for large pretrained models. – H-BERT v3 performs worse than H-BERT v0, but it is better than ALBERT base, showing that attn-to-sememe helps improve the generalization ability of pretrained models. In addition, adopting attn-to-sememe during fine-tuning is beneficial for downstream tasks. – Comparing H-BERT v2 with H-BERT v0, we can see that it is better to apply attn-to-sememes at the embedding layer of the encoder model. 4 https://github.com/pengming617/bert classification 5 https://embedding.github.io/evaluation/#extrinsic 6 https://biendata.com/competition/CCKS2017 2/
This article proposes to enhance the Chinese pre-trained language models with simple attention to sememes module. Experiments on 6 benchmark datasets shows that: From the architecture point of view, attn-to-sememes should be applied at the embedding layer; (2) attn-to-sememes are required at both pretraining and fine-tuning stages for better downstream performances. Our model can beat the vanilla ALBERT significantly across 6 datasets with roughly the same amount of parameters, showing that our model can effectively inject knowledge into PLMs.