Named Entity Recognition (NER) [1] and Relation Extraction (RE) [2] are well-known tasks in the field of information extraction research. Previous research has focused on diverse domain datasets, such as ACE05 1 from Newswire and online forums, and SciERC [3] from scientific papers. Both NER and RE models are based on either a general domain language model [4] or a domain-specific language model [5], [6], [7], depending on the dataset. And most of the works employ either a pipeline approach or a joint approach. A pipeline approach is training one model to extract entities and another model to classify relations between them. A joint approach is training the model for both tasks simultaneously.

ChEMU (Cheminformatics Elsevier Melbourne University) 2022 introduces 5 tasks to extract information from the snippet of chemical patents to support the drug discovery process. Among these tasks, we focus on NER task 1a and EE (event extraction) task 1b. Task 1a aims to extract chemical entities from the snippet. Task 1b aims to extract trigger words and relations between trigger words and entities from the snippet. Task 1b includes both NER and RE tasks. The extraction of trigger words is the NER task and the relation between trigger word and entity is the RE task. We develop context-aware NER and RE models based on the domain-specific language model and achieve state-of-the-art performance in ChEMU 2022, the exact match f1 score of task 1a is 96.33, and task 1b is 92.82. 2 In this paper, we explain our contributions to improving the performance of ChEMU 2022 : (1) domain-specific language model, (2) best performing NER and RE models, (3) context-aware model with input consisting of multiple sentences, (4) post-processing to the model prediction, (5) cross-validation and ensemble. Finally, we experiment and analyze our contributions in section 4.

For the domain-specific language model, we post-train the Bio-linkBert [7] model with various chemical corpora. We then select the best performing model from domain-specific benchmark datasets consisting of BLURB (Biomedical Language Understanding & Reasoning Benchmark) [8] and ChEMU 2020 [9]. Among the pipeline approach and the joint approach, we choose the pipeline approach because PURE [10] reports it gets higher performance than the joint approach. For the NER model, we experiment with two popular approaches, the sequence tagging approach [4], [11] and the span-based approach [10], [12]. And finally, we choose the sequence tagging approach that shows higher performance in ChEMU 2022 task 1a. The NER model is trained to predict both entities in task 1a and trigger words in task 1b. For the RE model, we train the model to classify the relation types or no relation between every pair of trigger words and entities in the snippet. In task 1b, the RE model predicts the relation between entities and trigger words predicted by the NER model. Furthermore, we train both models using inputs that contain multiple sentences rather than a single sentence so that the model can utilize contextual information. For the ensemble, we train the best-performing model with 10-fold cross-validation and then predict the results with soft-voting. Finally, we apply rule-based post-processing to the prediction results.

Named Entity Recognition (NER) [1] and Relation Extraction (RE) [2] are well-known tasks in the field of information extraction research. These tasks have lots of applications in various domains such as news, social media, biomedical and chemical domains. There are two areas of study. The first is to improve the language model, and the second is to improve the NER and RE models based on that model.

Recently the pre-trained language model such as BERT [4] and Roberta [13] have improved all NLP task performance. To improve the language model for NER and RE tasks, LUKE [14] and KeBioLM [15] use additional information such as named entity labels to pre-train the model. However, it is not easy to prepare a lot of label data in the chemical domain. Others approach such as BioBert [5], PubmedBert [6] and Bio-linkBert [7] are pre-trained using domain-specific corpora. Similarly, we train a domain-specific language model with a chemical domain corpus and ultimately improve the performance of the ChEMU 2022 task.

For NER tasks, there are the sequence tagging approach such as BERT [4] and the span-based approach such as PURE [10] and PL-Marker [12]. For the sequence tagging approach, BERT uses a BIO scheme to encode each token into a tag and trains a model to classify each token into its tag. For the span-based approach, PURE and PL-Marker generate entity span candidates whose length is shorter than the maximum span length and then train a model to classify them as entity type or no-entity. We experiment with both approaches and then select the model with the best performance in the ChEMU 2022 task.

For NER and RE tasks, there are the pipeline approach [10], [12] and joint approach [16]. In a pipeline approach, the NER model predicts entities and then the RE model predicts the relation between them. On the other hand, in the joint approach, a single model learns the NER and RE tasks simultaneously. Because PURE [10] reports the pipeline approach performs better than the joint approach, we adopt the pipeline approach.

We use transformer encoder-based models and these models are published in huggingface 3 , such as BERT, Roberta, BioBert, PubmedBert, Bio-megatron [17], and Bio-linkBert. Most of the publicly available pre-trained language models (PLM) are trained using general domain or biomedical domain knowledge. However, ChEMU 2022 data is composed of text based on chemical patents. Therefore, when fine-tuning publicly available pre-trained models, the gap between the chemical domain and other domains reduces the utilization of pre-trained knowledge. For example, if a word such as chemical compound is split into several tokens because the tokenizer is not trained with chemical domain texts, the language model may not capture its original meaning. And the understanding of the context is lowered due to a homonym problem in the different domains.

To overcome this problem, the one way is to learn a model from scratch using only the chemical domain texts, as PubmedBert and Bio-linkBert did in the biomedical domain, but it was difficult due to the lack of time. We try to solve the problems mentioned above by applying domain transfer to the set using the post-training method. In this paper, we post-train Bio-linkBert with various chemical corpora and then select the best performing model from domain-specific benchmark datasets consisting of BLURB (Biomedical Language Understanding & Reasoning Benchmark) and ChEMU 2020. Since we believe that the Pubmed dataset used by the pre-trained Bio-linkBERT has some chemical information, we want to put additional data information into the model without losing the already learned information. The same methodology as Mix-Review [18], a rehearsal-based continual learning approach, is applied for domain transfer.

Using the domain-specific language model mentioned above, we experiment with the sequence tagging approach and the span-based approach. And then we compare which is better for the entity and trigger word recognition of ChEMU 2022 tasks 1a and 1b. In the case of the sequence tagging approach, a bio scheme is used, each token is encoded with the BEGIN and IN tag of a specific entity or OUT tag. And then the model is trained to classify each token into its corresponding tag. To classify tags, the output representation of each token is simply fed into a linear layer. We also experiment with CRF or Bi-LSTM+CRF layers in appendix A.1, but there is no performance improvement. In the case of the span-based approach, we consider token sequences shorter than the maximum span length 4 as entity span candidates and then train the model to classify them to corresponding entity type or no entity. However, there are chemical entities much longer than the maximum sequence length in ChEMU 2022 task 1a. Therefore, we use several heuristic approaches to add long entity span candidates. One simple way we used is to add a space-split sequence of tokens. A span representation for classifying an entity is a concatenation of the first token, last token representation and width embedding to capture entity length information. 5 After experiments, we finally decide to go with the sequence tagging approach that shows higher performance.

According to the error analysis part of the ChEMU2020 [9], in some cases, contextual information from other sentences is necessary to extract trigger words. So we train the context-aware model with input including multiple sentences rather than a single sentence. It goes through several processing steps to generate the input data. First, we split the snippet into sentences with spacy 6 library. Second, by sliding the sentences from left to right, we generate inputs that contain as many sentences as possible without exceeding the maximum sequence length of the model. and trigger span, special tokens are inserted before and after the entity and trigger span. Each special token indicates the type of entity or trigger and whether it is inserted before or after the span. After input is fed to the model, output representations of special tokens before entity and before trigger are concatenated. And then, this concatenated representation is fed to the linear layer to classify relations.

Ensemble methods combine predictions from multiple models to improve performance. We train NER and RE models using 10-fold cross-validation on the merged training and development datasets. We apply the soft voting ensemble method to output results from 10 models.

We apply three post-processing methods to correct the results mispredicted by NER & RE models. The first method is to correct the entity misclassification of STARTING_MATERIAL as REAGENT_CATALYST. According to ChEMU 2020 [9], misclassifying STARTING_MATERIAL as REAGENT_CATALYST is one of the most common errors in the NER task. We design the rules according to the definition of the entities or trigger words. By definition, the difference between STARTING_MATERIAL and REAGENT_CATALYST is that STARTING_MATERIAL is consumed during the chemical reaction, while REAGENT_CATALYST is not consumed and only increases the reaction rate. In other words, unlike REACTION_CATALYST, the molecular structure of STARTING_MATERIAL is similar to REACTION_PRODUCT. Therefore, we measure similarity between STARTING_MATERIAL or REAGENT_CATALYST and REACTION_PRODUCT in the snippet and then correct the entity type if it appears to be misclassified. 7The second method is to correct mispredicted trigger word or entity spans. Sometimes the model predicts different spans for the same word in different sentences. For example, the model predicts "taken up" as a trigger word span, but sometimes only "taken" without "up" in other sentences. Sometimes this can happen because the labels for the same word are different from each other in the dataset. We apply post-processing that modifies all "taken" to "taken up". In the same manner, several spans of the entities are post-processed.

The third method is to correct the relation misclassification of WORKUP as REAC-TION_STEP. If the RE model predicts that a trigger word is related to REACTION_PRODUCT, YIELD_PERCENT, YIELD_OTHER at one time, the trigger word should be REACTION_STEP rather than WORKUP. The rule should capture the sentence at the end of the snippet that describes the material synthesis in which the product is finally formed. Thus, we force the trigger word WORKUP to be replaced by REACTION_STEP in this case.

We evaluate our domain-specific language model with the BLURB benchmark and ChEMU 2020 dataset. We then evaluate NER and RE models with ChEMU 2022 task 1a and 1b datasets. Since ). Among them, we focus on tasks 1a and 1b. Task 1a is to extract chemical entities and task 1b is to extract both the trigger words and the relations between trigger words and chemical entities. The dataset of task 1b is a superset of task 1a. Table 1 shows the overall statistics of the train and development datasets in ChEMU 2022 task 1b.

The BLURB benchmark dataset consists of six tasks as follows : named entity recognition, PICO (patient population, interventions, comparator, and outcomes), relation extraction, sentence similarity, document classification, and question answer. Among them, we use only NER and RE datasets, which are the target tasks of ChEMU 2022. Table 2 summarizes the dataset we use.

We post-train Bio-LinkBert with different corpus combinations and then select the bestperforming model. Based on the architecture and weight of Bio-LinkBert large 8 , we post-train Bio-LinkBert on a task of masked language modeling [13]. We experiment with three corpora:

(1) Google patent: 23 GB of chemical domain patents we crawled using chemical keywords, (2) Journal: 22 GB of chemical journal abstracts and body text (3) Pubmed abstract 9 : used by training BioBert [5], 38 GB of biomedical domain data. For the Pubmed abstract corpus, we use 12 GB, which is 30% of the total data. This is because Bio-LinkBert has already been trained with Pubmed abstract and the post-training aims to learn new information without losing what has been learned. The corpus used for our best-performing model is the combination of Journal and Pubmed abstract. We train our model for 15,000 steps (approx. 2 epochs) with sequence length 512, batch size 2k, weight reduction 0.01, warm-up 3000 steps, and learning rate 5e-5.

The training time is about 13 hours using DeepSpeed10 with 16 Nvidia A100 40GB GPUs.

We train a NER model to predict both entities and trigger words and a RE model to predict relations between them. At inference time, the RE model predicts the relation using the results predicted by the NER model. Although the code for ChEMU 2022 task 1a and task 1b is published, we implement all codes for pre-processing, post-processing, and modeling for NER and RE. As we will discuss in the 4.4 section, we achieve higher performance than the other participants in tasks 1a and 1b even using publicly available domain-specific language models such as For the ensemble, We train a model with 10-fold cross-validation. And these 10 trained models predict entities or relations by soft-voting. Finally, post-processing is applied to the prediction results of the ensemble model.

We post-train Bio-linkBert with various corpora to obtain the domain-specific language model that achieves high performance in ChEMU 2022 tasks 1a and 1b. The performance verification of this model is performed using the CHEMU 2020 dataset and the BLURB dataset.

Table 3 shows the entity and trigger extraction performance of post-trained models in ChEMU2020 task 1a. Table 4 shows the results of the NER and RE performance of the BLURB dataset. In Tables 3 and 4, the post-training with the journal and Pubmed abstract on Bio-linkBert large achieves the highest overall score, so we choose this model as our final model. We train these models with the input data generated from each paragraph. Training the model with input generated at the document level gives a slight performance improvement. However, in ChEMU 2022, the score eventually drops slightly, so it is not used.

Table 5 and Table 6 show the evaluation results of task 1a and 1b in ChEMU2022, respectively. The public and private scores are calculated from 30% and 70% of the test data set, respectively. 11Both exact match and relaxed match require predicted entity or trigger word type to match the label. For span, exact match requires that predicted span exactly matches gold span. However, the relaxed match only requires that the predicted span overlap the gold span. Both metrics use f1 score which is the harmonic mean of the precision and recall. In Table 5, our single model achieves public and private exact match f1 scores improvement of +2.32 and +1.74 compared to Hokkaido University which achieves the highest score among other participants. Also, our final model with ensemble and post-processing achieves +3.13 and +2.68.

In Table 6, the evaluation method of task 1b in ChEMU 2022 is very similar to 1a, except that the relation type must match the label as well. To check whether the performance of the RE model is higher than that of other participants, we predicted the relation using the entity and trigger prediction results of the model12 that achieved the lowest performance among the NER models we submitted. Even in this case, the public and private exact match f1 scores are +2.04 and +1.26 higher than the ChEMU Baseline, which has the highest performance among participants. Therefore, the proposed RE model also affects the performance improvement. The highest score is obtained by training an ensemble RE model using the prediction results of the best performing NER and applying post-processing to the prediction results. In this case, the public and private exact match f1 scores are +4.4 and +2.9 higher than the ChEMU Baseline.

This section explains how PLM and data pre-processing methods affect the performance of ChEMU 2022 task 1a and 1b. Table 7 shows the exact match f1 score of the publicly available PLMs and our language model, which is post-trained on Bio-linkBert large with journal and pubmed abstract data and the best performing model is used by measuring the Blurb and CHEMU20 performance at every 500 steps within 2 epochs. . In task 1a, our language model outperforms all other PLMs. However, although it is not common, the performance of Bio-linkBert large is lower than Bio-linkBert base. Therefore, post-training Bio-linkBert base as we did for Bio-linkBert large may improve the performance. In task 1b, our model outperforms the Bio-linkBert large model, but PubmedBert base gets the highest score. As a result, we use our language model for task 1a and PubmedBert base for task 1b.

Table 8 shows the comparison of the exact match f1 score in ChEMU 2022 according to the pre-processing methods that generates the input data to train the model. A single snippet txt file of ChEMU 2022 data consists of multiple lines. We apply pre-processing methods mentioned in section 3.2 and 3.3 with two different ways. The first is to pre-process the data line by line and the second is to pre-process the entire snippet. If the input is generated only on each line, the model cannot predict the result by referencing the context information given in the other lines. Furthermore, for task 1b, the first deals with relations that occur on a single line, while the second also includes relations that occur in multiple lines. Therefore, the second outperforms the first. Table 9 shows the ablation over three post-processing methods: (1) similarity: to correct the entity misclassification of STARTING_MATERIAL as REAGENT_CATALYST, (2) trigger & entity span: to correct mispredicted trigger word or entity spans. (3) strict relation: to correct the relation misclassification of WORKUP as REACTION_STEP in some cases. In the NER model, the similarity method improves public exact and relaxed f1 by +0.03 and +0.04, respectively. The entity span method improves public exact f1 by +0.03. As a result, the score after applying all post-processing methods shows +0.07 and +0.04 improvement in public exact and relaxed f1, respectively. In EE model public score, the most effective method is trigger span method which improves public exact f1 by +0.33. Following method is strict relation, improves public exact f1 by +0.17 and relaxed f1 by +0.16. This rule drops the private score a bit, but it is still effective in the public score. Similarity improves public exact f1 by +0.04 and relaxed f1 by +0.04 and entity span method improves public exact f1 by +0.04. Finally all of post-processing methods applied EE model shows +0.59, +0.21 in public exact, relaxed f1 score higher than EE base model.

In this paper, we present the domain-specific language model and context-aware NER and RE models for ChEMU 2022. For the best performing domain-specific language model, We post-train Bio-linkBert with various corpora. Based on this language model, we present the NER model using the sequence tagging method and the RE model using the PURE approach. Pre-processing methods where the input contains multiple lines of sentences help the model to be context-aware, which ultimately improves performance. Finally, we train the ensemble model and apply some rules as post-processing.

We achieve state-of-the-art performance on ChEMU 2022 tasks 1a and 1b and analyze contributions to performance. However, there is still room for improvement. First, because our language model has a maximum sequence length of only 512 tokens, the model can not predict entities and relations referencing the entire snippet. Second, some chemical entities consist of too many tokens, which can degrade the performance of the model. These will be our future works to develop improved language models.