The automated extraction of the chemical-reaction information in patents plays an important role in collecting chemical-reaction information in reaction databases for use by synthetic chemists. Chemical patents contain important information about new chemical discoveries because any new chemical compounds are usually published via patents [1]. With the number of patents increasing rapidly, manually collecting the information written in patents not only takes time and cost but also requires expertise in the subject matter of the patents.
Since 2020, the Cheminformatics Elsevier Melbourne University (ChEMU) laboratory has identified several tasks related to information extraction from chemical patents, including expression-level information extraction [2,3] and document-level information [4,5]. For 2022, the ChEMU laboratory is providing five tasks for ChEMU 2022 [6].
In recent years, deep learning has been recognized as a promising approach to information extraction from chemical literature. For example, pre-trained language models such as BioBERT [7] and ChemBERT [8] have shown high performance in information-extraction tasks. Moreover, the best systems in previous ChEMU tasks all employed deep-learning-based approaches, together with a small amount of rule-based post-processing [9,10]. In addition, both of these systems used a pipeline approach for relation extraction tasks [11,12], such as the event extraction and anaphora resolution tasks.
This paper describes our results for the three tasks at ChEMU 2022: Task 1a (named entity recognition, NER), Task 1b (event extraction, EE) and Task 1c (anaphora resolution, AR). We employed hybrid approaches that used deep learning models and a small set of post-processing rules. In addition, we propose post-processing methods for Task 1c that considered the results of Task 1a.
Prediction systems for the three tasks must solve two general tasks: the identification of the spans of entities and the labels (mention detection), together with the identification of relations between the spans (relation classification).
Task 1a, (i.e., NER), is a mention detection task that identifies one of the 10 entity types of labels. The set of labels contains these compounds (STARTING_MATERIAL, REAGENT_CATALYST, REACTION_PRODUCT, SOLVENT, OTHER_COMPOUND), conditions (TIME, TEMPERATURE), yields (YIELD_PERCENT, YIELD_OTHER), and a relation label (EXAMPLE_LABEL).
Task 1b, (i.e., EE), is a task that involves both mention detection and relation extraction. The mention detection task identifies events that have a relationship with entities in Task 1a and identifies relations between the events and the entities. Almost all events involve a label that is either REACTION_STEP or WORK_UP and some events involve both labels. The relation extraction task identifies relations between the events and compounds, which are annotated with ARG1, and relations between the events and conditions or yield are annotated with ARGM.
Task 1c, (i.e., AR), involves both mention detection and relation extraction tasks. The mention detection task identifies an antecedent as ENTITY and the anaphor as a label that represents their relationship. The relation extraction task identifies relations between antecedents and anaphors as a coreference relation (COREFERENCE) and four bridging relations (TRANSFORMED, RE-ACTION_ASSOCIATED, WORK_UP, and CONTAINED).
We developed systems for mention detection and relation classification tasks that used Chem-BERT [8], a pre-trained language model for chemistry-related documents. This approach was similar to those of the best systems adopted in previous tasks [11,12]. Figure 1 shows our pipeline method. First, we split a snippet into sentences by using ChemDataExtractor [13]. Then, ChemBERT predicted the labels for mentions/relations. post-processing methods were adopted for mention detection in Task 1a and relation detection in Task 1c, with the aim of addressing the document-level context.
We fine-tuned ChemBERT for Task 1a. In addition, we fine-tuned multiple ChemBERTs for Tasks 1b and 1c because these tasks are more complex than Task 1a. Table 1 shows the set of
For Task 1a, we trained a ChemBERT model and constructed a set of post-processing rules. First, ChemBERT was fine-tuned on the training set, excluding snippets that had overlapping mentions. A snippet was split into sentences by using ChemDataExtractor [13] and the sentences were split by a simple regex rule that used a particular tool1 by default. Then, IOB2 labels were assigned to the tokens. We used a linear classifier to predict the label of each tokens and the input for it was the output of the first sub-token [14]. Second, two postprocessing methods were applied. Because compounds in a heading section, which contains the product of the snippet (REACTION_PRODUCT) and/or the final product of the multistep reaction (OTHER_COMPOUND), cannot be distinguished without context, a post-processing method is required. The rule adopted is that if EXAMPLE_LABEL exists in a snippet, then the REACTION_PRODUCT that appears before the last EXAMPLE_LABEL is annotated with OTHER_COMPOUND. Then, if OTHER_COMPOUND appears in a heading and the same string appears as a REACTION_PRODUCT after the last source compound (STARTING_MATERIAL, REAGENT_CATALYST, or SOLVENT), then the entity is labeled as a REACTION_PRODUCT.
For Task 1b, we used the method in Task 1a for named entity recognition and trained a ChemBERT model for the detection of events in the same manner as for Task 1a.
For Task 1c, we trained one ChemBERT model for coreference and a second for bridging relations. The reason why we split the relations is because their mentions are partly different from each other. Therefore, we aimed to suppress false-positive relations caused by false-positive mentions from the other ones. In these models, single-label mention detection was performed and the relation labels were given in the relation classification step. Sentences were tokenized in the same manner as for Task 1a, with B, I, O and D labels from BIOHD [15] being used for tagging because Task 1c contains discontinuous mentions. When overlapping mentions were tokenized, we used the longer entities and discarded the shorter ones. Because the number of mentions in the training set for coreference was smaller than for other datasets used for mention detection, we augmented the number of positive examples in the training set by reusing sentences that contained one or more mentions five times.
For Task 1b, we trained a one ChemBERT model for ARG1 relations and a second for ARGM relations. The input to the relation classification was a sentence with the candidate pair for a relation between an event enclosed by [E1] and [/E1] tokens and a target enclosed by [E2] and [/E2] tokens. The output was a binary classification result indicating whether the pair has a relation or not. All candidate pairs in a sentence were classified and positive relations were annotated by the system. For example, if two events and three targets are included in a sentence, the number of candidate pairs would be six. This approach is similar to the Melax Tech system [11], which performed best in the ChEMU 2020 task [9]. This approach was also discussed in a general framework [16]. In the training stage, we used not only gold-standard entities but also predicted events that were generated by five systems trained on 80% of the training set, similarly to five-fold cross-validation.
For Task 1c, we trained one ChemBERT model for coreference and a second for bridging relations. The input to the relation classification was a pair of sentences representing a candidate pair for a relation between an anaphor enclosed by [E1] and [/E1] tokens and an antecedent enclosed by [E2] and [/E2]. The reason for using a pair of sentences, different from Task 1b, was that relations in Task 1c were often across sentences. If a mention was discontinuous, the first block of the mention was enclosed. The output was the label of the relation or a NO_RELATION label.
We applied two post-processing methods for Task 1c because relations that involve more than two sentences were included in the snippets. First, for coreference relations, when RE-ACTION_PRODUCT appeared multiple times in a snippet and the sentence-level distance of the mentions was more than two, which means it cannot be found by ChemBERT, we assign a COREFERENCE relation. Second, for bridging relations, when a candidate for an antecedent did not have any anaphors, we searched for antecedent candidates for the anaphor by finding words that started with "the" and were an anaphor for another antecedent. The candidate for the anaphor that was closest to the antecedent was then selected as the anaphor. If the antecedent contained STARTING_MATERIAL, REAGENT_CATALYST or SOLVENT, the relation was annotated with REACTION_ASSOCIATED. Otherwise, the relation was annotated with a label that was the same as the already annotated anaphor after the antecedent.
In addition to the above methods, we used a post-processing tool distributed by the task organizer 2 , which generates a coreference between A and C when coreferences between A and B and between B and C already exist.
We used ChemBERT v3.0 [8] for the mention detection and relation classification models. ChemBERT was implemented by using AllenNLP [17] and HuggingFace Transformers [18]. We used the AdamW optimizer [19] and cross entropy loss for optimization. The models were trained on the training set for the task and evaluated on the development set and both public and private test sets. Hyperparameter values were set as follows: max sequence length=384 (covering all sequences contained in the training and development sets), batch size=16, learning rate=1e-5, and patience=7. Because the relation classification in Task 1c accepts a pair of sentences as its input, a maximum sequence length of 512 was used for this task. The validation metric for early stopping of mention detection in the development set was the F-score. For relation classification, it was the validation loss.
The performances of the various systems were evaluated with respect to both exact and relaxed matching for precision, recall, and F-score.
We submitted a system with post-processing for Task 1a before the deadline and a system without post-processing after the deadline. Table 2 shows our results for Task 1a on the private set. Our system obtained an exact match F-score of 0.9412 and a relaxed match F-score of 0.9572. Table 3 shows our results for the private set in detail. We submitted one system for Task 1b before the deadline. We also submitted a corrected version after the deadline, having found an error related to the sequence length of the input to the system. Table 4 shows our results for Task 1b on the private set. The corrected system obtained an exact match F-score of 0.8865 and a relaxed match F-score of 0.9027. Table 5 shows our results for the private set in detail.
We submitted three systems for Task 1c before the deadline. We also submitted a corrected version after the deadline, having found an error related to the sequence length of the input to the system. Table 6 shows our results for Task 1c on the private set. The corrected system obtained an exact match F-score of 0.7232 and a relaxed match F-score of 0.8053. Table 7 shows our results on the private set in detail.
Selecting "the best model" of the various models in training was difficult. Table 8 shows the F-scores on the development and private sets. We used models that showed the best F-score on the development set for mention detection and the best validation loss for relation classification; therefore, it is not surprising that F-scores on the development sets were better than those for the private sets. In particular, the result for Task 1c (greater than 0.09) represented a large gap. An explanation for this larger gap could be that the predictions by the models were unstable because Task 1c was more difficult than the other tasks. Therefore, we must reconsider training methods when seeking a better model.
In Task 1b, all relations whose recalls were zero had fewer than five gold-standard examples (Table 5). It is quite difficult for our machine learning framework to identify such relations with a small amount of examples.
Although our system showed good results for Tasks 1a and 1b, we found errors caused by a lack of document-level information. Figure 2 shows a confusion matrix for Task 1a. Because the role of a compound depends on a reaction, it is difficult to identify the label of compounds without document-level information. Examples included errors among STARTING_MATERIAL, REAGENT_CATALYST, and SOLVENT and errors between REAC-TION_PRODUCT and OTHER_COMPOUND. In addition, errors between a compound for a reaction (REAGENT_CATALYST, SOLVENT, STARTING_MATERIAL) and one for a work-up 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 *ARG1|WORKUP|REAGENT_CATALYST 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 *ARG1|WORKUP|SOLVENT 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 *ARG1|WORKUP|STARTING_MATERIAL 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 (OTHER_COMPOUND) were found because distinguishing between them from just one sentence was sometimes difficult. Errors between REACTION_STEP and WORK_UP for Task 1b were also found to be caused by the same difficulty.
However, adopting a post-processing method for Task 1a mitigated the errors between REACTION_PRODUCT and OTHER_COMPOUND (Table 2). The post-processing methods employed in Task 1c were also useful in mitigating these errors (Table 6). However, we should note that using these methods adversely affected the precision. For example, our post-processing method for coreference generated false relations when false positive REACTION_PRODUCT existed. Therefore, we must be careful when using post-processing methods. With the aim of improving our systems, we tried to consider the relationships between the tasks in some preliminary experiments. For example, we tried to construct post-processing rules for named entity mentions in Task 1a by on the results for Task 1b. However, these rules did not improve the results because it was difficult to determine which prediction (named entity or event) was correct. In addition, we tried to use the relationships not only in the forward direction (i.e., Task 1a to Task 1c), but also in the backward direction (Task 1c to Task 1a). However, improving the performance in the backward direction was also difficult because the performance of the later task was lower than in the earlier task. Despite these difficulties, the post-processing methods for Task 1c that considered the results for Task 1a did improve our system's performance. Therefore, we must conduct a more detailed analysis of the relationships between the tasks if we are to improve our systems via this approach.
Table 9 shows the results for mention detection in Task 1c on the development set. First, the detection of coreference mentions was difficult compared to identifying bridging relations. The reasons were that coreference mentions had only a small number of mentions in the training data and sometimes required inter-sentence information to extract antecedents. The significance of the data augmentation was not clear. Therefore, we must reconsider the ratios used for positive mentions.
Several errors were caused by failures in sentence splitting. For example, START-ING_MATERIAL "Ex. 18A" caused a split into two sentences by the ChemDataExtractor sentence splitter because of the ". " in "Ex. 18A". A solution to this problem would involve applying a set of rules involving dependency parsing and trigger words.
This paper has reported our results for the three tasks at ChEMU 2022. We proposed hybrid methods that used ChemBERT and a small set of post-processing rules for these tasks. We employed a pipeline approach for Tasks 1b and 1c that combined mention detection and relation classification. Because we used only one or two sentences as the input to ChemBERT, this lack of document-level information suppressed the performance of the system. Although we confirmed that adopting a set of post-processing rules was effective in considering documentlevel information, we also confirmed that the set of rules we used was insufficient. In addition, although we tried to use relationships between the tasks to improve performance, it was difficult to construct rules that did achieve improvements. Therefore, we must conduct more detailed analyses about the relationships between the tasks.
We would like to thank the ChEMU team for providing the datasets. This work was partially supported by JSPS KAKENHI Grant Number 21K19814.