In this paper, we provide an overview of the Cheminformatics Elsevier Melbourne University (ChEMU) evaluation lab 2022, part of the Conference and Labs of the Evaluation Forum 2022 (CLEF 2022). The ChEMU campaign focuses on information extraction tasks over chemical reactions in patents. The ChEMU 2020 lab provided two information extraction tasks, named entity recognition and event extraction. The ChEMU 2021 lab introduced one more task, anaphora resolution. This year, we re-run all the three tasks with new test data. Together, the tasks support comprehensive automatic chemical patent analysis. Herein, we describe the resources created for these tasks and the evaluation methodology adopted. We also provide a brief summary of the methods employed by participants of this lab and the results obtained across 22 runs from 3 teams, finding that several submissions achieve better results than the baseline methods prepared by the organizers.
The discovery of new chemical compounds is a key driver of the chemistry and pharmaceutical
industries. Patents serve as a critical source of information about new chemical compounds,
providing timely and comprehensive information about new chemical compounds [
The ChEMU campaign focuses on information extraction tasks over chemical reactions in
patents. The ChEMU2020 lab [
In collaboration with chemical domain experts, we have built upon the datasets used in ChEMU 2020/2021 (1500 snippets) and prepared 500 snippets from selected chemical patents that specifically target all three tasks. For the NER and the EE tasks, three chemical experts were hired to manually annotate the corpus, labeling named entities and event steps in these text segments. Two of them reviewed all text segments independently and the third annotator acted as an adjudicator who resolved their disagreements and merged their annotations into the final gold-standard corpus. For the AR task, two chemical experts, a PhD candidate and a ifnal year bachelor student in Chemistry were hired to annotate the same set of snippets. The dataset was first annotated by the two annotators individually, and then their annotations were compared and combined by an adjudicator.
The ChEMU2022 lab has received considerable interest, attracting 54 registrants. Specifically, we received 8 runs from 3 teams in the NER task, 11 runs from 3 teams in the EE task, and 3 runs from 1 team in the AR task, respectively. Several submissions achieved exciting results, with a few of them outperforming baseline models significantly.
The rest of the paper is structured as follows. We first discuss related work and shared tasks in Section 2 and introduce the corpus we created for use in the lab in Section 3. Then we give an overview of the tasks in Section 4 and detail the valuation framework of ChEMU in Section 5 including the evaluation methods and baseline models. We present the evaluation results in Section 7 and finally conclude this paper in Section 8.
To assess and advance the natural language processing (NLP) techniques in the biochemical
domain, many shared tasks/labs have been organized, including n2c21, TREC2, BioCreative3,
BioNLP4, and CLEF workshops5. These shared tasks have covered a range of benchmark text
mining tasks: information retrieval, such as document retrieval (CLEF eHealth 2014 [
Nevertheless, most of these shared tasks/labs did not focus on the domain of chemical patents. These shared tasks mainly focused on the text mining over biomedical texts (e.g., scientific literature, such as PubMed abstracts) or clinical data (e.g., clinical health records). Text mining techniques that are developed for biomedical or biochemical texts, such as scientific journals and clinical records may not be efective for chemical patents. This is because their purpose is distinct—chemical patents are written for protection of intellectual property related to chemical compounds—and their content has diferent scope and characteristics, including variations in linguistic structures. Thus, it is critical to develop text mining techniques that are tailored for chemical patents.
Only two shared tasks have previously considered chemical patents. TREC 2009 [
The ChEMU labs also contribute new corpus on chemical text mining for the research
community6. Most existing benchmark datasets for biochemical text mining focus on biomedical
texts, i.e., texts that consider the interaction of chemicals with molecular biology or human
1https://n2c2.dbmi.hms.harvard.edu/
2https://trec.nist.gov/
3https://biocreative.bioinformatics.udel.edu/
4https://2019.bionlp-ost.org/
5https://sites.google.com/site/clefehealth/
6https://chemu.eng.unimelb.edu.au/
disease. CHEMProt [
The number of public datasets that focus on the chemistry domain is limited. Further, several
existing chemical datasets are based on structured/semi-structured texts rather than free, natural
language, texts. For example, the ZINC 15 250k corpus9 is a collection of 250,000 molecules with
their Simplified Molecular Input Line Entry System (SMILES) strings. The Tox21 dataset contains
roughly 7,000 molecules and typical 120 characteristics, such as atomic number, aromicity, donor
status. There are two datasets that are constructed from free patent texts: (1) the dataset released
by the ChemDNER patents task and (2) the dataset created by Akhondi et al. [
Despite the limited number of shared tasks on chemical patent mining, there is an increasing
interest in developing information extraction models for patents in general research communities
[
In this section, we explain how the dataset is created for our shared tasks. The complete annotation guidelines are made available on our website10.
The ChEMU chemical reaction corpus was built with the aid of Elsevier Reaxys® database.11 Reaxys® is a rich information resource for chemical reactions, which contains detailed descriptions of chemical reactions that are extracted via an “excerption” process, i.e., manual selection of information from literature sources, such as patents and scientific publications. 7https://pubmed.ncbi.nlm.nih.gov/ 8https://go.drugbank.com/ 9https://github.com/aspuru-guzik-group/chemical_vae/tree/master/models/zinc 10http://chemu2022.eng.unimelb.edu.au/ 11Reaxys® Copyright ©2022 Elsevier Life Sciences IP Limited except certain content provided by third parties. Reaxys is a trademark of Elsevier Life Sciences IP Limited, used under license. https://www.reaxys.com [Step 4] Synthesis of N-((5-(hydrazinecarbonyl)pyridin-2-yl)methyl)-1-methylN-phenylpiperidine-4-carboxamide Methyl 6-((1-methyl-N-phenylpiperidine-4carboxamido)methyl)nicotinate (0.120 g, 0.327 mmol), synthesized in step 3, and hydrazine monohydrate (0.079 mL, 1.633 mmol) were dissolved in ethanol (10 mL) at room temperature, and the solution was heated under reflux for 12 hours, and then cooled to room temperature to terminate the reaction. The reaction mixture was concentrated under reduced pressure to remove the solvent, and the concentrate was purified by column chromatography (SiO2, 4 g cartridge; methanol/dichloromethane = from 5% to 30%) and concentrated to give the title compound (0.115 g, 95.8%) as a foam solid.
In ChEMU 2020, we selected 180 English patents from the European Patent Ofice and the United States Patent and Trademark Ofice, for which information had been included in the Reaxys database. From these patents, 1500 text segments were sampled from chemical reaction descriptions pre-identified by expert domain annotators, available as a product of the process used to populate information in Reaxys® . We refer to each text segment as a patent “snippet” and use the two expressions interchangeably in the remainder of this paper. The 1500 snippets were annotated for the named entity recognition (NER) and the event extraction (EE) tasks. In ChEMU 2021, we annotated the same 1500 snippets for the anaphora resolution (AR) task. In ChEMU 2022, we further collect 500 snippets from the selected patents and annotate them for all three tasks.
We present an example of a patent snippet in Figure 1. This snippet describes the synthesis of a particular chemical compound, N-((5-(hydrazinecarbonyl)pyridin-2-yl)methyl)-1-methylN-phenylpiperidine-4-carboxamide. The synthesis process consists of an ordered sequence of reaction steps: 1. dissolving the chemical compound synthesized in step 3 and hydrazine monohydrate in ethanol; 2. heating the solution under reflux; 3. cooling the solution to room temperature; 4. concentrating the cooled mixture under reduced pressure; 5. purification of the concentrate by column chromatography; 6. concentration of the purified product to get the title compound.
Our shared tasks aim at extraction of chemical reactions from chemical patents, e.g., extracting the above synthesis steps given the patent snippet in Figure 1. To achieve this goal, it is crucial for us to first identify the entities that are involved in these reaction steps (e.g., hydrazine monohydrate and ethanol) and then determine the relations between the involved entities (e.g., hydrazine monohydrate is dissolved in ethanol).
Furthermore, our shared tasks also aim at resolving the reference in the chemical reactions. For example, the solution in the second step refers to the title compound (0.120 g, 0.327 mmol), hydrazine monohydrate (0.079 mL, 1.633 mmol), and ethanol (10 mL).
Four categories of entities are annotated over the corpus: (1) chemical compounds that are involved in a chemical reaction; (2) conditions under which a chemical reaction is carried out; (3) yields obtained for the final chemical product; and (4) example labels that are associated with reaction specifications. Ten labels are further defined under the above four categories. We define five diferent roles that a chemical compound can play within a chemical reaction, corresponding to five labels under this category: STARTING MATERIAL, REAGENT CATALYST, REACTION PRODUCT, SOLVENT, and OTHER COMPOUND. We also define two labels under the category of conditions: TIME and TEMPERATURE; and two labels under the category of yields: YIELD PERCENT and YIELD OTHER.
The definitions of all resultant labels are summarized as follows: 1. Reaction product: A substance that is formed during a chemical reaction. 2. Starting material: A substance that is consumed in the course of a chemical reaction providing atoms to products. 3. Reagent catalyst: A compound added to a system to cause or help with a chemical reaction.
Compounds like catalysts, bases to remove protons or acids to add protons must be also annotated with this tag. 4. Solvent: A chemical entity that dissolves a solute resulting in a solution. 5. Other compound: Other chemical compounds that are not the products, starting materials, reagents, catalysts and solvents. 6. Example label: A label associated with a reaction specification. 7. Temperature: The temperature at which the reaction was carried out. 8. Time: The reaction time of the reaction.
9. Yield percent: Yield given in percent values.
10. Yield other: Yields provided in other units than %.
A chemical reaction process is usually a sequence of steps, and these steps can be categorized into two types: (1) reaction steps, i.e., the steps required to convert the starting materials to the target reaction product; and (2) work-up steps, i.e., the manipulations required to purify or isolate a chemical product. For example, in Figure 1, the step of heating the solution under reflux for 12 hours is a reaction step while the step of cooling it to room temperature is a work-up step.
We define two types of trigger words: WORKUP which refers to an event step where a chemical compound is isolated/purified, and REACTION STEP which refers to an event step that is involved in the conversion from a starting material to an end product. When labelling event arguments, we adapt semantic argument role labels Arg1 and ArgM from the Proposition Bank to label the relations between the trigger words and other arguments. Specifically, the label Arg1 refers to the relation between an event trigger word and a chemical compound. Here, Arg1 represents argument roles of being causally afected by another participant in the event. ArgM represents adjunct roles with respect to an event, used to label the relation between a trigger word and a temperature, time or yield entity. The definitions of trigger word types and relation types are summarized as follows: 1. Workup: An event step which is a manipulation required to isolate and purify the product of a chemical reaction. 2. Reaction step: An event within which starting materials are converted into the product. 3. Arg1: The relation between an event trigger word and a chemical compound. 4. ArgM: The relation between an event trigger word and a temperature, time, or yield entity.
We aim to capture anaphora in chemical patents, with a focus on identifying chemical compounds
during the reaction process. Consistent with other anaphora corpora [
1. Chemical names: the formal name of chemical compounds. 2. Identifiers : identifiers or labels that uniquely represent chemical compounds which occur earlier in the text. 3. Phrases and noun types: pronouns that refer to a previously mentioned chemical compounds, e.g. they or it, and definite and indefinite noun phrases that refer to chemical compounds, e.g. the solvent, the title compound, the mixture, and a white solid, a crude product.
It should be noted that verbs (e.g. mix, purify, distil) and descriptions that refer to events (e.g. the same process, step 5) are not annotated in this corpus.
Unlike many annotation schemes, our annotation allows discontinuous mentions. For example, the underlined spans of the fragment 114 mg of 4-((4aS,7aS)-6-benzyloctahydro-1-pyrrolo[3,4b]pyridine-1-yl)-7H-pyrrolo[2,3-d]pyrimidine was obtained with a yield of about 99.1% are treated as a single discontinuous mention. This introduces further complexity into the task and helps to capture more comprehensive anaphora phenomena.
There are some diferences in the definitions of entities for the NER task and the AR task. For the NER task, entity annotations identify chemical compounds (i.e. REACTION_PRODUCT, STARTING_MATERIAL, REAGENT_CATALYST, SOLVENT, and OTHER COMPOUND), reaction conditions (i.e. TIME, TEMPERATURE), quantity information (i.e. YIELD_PERCENT, YIELD_OTHER), and example labels (i.e. EXAMPLE_LABEL). There is overlap with our definition of mention for the labels relating to chemical compounds. However, in AR annotation, chemical names are annotated along with additional quantity information, as we consider this information to be an integral part of the chemical compound description. Furthermore, the original entity annotations do not include generic expressions that corefer with chemical compounds such as the mixture, the organic layer, or the filtrate , and neither do they include equipment descriptions.
Anaphora resolution subsumes both coreference and bridging. In the context of chemical patents, we define four sub-types of bridging, incorporating generic and chemical knowledge. 1. Coreference: two expressions/mentions that refer to the same entity. 2. Bridging: a) Transformed: two chemical compound entities that are initially based on the same chemical components and have undergone possible changes through various conditions (e.g., pH and temperature). b) Reaction-associated: the relationship between a chemical compound and its immediate sources via a mixing process. The immediate sources do need to be reagents, but they need to end up in the corresponding product. The source compounds retain their original chemical structure. c) Work-up: the relationship between chemical compounds that were used for isolation or purification purposes, and their corresponding output products. d) Contained: the association holding between chemical compounds and the related equipment in which they are placed. The direction of the relation is from the related equipment to the previous chemical compound.
A referring mention which cannot be interpreted on its own, or an indirect mention, is called an anaphor, and the mention which it refers back to is called the antecedent. In relation annotation, we preserve the direction of the anaphoric relation, from the anaphor to the antecedent. Following similar assumptions in recent work, we restrict annotations to cases where the antecedent appears earlier in the text than the anaphor.
To facilitate the annotation process, a silver standard set was first prepared based on information captured in the Elsevier Reaxys® database. The extracted records from Reaxys® are linked to the IDs of their source patents. However, the precise locations of the key entity and relation information in these records in source patents are needed to construct the gold-standard corpus. The silver-standard dataset was prepared by automatically mapping elements of the records in the Reaxys® database to the source patents from which the records were extracted. This mapping process was performed by scanning patent texts and searching for excerpted entity mentions.
For the NER and the EE tasks, three chemical experts were hired to prepare the gold standard corpus. They manually reviewed all texts and pre-annotations in the silver-standard dataset to add or correct precise locations of the relevant entities and relations in the texts, according to annotation guidelines in Section 3.2.1. Two of the experts first independently reviewed and updated the silver standard annotations. Then, a third chemical expert served as an adjudicator who resolved their disagreements to produce the final gold-standard corpus. For the AR task, one of the chemical experts who had annotated for the anaphora resolution task in ChEMU 2021 was hired to annotate the same set of snippets.
The annotation process was conducted using the BRAT annotation tool,12 which is an interactive web-based tool for adding annotations to input texts. Continuing with the example snippet shown in Figure 1, a visualization of the snippet after annotation is presented in Figure 2 for the NER and the EE tasks, and Figure 3 for the AR task.
We combine the training/development/test sets for ChEMU 2020/2021 (1500 snippets) and use it as the training set for ChEMU 2022. The 500 new snippets that we annotated for ChEMU 2022 are used as the test set.
In ChEMU 2020 and 2021, the evaluation results of all submissions to the test set were only available when the shared tasks ended. This year, we run all shared tasks in a Kaggle-style where the test set (500 snippets) is randomly partitioned into two splits public/private with a ratio of 30%/70%, and the participants will get immediate feedback on the public test set (150 snippets) after making a submission, while the evaluation results on the private test set (350 snippets) remain secret until the end of the shared tasks. Note that the participants are not aware of the specific split of public and private test sets.
The three tasks, named entity recognition, event extraction, and anaphora resolution, are all snippet-level tasks since they only consider entities or relations between them within a few consecutive sentences. In our ChEMU corpus, every snippet has been annotated for all three tasks, which opens the opportunity to explore multi-task learning since the input data is the same for all three tasks, as illustrated in Table 1.
In order to understand and extract a chemical reaction from natural language texts, the first essential step is to identify the entities that are involved in the chemical reaction. The first task aims to accomplish this step by identifying the ten types of entities described in Section 3.2.1. The task requires the detection of the entity names in patent snippets and the assignment of correct labels to the detected entities. For example, given a detected chemical compound, the task requires the identification of both its text span and its specific type according to the role in which it plays within a chemical reaction description.
The title compound was used without purification (1.180 g, 95.2%) as yellow solid. NER EE AR
The title compound was used without purification (1.180 g, 95.2%) as yellow solid. REACTION_PRODUCT: title compound YIELD_OTHER: 1.180 g YIELD_PERCENT: 95.2% The title compound was used without purification (1.180 g, 95.2%) as yellow solid. REACTION_STEP: used → REACTION_PRODUCT: title compound REACTION_STEP: used → YIELD_OTHER: 1.180 g REACTION_STEP: used → YIELD_PERCENT: 95.2%
A chemical reaction usually consists of an ordered sequence of event steps that transforms a starting product to an end product, such as the six reaction steps in the synthesis process of the chemical compound described in the example in Figure 1. The event extraction task (Task 2) targets identifying these event steps. Similarly to conventional event extraction problems, the EE task involves three subtasks: event trigger word detection, event typing and argument prediction. First, it requires the detection of event trigger words and assignment of correct labels for the trigger words. Second, it requires the determination of argument entities that are associated with the trigger words, i.e., which entities identified in the NER task participate in event or reaction steps. This is done by labelling the connections between event trigger words and their arguments. Given an event trigger word and a set of arguments that participate in , the EE task requires the creation of || relation entries connecting to an argument entity in . Here, || represents the cardinality of the set . Finally, this task requires the assignment of correct relation type labels (Arg1 or ArgM) to each of the detected relations.
This task requires the resolution of anaphoric dependencies between expressions in chemical patents. The participants are required to find five types of anaphoric relationships in chemical patents, i.e. coreference, reaction-associated, work-up, contained, and transform.
Taking the text snippet in Figure 4 as an example, several anaphoric relationships can be extracted from it. [The mixture]4 and [the mixture]3 refer to the same “mixture” and thus, form a coreference relationship. The two expressions [The mixture]1 and [the mixture]2 are initially based on the same chemical components but the property of [the mixture]2 changes after the “stir” and “cool” action. Thus, the two expressions should be linked as “Transformed”. The expression [The mixture]1 comes from mixing the chemical compounds prior to it, e.g., [water (4.9 ml)]. Thus, the two expressions are linked as “Reaction-associated”. The expression [Acetic acid (9.8 ml)] and [water (4.9 ml)] were added to [the solution] in [a flask]. [The mixture]1 was stirred for 3 hrs at 50°C and then cooled to 0°C . 2Nsodium hydroxide aqueous solution was added to [the mixture]2 until the pH of [the mixture]3 became 9. [The mixture]4 was extracted with [ethyl acetate] for 3 times. [The combined organic layer] was washed with water and saturated aqueous sodium chloride.
ID AR1 AR2 AR3 AR4 AR5
Relation type Coreference Transformed Reaction_associated Work-up Contained
Anaphor [The mixture]4 [the mixture]2 [The mixture]1 [The combined organic layer] [a flask]
Antecedent [The combined organic layer] comes from the extraction of [ethyl acetate]. Thus, they are linked as “Work-up”. Finally, the expression [the solution] is contained by the entity [a flask] , and the two are linked as “Contained”.
We use BRATEval13 to evaluate all the runs that we receive. Three metrics are used to evaluate the performance of all the submissions: Precision, Recall, and 1 score. We use two diference matching criteria, exact matching and relaxed matching (approximate matching), as in some practical applications it also makes sense to understand if the model can identify the approximate region of mentions.
Formally, let = (, , ) denote an entity where is the type of , and are the beginning position (inclusive) and end position (exclusive) of the text span of . Then two entities 1 and 2 are exactly matched (1 = 2), if 1 = 2, 1 = 2, and 1 = 2. While two entities 1 and 2 are approximately matched (1 ≈ 2) if 1 = 2, 2 < 1, and 1 < 2, i.e. the two spans [1, 1) and [2, 2) overlaps.
Furthermore, let = (, , ) be a relation where is the type of , the anaphor of , the antecedent of . Then 1 and 2 are exactly matched (1 = 2) if 1 = 2, 1 = 2, and 1 = 2. While 1 and 2 are approximately matched (1 ≈ 2) if 1 = 2, 1 ≈ 2, and 1 ≈ 2.
In summary, we require strict type match in both exact and relaxed matching, but are lenient in span matching.
13https://bitbucket.org/nicta_biomed/brateval/src/master/ 4 3 2 1 0 4 3 2 1 0 4 3 2 1 0 4 3 2 1 0 4 3 2 1 0 4 3 2 1 0 (a) All pairs matched in relaxed setting (b) A non-optimal bipartite matching ( = 3) (c) A maximum bipartite matching ( = 4)
With the above definitions, the metrics for exact matching can be easily calculated. The true positives (TP) are exact matching pairs found in gold relations and predicted relations. Then false positives (FP) are the predicted relations that don’t have a match, i.e. = # − , where # is the number of predicted relations. Similarly, false negatives are the gold relations that are not matched by any predicted relations, i.e. = # − where # is the number of gold relations. Finally Precision = /( + ), Recall = /( + ), and 1 = 2/(1/ + 1/).
Unlike exact matching, relaxed matching is not well-defined and metrics in this setting have more than one way to calculate, therefore we need to clearly define all the metrics. 5 Let consider an example shown in Figure 5a where nodes {}=1 are predicted relations, 5 {}=1 are gold relations, and every edge between a node and a node means they are approximately matched. At first glance, one may think that = = 0 because every gold relation has at least a match and so does every predicted relation. However, it is impossible to ifnd 5 true positive pairs from this graph without using one node more than once. Therefore, if = = 0, then + ̸= # = 5 and + ̸= # = 5, which is inconsistent with the formulas in exact setting.
So, instead of defining as the number of gold relations that don’t have a match, we just define = # − . Similarly is defined as # − . Then the problem remained is how to calculate . Actually, finding true positive pairs can be considered as bipartite matching. Figure 5b shows a matching with = 3 but is not optimal. Figure 5c shows one possible maximum bipartite matching with = 4. Another optimal matching is replacing edge 0 − 0 with 0 − 1.
In summary, we define as the maximum bipartite matching for the graph constructed by all approximately matched pairs, then = # − and = # − , finally Precision = /( + ), Recall = /( + ), and 1 = 2/(1/ + 1/). This has been implemented in the latest BRATEval.
We consider two types of coreference linking, i.e. (1) surface coreference linking and (2) atomic coreference linking, due to the existence of transitive coreference relationships. By transitive coreference relationships we mean multi-hop coreference such as a link from an expression T1 to T3 via an intermediate expression T2, viz., “T1→T2→T3”. Surface coreference linking will restrict attention to one-hop relationships, viz., to: “T1→T2” and “T2→T3”. Whereas atomic coreference linking will tackle coreference between an anaphoric expression and its first antecedent, i.e. intermediate antecedents will be collapsed. Thus, these two links will be used for the above example, “T1→T3” and “T2→T3”. Note that we only consider transitive linking in coreference relationships.
Note that {T1→T2,T2→T3} infers {T1→T3,T2→T3}, but the reverse is not true. This leads to a problem about how to score a prediction {T1→T3,T2→T3}, when the gold relation is {T1→T2,T2→T3}. Both T1→T3 and T2→T3 are true, but some information is missing here.
Our solution is to first expand both the prediction set and gold set where all valid relations that can be inferred will be generated and added to the set, and then to evaluate the two sets normally. In the above example, the gold set will be expanded to {T1→T2,T2→T3,T1→T3}, and then the result is = 2, = 1. Likewise, when evaluate {T1→T4,T2→T4,T3→T4} against {T1→T2,T2→T3,T3→T4}, the gold set will be expanded into 6 relations, while the prediction set won’t be expanded as no new relation can be inferred. So the evaluation result will be = 3, = 3. One may worry that if there is a chain of length then its expanded set will be in (2), when is large, this local evaluation result will have too much influence on the overall result. But we find in practice that coreference chains are relatively short, with 3 or 4 being the most typical lengths, so it is unlikely to be a big issue.
We use a joint model for recognizing named entities and classifying relations between them. The model first processes the input snippet using a BERT model to obtain the contextualized word representations. We adopt the BIO tagging schema for training the NER classifier which classifies every word into entity tags. Then a list of identified entities is created based on the output of the NER classifier. For each entity in the list, the contextualized word representations are max pooled to obtain the representation for the entity. Then the model enumerates all possible pairs of entities and provides them to a relation classifier which classifies every pair of entities by concatenating the representations of both entities.
Our baseline model adopts an end-to-end architecture for coreference resolution [
We received paper submissions from all the participating teams, i.e. the LG team, the HUKB
team, and the VCU team.
6.1. LG Team
The LG team developed context-aware NER and RE models based on the domain-specific
language model with pipeline approach. For the domain-specific language model, they
posttrain the BiolinkBert[
The HUKB team participated in all three tasks. For the NER task, they used ChemBERT[49], a pre-trained language model for chemistry-related documents, and a set of post-processing rules that considers document-level context. First, ChemBERT predicted mentions using a sentence as the input. Then, two post-processing methods were applied. They submitted a result and it obtained an exact match F-score of 0.9412 and a relaxed match F-score of 0.9572.
For the EE task, they adopted a pipeline approach for relation extraction. In addition to named entities detected by the NER task, they also used ChemBERT for event detection as the NER task. After detecting those mentions, they also used ChemBERT for relation classification. In this classification process, all candidate pairs in a sentence were classified and positive relations were annotated by the system. They fine-tuned one ChemBERT model for ARG1 and a second for ARGM relations. In the training stage, they used not only gold-standard entities but also predicted events that were generated by five systems trained on 80% of the training set, similarly to five-fold cross-validation. They submitted a result and it obtained an exact match F-score of 0.8868 and a relaxed match F-score of 0.9028. They also submitted a result after the evaluation phase that corrected the sequence length of an input and the result obtained an exact match F-score of 0.8865 and a relaxed match F-score of 0.9027.
For the AR task, they adopted a pipeline approach for relation extraction, where the candidates of mentions are first detected and then relations between them are classified. In addition, they used a set of post-processing rules that considers document-level context. For mention detection, they fine-tuned one Chem-BERT model for coreference and a second for bridging relations. They augmented the number of positive examples for coreference relations in the training set by reusing sentences that contained one or more mentions five times because the number of mentions in the training set for coreference was smaller than for other datasets used for mention detection. For relation classification, Chem-BERT detected a relation for each pair of mentions in two continuous sentences. They trained two models for coreference and bridging relations. They applied two post-processing methods for each relations that consider document-level context. Both of them used results in the NER task; for example, REACTION PRODUCT in the NER task was used for the detection of corefenrence relations. They submitted three results: (1) without post-processing rules, (2) with a post-processing rule for bridging relations and (3) with post-processing rules both coreference and bridging relations. The third result obtained the best performance that was an exact match F-score of 0.7085 and an F-score of 0.7893. They also submitted a result after the evaluation phase that corrected the sequence length of an input and the result obtained an exact match F-score of 0.8865 and a relaxed match F-score of 0.9027.
The VCU team participated in the NER and the EE tasks. For the NER task, they evaluated two
methods for identifying the experimental parameters and triggers. The first was a
transformerbased architecture and the second a bidirectional Long Short Term Memory (biLSTM) units, both
with a Conditional Random Field (CRF) output layer. The input to their biLSTM model were
pretrained word embeddings[
They treat the EE task as a binary classification task building a separate model for each trigger word-entity type to determine whether a relationship exists between them. They first identify the sentence where the trigger word-entity pair is located, then they replace the non-targeted trigger word-entity pairs with ‘X’ from the input sentence except for the targeted trigger wordentity pair. Here, BERT captures the contextual information within a sentence, whereas GCN captures the global information. They use the BERT tokenizer for word tokenization, and then they generate a vocabulary graph = (, ) where the word nodes in the graph are denoted by mapped integers. Next, the combined input of mapped word indices with the generated graph embeddings is passed through BERT, and the final embedding representation is fed into a fully connected layer for classification. This method obtained an overall with an exact precision, recall and F1 score of 0.82, 0.68, and 0.75 respectively, and a relaxed precision, recall and F1 score of 0.88, 0.73, and 0.79 respectively.
A total of 54 participants registered on our submission website for the shared tasks. Among them, we finally received 22 submissions from 3 teams on the test set. The 3 teams are LG AI Research (LG), Hokkaido University (HUKB), and Virginia Commonwealth University (VCU). In this section, we report their results along with the performance of our baseline systems.
We report the overall performance of all runs in Table 2. The baseline achieves 0.9367 in F1-score under exact-match. Four runs outperform the baseline in terms of F1-score under exact-match. The best run was submitted by team LG AI Research, achieving a high F1-score of 0.9673. The F1-scores for submissions from team VCU in relaxed match are 10%-15% higher than those in exact-match. This diference between exact-match and relaxed-match may be related to the long text spans of chemical compounds, which is one of the main challenges in NER tasks in the domain of chemical documents.
The overall performance of all runs is summarized in Table 3 in terms of Precision, Recall, and F1-score under both exact-match and relaxed-match. The rankings of diferent systems are
R
F
P
R
F almost fully consistent across all metrics. Our baseline obtains 0.9088 F1-score under exactmatch and the best run is from team LG AI Research which archieves 0.9199 F1-score under exact-match. The performance gap between our baseline and the best run indicates the dificulty of the event extraction task comparing to the NER task. We also notice that recall scores of most runs are consistently lower than their precision scores, which may reveal that the task of identifying a relation from a chemical patent is harder than the task of typing an identified relation.
The evaluation results of all submission to the anaphora resolution task are shown in Table 4. The first run from the Hokkaido University team achieves an F1-score of 0.7085 in exact-match, outperforming our baseline which gets 0.687. The lead of the best run is even larger in relaxed matching, with an F1-score of 0.7893, about 6 points higher than our baseline. This shows the potential of the model built by the Hokkaido University team and indicates that the performance in exact matching may be further boosted if the boundary errors of their model could be corrected in a post-processing step. Our baseline has higher precision in the exact setting, which indicates that our model is more conservative and could possibly be enhanced by making more aggressive predictions to improve recall.
This paper presents a general overview of the activities and outcomes of the ChEMU 2022 evaluation lab. As the third instance of our ChEMU lab series, ChEMU 2022 targets three tasks focusing on chemical reaction information extraction from chemical patents. The evaluation result includes diferent approaches to tackling the shared task, with several submissions outperforming our baseline methods. We look forward to fruitful discussion and deeper understanding of the methodological details of these submissions at the workshop.
We are grateful for the detailed excerption and annotation work of the domain experts that support Reaxys, and the support of Ivan Krstic, Director of Chemistry Solutions at Elsevier. Funding for the ChEMU project is provided by an Australian Research Council Linkage Project, project number LP160101469, and Elsevier. We acknowledge the support of annotators for the anaphora resolution task, Dr. Sacha Novakovic and Colleen Hui Shiuan Yeow at the University of Melbourne. S. Muresan, P. Nakov, A. Villavicencio (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, Association for Computational Linguistics, 2022, pp. 4904–4917.
URL: https://aclanthology.org/2022.acl-long.337. [49] J. Guo, A. S. Ibanez-Lopez, H. Gao, V. Quach, C. W. Coley, K. F. Jensen, R. Barzilay, Automated chemical reaction extraction from scientific literature, J. Chem. Inf. Model. 62 (2022) 2035–2045. URL: https://doi.org/10.1021/acs.jcim.1c00284. doi:10.1021/acs.jcim. 1c00284.