In this paper, we provide an overview of the Cheminformatics Elsevier Melbourne University (ChEMU) evaluation lab 2021, part of the Conference and Labs of the Evaluation Forum 2021 (CLEF 2021). The ChEMU evaluation lab focuses on information extraction over chemical reactions from patent texts. As the second instance of our ChEMU lab series, we build upon the ChEMU corpus developed for ChEMU 2020, extending it for two distinct tasks related to reference resolution in chemical patents. Task 1 Chemical Reaction Reference Resolution - focuses on paragraph-level references and aims to identify the chemical reactions or general conditions specified in one reaction description referred to by another. Task 2 - Anaphora Resolution - focuses on expression-level references and aims to identify the reference relationships between expressions in chemical reaction descriptions. Herein, we describe the resources created for these tasks and the evaluation methodology adopted. We also provide a brief summary of the results obtained in this lab, finding that one submission achieves substantially better results than our baseline models.
The discovery of new chemical compounds is perceived as a key driver of the chemical industry
and many other industrial sectors, and information relevant for this discovery is found in
chemical synthesis descriptions in natural language texts. In particular, patents serve as a critical
source of information about new chemical compounds. Compared with journal publications,
patents provide more timely and comprehensive information about new chemical compounds [
Despite the significant commercial and research value of the information in patents, manual
extraction of such information is costly, considering the large volume of patents available [
The ChEMU campaign focuses on information extraction tasks over chemical reactions
in patents1. ChEMU 2020 [
Unfortunately, we didn’t receive any submissions to Task 1, chemical reaction reference resolution. The complexity of this task in particular combined with relatively short time periods for people to develop their systems may have made it dificult for people to participate. We plan to re-run it in 2022, to give the opportunity for more people to participate since the data and task definitions will have been around for a longer period of time. As a result, the remainder of this paper will focus on the second task, anaphora resolution.
The rest of the paper is structured as follows. We first discuss related work and shared tasks in Section 2 and introduce the corpus we created for use in the lab in Section 3. Then we give an overview of the task in Section 4 and detail the valuation framework of ChEMU in Section 5 including the evaluation methods and baseline models. We present the evaluation results in Section 6 and finally conclude this paper in Section 7.
Several shared tasks have addressed reference resolution in scientific literature. BioNLP2011
hosted a subtask on protein coreference [
First, previous shared tasks considered diferent domains of scientific literature. For example,
the dataset used in BioNLP2011 is derived from the GENIA corpus [
Secondly, our reference resolution tasks include both paragraph-level and entity-level reference phenomena. Our first task aims at identification of reference relationships between reaction descriptions, i.e. paragraph-level. This task is challenging because a reaction description may refer to an extremely remote reaction and thus requires processing of very long documents. Our second task aims at anaphora resolution, similarly to previous entity-level coreference tasks. However, a key diference is that we extend the scope of this task by including both coreference and bridging phenomena. That is, we not only aim at finding expressions referring to the same entity, but also expressions that are semantically related or associated.
In this section, we explain how the dataset is created for the anaphora resolution task. The
complete annotation guidelines are made available at [
We build on the ChEMU corpus [
2Reaxys® Copyright ©2021 Elsevier Life Sciences IP Limited except certain content provided by third parties. Reaxys is a trademark of Elsevier Life Sciences IP Limited, used under license. https://www.reaxys.com
We aim to capture anaphora in chemical patents, with a focus on identifying chemical compounds
during the reaction process. Consistent with other anaphora corpora [
Furthermore, there are a few types of NPs that need specific handling in chemical patents: 1. Quantified NPs: Chemical compounds are usually described with a quantity. NPs with quantities are considered as atomic mentions if the quantities are provided, e.g. 398.4 mg of the compound 1. 2. NPs with prepositions: Chemical NPs connected with prepositions (e.g. in, with, of ) can be considered as a single mention. For example, the phrase 2,4-dichloro-6-(6-triuoromethylpyridin2-yl)-1,3,5-triazine (5.0 g, 16.9 mmol) in tetrahydrofuran (100 mL) is a single mention, as it describes a solvent that contains 2,4-dichloro-6-(6-triuoromethylpyridin-2-yl)-1,3,5triazine (5.0 g, 16.9 mmol) and tetrahydrofuran (100 mL).
NPs describing chemical equipment containing a compound may also be relevant to anaphora resolution. This generally occurs when the equipment that contains the compound undergoes a process that also afects the compound. Thus, equipment expressions such as the flask and the autoclave can also be mentions if they are used to implicitly refer to a contained compound.
Unlike many annotation schemes, our annotation allows discontinuous mentions. For
example, the underlined spans of the fragment 114 mg of
4-((4aS,7aS)-6-benzyloctahydro-1-pyrrolo[3,4b]pyridine-1-yl)-7H-pyrrolo[2,3-d]pyrimidine was obtained with a yield of about 99.1% are treated
as a single discontinuous mention. This introduces further complexity into the task and helps
to capture more comprehensive anaphora phenomena.
3.2.4. Relationship to ChEMU 2020 entities
Since this dataset is built on the ChEMU 2020 corpus [
In the original ChEMU 2020 corpus, entity annotations identify chemical compounds (i.e. REACTION_PRODUCT, STARTING_MATERIAL, REAGENT_CATALYST, SOLVENT, and OTHER COMPOUND), reaction conditions (i.e. TIME, TEMPERATURE), quantity information (i.e. YIELD_PERCENT, YIELD_OTHER), and example labels (i.e. EXAMPLE_LABEL). There is overlap with our definition of mention for the labels relating to chemical compounds. However, in our annotation, chemical names are annotated along with additional quantity information, as we consider this information to be an integral part of the chemical compound description. Furthermore, the original entity annotations do not include generic expressions that corefer with chemical compounds such as the mixture, the organic layer, or the filtrate , and neither do they include equipment descriptions.
Anaphora resolution subsumes both coreference and bridging. In the context of chemical patents, we define four sub-types of bridging, incorporating generic and chemical knowledge.
A referring mention which cannot be interpreted on its own, or an indirect mention, is
called an anaphor, and the mention which it refers back to is called the antecedent. In relation
annotation, we preserve the direction of the anaphoric relation, from the anaphor to the
antecedent. Following similar assumptions in recent work, we restrict annotations to cases
where the antecedent appears earlier in the text than the anaphor.
3.3.1. Coreference
Coreference is defined as expressions/mentions that refer to the same entity [
2. Multiple Antecedents: the anaphor refers to multiple antecedents, e.g. start materials refers to all the chemical compounds or materials that are used at the beginning. It is possible for there to be ambiguity as to which mention of a given antecedent an anaphor refers to (where the mention is identical); in these cases the closest mention is selected. 3.3.2. Bridging As stated above, when we consider the anaphora relations, we take the chemical properties of the mention into consideration. Coreference is insuficient to cover all instances of anaphora in chemical patents, and bridging occurs frequently. We define four bridging types: TRANSFORMED Links between chemical compounds that are initially based on the same components, but which have undergone a change in condition, such as pH or temperature. Such cases must be one-to-one relations (not one-to-many). As shown in Figure 2, the mixture in line 2 and the first-mentioned mixture in line 3 have the TRANSFORMED relation, as they have the same chemical components but diferent chemical properties.
REACTION-ASSOCIATED The relationship between a chemical compound and its immediate source compounds is via a mixing process, where the source compounds retain their original chemical structure. This relation is one-to-many from the anaphor to the source compounds (antecedents). For example, the mixture in line 2 has REACTION-ASSOCIATED links to three mentions on line 1 that are combined to form it: (1) the solution of Compound (4) (0.815 g, 1.30 mmol) in THF (4.9 ml); (2) acetic acid (9.8 ml); and (3) water (4.9 ml)).
WORK-UP Chemical compounds are used to isolate or purify an associated output product, in a one-to-many relation, from the anaphor to the compounds (antecedents) that are used for the work-up process. As demonstrated in Figure 2, The combined organic layer in line 5 comes from the extraction of The mixture and ethyl acetate in line 4, and they are hence annotated as WORK-UP.
CONTAINED A chemical compound is contained inside equipment. It is a one-to-many relation from the anaphor (equipment) to the compounds (antecedents) that it contains. An example of this is a flask and the solution of Compound (4) (0.815 g, 1.30 mmol) in THF (4.9 ml) on line 1, where the compound is contained in the flask.
For the corpus annotation, we use the BRAT text annotation tool.3 In total 1500 snippets have
been annotated by two chemical experts, a PhD candidate and a final year bachelor student
in Chemistry. A draft of the annotation guideline was created and refined with chemical
experts, then four rounds of annotation training were completed prior to beginning oficial
annotation. In each round, the two annotators individually annotated the same 10 snippets
(diferent across each round of annotation), and their annotations were compared and combined
by an adjudicator; annotation guidelines were then refined based on discussion. After several
rounds of training, we achieved a high inner-annotator agreement of Krippendorf’s = 0.92
[
We randomly partitioned the whole dataset into three splits for training, development, and test purposes, with a ratio of 0.6/0.15/0.25. The training and development sets were released to participants for model development. Note that participants are allowed to use the combination of training and development sets and to use their own partitions to build models. The test set is withheld for use in the formal evaluation. The statistics of the three splits including their number of snippets, total number of sentences, and average number of tokens per sentence, are summarized in Table 1.
To ensure the snippets included in the training, development, and test splits have similar distributions, we compare the distribution of relation types (five types of relations in total). Based on the numbers in Table 1, we confirm that the label distribution in the three splits are similar, with very little variation (≤ 2%) across the three splits observed for each relation type.
4With the lowest agreement being = 0.89 for coreference mentions.
This task requires the resolution of general anaphoric dependencies between expressions in chemical patents. Five types of anaphoric relationships are defined: 1. Coreference: two expressions/mentions that refer to the same entity. 2. Transformed: two chemical compound entities that are initially based on the same chemical components and have undergone possible changes through various conditions (e.g., pH and temperature). 3. Reaction-associated: the relationship between a chemical compound and its immediate sources via a mixing process. The immediate sources do need to be reagents, but they need to end up in the corresponding product. The source compounds retain their original chemical structure. 4. Work-up: the relationship between chemical compounds that were used for isolation or purification purposes, and their corresponding output products. 5. Contained: the association holding between chemical compounds and the related equipment in which they are placed. The direction of the relation is from the related equipment to the previous chemical compound.
Taking the text snippet in Figure 3 as an example, several anaphoric relationships can be extracted from it. [The mixture]4 and [the mixture]3 refer to the same “mixture” and thus, form a coreference relationship. The two expressions [The mixture]1 and [the mixture]2 are initially based on the same chemical components but the property of [the mixture]2 changes after the “stir” and “cool” action. Thus, the two expressions should be linked as “Transformed”. The expression [The mixture]1 comes from mixing the chemical compounds prior to it, e.g., [water (4.9 ml)]. Thus, the two expressions are linked as “Reaction-associated”. The expression [The combined organic layer] comes from the extraction of [ethyl acetate]. Thus, they are linked as “Work-up”. Finally, the expression [the solution] is contained by the entity [a flask] , and the two are linked as “Contained”.
[Acetic acid (9.8 ml)] and [water (4.9 ml)] were added to [the solution] in [a flask]. [The mixture]1 was stirred for 3 hrs at 50°C and then cooled to 0°C . 2N-sodium hydroxide aqueous solution was added to [the mixture]2 until the pH of [the mixture]3 became 9. [The mixture]4 was extracted with [ethyl acetate] for 3 times. [The combined organic layer] was washed with water and saturated aqueous sodium chloride.
ID AR1 AR2 AR3 AR4 AR5
Relation type Coreference Transformed Reaction_associated Work-up Contained
Anaphor [The mixture]4 [the mixture]2 [The mixture]1 [The combined organic layer] [a flask]
Antecedent
We use BRATEval5 to evaluate all the runs that we receive. Three metrics are used to evaluate the performance of all the submissions: Precision, Recall, and 1 score. We use two diference matching criteria, exact matching and relaxed matching (approximate matching), as in some practical applications it also makes sense to understand if the model can identify the approximate region of mentions.
Formally, let = ( , , ) denote an entity where is the type of , and are the beginning position (inclusive) and end position (exclusive) of the text span of . Then two entities 1 and 2 are exactly matched (1 = 2), if 1 = 2, 1 = 2, and 1 = 2. While two entities 1 and 2 are approximately matched (1 ≈ 2) if 1 = 2, 2 < 1, and 1 < 2, i.e. the two spans [1, 1) and [2, 2) overlaps.
Furthermore, let = ( , , ) be a relation where is the type of , the anaphor of , the antecedent of . Then 1 and 2 are exactly matched (1 = 2) if 1 = 2, 1 = 2, and 1 = 2. While 1 and 2 are approximately matched (1 ≈ 2) if 1 = 2, 1 ≈ 2, and 1 ≈ 2.
In summary, we require strict type match in both exact and relaxed matching, but are lenient in span matching. 5.1.1. Exact Matching With the above definitions, the metrics for exact matching can be easily calculated. The true positives (TP) are exact matching pairs found in gold relations and predicted relations. Then false positives (FP) are the predicted relations that don’t have a match, i.e. = # − , where # is the number of predicted relations. Similarly, false negatives are the gold relations that are not matched by any predicted relations, i.e. = # − where # is the 5https://bitbucket.org/nicta_biomed/brateval/src/master/ 4 3 2 1 0 4 3 2 1 0 4 3 2 1 0 4 3 2 1 0 4 3 2 1 0 4 3 2 1 0 (a) All pairs matched in relaxed setting (b) A non-optimal bipartite matching ( = 3) (c) A maximum bipartite matching ( = 4) number of gold relations. Finally Precision = /( + ), Recall = /( + ), and 1 = 2/(1/ + 1/). 5.1.2. Relaxed Matching Unlike exact matching, relaxed matching is not well-defined and metrics in this setting have more than one way to calculate, therefore we need to clearly define all the metrics. 5 Let consider an example shown in Figure 4a where nodes {}=1 are predicted relations, 5 {}=1 are gold relations, and every edge between a node and a node means they are approximately matched. At first glance, one may think that = = 0 because every gold relation has at least a match and so does every predicted relation. However, it is impossible to ifnd 5 true positive pairs from this graph without using one node more than once. Therefore, if = = 0, then + ̸= # = 5 and + ̸= # = 5, which is inconsistent with the formulas in exact setting.
So, instead of defining as the number of gold relations that don’t have a match, we just define = # − . Similarly is defined as # − . Then the problem remained is how to calculate . Actually, finding true positive pairs can be considered as bipartite matching. Figure 4b shows a matching with = 3 but is not optimal. Figure 4c shows one possible maximum bipartite matching with = 4. Another optimal matching is replacing edge 0 − 0 with 0 − 1.
In summary, we define as the maximum bipartite matching for the graph constructed by all approximately matched pairs, then = # − and = # − , finally Precision = /( + ), Recall = /( + ), and 1 = 2/(1/ + 1/). This has been implemented in the latest BRATEval.
We consider two types of coreference linking, i.e. (1) surface coreference linking and (2) atomic coreference linking, due to the existence of transitive coreference relationships. By transitive coreference relationships we mean multi-hop coreference such as a link from an expression T1 to T3 via an intermediate expression T2, viz., “T1→T2→T3”. Surface coreference linking will restrict attention to one-hop relationships, viz., to: “T1→T2” and “T2→T3”. Whereas atomic coreference linking will tackle coreference between an anaphoric expression and its first antecedent, i.e. intermediate antecedents will be collapsed. Thus, these two links will be used for the above example, “T1→T3” and “T2→T3”. Note that we only consider transitive linking in coreference relationships.
Note that {T1→T2,T2→T3} infers {T1→T3,T2→T3}, but the reverse is not true. This leads to a problem about how to score a prediction {T1→T3,T2→T3}, when the gold relation is {T1→T2,T2→T3}. Both T1→T3 and T2→T3 are true, but some information is missing here.
Our solution is to first expand both the prediction set and gold set where all valid relations that can be inferred will be generated and added to the set, and then to evaluate the two sets normally. In the above example, the gold set will be expanded to {T1→T2,T2→T3,T1→T3}, and then the result is = 2, = 1. Likewise, when evaluate {T1→T4,T2→T4,T3→T4} against {T1→T2,T2→T3,T3→T4}, the gold set will be expanded into 6 relations, while the prediction set won’t be expanded as no new relation can be inferred. So the evaluation result will be = 3, = 3. One may worry that if there is a chain of length then its expanded set will be in (2), when is large, this local evaluation result will have too much influence on the overall result. But we find in practice that coreference chains are relatively short, with 3 or 4 being the most typical lengths, so it is unlikely to be a big issue.
Our baseline model adopts an end-to-end architecture for coreference resolution [
We released the code for training our baseline models to help the participants to get started
on the shared task.6 Two variants of the baseline model are evaluated on the test set, one using
the ELMO embeddings as input to the BiLSTM component, while the other used pretrained
ChELMO, based on the embeddings of [
A total of 19 teams registered on our submission website for the shared task. Among them, we ifnally received 2 submissions on the test set. One team is from Carnegie Mellon University, US (CMU) and the other one is from Hokkaido University, Japan (HUKB). More details about their systems are provided in Section 7. In this section, we report their results along with the performance of our two baseline systems.
We report the overall performance of all runs in Table 2. The rankings of diferent systems are fully consistent across all metrics. The CMU team achieves an 1 score of 0.7847 in exact matching, outperforming our two baselines which get 0.7633 and 0.7453, followed by the HUKB team who obtains 0.6907. The lead of the CMU team is even larger in relaxed matching, with an 1 score of 0.8723, about 7 points higher than our baselines. This shows the potential of the CMU model and indicates that the performance in exact matching may be further boosted if the boundary errors of their model could be corrected in a post-processing step.
Our baselines have higher precision in the exact setting and precision in relaxed setting is also very close to the best, which indicates that our models are more conservative and could possibly be enhanced by making more aggressive predictions to improve recall. The use of domain-pretrained embeddings (ChELMO vs. ELMO) does, as expected, benefit performance.
Table 3 provides more details about the performance of all models for each relation type. The CMU team outperforms others on TRANSFORMED relation by a large margin. While our baselines performs the best on CONTAINED relation type. For the other three relation types, the CMU model wins 1 score and recall, while our models achieve the highest precision, which is similar to our observation on the overall results. Given that the models perform very diferently, it would be very interesting to do more analysis when the details of all the models
are disclosed, and hopefully every team can borrow ideas from others and further improve the performance.
We received paper submissions from both the participating teams, i.e. the HUKB team and the CMU team. We first describe their approaches, then summarize the same and diferent aspects of them. 7.1. HUKB The HUKB team used a two-step approach for the anaphora resolution task, where mentions including both antecedent and anaphor are first detected, then mentions are classified into diferent types and relations between them are determined. For step one, they found that although existing parsers such as Chemical Tagger can generate useful features, they are not enough for mention detection. Therefore they trained a BioBERT model to find candidate mentions. This is done by treating mention detection as a NER task, where a BIOHD format is used to convert gold spans into sequence of labels. The BIOHD format is an extension to the well-known BIO format to support discontinuous spans which exist in this amphora resolution task. In the second step, diferent types of relations are determined based on diferent rules. COREFERENCE is first detected by 5 regular expression rules. The remaining relations are detected based on the features generated by ChemicalTagger. When no more relations can be found, a post-processing step is carried out to handle the transitivity property of COREFERENCE relations, i.e. enumerating all valid COREFERENCE relations based on the transitivity property. 7.2. CMU The CMU team proposed a pipelined system for anaphora resolution, where mentions are first extracted and then relations between them are determined. The first step is done by a BERT-CRF model which is trained using BIO tagging. In the second step, for each pair of mentions, the sequence of sentences that contain the pair is fed to a BERT model to obtain the encoded representation of every token, then the representation of a mention is simply the mean of all tokens in it, and finally the representations of two mentions are concatenated and classified into 6 classes using a linear layer (5 relations + a class for no relation). To correct boundary errors in mention detection, a rule-based post processing is done between step 1 and 2. Furthermore, ensembling of 5 models are used in both step 1 and 2 to improve performance.
Both of them adopted a two-step approach where mentions are first detected and then relations between them are determined. They also both relied on BERT-like models to extract contextualized representations for mention detection. While the CMU team used a BERT-like model in the relation extraction, the HUKB team chose a rule-based method. In addition, the CMU team used ensembling of 5 models with majority voting in both mention detection and relation extraction, as well as a post-processing step between them to correct potential boundary errors in mention detection, where both techniques contribute to their superior overall performance.
This paper presents a general overview of the activities and outcomes of the ChEMU 2021 evaluation lab. As the second instance of our ChEMU lab series, ChEMU 2021 targets two new tasks focusing on reference resolution in chemical patents. Our first task aims at identification of reference relationships between chemical reaction descriptions, and our second task aims at identification of reference relationships between expressions in chemical reactions. The evaluation result includes diferent approaches to tackling the shared task, with one submission clearly outperforming our baseline methods. We look forward to fruitful discussion and deeper understanding of the methodological details of these submissions at the workshop.
Funding for the ChEMU project is provided by an Australian Research Council Linkage Project, project number LP160101469, and Elsevier. We acknowledge the support of our ChEMU-Ref annotators, Dr. Sacha Novakovic and Colleen Hui Shiuan Yeow at the University of Melbourne, and the annotation teams supporting the reaction reference task annotation.