patents. In this paper, we propose a new set of annotations for the ChEMU Chemical Reaction Corpus. Our annotations (ChemProp) non-trivially incorporate the signals from ChEMU 2020 and 2021 schema to extract the instructional structure from chemical patents with details about inputs, outputs, and reaction attributes for each event in the reaction snippet. We propose a semi-automatic algorithm to create ChemProp and benchmark state-of-the-art models proposed for ChEMU 2020 and ChEMU 2021 on it. We hope that ChemProp can play an important part in modeling the instructional language present in chemical
CEUR ceur-ws.org
chemical processes and synthesis, which are often
deChemical research relies heavily on the knowledge of puts) of the underlying chemical patent. For example,
theentire instructional structure (e.g. a chronological
sequence of inputs, reaction-steps, conditions, and
outeven though ChEMU 2020 schema tries to relate
reacabout new compounds [
CLEF ChEMU shared-task series released the ChEMU work for converting natural language chemical patents Chemical Reaction Corpus that contains reaction snip-into structured recipes. Instructional language is a usepets extracted from chemical patents.
For ChEMUful structure that comprises of step-by-step instructions
duces five domain-specific relations (including bridging
2020 [
None of these shared tasks however fully capture (entities) that depict the (i) INPUT, (ii) OUTPUT, and (iii) tasks. nEvelop-O LGOBE (C. Rose) htp:/ceur-ws.org ISN1613-073
Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop Proceedings (CEUR-WS.org) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License annotate the trigger event verb, and the noun phrases reaction-attributes (RXN_ATTR) of that reaction step.
by (i) automatically combining the annotations of CLEF
incorporating events/entities that are missed by the two
annotation schemes. 2. Furthermore, CC20 does not capture relationships
Furthermore, we show the significance of these aug- between reaction steps (events) and noun-phrase mentations by evaluating the performance of the best mentions that denote combinations/ mixtures performing models on ChEMU 2020 and 2021 shared- (e.g., the reaction mixture) or coreferent exprestasks on ChemProp. Our experiments show that models sions (e.g., the product). trained on ChemProp training data only achieve 0.69 Micro-F1 points on the test data, thus highlighting thCeLEF ChEMU 2021 Annotation Schema (CC21). room for improvement. We also show that ChemProp Next year, He et al[.4] proposed an additional layer of contains novel entities and relationships that are noatnnotation to the patents corpus, which focuses on the present in ChEMU shared-tasks thus making it beneficial identification of anaphoric references. The new corpus as a standalone benchmark for instructional language contains annotations for both COREFERENCE and bridgmodeling from chemical patents. ing relations (Figure2). The authors define four domainspecific sub-types for bridging: TRANSFORMED, RE2. Prior Art ACTION_ASSOCIATED, WORK_UP, CONTAINED. As a standalone schema, CC21 sufers from the following In this section, we brielfy describe the ChEMU Chemical issues: Reaction Corpus and the two state-of-the-art annotation 1. CC21 does not contain explicit information about schemes proposed during ChEMU shared-tasks ’20 & ’21. reaction steps. Therefore, it is less useful, in isolation, for information extraction from chemical patents. 2. Furthermore, CC21 difers from CC20 on its definition ofmentions and therefore makes the combination of two annotations non-trivial. In the next section, we describe the algorithm to handle these ambiguities.
He et al.[
Their annotation schema aims at extraction of chemical reactions (i.e. REACTION_STEP, WORKUP) from patent snippets. It identifies trigger words that describe reaction steps and relates them to named-entities linked to the step (i.e. chemical compounds, time, temperature, and3. ChemProp: Annotation yields; Figure 1). Despite being a comprehensive annotation schema, CC20 sufers from two major drawbacks: CC20 contains relationships between events and named1. CC20 does not annotate reaction steps that deontities, whereas CC21 connects noun phrases (including not relate to any named-entity in the discoursenamed-entities) based on their anaphoric relationships. snippet. E.g., as shown in Figure1, CC20 does Together, CC20 and CC21 provide somewhat complenot annotate the evencotncentrated (in line 7). mentary information about each reaction snippet in the
ChEMU corpus. In this section, we describe the steps we3.1.1. Pre-processing take to merge these somewhat heterogeneous annotation schema to create our new dataset ChemProp. As a pre-processing step, we setup data-structures that help with the conversion algorithm. We create First, we present our algorithm that automatically merges signals from CC20 and CC21 and extracts lexical spans in the discourse, that most closely represent the inputs, outputs, & reaction-attributes for each reaction step.
to CC21 named-entity annotations to tackle the small annotation diferences between the two schemas. The map is many-to-one because CC21 annotates entire solutions, whereas CC20 keeps the individual compounds/elements, e.g., [a solution ofethanol 20 and water 20 ] 21 .
ethanol20 wate r20
↔ a solution of ethanol and wat e2r1 ↔ a solution of ethanol and wat e2r1 .
oriented relations (i.e., REACTION_ASSOCIATED (R_ASSOC), WORK_UP, and TRANSFORMED (TRANS)) to their corresponding ARG2 mentions. For example, for the relation TRANSFORMED betweenThe reaction mixture and The reaction in Figure 2 (line 4, 5), we store The reaction mixture21
Rxn:TRANS −−−−−−−−−→ The reactio n21 .
1. If ARG2 21 is also present in the CC20 annotation, we use thisARG2 21 as a pivot to combine the two annotations. We extract the CC20 relation (Rxn 20 ) it is linked with. 2. If ARG2 21 is not present in CC20, it means that
ARG2 21 is a non named-entity noun-phrase and therefore needs to be resolved further. To ground such cases, we rely on the fact that the starting compounds in each reaction snippet starts are named-entities. Therefore, we can assume that in order to reach currenAtRG1 21 , all otherARG2 21 s that also appeared asARG1 21 (noun-phrases) have been resolved in earlier iterations of this algorithm. Therefore, the latestRxn 20 in the patent snippet between the CC20 reaction event associated witAhRG2 21 (i.e. ARG2 21 →− Rxn 20 ) and ARG1 21 is returned. 3. The mapping from CC20 events (ARG1) to their From this list of relations, we consider thRexn 20 that corresponding argument mentions (ARG2). For is closest toARG1 21 , but occurs before it, to be the example, for the eventstirred in Figure 1 (line 4), lexical event trigger that outputs AR G211 : we store stirre d20 stirre d20
Event:TEMP ←−−−−−−−−→ r.t.20 Event:TIME ←−−−−−−−−→ 2 h 20 .
Cand_Rxn 20 Rxn 20 ARG1 21 = [Rxn 20 , Rxn220 , ..., Rxn 20 , ...]
1 = _
(ARG1, Cand_Rxn 20 ) Output:Rxn ←−−−−−−−→ Rxn 20 .
relationships between diferent CC21 mentions. Consider the case whereARG1 21 is The combined orWe create a separate map for coreference as they ganic phases (Figure 2; line 6). It is related to four mendenote that both mentions in the pair are equivt-ions (ARG2 21 s) {The reaction mixture, a pad of celite, alent i.e., point to the same underlying entity inwater, ethyl acetate} by the relation WORK_UPR( xn 21 ). the discourse: Three of these mentions{a pad of celite, water, ethyl acetate} are also present in the mapping created in preARG1 21 ←C−o−r−→ef ARG2 21 . processing step 1, whereas{The reaction mixture} is not.
As discussed earlier, forARG2 21 s present in CC20, we The above mappings store the relationships annotated infirst create a list ofRxn 20 s (Cand_Rxn 20 ) they corCC20 and CC21 that are relevant to creating ChemPropr.espond to{filtered, diluted, extracted} (Figure 1). For The reaction mixture, the correspondingRxn 20 , based on 3.1.2. Algorithm its resolution in the previous step,Tishe reaction mixture ←→ stirred.
We allow for three relationships between reaction events The latest CC20 relation betweenstirred and and entities in our new ChemProp benchmark – IN- ARG2 21 , extracted, is then inserted intCoand_Rxn 20 . PUT, OUTPUT, RXN_ATTR. CC20 already annotates Combining the four relations we ge{textracted, filtered, RXN_ATTRs (reaction-attributes) and named-entity IN
diluted, extracted}. From these,extracted occurs closest PUTS/ OUTPUTS. Therefore, to complete the schema, to the current ARG1 while occurring before it. Therewe devise an algorithm that can find a mapping between fore, The combined organic phases is considered to be the each non named-entity noun-phrase in CC21 to a reac- OUTPUT of extracted (The combined organic phases ←→ tion event in CC20 (AR G21 ←→ Rxn 20 ). extracted; Figure 3).
For each CC21 entity A(RG1 21 ) that is related to other CC21 entitiesA(RG2 21 ) (that occur before it Exceptional Cases. Although, most of the events can in the discourse) via one of the three reaction-orientedbe fully annotated in ChemProp format using the above CC21 relations (Rxn 21 ) i.e., REACTION_ASSOCIATED, WORK_UP, and TRANSFORMED, two possibilities need steps, there are certain exceptions that arise due to the mismatch between the motivation of CC20 and CC21 to be considered. For each ARG 221 in ARG2 21 : schemas. about the chemical patent snippets that will be systemat3.2. Manual Quality Assurance ically missed by the systems trained on CC20. Next, we manually go through the development and test We observe thatCC21_best (gold entities) achieves data to fix the exceptions (described in the previous sec- an F1 score of 0.86 on ChemProp test set. This suggests tion). that the model is able to somewhat reliably figure out
As discussed earlier, the CC20 annotation does notwhich named-entities/ noun phrases are related to which annotate reaction steps that do not relate to any namerdea-ction event in the snippetC.C21_best (end-to-end), in entity in the snippet. However, in our case such eventsaddition to relation classification, also extracts mentions are equally relevant and need to be extracted to get farom raw patent snippets, and therefore expectedly per
Trained on ChemProp CC21_best (gold-entities) CC21_best (end-to-end) 0.74 0.62 forms much worse thanCC21_best (gold entities) with an overall F1 score of 0.69.
In this work, we propose a new corpus ChemProp that non-trivially combines properties from ChEMU 2020 and 2021 annotation schema to extract the instructional structure from chemical patents. We provide a semi-automatic algorithm to create ChemProp. Evaluating state-of-theart models on the our new dataset suggests that there is still room for improvement in extracting relevant instructional triggers from patent text. We believe that ChemProp can act as an important benchmark for instructional language modeling.