Many new chemical compounds are reported each year in patent documents, leading to increasing demand for methods for automatic information extraction of chemical compounds and reactions from patents. Chemical patents often detail a number of similar compounds that have a common substructure and can be synthesized in analogous ways, and therefore contain many references connecting descriptions of similar chemical reactions, to avoid redundancy in describing common reaction conditions. This leads to the problem of reaction reference resolution, where, given a reaction description, we need to identify links to other reaction descriptions it refers to. In this paper, we formally introduce the task and propose baseline methods to address it in analogy with co-reference resolution. To evaluate the performance, we create a large-scale silver-standard dataset based on a commercial database of chemical reactions. The experimental results show that the approach based on a state-ofthe-art co-reference resolution method struggles to outperform a simple heuristic in detecting reference links, demonstrating the difficulty of the proposed task and its fundamentally different nature to co-reference resolution.
• Computing methodologies → Information extraction. information extraction, reaction reference resolution, natural language processing *Also with The University of Melbourne. †Also with The University of Melbourne. ‡Also with The University of Melbourne. 1
Patents represent a critical source of information about new
chemical compounds, and lead journal publications in both volume and
time [
A text mining system for chemical reaction information extraction
should aim to obtain details of each reaction, requiring: (1) the
identification of specific spans of text describing reactions; and (2)
the extraction of the compounds participating in, and the conditions
of, a reaction. The second step involves named entity recognition [
The first step of locating all information relevant to a chemical
reaction has received relatively little attention. This step is a challenge
as it requires processing very long and semantically unstructured
documents, in which a number of similar chemical reactions are
presented. Yoshikawa et al. [
To fill this gap in previous work and take a step towards an
end-to-end method for chemical reaction information extraction,
this paper focuses on the task of reaction reference resolution. Our
contributions are three-fold:
• We define the new task of reaction reference resolution, where,
given a reaction description, we aim to identify other reaction
descriptions it refers to.
• In the absence of a large-scale gold-standard dataset, we
leverage Reaxys®, a large commercial chemical reaction database,
to create a silver-standard dataset, of a size that supports the
development of not only traditional rule-based methods but
also neural network models.
• We propose several baseline methods for the task and
evaluate them on our silver-standard dataset. The baselines
include an extension of the state-of-the-art end-to-end neural
co-reference resolution method of Lee et al. [
In this section, we formally introduce the reaction reference
resolution task. We assume a pipeline system where, given the Description
section of a patent, text spans corresponding to individual reaction
descriptions are extracted in advance (following the methods of
Jessop et al. [
An input document is represented by a tuple = (, ), where is a sequence of paragraphs = (1, . . . , | | ) and is a sequence of all textual spans of reaction descriptions = (1, . . . , | | ), dubbed reaction spans. A reaction span is a sequence of consecutive paragraphs, represented by the first and last indices of paragraphs of the span = (start(), end()). For each reaction span , the task is to: (1) predict whether the reaction refers to one or more previous reaction descriptions; and if so, (2) detect the reaction span(s) (for < ) that refers to. Hereinafter, we call the parent reaction and the child reaction.
Figure 1 shows typical examples of reaction references. A reference relation is often indicated by an example ID, as in example (a), or the compound label used in the previous reaction description, as in example (b). However, it can also be the case that there is no direct referential expression between reaction descriptions, as in example (c). Therefore, we formulate the task as reference resolution between reaction descriptions (paragraphs) rather than between a referring expression such as Example 1 and its referent, as in traditional coreference resolution tasks. Subtasks (1) and (2) are analogous to the mention detection and mention clustering subtasks of co-reference resolution.
In this section, we introduce a baseline neural model that captures the
context and detects reference relations between reaction descriptions.
As illustrated in Figure 2, our model is based on the end-to-end
neural co-reference model of Lee et al. [
Our model has a hierarchical architecture that first encodes each paragraph and then encodes the entire document to obtain a contextualized representation of each paragraph. We first encode each input paragraph to obtain a context-independent paragraph encoding. A paragraph consists of a sequence of words = { 1() , . . . , () . | | } Each word is represented by: , = WE(() ) ⊕ ELMo( )
⊕ CC(() ) ⊕ NER( ) ,
where WE(() ) ∈ RWE is a pretrained word embedding, ELMo( ) ∈
RELMo is a contextualized word embedding [
= −−−,−|→| ⊕ ←−,−1, (2) −−→, = LSTM(, , −−−,−→−1; PF), ←−,− = LSTM(, , ,+ ←−−−−1; PB), where PF and PB denote model parameters of the forward and the backward LSTMs, respectively.
To obtain the representation of each reaction span, we first obtain a contextualized representation of each paragraph using a documentlevel bidirectional LSTM: ∗ = →−∗ ⊕ ∗,
←− →− ∗ = LSTM( , −−∗−→−1; DF), ←−∗ = LSTM( , ←∗−+−−1; DB), where DF and DB denote model parameters of the forward and the backward LSTMs, respectively. These paragraph representations are combined to obtain each reaction span representation:
= s∗tart() ⊕ e∗nd() . 3.2
Our goal is to identify the parent reaction span for each child reaction span. As described in Section 2, this can be further divided into two subtasks: predicting whether the given span is a child of some reaction span, and linking the child spans to their parent spans.
(1) (3) (4) (5) (6) (7) (8)
ID Text (a) Reference with the example ID RX1 1E,x1a-Dmipolxeo2-36,31-1d-iDbuiotyxlo--53-,p3h-ednibyul-t7y-l-m5e-pthhyelnthyilo-7-8-m-[eNth-(y(lRth)-io--8c-a(rNb-o{x(yR-)4--h-y[dNro-(x(ySb)-e1n-zcyalr)bocxayrb-(aRm)o-hyylmdreotxhyopxryo]p-y2l,)3c,a4r,b5a-mteotryalh]y-4d-r.o..-1,2,5-benzothiadiazepine (Example 18; 100mg, 0.152mmol) was dissolved in ...
RX2 ETxhaemtiptllee 2co7m1,p1o-Duniodxwo-a3s,3s-ydnibthuetysli-z5e-dphbeyntyhl-e7p-mroectehdyultrheiod-8e-s(cNri-b{e(Rd)i-nE-[xNa-m((Sp)le-12-6casrtbaortxiyn-g2f-r,ommet.h..ylpropyl)carbamoyl]-4(b) Reference with the compound label RX1 A mixture of the obtained ester, ... was stirred under argon and heated at 110◦ C. for 24 h. ... Column chromatography of the residue (silica gel-hexane/ethyl acetate, 9:1) gave Compound
B11, ...
RX2 .U..sing 2-ethoxyethanol and following the procedure for Compound B11 gave Compound B13, bis(2-ethoxyethyl) 3,3’-((2-(bromomethyl)-2-((3-((2ethoxyethoxy)carbonyl)phenoxy)methyl)propane-1,3-diyl)bis(oxy))dibenzoate, (3.82 g, 35% yield). ... (c) Reference with no direct referential expression RX1 III: 5-fluoro-N2-(4-methyl-3-propionylaminosulfonylphenyl)-N4-[4-(prop-2-ynyloxy)phenyl]-2,4- pyrimidinediamine mono-sodium salt 5-Fluoro-N2-(4-methyl-3-propionylaminosulfonylphenyl)-N4-[4-(prop-2-ynyloxy)phenyl]-2,4-pyrimidinediamine, II, (0.125 g, 0.258 mmol) was suspended in ...
The following compounds were made in a similar fashion to those above.
RX2 IV: 5-Fluoro-N2-[4-methyl-3-(N-propionylaminosulfonyl)phenyl]-N4-[4-(2-propynyloxy)phenyl]-2,4-pyrimidinediamine Potassium Salt ...
RX3 V: 5-Fluoro-N2-[4-methyl-3-(N-propionylaminosulfonyl)phenyl]-N4-[4-(2-propynyloxy)phenyl]-2,4-pyrimidinediamine Calcium Salt ...
To evaluate the difficulty of these individual subtasks, we propose several model variants that differ in training objectives and decoding methods. Specifically, we train our model using either a binary classification objective that only models whether a reaction span is child or not, or an end-to-end objective that also models which span the child refers to. In decoding, our model uses either a heuristic method or the learned end-to-end model to identify the parent reaction for each child reaction span.
Binary classification objective. The notation = 0 indicates the special case where the span does not refer to any preceding spans.
The binary classification objective models whether or not a reaction span is a child of at least one preceding reaction span:
NN () := Pr( ≠ 0 | ) = ( ()), (9) where denotes the sigmoid operator, and () is a span scoring function that computes the likelihood of the -th span being a child, using a feed-forward network:
() = C⊤-NN FFNN( ) + C-NN. (10)
Given the training dataset = (1, . . . , | | ), the model is trained to minimize the following binary cross-entropy loss:
LBIN = − Õ || Õ 1 ≠0 log NN () =(,)∈ =1 +1 =0 log(1 − NN ()) .
(11)
End-to-end objective. The end-to-end objective models not only
the likelihood of a span being a child, but also which span the child
span refers to. Analogous to Lee et al. [
| | Pr(1, . . . , | | | ) = Ö Pr( | ). (12)
=1 The probability of each preceding span being the parent of the given span is computed as follows: NN ( , ) := Pr( = | )
−1 =exp( ( , ))/Õ exp( ( ′, )),
′=0 where ( , ) denotes the score of the likelihood that span refers to preceding span . The score is computed using the pair of corresponding span representations: ( , ) = P⊤-NN , + P-NN,
, = FFNN( ⊕ ⊕ ( ◦ ) ⊕ Dist( , ))
for > 0, where ◦ denotes element-wise multiplication. Dist( , )
denotes the distance embedding, whose definition follows that of
Lee et al. [
Given the training dataset = (1, . . . , | | ), the model is trained to minimize the negative log-likelihood loss against the true parent spans, where ∗ denotes the set of all correct parent span indices:
LE2E = −
Õ Õ|| log Õ Pr( = | ). =(,)∈ =1 ∈∗
(16)
Note that our model is trained to predict only one parent for each child, although a child can have multiple parent spans. We observed that more than 90% of the child reaction spans in our dataset have only a single parent, and this simplification has very limited effect on performance. We can easily extend our model to allow multiple parent spans by modeling the link probability for each candidate parent span independently. Specifically, we can replace the softmax operation in Equation (13) with the sigmoid activation, and the loss function Equation (16) with the binary cross-entropy loss.
Decoding. To decode with the model trained with the end-toend objective, we use the maximum index (including 0, i.e. no reference) as the predicted parent index of a reaction span; we refer to this method as “E2E”. For the model trained with the binary classification objective (Equation (11)), we use a simple heuristic to predict the parent reaction spans, called “IMMPREV”: reaction spans that are classified as a child reaction span are linked to the immediately-preceding previous span that is not classified as a child span. 4
To the best of our knowledge, there is no publicly available dataset for reaction reference. Manually annotating a dataset of chemical reaction reference requires expert knowledge of chemistry and involves document-wise annotation of patents, with a single document often containing hundreds of paragraphs. As a more viable alternative, we create a silver-standard dataset from Reaxys®, a large-scale commercial chemical reaction database which contains a large quantity of reaction information associated at the document level with patents. While our dataset relies on a proprietary database, we believe that our methodology can generalize to similar patent silver standards generated using comparable reaction databases.
Reaxys® contains information of chemical reactions associated with their location information, i.e., the patents and the paragraphs where their reaction processes are described. To avoid exhaustive manual extraction of the full reaction details for all compounds described in a patent, construction process of Reaxys® database is streamlined through reuse of the conditions of a reaction for similar reactions described in the same patent. For this reason, the database includes internal reference links between similar reactions that share reaction conditions. As such, the mapping process from the database to our silver-standard dataset is straightforward: we simply map each reaction into one or more paragraph sequences (reaction spans) using 2 ( | |2), where | | < 50 for more than 90% of the dataset. # Documents # Paragraphs # Reaction spans # Child reaction spans # Tokens / Paragraph
143 39,437 3,072 549 73.7 the location information and then use the reference links to label corresponding pairs of reaction spans. There are some limitations due to the fact that the database is not originally intended to be used for document processing. For example, the link information is not always complete, especially when multiple references are involved.3 It is for this reason that we consider the annotations a silver-standard, with moderate recall and high precision.
We follow the work of Yoshikawa et al. [
Pattern matching baseline. We observe that there are high-precision patterns in child reactions that indicate their parent reaction spans. Namely, the parent reaction is often mentioned by its “example ID” such as Example 1, Preparation 1, or Step 1. An example is given in Figure 1(a). Based on this observation, we develop a rule-based baseline using regular expressions to detect such example ID patterns. For each reaction span that contains such a pattern, we search the preceding spans in reverse chronological order until we find a span that contains the example label in its heading or at the beginning of the text body.
Immediate previous baseline. Another naïve baseline is to link all reaction spans except the first, to the immediately previous span, i.e. = − 1. We label this method “NAIVE-IMMPREV”. We also include an oracle variant, “ORACLE-IMMPREV”, which uses the true labels for the subtask of child span detection (i.e. the oracle provides a perfect decision as to whether a reaction span is a child of other reaction spans or not), and performs parent–child linking by linking 3Reactions that refer to multiple reaction steps usually have a link to only the final step of the reaction sequence. par_id parent_id
text 117 118 each given child span to its immediately preceding non-child span, similarly to the IMMPREV decoding method from Section 3.2. By assuming the annotations for child spans as given, we can measure the complexity of the parent–child linking task separately from the child span detection task.
Bag-of-words baseline. We also implement a simple bag-of-words classifier. For each reaction span , we construct a vector of vocabulary size | |, where each element is the frequency of the corresponding term in the paragraph. We calculate the score of each span being a child as follows: BOW () := Pr( ≠ 0 | ) = ( C⊤-BOW + C-BOW), (17) where denotes the sigmoid function. For each span that is classified as a child, we calculate the score of every preceding span being the parent of as: BOW ( , ) := Pr( = | ≠ 0, )
−1 =exp(BOW ( , ))/Õ exp(BOW ( ′, )),
′=1 BOW ( , ) = P⊤-BOW ( ⊕ ⊕ , ) + P-BOW, where , is a one-hot vector of the words appearing in both and . The model is trained to optimize the following joint loss function, with an L2 regularization term:
Õ LBOW =
(bin (, ) + link (, )), bin (, ) = − link (, ) = − (,)∈ || Õ 1 ≠0 BOW () =1
+1 =0 (1 − BOW ()) , Õ log Õ BOW ( , ).
: ≠0 ∈∗
Each paragraph is tokenized using the OSCAR4 tokenizer [
Pattern matching
Bag-of-words
RREF BINARY
RREF E2E
30 trigram filters to generate 25d character embeddings. For the NER
features, we use a named entity recognizer based on the Reaxys®
Gold chemical tag set [
The models are trained using the Adam optimizer [
Child span detection. Table 2 shows the results of the baseline methods on child span detection, which is the performance in terms of binary classification predicting whether each reaction span is a child of another reaction span or not. The neural methods obtain high F-scores over 0.8, whereas the bag-of-words classifier shows only a small gain over the rule-based baseline. This suggests that this
End-to-end performance. Table 3 shows the performance on the end-to-end task. As our methods assume a single parent for each child span, we evaluate the performance with two different measures. As the primary measure, we calculate the precision (P), recall (R), and F-scores (F1) by regarding the prediction as correct if the model predicts one of the true parents of the target reaction span. We also report the recall (Rm) and F-score (F1m) based on the multiple reference scenario, in which, if the target reaction span has multiple true parents, the recall is calculated based on the total number of the parents in the denominator.4
For the end-to-end task, RREF BINARY-IMMPREV performs better than RREF E2E model, even though it is trained only with the binary classification objective and not trained on parent–child linking. The RREF E2E model struggles to detect parent–child links effectively, and achieves lower performance than the simple IMMPREV decoding. Considering the fact that the RREF E2E model performs comparably with RREF BINARY at child span detection, we hypothesize that even if the model is trained with the end-to-end objective, it primarily learns to detect child reaction spans from non-child ones, largely ignoring relations between other reaction spans.
These results reveal that it is the parent–child linking step rather than the child detection step that makes reaction reference resolution difficult. IMMPREV decoding is a strong baseline (0.706 F-score given oracle child reaction spans), but there is significant room for improvement. Despite its success in the traditional co-reference resolution task, the RREF E2E method struggles to learn the pairwise relations between reaction spans. One of the main differences between the traditional co-reference resolution task and the reaction reference resolution task is text length: the target spans are paragraph sequences in a long document rather than short word 4 P = #(correct refs) / #(predicted refs), R = #(correct refs) / #(spans with at least one true refs), F1 = 2P R / ( P + R), RF1mm == #F(1co=rr2ecPt Rremfs)//(#P(a+ll Rtrume)refs), sequences within a few sentences. A model that can more effectively learn to capture relations between long text spans is left for future work. 6.2
Examples of the system output are shown in Figure 4, which corresponds to the typical reference cases introduced in Figure 1, namely (a) reference with the example ID, (b) reference with the compound label, and (c) reference with no direct referential expression. Obviously, the pattern matching baseline fails when the target span does not mention the example label of its parent span. While the performance of ORACLE-IMMPREV and RREF E2E is less tied to specific reference cases, they can fail in many “easy” cases where there is an explicit mention of the parent example ID, indicating that it is difcfiult to learn such direct references without explicit training signal that forces the model to attend to particular patterns. The example in Figure 4 (c) is a special case that involves general conditions: the first paragraphs describe the general procedure to synthesize multiple similar compounds, and the examples of specific compound names are listed in subsequent paragraphs. We can see that RREF E2E fails to detect references to general conditions that ORACLE-IMMPREV detects successfully. In some cases, RREF E2E even fails to recognize them as a child reaction, which might be because reference to general conditions is likely to be implicit, i.e. the child description does not explicitly mention the example label of its parent description. 7
Krallinger et al. [
To our knowledge, the first study to take into account references in
reaction descriptions was by Ai et al. [
Example 24 ... ...
Example 26 ...
Example 27 ...
The title compound was synthesized by the procedure described in Example 26 starting from ... system is reported to produce results in the range of 80–90% for simple synthesis paragraphs, and 60–70% for complex paragraphs, details of their data set and evaluation procedures are not provided. It is unclear whether these results would generalise to larger data sets with more variability, and we cannot directly compare their approach with ours.
Finally, we compare our task formulation with text alignment
task. Text alignment task [
We have introduced the chemical reaction reference resolution problem, a largely unexplored yet critical step in information extraction of complete reaction details presented in chemical patents. We provided a formal specification of this task, built an evaluation corpus, and adapted a neural co-reference architecture to this task, as well as introducing several other baseline methods. Our neural method achieved promising results for detecting (child) reaction descriptions containing references, but identifying the precise (parent) reference of child spans is challenging for the model. Key issues for future work are more effective learning of reference patterns in reaction descriptions, and better handling of long text spans, as well as longdistance relations.
Some aspects of reaction reference are not well explored in this
paper. For example, we did not observe many examples involving
multiple references and references to general conditions in our
silverstandard dataset. This is partly because the database used to construct
our corpus is not designed for this use. Very recently, ChEMU [
This project is supported with funding to the ChEMU
(Cheminformatics Elsevier Melbourne University) project from the Australian
Research Council Linkage Project LP160101469, with partner
contributions from Elsevier. Some of the computations in this paper
were performed using the Spartan HPC-Cloud Hybrid [