This paper describes our team's participation in the Tracks 1 & 2 from Conference and Labs of the Evaluation Forum (CLEF 2020) Challenge organized by Cheminformatics Elsevier Melbourne University for extracting information over chemical reactions from patents. We discuss our systems: MedaCy, a python-based supervised multi-class entity recognition system, and RelEx, a python-based relation extraction system which includes rule-based and supervised learning pipelines. Our best model for Task 1 obtained an overall relaxed precision of 0.95 and exact precision of 0.87; relaxed recall of 0.99 and exact recall of 0.86; and relaxed F1 score of 0.97 and exact F1 score of 0.87. Our best model for Task 2 obtained an overall precision of 0.80; recall of 0.54; and F1 score of 0.65.
Chemical Patents are a primary source for information about novel chemicals
and chemical reactions. With the increasing volume of such patents, the
dissemination of information about these chemicals and chemical reactions has become
even more labor and time intensive. This information can be used to discover
new chemicals and synthetic pathways[
The process of extracting relevant information from chemical patents has
been referred to as chemical reaction detection [
The CLEF 2020 ChEMU [
In this paper, we describe our participation in the CLEF 2020 ChEMU Task 1 and Task 2 Challenge. For this challenge, we used our python framework MedaCy 1 to automatically identify the experimental parameters associated with the reaction including the trigger words used to link the parameters; and RelEx 2 to automatically link the trigger words with the experimental parameters to provide the sequence of steps within the reaction. MedaCy contains a number of supervised multi-label sequence classi cation algorithms for NER. RelEx contains a rule-based and supervised learning-based algorithms to identify relations between entities. Our best models for Task 1 obtained an overall relaxed precision of 0.95 and exact precision of 0.87; relaxed recall of 0.99 and exact recall of 0.86; and relaxed F1 score of 0.97 and exact F1 score of 0.87. Our best model for Task 2 obtained an overall precision of 0.80; recall of 0.54; and F1 score of 0.65. 1 https://github.com/NLPatVCU/medaCy/ 2 https://github.com/NLPatVCU/RelEx/tree/CLEF 2020 includes 10 di erent entity labels described as shown in the Table 1. The ARG1 event label corresponds to relations between a trigger word (REACTION STEP, WORKUP) and chemical compound entities. Table 2 shows the event statistics of the training dataset. The ARGM event label corresponds to the relations between a trigger word and temperature, time, or yield entities.
TEMPERATURE
Triggers RWEOARCKTUIOPN STEP
To identify the experimental parameters and triggers from the data, we use
MedaCy's bidirectional Long Short Term Memory (LSTM) units with a
Conditional Random Field (CRF) output layer implemented in PyTorch [
The input to our biLSTM+CRF model in this work is pre-trained word
embeddings [
The word2vec [
In this work, for each Trigger word-Entity pair we perform a binary classi cation to identify whether there is a relation between the trigger word and the entity or not. First, we identify and extract the sentence where a Trigger word-Entity pair pair lies and based on where the text spans are located in the sentence, we divide the sentence into segments as follows: { preceding - tokenized words before the rst concept { concept 1 - tokenized words in the rst concept { middle - tokenized words between the 2 concepts { concept 2 - tokenized words in the second concept { succeeding - tokenized words after the second concept
Figure 2 shows an abstract view of the construction of the CNN-based model. A segment is represented by a matrix of k N where k is the dimension of the word embeddings and N is the number of words in a segment. In this work, we use ChemPatent pre-trained word embeddings. We construct separate convolution units for each segment and concatenate before the xed-length vector is fed to the dense layer that performs the classi cation. Each convolution unit applies a sliding window that processes the segment and feeds the output to the max-pooling layer to extract important features independent of their location. The output features of the max-pooling layer of each segment are then attened and concatenated into a vector before feeding it into the fully connected feedforward layer. The vector is nally fed into a softmax layer to perform the binary classi cation whether the relationship exists or not. 3.3
Word Embeddings . We explore two pre-trained word embeddings: 1) ChemPatent
embeddings [
MedaCy . We used PyTorch [
For Tasks 1 and 2, we report the precision, recall, and F1 scores. Precision is the ratio between correctly predicted mentions over the total set of predicted mentions for a speci c entity; recall is the ratio of correctly predicted mentions over the actual number of mentions, and F1 is the harmonic mean between precision and recall. For Task 1, we report both the exact and relaxed results for each entity category. In exact evaluation, two annotations are equal only if they have the same tag with exactly matching spans. With the relaxed evaluation, two annotations are equal if they share the same tag and their spans overlap with each other.
In this section, we discuss the results for Task 1 and 2. 4.1
Results. Tables 3 - 5 show the exact and relaxed precision, recall, and F1
scores obtained over the testing set for identifying the named entities in each of
our three runs. Run 1 model was trained over the training data using the
biLSTM+CRF with the CheMU Patent embeddings; run 2 model was trained over
the training data using the biLSTM+CRF with the WikiPubmed embeddings;
and run 3 model was trained over the training and development data combined
with the biLSTM+CRF using the WikiPubmed embeddings. Table 6 shows the
baseline results using the CRF-based NER system BANNER [
Although the exact results for the models performed slightly worse than baseline, each of the models performed better on the relaxed results, with the model trained over patent embeddings performing best. This discrepancy may be due to the way that MedaCy handles entity classi cation. Within MedaCy, each Entity EXAMPLE LABEL OTHER COMPOUND REACTION PRODUCT REAGENT CATALYST SOLVENT STARTING MATERIAL TEMPERATURE TIME YIELD OTHER YIELD PERCENT System Entity EXAMPLE LABEL OTHER COMPOUND REACTION PRODUCT REAGENT CATALYST SOLVENT STARTING MATERIAL TEMPERATURE TIME YIELD OTHER YIELD PERCENT System trained over training data with WikiPubmed embeddings trained over training and development data with WikiPubmed embeddings individual token is given its own label (`O' for unlabelled entities), so for entities with spans long than one token, the entity may have only been partially labelled. For instance, in many cases of the TEMPERATURE label, MedaCy labeled `C' or ` C,' excluding the number preceding the temperature symbol. This may also account for why each model performed poorly for the TEMPERATURE label when evaluating in exact mode, but performed well when evaluating in relaxed mode.
Error Analysis. Confusion matrices for the three runs over the testing dataset are shown in Figures 3 - 5. Rows in the matrix represent annotated entities and columns represent predicted entities. For instance, in 3, YIELD OTHER (Y.O) was misidenti ed as YIELD PERCENT (Y.P.) 28 times. Table 7 shows the acronym of each of the labels used in the confusion matrices. The colors in the matrix indicate the density of the entities and the system annotations. The bottom right corner of each matrix is darker because of the large number of OTHER COMPOUND (O.C) entities in the dataset.
The majority of mislabeling occurred when more speci c entity labels, such as STARTING MATERIAL (S.M.), REAGENT CATALYST (R.C.), or REACTION PRODUCT (R.P.), were predicted to be OTHER COMPOUND (O.C.). This may be because the models were able to predict that certain spans contained chemical named, but were too general and unable to predict the speci c label. Additionally, spans annotated as OTHER COMPOUND (O.C.) were consistently predicted to be more speci c types of compounds. It seems that while the models are able to predict which spans contain chemical compounds, they are less able to distinguish between the types of compounds. 4.2
Results. Tables 8 - 10 show the exact match precision, recall, and F1 scores obtained over the testing set for each of our three runs. Run 1 used our RelEx's CNN-based system trained over the ChemPatent embeddings with the trigger words identi ed using medaCy's biLSTM+CRF trained over the ChemPatent embeddings. Run 2 used our RelEx's rule-based system with the trigger words identi ed using medaCy's biLSTM+CRF trained with ChemPatent embeddings. Run 3 used our rule-based system with the trigger words identi ed using medaCy's biLSTM+CRF trained with WikiPubmed embeddings. Table 11 shows the comparison with the co-occurrence baseline provided by the organizers of the ChEMU challenge and the overall results from each of our runs.
The overall results show that all three runs obtain a higher precision and F1 score than the baseline but not recall. The system results show that the CNN-based (Run 1) model obtains a higher overall F1 score than both the rulebased (Run 2 & 3) models. When training with CNN the overall precision of the predictions is high but the recall is low, this shows that CNN failed to classify all instances but was able to classify most of the predicted instances correct. Also, we can see the performance of each event class (Trigger wordEntity pair) in Run 1 is proportional to the number of instances in the training set. For example, event classes, REACTION STEP-REAGENT CATALYST tem with trigger words identi ed using medaCy trained with CheMU patent embeddings tem with trigger words identi ed using medaCy trained with CheMU patent embeddings WORKUP WORKUP
Entity OTHER COMPOUND REACTION PRODUCT REAGENT CATALYST SOLVENT STARTING MATERIAL OTHER COMPOUND REACTION PRODUCT SOLVENT STARTING MATERIAL TEMPERATURE TIME YIELD OTHER YIELD PERCENT TEMPERATURE
TIME
System WORKUP WORKUP
Entity OTHER COMPOUND REACTION PRODUCT REAGENT CATALYST SOLVENT STARTING MATERIAL OTHER COMPOUND REACTION PRODUCT REAGENT CATALYST SOLVENT STARTING MATERIAL TEMPERATURE TIME YIELD OTHER YIELD PERCENT TEMPERATURE TIME YIELD OTHER
YIELD PERCENT and REACTION STEP-STARTING MATERIAL, have more training instances and obtain a high F1 score, whereas the event classes, WORKUP-SOLVENT and WORKUP-STARTING MATERIAL, have a very few instances and obtain an F1 score of zero.
The rule-based models (Run 2 & 3) obtain comparatively high recall and low precision. The rule-based methods predicts all the closest occurrences of the trigger words of the entity compounds in the traversal area, however many WORKUP WORKUP
Entity OTHER COMPOUND REACTION PRODUCT REAGENT CATALYST SOLVENT STARTING MATERIAL OTHER COMPOUND REACTION PRODUCT REAGENT CATALYST SOLVENT STARTING MATERIAL TEMPERATURE TIME YIELD OTHER YIELD PERCENT TEMPERATURE TIME YIELD OTHER
YIELD PERCENT System predictions are false positives. Since the number of instances in the training set does not a ect the rule-based methods, the performance of the event classes that have few instances performs better. For example, the event classes, WORKUPTIME and REACTION STEP-OTHER COMPOUND, obtained zero F1 score with CNN-based model but performed better with the rule-based models obtaining F1 scores of 0.43 and 0.88, respectively.
Table 12 shows the arithmetic mean and weighted arithmetic mean of the precision, recall, and F1 score for both trigger word classes for each run. Bold terms indicate the best performance for each trigger word. We can see the CNNbased method (Run 1) performs well with the REACTION STEP classes and poor with WORKUP classes. This is because most of the REACTION STEP classes have more instances for the CNN to train on but most of the WORKUP classes have few instances. This is the same reason the rule-based methods (Run 2 & 3) perform better with those classes. The weighted arithmetic mean results contradict with the arithmetic mean results, as we can see a notable di erence in the F1 score when comparing the classes of REACTION STEP and WORKUP. The WORKUP event class obtains a better performance due to the signi cant imbalance between the individual event classes. The weighted arithmetic mean WORKUP
Entity Run 1 Run 2 Run 3 Run 1 Run 2 Run 3 allocates more weight to the classes that have more instances and vice versa we see an improvement in the performance of both classes.
Error Analysis. Tables 13 and 14 show a detailed error analysis of the CNNbased (Run 1) and the rule-based method (Run 2) respectively where the trigger words are trained with ChemPatent embeddings. Here we report the number of true positives (tp), false positives (fp), and false negatives (fn) and also "fpm" and "fnm", two metrics that represent the number of false positives and false negatives, of which the corresponding entities are missing.
The results are consistent with the previous observations from the tables 8, 9 and 10. We can see REACTION STEP classes performed better than the WORKUP classes. It is safe to say that, class imbalance plays a signi cant role in the miss-annotation of the instances. The results also show that the rulebased model signi cantly over annotates given the number of false positives. For example, the rule-based model (Run 2) identi ed 379 instances of the WORKUPREACTION PRODUCT event class with only four being true positives. WORKUP WORKUP
System
Entity OTHER COMPOUND REACTION PRODUCT REAGENT CATALYST SOLVENT STARTING MATERIAL OTHER COMPOUND REACTION PRODUCT REAGENT CATALYST SOLVENT STARTING MATERIAL TEMPERATURE TIME YIELD OTHER YIELD PERCENT TEMPERATURE TIME YIELD OTHER
YIELD PERCENT WORKUP WORKUP
System 5
Entity OTHER COMPOUND REACTION PRODUCT REAGENT CATALYST SOLVENT STARTING MATERIAL OTHER COMPOUND REACTION PRODUCT REAGENT CATALYST SOLVENT STARTING MATERIAL TEMPERATURE TIME YIELD OTHER YIELD PERCENT TEMPERATURE TIME YIELD OTHER YIELD PERCENT We trained three biLSTM+CRF models over di erent pre-trained word embeddings, as well as di erently sized datasets. Results show that while these models did not outperform the baseline model when evaluating exact span matches, the models outperformed the baseline when evaluating in relaxed mode. A model trained using word embeddings trained over chemical patents performed best when evaluating in relaxed mode, while a model trained using biomedical word embeddings and a combination of the training and development datasets performed best when evaluated on exact span matches. Errors primarily occurred because of issues with the model distinguishing between di erent entity labels, such as models mislabeling entities annotated as OTHER COMPOUND for more speci c labels, like REACTION PRODUCT or STARTING MATERIAL. Additionally, the way that MedaCy predicts entity labels may have contributed to errors with labeling entity spans fully. Future work will focus on better distinguishing between di erent types of chemical compounds, as well as looking into models based on language models.
We used one CNN-based model and two rule-based models to extract events and according to the results, all three models outperformed the baseline model. Results show that the CNN-based method outperforms the rule-based methods, especially with the REACTION STEP classes as those classes have more instances to train on. Meanwhile, as the rule-based methods do not require training instances to train they perform better with WORKUP classes. In the future, we plan to explore building a hybrid model with both CNN and rule-based methods to increase the performance.