Abstract. The discovery of new chemical compounds is perceived as a key driver of the chemistry industry and many other economic sectors. The information about the new discoveries are usually disclosed in scienti c literature and in particular, in chemical patents, since patents are often the rst venues where the new chemical compounds are publicized. Despite the signi cance of the information provided in chemical patents, extracting the information from patents is costly due to the large volume of existing patents and its drastic expansion rate. The Cheminformatics Elsevier Melbourne University (ChEMU) evaluation lab 2020, part of the Conference and Labs of the Evaluation Forum 2020 (CLEF2020), provides a platform to advance the state-of-the-arts in automatic information extraction systems over chemical patents. In particular, we focus on extracting synthesis process of new chemical compounds from chemical patents. Using the ChEMU corpus of 1500 \snippets" (text segments) sampled from 170 patent documents and annotated by chemical experts, we de ned two key information extraction tasks. Task 1 targets at chemical named entity recognition, i.e., the identi cation of chemical compounds and their speci c roles in chemical reactions. Task 2 targets at event extraction, i.e., the identi cation of reaction steps, relating the chemical compounds involved in a chemical reaction. In this paper, we provide an overview of our ChEMU2020 lab. Herein, we describe the resources created for the two tasks, the evaluation methodology adopted, and participants results. We also provide a brief summary of the methods employed by participants of this lab and the results obtained across 46 runs from 11 teams, nding that several submissions achieve substantially better results than the baseline methods prepared by the organizers. 1
Chemical patents represent as an indispensable source information about new
discoveries in chemistry. They are usually the rst venues where new chemical
compounds are disclosed [
Although the information in chemical patents are of signi cant research and
commercial value, extracting such information is nontrivial, since the large
volCopyright c 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0). CLEF 2020, 22-25
September 2020, Thessaloniki, Greece.
ume of existing patents and its drastic expansion rate has made manual
annotation costly and time-consuming [
The ChEMU (Cheminformatics Elsevier Melbourne University) lab aims to
provide a platform for worldwide experts in both NLP and chemistry to
develop automated information extraction methods over chemical patents, and
to advance the state-of-the-arts in this area. As a rst running of ChEMU, our
ChEMU2020 lab focuses on extraction of chemical reactions from patents [
In collaboration with chemical domain experts, we have prepared a highquality annotated data set of 1,500 segments of chemical patent texts speci cally targeting these two tasks. The 1,500 segments are sampled from 170 chemical patents, and each segment contains a meaningful chemical reaction. Annotations including entities and event steps are rstly prepared by three chemical experts and then merged to gold-standards.
The ChEMU2020 lab has received considerable interest, attracting 37 registrants from 13 countries including Portugal, Switzerland, Germany, India, Japan, United States, China, and United Kingdom. Speci cally, we received 26 runs (1 post-evaluation submission) from 11 teams in Task 1, 10 runs from 5 teams in Task 2, and 10 runs from 4 teams in the task of end-to-end systems (a pipeline combining Task 1 and 2), respectively. Several teams achieved exciting results, outperforming baseline models signi cantly. In particular, submissions from a team from the company Melax Technologies (from Houston, TX, USA) ranked rst in all 3 tasks.
The rest of the paper is structured as follows. We rst introduce the corpus
we created for use in the lab in Sect. 2. Then we give an overview of the tasks
and tracks in Sect. 3, and discuss the evaluation framework used in the lab in
Sect. 4. We present the overall evaluation results in Sect. 5 and introduce the
participants' approaches in Sect. 6, comparing them in Sect. 7. Conclusions are
presented in Sect. 8. Note that this paper is an extension of our previous overview
paper [
The annotated corpus prepared for the ChEMU shared task consists of 1,500 patent snippets (text segments) that were sampled from 170 English document patents from the European Patent O ce and the United States Patent and Trademark O ce. Each snippet contains a meaningful description of a chemical reaction [47].
The corpus was based on information captured in the Reaxys R database.1
This resource contains details of chemical reactions identi ed through a mostly
manual process of extracting key reaction details from sources including patents
and scienti c publications, dubbed \excerption" [
To prepare the gold-standard annotations for the extracted patent snippets,
multiple domain experts with rich expert knowledge in chemistry were invited to
assist with corpus annotation. A silver-standard annotation set was rst generated
by mapping the records from the Reaxys database back to the source patents
from which the records were originally extracted. This was done by scanning
the patent texts for mentions of relevant entities. Since the original records are
only linked to the IDs of source patents and do not provide the precise locations
of excerpted entities or event steps, these annotations needed to be manually
reviewed to produce higher-quality annotations. Two domain experts manually
and independently reviewed all patent snippets, correcting location information
of the annotations in silver-standard annotations and adding more annotations.
Their annotations were then evaluated by measuring their inter-annotator
agreement (IAA) [
We present an example of a patent snippet in Fig. 1. This snippet describes the synthesis of a particular chemical compound, named N-((5-(hydrazinecarbonyl) pyridin-2-yl)methyl)-1-methyl-N-phenylpiperidine-4-carboxamide. The synthesis process consists of an ordered sequence of reaction steps: (1) dissolving the chemical compound synthesized in step 3 and hydrazine monohydrate in ethanol; (2) heating the solution under re ux; (3) cooling the solution to room temperature; (4) concentrating the cooled mixture under reduced pressure; (5) puri cation of the concentrate by column chromatography; and (6) concentration of the puri ed product to get the title compound.
This shared task aims at extraction of chemical reactions from chemical patents, e.g., extracting the above synthesis steps given the patent snippet in Fig. 1. To achieve this goal, it is crucial for us to rst identify the entities that are involved in these reaction steps (e.g., hydrazine monohydrate and ethanol) 1 https://www.reaxys.com Reaxys R Copyright c 2020 Elsevier Limited except certain content provided by third parties. Reaxys is a trademark of Elsevier Limited.
An example snippet [Step 4] Synthesis of N-((5-(hydrazinecarbonyl)pyridin-2-yl)methyl)-1-methylN-phenylpiperidine-4-carboxamide Methyl 6-((1-methyl-N-phenylpiperidine4-carboxamido)methyl)nicotinate (0.120 g, 0.327 mmol), synthesized in step 3, and hydrazine monohydrate (0.079 mL, 1.633 mmol) were dissolved in ethanol (10 mL) at room temperature, and the solution was heated under re ux for 12 hours, and then cooled to room temperature to terminate the reaction. The reaction mixture was concentrated under reduced pressure to remove the solvent, and the concentrate was puri ed by column chromatography (SiO2, 4 g cartridge; methanol/dichloromethane = from 5% to 30%) and concentrated to give the title compound (0.115 g, 95.8%) as a foam solid. and then determine the relations between the involved entities (e.g., hydrazine monohydrate is dissolved in ethanol). Thus, our annotation process consists of two steps: named entity annotations and relation annotations. Next, we describe the two steps of annotations in Sect. 2.2 and Sect. 2.3, respectively. 2.2
Four categories of entities are annotated over the corpus: (1) chemical compounds that are involved in a chemical reaction; (2) conditions under which a chemical reaction is carried out; (3) yields obtained for the nal chemical product; and (4) example labels that are associated with reaction speci cations.
Ten labels are further de ned under the above four categories. We de ne ve di erent roles that a chemical compound can play within a chemical reaction, corresponding to ve labels under this category: STARTING MATERIAL, REAGENT CATALYST, REACTION PRODUCT, SOLVENT, and OTHER COMPOUND. For example, the chemical compound \ethanol" in Fig. 1 must be annotated with the label \SOLVENT".
We also de ne two labels under the category of conditions: TIME and
TEMPERATURE; and two labels under the category of yields: YIELD PERCENT
and YIELD OTHER. The de nitions of all resultant labels are summarized in
Table 1. Interested readers may nd more information about the labels in [
A chemical reaction step typically involves an action and chemical compound(s) on which the action takes e ect. We therefore treat the extraction of a reaction step as a two-stage task: (1) identi cation of a trigger word that indicates a chemical reaction step; and (2) identi cation of the relation between a trigger word and chemical compound(s) that is(are) linked to the trigger word. In addition, we observe that it is also crucial for us to link an action to the conditions under which the action is carried out, and resultant yields from the action, in order to fully quantify a reaction step. Thus, annotations in this step are performed to identify the relations between actions (trigger words) and all arguments that are involved in the reaction steps, i.e., chemical compounds, conditions, and yields. REACTION PRODUCT SOLVENT OTHER COMPOUND TIME TEMPERATURE YIELD PERCENT YIELD OTHER
EXAMPLE LABEL Relation Annotations
WORKUP Arg1 ArgM
De nition A substance that is consumed in the course of a chemical reaction providing atoms to products is considered as starting material.
A reagent is a compound added to a system to cause or help with a chemical reaction.
A product is a substance that is formed during a chemical reaction.
A solvent is a chemical entity that dissolves a solute resulting in a solution.
Other chemical compounds that are not the products, starting materials, reagents, catalysts and solvents.
The reaction time of the reaction.
The temperature at which the reaction was carried out.
Yield given in percent values.
Yields provided in other units than %.
A label associated with a reaction speci cation.
An event step which is a manipulation required to isolate and purify the product of a chemical reaction.
An event within which starting materials are converted into the product.
The relation between an event trigger word and a chemical compound.
The relation between an event trigger word and a temperature, time, or yield entity.
O sets
We de ne two types of trigger words: WORKUP which refers to an event
step where a chemical compound is isolated/puri ed, and REACTION STEP
which refers to an event step that is involved in the conversion from a starting
material to an end product. When labelling event arguments, we adapt semantic
argument role labels Arg1 and ArgM from the Proposition Bank [
The gold-standard annotations for the data set were delivered in the BRAT stando format [42]. For each snippet, two les were delivered: a text le (.txt) containing the original texts in the snippet, and a paired annotation le (.ann) containing all the annotations that have been made for that text, including entities, trigger words, and event steps. Continuing with the above snippet example, we present the formatted annotations for the highlighted sentence in Tables 2 and 3. For ease of presentation, we show the annotated named entities and trigger words in Table 2 and the annotated event steps in Table 3. Speci cally, two entities (i.e., T1 and T2) and one trigger word are included in Table 2, and two event steps are included in Table 3. We randomly partitioned the whole data set into three splits for training, development and test purposes, with a ratio of 0.6/0.15/0.25. The training and development sets were released to participants for model development. Note that participants are allowed to use the combination of training and development sets and to use their own partitions to build models. The test set is withheld for use in the formal evaluation. The statistics of the three splits including their number of snippets, total number of sentences, and number of words per snippet, are summarized in Table 4.
To ensure a fair split of data as much as possible, we conduct two statistical tests on the resultant train/dev/test splits. In the rst test, we compare the distributions of entity labels (ten classes of entities in Task 1 and two classes of trigger words in Task 2) within train/dev/test sets, to make sure that the three sets of snippets have similar distributions over labels. The distributions are summarize in Table 5, where each cell represents the proportion (e.g., 0.038) of an entity label (e.g., EXAMPLE LABEL) in the gold annotations of a data split (e.g., Train). The results in Table 5 con rm that the label distributions in the three splits are similar. Only some slight uctuations (6 0:004) across the three splits are observed for each label.
We further compare the International Patent Classi cation (IPC) [
We provide two tasks in ChEMU lab: Task 1|Named Entity Recognition (NER), and Task 2|Event Extraction (EE). We also host a third track where participants can work on building end-to-end systems addressing both tasks jointly. In order to understand and extract a chemical reaction from natural language texts, the rst essential step is to identify the entities that are involved in the chemical reaction. The rst task aims to accomplish this step by identifying the ten types of entities described in Sect. 2.2. The task requires the detection of the entity names in patent snippets and the assignment of correct labels to the detected entities (see Table 1). For example, given a detected chemical compound, the task requires the identi cation of both its text span and its speci c type according to the role in which it plays within a chemical reaction description. 3.2
A chemical reaction usually consists of an ordered sequence of event steps that transforms a starting product to an end product, such as the ve reaction steps in the synthesis process of the chemical compound described in the example in Figure 1. The event extraction task (Task 2) targets identifying these event steps.
Similarly to conventional event extraction problems [
In the track for Task 2, the gold standard entities in snippets are assumed to be known input. While in a real-world use of an event extraction system, gold standard entities would not typically be available, this framework allowed participants to focus on event extraction in isolation of the NER task. 3.3
We also hosted a third track which allows participants to develop end-to-end systems that address both tasks simultaneously, i.e., the extraction of reaction events including their constituent entities directly from chemical patent snippets. This is a more realistic scenario for an event extraction system to be applied for large-scale annotation of events.
In the testing stage, participants in this track were provided only with the text of a patent, and were required to identify the named entities de ned in Table 1, the trigger words de ned in Sect. 3.2, and the event steps involving the entities, that is, the reaction steps. Proposed models in this track were evaluated against the events that they predict for the test snippets, which is the same as in Task 2. However, a major di erence between this track and Task 2 is that the gold named entities were not provided but rather had to be predicted by the systems. 3.4
We illustrate the work ows of the three tracks in Fig. 2 using as example the sentence highlighted in Fig 1. In Task 1|NER|, participants need to identify entities that de ned in Table 1, e.g., the text span \ethanol" is identi ed as \SOLVENT". In Task 2|EE|, participants are provided with the three gold standard entities in the sentence. They are required to rstly identify the trigger words and their types (e.g., the text span \dissolved" is identi ed as \REACTION STEP") and then identify the relations between the trigger words and the provided entities (e.g., a directed link from \dissolved" to \ethanol" is added and labeled as \ARG1"). In the track of end-to-end systems, participants are only provided with the original text. They are required to identify both the entities and the trigger words, and predict the event steps directly from the text. Training stage. In the training stage, the training and development data sets were released to all participants for model development. To accommodate the needs of participants in di erent tracks, two di erent versions of training data, namely Data-NER and Data-EE, were provided. Data-NER was prepared for participants in Task 1, where the gold-standard entities de ned in Table 1 were included. Data-EE was prepared for Tasks 2 and 3, where both the gold-standard entities, annotated trigger words and entity relations were included. Testing stage. Since the gold-standard entities need to be provided to participants in Task 2, the testing stage of Task 2 was delayed until after the testing of Tasks 1 and 3 are completed, in order to prevent any leakage of information. Therefore, the testing stage consists of two phases. In the rst phase, the text (.txt) les of all test snippets were released. Participants in Task 1 are required to use the released patent texts to predict the entities as de ned in Table 1. Participants in Task 3 were required to also predict the trigger words and entity relations de ned in Sect. 3.2. In the second phase, the gold-standard entities of all test snippets were released. Participants in Task 2 can use the released goldstandard entities, along with the text les released in the rst phase, to predict the event steps in test snippets.
Submission website. A submission website has been developed, which allows participants to submit their runs during the testing stage.2 In addition, the website o ers several important functions to facilitate organizing the lab.
First, it hosts the download links for the training, development, and test
data sets so that participants can access the data sets conveniently. Second, it
allows participants to test the performance (against the development set) of their
models before the testing stage starts, which also o ers a chance for participants
to familiarize themselves with the evaluation tool BRATEval [
In this section, we describe the evaluation framework of the ChEMU lab. We introduce three baseline algorithms for Task 1, Task 2, and end-to-end systems, respectively. 4.1
We use BRATEval [
In addition, we exploit two di erent matching criteria, exact-match and relaxed-match, when comparing the texts spans of two entities. Here, the exactmatch criterion means that we consider that the text span of an entity matches with that of another entity if both the starting and the end o sets of their spans match. The relaxed-match criterion means that we consider that the text span of one entity matches with that of another entity as long as their text spans overlap.
The submissions for Task 2 and end-to-end systems are evaluated using Precision, Recall, and F1-score by comparing the predicted events and gold standard 2 http://chemu.eng.unimelb.edu.au/ events. We consider two events as a match if (1) their trigger words and event types are the same; and (2) the entities involved in the two events match. Here, we follow the method in Task 1 to test whether two entities match. This means that the matching criteria of exact-match and relaxed-match are also applied in the evaluation of Task 2 and of end-to-end systems. Note that the relaxed-match will only be applied when matching the spans of two entities; it does not relax the requirement that the entity type of predicted and ground truth entities must agree. Since Task 2 provides gold entities but not event triggers with their ground truth spans, the relaxed-match only re ects the accuracy of spans of predicted trigger words.
To somewhat accommodate a relaxed form of entity type matching, we also
evaluate submissions in Task 1|NER using a set of high-level labels shown
in the hierarchical structure of entity classes in Fig. 3. The higher-level labels
used are highlighted in grey. In this set of evaluations, given a predicted
entity and a ground-truth entity, we consider that their labels match as long as
their corresponding high-level labels match. For example, suppose we get as
predicted entity \STARTING MATERIAL, [335, 351), boron tribromide" while
the (correct) ground-truth entity instead reads \REAGENT CATALYST, [335,
351), boron tribromide", where each entity is presented in the form of \TYPE,
SPAN, TEXT". In the evaluation framework described earlier this example will
be counted as a mismatch. However, in this additional set of entity type relaxed
evaluations we consider the two entities as a match, since both labels
\STARTING MATERIAL" and \REAGENT CATALYST" specialize their parent label
\COMPOUND".
We released one baseline method for each task as a benchmark method. Speci
cally, the baseline for Task 1 is based on retraining BANNER [
BANNER. BANNER is a named entity recognition tool for bio-medical data. In this baseline, we rst use the GENIA Sentence Splitter (GeniaSS) [38] to split input texts into separate sentences. The resulting sentences are then fed into BANNER, which predicts the named entities using three steps, namely tokenization, feature generation, and entity labelling. A simple tokenizer is used to break sentences into either a contiguous block of letters and/or digits or a single punctuation mark. BANNER uses a conditional random eld (CRF) implementation derived from the MALLET toolkit3 for feature generation and token labelling. The set of machine learning features used consist primarily of orthographic, morphological and shallow syntax features.
Co-occurrence Method. This method rst creates a dictionary De for the observed trigger words and their corresponding types from the training and development sets. For example, if a word \added" is annotated as a trigger word with the label of \WORKUP" in the training set, we add an entry hadded; WORKUPi to De. In the case where the same word has been observed to appear as both types of \WORKUP" and \REACTION STEP", we only keep as entry in D its most frequent label. The method also creates an event dictionary Dr for the observed event types in the training and development sets. For example, if an event hARG1; E1; E2i is observed where \E1" corresponds to trigger word \added" of type \WORKUP" and \E2" corresponds to entity \water" of type \OTHER COMPOUND", we add an entry hARG1; WORKUP; OTHER COMPOUNDi to Dr.
To predict events, this method rst identi es all trigger words in the test set using De. It then extracts two events hARG1; T1; T2i and hARGM; T1; T2i for a trigger word \E1" and an entity \E2" if (1) they co-occur in the same sentence; and (2) the relation type hARGx; T1; T2i is included in Dr. Here, \ARGx" can be \ARG1" or \ARGM", and \T1" and \T2" are the entity types of \E1" and \E2" respectively.
BANNER + Co-occurrence Method. The above two baselines are combined to form a two-stage method for end-to-end systems. This baseline rst uses BANNER to identify all the entities in Task 1. Then it utilizes the co-occurrence method to predict events, except that gold standard entities are replaced with the entities predicted by BANNER in the rst stage. 3 http://mallet.cs.umass.edu/
In total, 39 teams registered for the ChEMU shared task, of which 36 teams registered for Task 1, 31 teams registered for Task 2, and 28 teams registered for both tasks. The 39 teams are spread across 13 di erent countries, from both the academic and industry research communities. In this section, we report the results of all the runs that we received for each task. Task 1 received considerable interest with the submission of 25 runs from 11 teams. The 11 teams include 1 team from Germany (OntoChem), 3 teams from India (AUKBC, SSN NLP and JU INDIA), 1 team from Switzerland (BiTeM), 1 team from Portugal (Lasige BioTM), 1 team from Russia (KFU NLP), 1 team from the United Kingdom (NextMove Software/Minesoft), 2 teams from the United States of America (Melaxtech and NLP@VCU), and 1 team from Vietnam (VinAI). We evaluate the performance of all 25 runs, comparing their predicted entities with the ground-truth entities of the patent snippets in the test set. We report the performances of all runs under both matching criteria in terms of three metrics, namely Precision, Recall, and F1-score.
We report the overall performance of all runs in Table 7. The baseline of Task 1 achieves 0.8893 in F1-score under exact match. Nine runs outperform the baseline in terms of F1-score under exact match. The best run was submitted by team Melaxtech, achieving a high F1-score of 0.9570. There were sixteen runs with an F1-score greater than 0.90 under relaxed-match. However, under exact-match, only seven runs surpassed 0.90 in F1-score. This di erence between exact-match and relaxed-match may be related to the long text spans of chemical compounds, which is one of the main challenges in NER tasks in the domain of chemical documents.
Next, we evaluate the performance of all 25 runs using the high-level labels in Fig. 3 (highlighted in grey). We report the performances of all runs in terms of Precision, Recall, and F1-score in Table 8. We received 10 runs from ve teams. Speci cally, the ve teams include 1 team from Portugal (Lasige BioTM), 1 team from Turkey (BOUN REX), 1 team from the United Kingdom (NextMove Software/Minesoft) and 2 teams from the United States of America (Melaxtech and NLP@VCU). We evaluate all runs using the metrics Precision, Recall, and F1-score. Again, we utilize the two matching criteria, namely exact-match and relaxed-match, when comparing the trigger words in the submitted runs and ground-truth data.
The overall performance of each run is summarized in Table 9.4 The baseline (co-occurrence method) scored relatively high in Recall, i.e, 0.8861. This was 4 The run that we received from team Lasige BioTM is not included in the table due to a technical issue found in this run.
F
P
F
P
F expected, since the co-occurrence method aggressively extracts all possible events within a sentence. However, the F1-score was low due to its low Precision score. Here, all runs outperform the baseline in terms of F1-score under exact-match. Melaxtech ranks rst among all o cial runs in this task, with an F1-score of 0.9536. We received 10 end-to-end system runs from four teams. The four teams include The four teams include 1 team from Turkey (BOUN REX), 1 team from the United Kingdom (NextMove Software/Minesoft) and 2 teams from the United States of America (Melaxtech and NLP@VCU).
The overall performance of all runs is summarized in Table 10 in terms of Precision, Recall, and F1-score under both exact-match and relaxed-match.5 Since gold entities are not provided in this task, the average performance of the runs in this task are slightly lower than those in Task 2. Note that the Recall scores of most runs are substantially lower than their Precision scores. This may reveal that the task of identifying a relation from a chemical patent is harder 5 The run that we received from the Lasige BioTM team is not included in the table as there was a technical issue in this run. Two runs from Melaxtech, Melaxtechrun2 and Melaxtech-run3, had very low performance, due to an error in their data pre-processing step. than the task of typing an identi ed relation. The rst run from Melaxtech team ranks best among all runs received for this task. 6
We received 8 paper submissions from participating teams, namely BiTeM,
VinAI, BOUN-REX, NextMove/Minesoft, NLP@VCU, AU-KBC, LasigBioTM,
and MelaxTech. In this section, we present an overview of the approaches
proposed by these teams. We start by introducing the approach of each team rst.
Then we discuss the di erences between these approaches.
To tackle the complexities of chemical patent narratives, the BiTeM team
explored the power of ensemble of deep neural language models based on
transformer architectures to extract information in chemical patents [
Lowe and May eld [
VLP@VCU
VLP@VCU team participated in two tasks: Task 1|NER and Task 2{EE. For
Task 1, the VLP@VCU team identi ed the named entities using BiLSTM units
with a Conditational Random Graph (CRF) output layer. The inputs to this
model are pre-trained word embeddings [
For Task 2|EE, the VLP@VCU team explored two methods to identify
the chemical arguments between the trigger words and the entities. First, a
rule-based method was explored, which uses a breadth- rst search to nd the
closest occurrence of the trigger word on either side of the entity. Second, a
CNNbased model was explored. This model performs a binary classi cation to identify
whether there is a relation or not for each Trigger word-Entity pair. The
sentence containing the Trigger word-Entity pair is rst extracted and then divided
into ve segments, where each segment is represented by a k N matrix. Here,
k represents the latent dimensionality of the pre-trained word embeddings [
Di erent approaches were explored by the participating teams. In Table 11, we summarize the key strategies in terms of three aspects: tokenization method, token representations, and core model architecture.
For teams who participated in Tasks 2 and 3, a common two-step strategy was adopted for relation extraction: (1) identify trigger words; and (2) extract the relation between identi ed trigger words and entities. The rst step is essentially an NER task, and the second step is a relation extraction task. As such, NER models were used by all these teams for Tasks 2 and 3 as well as by the teams participating in Task 1. Therefore, in what follows, we rst discuss and compare the approaches of all teams without considering the target tasks, subsequently considering relation extraction approaches.
Chemistry domain-speci c Representation
Embeddings
Character-level Pre-trained
Chemistry domain-speci c Features
PoS
Phrase Model Architecture
Transformer
Bi-LSTM
CNN
MLP
CRF
FSM
Rule based
Tokenization is an important data pre-processing step that splits input texts
into words/subwords, i.e., tokens. We identify three general types of
tokenization methods used by participants: (1) rule based tokenization; (2) dictionary
based tokenization; and (3) subword based tokenization. Speci cally, rule based
tokenization applies pre-de ned rules to split texts into tokens. The rules applied
can be as simple as \white-space tokenization", but can also be a complex
mixture of a set of carefully designed rules (e.g., based on language-speci c grammar
rules and common pre xes). Dictionary based tokenization requires the
construction of a vocabulary and the text splitting is performed by matching the input
text with the existing tokens in the constructed vocabulary. Subword
tokenization allows a token to be a sub-string of a word, i.e., subword units. It relies
on the principle that most common words should be left as is, but rare words
should be decomposed in meaningful subword units. Popular subword
tokenization methods include WordPiece [39] and Byte Pair Encoding (BPE) [
Four teams utilized tokenization methods that are purely rule-based.
Specifically, VinAI used Oscar4 tokenizer [
Four of the above teams participated in Task 2 or Task 3. As mentioned before, these teams utilized their NER models for trigger word detection. Thus, here, we only discuss their approaches for relation extraction assuming that the trigger words and entities are known.
NextMove/Minesoft again made use of ChemicalTagger for event extraction. ChemicalTagger is able to recognize WORKUP and REACTION STEP words, thus, assignment of relationships were achieved by associating all entities in a ChemicalTagger action phrase with the trigger word responsible for the action phrase. A set of post-processing rules were also applied to enhance the accuracy of ChemicalTagger.
LasigBioTM, NLP@VCU, and MelaxTech formulated the task of relation extraction as a binary classi cation problem. That is, given each candidate pair of trigger word and named entity that co-locate within an input sentence, the goal of the task is to determine whether the candidate pair of entities are related or not.
LasigBioTM developed a BioBERT-based model to accomplish this classi cation. The input of BioBERT is the sentence containing the candidate pair but the trigger word and named entity of the candidate pair were replaced with the tags \@TRIGGER$" and \@LABEL$", respectively. The output of BioBERT is modi ed as a binary classi cation layer which aims to predict the existence of relation for the candidate pair.
NLP@VCU proposed two systems for relation extraction. Their rst system is a rule-based system. Given a named entity, a relation is extracted between the named entity and its nearest trigger word. Their second system is developed based on CNNs. They split the sentence containing the candidate pair into ve segments: the sequence of tokens before/between/after the candidate pair, the trigger word, and the named entity of the candidate pair. Separate convolutional units were used to learn the latent representations of the ve segments, and a nal output layer was used to determine if the candidate pair is related or not.
MelaxTech continued the use of the BioBERT model re-trained on the patent texts released during the test phase. Similar to LasigBioTM, the input to their model is the sentence containing the candidate pair but only the candidate named entity is generalized by its semantic type in the sentences. Furthermore, rules were also applied in the post-processing step to recover false negative relations with a long distance, including relations across clauses and across sentences. 7.5
The various approaches adopted by teams and the resulting performances have provided us with valuable experiences in how to address the tasks and what choices of methods are more suitable for our tasks.
Tokenization. In general, domain-speci c tokenization tools perform better than tokenization methods that are for general English corpus. This is as expected since the vocabulary of chemical patents contains a large number of domain-speci c terminology, and a machine can better understand and learn the characteristics of input texts if the texts are split into meaningful tokens. Another observation is that subword-based tokenization may contribute to overall accuracy. Chemical names are usually long, which make subword-based tokenization a suitable method for breaking down long chemical names. But further investigation is needed to support this claim.
Representation. Pre-trained embeddings are shown to be e ective in
enhancing system performances. Speci cally, the Melaxtech and Lasige BioTM
systems are based on BioBERT model [
Model Architecture. The most popular choice of model is BERT [
This paper presents a general overview of the activities and outcomes of the ChEMU 2020 evaluation lab. The ChEMU lab targets two important information extraction tasks applied to chemical patents: (1) named entity recognition, which aims to identify chemical compounds and their speci c roles in chemical reactions; and (2) event extraction, which aims to identify the single event steps that form a chemical reaction.
We received registrations from 39 teams and 46 runs from 11 teams across all tasks and tracks, and 8 teams have contributed detailed system descriptions for their methods. The evaluation results show that many e ective solutions have been proposed, with systems achieving excellent performance on each task, up to nearly 0.98 macro-averaged F1-score on the NER task (and up to 0.99 F1-score on a relaxed match), 0.95 F1-score on the isolated relation extraction task, and around 0.92 F1-score for the end-to-end systems. These results strongly outperformed baselines.
We are grateful for the detailed excerption and annotation work of the domain experts that support Reaxys, and the support of Ivan Krstic, Director of Chemistry Solutions at Elsevier. Funding for the ChEMU project is provided by an Australian Research Council Linkage Project, project number LP160101469, and Elsevier. 37. Ruas, P., Lamurias, A., Couto, F.M.: LasigeBioTM team at CLEF2020 ChEMU evaluation lab: Named Entity Recognition and Event extraction from chemical reactions described in patents using BioBERT NER and RE. In: Working Notes of CLEF 2020|Conference and Labs of the Evaluation Forum (2020) 38. S tre, R., Yoshida, K., Yakushiji, A., Miyao, Y., Matsubayashi, Y., Ohta, T.: AKANE system: protein-protein interaction pairs in BioCreAtIvE2 challenge, PPI-IPS subtask. In: Proceedings of the second BioCreative challenge workshop. vol. 209, p. 212. Madrid (2007) 39. Schuster, M., Nakajima, K.: Japanese and korean voice search. In: 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5149{5152. IEEE (2012) 40. Senger, S., Bartek, L., Papadatos, G., Gaulton, A.: Managing expectations: assessment of chemistry databases generated by automated extraction of chemical structures from patents. Journal of Cheminformatics 7(1), 1{12 (2015) 41. Soysal, E., Wang, J., Jiang, M., Wu, Y., Pakhomov, S., Liu, H., Xu, H.: CLAMP{ a toolkit for e ciently building customized clinical natural language processing pipelines. Journal of the American Medical Informatics Association 25(3), 331{ 336 (2018) 42. Stenetorp, P., Pyysalo, S., Topic, G., Ohta, T., Ananiadou, S., Tsujii, J.: BRAT: a web-based tool for NLP-assisted text annotation. In: Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics. pp. 102{107 (2012) 43. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. pp. 5998{6008 (2017) 44. Verspoor, C.M.: Contextually-dependent lexical semantics (1997) 45. Verspoor, K., Nguyen, D.Q., Akhondi, S.A., Druckenbrodt, C., Thorne, C., Hoessel, R., He, J., Zhai, Z..: ChEMU dataset for information extraction from chemical patents. https://doi.org/10.17632/wy6745bjfj.1 46. Wang, J., Ren, Y., Zhang, Z., Zhang, Y.: Melaxtech: A report for CLEF 2020 { ChEMU Task of Chemical Reaction Extraction from Patent. In: Working Notes of CLEF 2020|Conference and Labs of the Evaluation Forum (2020) 47. Yoshikawa, H., Nguyen, D.Q., Zhai, Z., Druckenbrodt, C., Thorne, C., Akhondi, S.A., Baldwin, T., Verspoor, K.: Detecting Chemical Reactions in Patents. In: Proceedings of the 17th Annual Workshop of the Australasian Language Technology Association. pp. 100{110 (2019) 48. Zhai, Z., Nguyen, D.Q., Akhondi, S., Thorne, C., Druckenbrodt, C., Cohn, T., Gregory, M., Verspoor, K.: Improving Chemical Named Entity Recognition in Patents with Contextualized Word Embeddings. In: Proceedings of the 18th BioNLP Workshop and Shared Task. pp. 328{338. Association for Computational Linguistics (2019)