This paper describes our system developed for ChEMU @ CLEF Cheminformatics Elsevier Melbourne University lab, Named Entity Recognition (NER) task for identifying chemical compounds as well as their types in context, i.e., to assign the label of a chemical compound according to the role which the compound plays within a chemical reaction from patent documents. We have presented two systems which use Conditional random elds (CRFs) algorithms and Arti cial Neural Networks (ANN). In this work we used feature set that includes linguistic, orthographical and lexical clue features. In the development of systems, we have used only the training data provided by the track organizers and no other external resources or embedding models were used. We obtained an F-score of 0.6640 using CRFs and F-Score of 0.3764 using ANN on the test data.
CLEF 2020 ChEMU NER task aims to automatically identify chemical
compounds and their speci c types, i.e. to assign the label of a chemical compound
according to the role it plays in a chemical reaction. In addition to chemical
compounds, the task also requires identi cation of the temperature and the reaction
time at which the chemical reaction was carried out, the yields obtained for the
nal chemical product and the label of the reaction. The focuses of this task is
mainly on information extraction from chemical patents. This is a challenging
task as patents are written very di erently as compared to scienti c literature.
When writing scienti c papers, authors strive to make their words as clear and
straightforward as possible, whereas patent authors often seek to protect their
knowledge from being fully disclosed [
We pre-processed the data provided by the task organizers to the required format to develop our NER systems. Subsequently, features were extracted and trained for the identi cation of the entities from the corpus using Machine learning (ML) algorithms. In section 2, we brie y review the recent literature. In the following section 3, features and the method used to develop the language models are described. Results are discussed in section 4. The paper ends with the conclusion. 2
In recent years Deep Learning is ourishing as a well-known ML methodology
for NLP applications. By using the multilayer neural architecture it can learn
the hidden patterns from the enormous amount of data and handles the complex
problems. This section brie y explains the recent research works in the eld of
NER using Deep Learning. The earlier work on neural network was done by Gallo
et.al [
In this section we present our systems developed using Condition Random Fields (CRFs) and Arti cial Neural Networks (ANN). For our work we use CRF++ tool and Scikit python package. CRF++1 tool is an open source implementation of CRFs and is a general purpose tool. Our NER system follows a pipeline architecture, where the data is rst pre-processed to required format that is needed to train the system. After training the system the NEs are automatically identi ed from the test set. 3.1
Feature selection is an important step in the ML approach for NER. Features play an important role in boosting the performance of the system. Features selected must be informative and relevant. We have used word level features, grammatical features, functional terms features that are detailed below: 1. Word level features: Word level features include Orthographical features and Morphological features.
(a) Orthographical features contain Capitalization, combination of digits, symbols and words and Greek words.
(b) Pre x and su x of chemical entities is considered as morphological features.
2.Grammatical features: Grammatical features include word, PoS, chunks and combination of word, PoS and chunk.
3. Functional term feature: Functional term helps to identify the biological named entities and categorize them to various classes. Example: Alkyl, acid, alkanylene.
Grammatical features of Part-of-Speech (PoS) and chunk information are obtained using automatic tools. More details about the tools are given in next sub-section. The morphological features are obtained by extracting `n' last and starting characters of chemical entities. After performing a few experiments, it was identi ed that n=4 is the optimal one. A Functional terms lexicon was collected from online sources. 3.2
The data is pre-processed using a sentence splitter and tokenizer and is converted into column format with entities tagged using the le containing detailed chemical mention annotation (BRAT format annotation le). The sentence splitter and the tokenizer used are rule based engines, developed in-house. In these 1 https://taku910.github.io/crfpp/ engines we have done a modi cation by adding a special processing to accommodate long entity names with more than 200 characters. We have split these long names into two tokens and then combined it as one after PoS tagging and Phrase chunking is completed.
Then the data is annotated for syntactic information of Part-of-speech (PoS)
and Phrase Chunk information (Noun Phrase, Verb phrase) using fnTBL tool
[
The features from the pre-processed data are extracted as explained in section 3.1.The data format is similar for both the algorithms. The systems are trained using the same features extracted from the data. Using these models the chemical named entities from the test data were automatically identi ed. Chemical entity mention in patents requires the detection of the start and end indices corresponding to all chemical entities. Hence we converted the output from the system to the required BRAT format for task submission. The NER language models developed using CRFs, used the features as explained in section 3.1 from training. Using the NER model the NEs are automatically identi ed from the testing corpus. All the features were extracted from the training corpus provided by the organizers and no other external resources have been used. Similarly, we have followed for the ANN system also. ANN system is described below. Arti cial Neural Networks (ANN) A Multi-Layer Perceptron (MLP) is a feed forward Arti cial Neural Network (ANN). The input layer receives the input data in the numerical form, the output layer takes the decision about the input and the hidden layers exists between these two acts as the computational engine.
The three important steps involved in neural network are 1) each input is multiplied by a weight 2) all the weighted inputs are added together with the bias and 3) the sum is passed through the activation function. The input node accepts the information in a numerical form and depending on the weight and the transfer function, the activation value which is the weighted sum of inputs will be passed to the next node. The activation function is used to monitor the threshold level and convert the unbounded value into a bounded one. Each node in the network runs the activation value and tweak the sum based on its transfer function. The activation function runs through the entire network until it reaches an output node. The traditional systems used sigmoid or the hyperbolic tangent activation function.
In this work, Multilayer Perceptron (MLP) network is used. MLP is a combination of layers of perceptrons connected together. The rst layer's output goes as the input to the next layer until it reaches the output layer. The hidden layer is the layer that exists between the input and output layer Feed forward networks like MLP have two passes, namely forward and backward passes. In the forward pass, propagate forward to get the output and compare it with the real value to get the error. In order to reduce the error multilayer perceptrons propagate backward and adjust the weights. This process of back-propagation is used to adjust the weight and bias relative to the error. This process continues until the estimated output is obtained. ReLu activation function is used in MLP. The feed forward network consists of two motions namely forward and backward pass. The training process comprises of 3 steps, they are forward pass, error calculation and backward pass. In forward pass the input data is multiplied by its weight and added to the bias at each node and passes through the hidden layer to the output layer. The cost function is used to predict the performance of the model, which can be computed as the di erence between the predicted and the expected value. Once the loss is calculated we have to back propagate the loss in order to update the weights of each node using the gradient descent function. The weights are being tweaked according to the gradient ow in that direction. The main intention here is to minimize the loss.
In this work we have used the python package of Scikit-Learn's Multi-Layer Perceptron. The process of converting the input data into numerical feature vectors is called as vectorization and it involves mainly three steps namely tokenization, counting and normalization. The resultant data is called as Bag of words representation. In this form rather than the relative position of the words in the document the input text is represented using the word occurrences. We have used the countvectorizer to represent the data into bag of words format. It converts the text data into the numerical features. The TFIDFVectorizer is used to convert the bag of words into the matrix of TFIDF features. After initiating the size of the hidden layer and determining the activation and optimization the data is given for the training process. The ReLu activation function is used for the hidden layers of the present MLP implementation. The stochastic gradient optimizer Adam is used for the weight optimization.
The number of layers we used for the hidden layer is 30, the activation function utilized for the hidden layer is ReLu (Recti ed Linear unit function), `adam' stochastic gradient-based optimizer is used as a solver for the weight optimization. 'Alpha' regularization parameter is set to 0.0001. The learning rate schedule used for weight updation is 'constant'. It is the constant learning rate provided by the initial learning rate which helps to control the step-size in updating the weights and the 'learning rateinit' value is set as 0.001. Batch size refers to the number of training examples used in one iteration. The batch size is assigned as mini batches for stochastic optimizers and it is set to 200 by default. The architecture of MLP implementation is shown in gure 1. 4
The system outputs on the test data were evaluated by the track organizers, precision, recall and F score were calculated. The test results are tabulated in Table 1 and 2. Table 1 provides the test results of the system developed using CRFs and Table 2 presents results of system using ANN.
The system based on CRFs had given a very good precision. The recall is
low and especially for the entities \YIELD OTHER" and \YIELD PERCENT".
This could have been improved by using post processing rules. The results
obtained using ANN are lower than the CRFs based system. This clearly shows that
the training data size is not su cient for ANN. The ANN system requires used
of external resources such as pre-trained word embeddings and other available
annotated resources.
We submitted two systems developed using Machine Learning (ML) techniques,
Condition Random Fields (CRFs) and Arti cial Neural Networks (ANN). A two
stage pre-processing is done on development data 1) the formatting stage, where
the sentence splitting, tokenizing and the data conversion to column format and
2) the data annotation stage, where the data is annotated for syntactic
information of Part-of-speech (POS) and Phrase Chunk information (Noun Phrase,
Verbphrase) are performed. For both POS and Chunk information, fnTBL [