Natural language processing can assist scientists to leverage the increasing amount of information contained in scientific bibliography. The current trend, based on deep learning and embeddings, uses representations at the (sub)word level that require large amounts of training data and neural architectures with millions of parameters to learn successful language models, like BERT. However, these representations may not be well suited for the scientific domain, where it is common to find complex terms, e.g. multi-word, with a domain-specific meaning in a very specific context. In this paper we propose an approach based on a linguistic analysis of the corpus using a knowledge graph to learn representations that can unambiguously capture such terms and their meaning. We learn embeddings from diferent linguistic annotations on the text and evaluate them through a classification task over the SciGraph taxonomy, showing that our representations outperform (sub)word-level approaches.
• Computing methodologies → Natural language processing; Neural networks; Semantic networks; Machine learning approaches.
Nowadays scholarly communications are evolving, thanks to the
efort of research communities, funding agencies and publishers,
beyond the conventional delivery method based on documents
to gain better visibility, reuse capabilities and to foster a broader
data accessibility[
Aligned with the FAIR principles, in particular with the goal of
assisting humans and machines in managing data, research objects
[
In fact NLP technology is progressing at a fast pace thanks to
word embeddings [
Scientific terminology is domain specific and scarce in a general corpus and hence accumulating the necessary amount of evidence from documents to identify it a as single entity with a specific meaning is very unlikely if we analyse single words and sub words representations. On the other hand, precisely in the scientific domain, the amount of structured resources, including catalogs, taxonomies and knowledge graphs with specific terminology and their corresponding definitions is available. Thus, the question raises, what is the minimum information unit or their combination thereof, which allows for eficient representations in vector form and at the same time can be linked to a semantically significant concept?
In this paper we propose to generate embeddings using surface forms, lemmas and concepts that are able to represent complex terms consisting of more than one word. The linguistic information is the result of applying a linguistic analysis that relies on a knowledge graph where linguistic knowledge is encoded. The linguistic analysis performs a grammatical, syntactical and semantic analysis to recognize and disambiguate terms that can consist of more than 2SciGraph homepage: https://www.springernature.com/gp/researchers/scigraph 3According to Wikipedia a Molecularly imprinted polymer is a polymer that has been processed using the molecular imprinting technique one word. We generate embeddings from a scholarly communications corpus for single and joined representations (surface forms, lemmas, part-of-speech, and concepts). We experiment with these embeddings in a text classification task where the goal is to classify academic publications in a topic taxonomy.
Our results show that using linguistic annotation embeddings helps to learn better classifiers when compared to those learned only with words or subword embeddings. According to our experimentation the best approach is to use surface form and lemma embeddings jointly. When surface form and lemma embeddings are enriched with gramma information embeddings, like part-of-speech tag embeddings, the classifier with the greatest precision is learned. On the other hand, concept embeddings results were mixed probably due to the general-purpose annotator used in the experiments with a limited coverage of the scientific domain vocabulary.
This papers is structured as follows. Section 2 describes the related work and the paper contributions. Section 3 summarizes the approach to learn the embeddings for linguistic annotations. Next, Section 4 presents the experimental work where we evaluate the embeddings in a text classification task. Finally section 5 presents the conclusions and future lines of work. 2
Recent work in distributional representation of words has moved
from static [
In parallel researchers have proposed to learn jointly concepts
and word embeddings as an alternative approach to cope with the
ambiguity of the language. For example Camacho-Collados et al.
[
In the scientific domain, Wang et al . [
The main contribution of this paper is a comprehensive experimentation in the scientific domain with Vecsigrafo embeddings jointly learned from linguistic annotations and compare them with word and subword embeddings. 3
To learn embeddings from diferent linguistic annotations we use
Vecsigrafo [
In contrast to simple tokens produced by space separation tokenization, linguistic annotations used in Vecsigrafo are based on terms that are related to one or more words. Surface forms are terms as they appear in the text, and lemmas are the base form of these terms. Source forms and lemmas can refer to concepts in a knowledge graph. For example, table 1 shows the linguistic annotations added to a text excerpt taken from a publication. Note how at the surface form level some tokens are grouped into terms like local anesthetic and phrenic nerve, and at the lemma level some surface forms such as concerns and relating are turned into their base form concern and relate. The grammar information indicates the role of the terms as nouns (N), verbs(V), noun and verb phrases (NP, VP), prepositions (P) and punctuation marks (PNT). In addition some of the terms are related to the concepts like like local anesthetic that is annotated with the concept en%23107824862 that is defined as An anesthetic that numbs a local area.
Formally, Vecsigrafo generates, from a corpus an embedding space Φ = {(x, e) : x ∈ SF ∪ L ∪ G ∪ C, e ∈ Rn } where SF , L, G, and C are sets of surface forms, lemmas, grammar types, and concepts. One of the benefits of Vecisgrafo is that concept embeddings contribute to identifying the intended meaning of ambiguous terms in the corpus since the term and concept embeddings are learned jointly. To use Vecsigrafo embeddings in Φ we need to annotate the target corpus with the linguistic elements used to learn the embeddings. Note that embeddings representing linguistic annotations for the same term can be merged to generate a single embedding for the term, for example, by applying vector operations such as concatenation or averaging, or dimensional reduction techniques like PCA or SVD. 4
In this section we describe the scholarly communication corpus used to learn the linguistic embeddings, the NLP toolkit used to annotate the corpus, the neural network that uses the linguistic embeddings to classify the research publications, and report the evaluation results of the classifiers. 4.1
SciGraph [
BY 4.0 License (i.e., attribution) with the exception of abstracts and grant metadata, which are available under CC BY-NC 4.0 License (i.e., attribution and non-comercial) A core ontology expressed in OWL encodes the semantics of the data in the knowledge graph consisting of 47 classes and 253 properties. From SciGraph we extract publications including articles and book chapters published from 2001 to 2017. We use the titles and abstracts of the publications to generate the corpus with roughly 3.2 million publications, 1.4 million distinct words, and 700 million tokens.
Next we use Expert System NLP suit (Cogito) to parse the text and add linguistic annotations. Cogito disambiguator relies on its own knowledge graph called Sensigrafo, that encodes the linguistic knowledge in a way similar to WordNet, and applies a rule-based approach to disambiguation. The Sensigrafo contains about 400K, lemmas and 300K concepts interlinked via 61 relation types. Note that we could have used any other NLP toolkit as long as it generates the linguistic annotations used in this work. The corpus parsing and annotations generated by Cogito are reported in table 2.
For each linguistic element we learned an initial set of
embeddings with 300 dimensions using Vecsigrafo. The diference between
the number of learned embeddings and the linguistic annotations
is due to a filter that we applied based on previous results [
Normal Distribution Optimized by CNN Publications in Scigraph have one or more field of research codes that classify the documents in 22 categories such as Mathematical Sciences, Engineering or Medical and Health Sciences. Thus, we can formulate a multi-label classification task that aims at predicting one or more of these 22 first level categories for each publication.
Embeddings are the natural numerical representation of text
for neural networks. Kim [
To learn the classifier we use an of the shelf CNN implementation available in Keras, with 3 convolutional layers, 128 filters and a 5-element window size. As corpus we use 187795 articles available in SciGraph published in 2011. To evaluate the classifiers we use ten-fold cross-validation and precision, recall and f-measure as metrics. We use a vocabulary with maximum 20K entries, and sequences size 1000.
As baseline, we train a classifier that learns from embeddings generated randomly following a normal distribution. As upper bound we learn a classifier that is able to optimize the embeddings in the learning process. The evaluation of baseline and upper bound classifiers are presented in table 3. 4.3
We train classifiers using single Vecsigrafo embeddings for each linguistic annotation (sf, l, c) and for the ten 2, and 3 size combinations of (sf, l, g, c). Grammar embeddings were not evaluated independently due to the low number of distinct grammar types used to annotate the terms. When using embeddings of two or three linguistic annotations two diferent approaches are used. The ifrst approach relies on a single vocabulary containing at most 20K entries per each linguistic annotations in the text, and no merging operation is carried out, while in the second one embeddings are merged using concatenation or average. Evaluations results are reported in table 4. 4.4
Regarding single linguistic annotations, lemma l and surface form sf embeddings contribute to learn the better classifier than using token t embeddings respectively. This shows that the classifier learning process benefits from the conflation of diferent term and word variations (sf, t) into a base form (l). However, grouping raw tokens into terms (sf ) only generates a slight improvement in the classifier performance with respect to using only tokens (t). On the other hand, concept (c) embeddings performance in this task is worst than t embeddings. The low number of c embeddings (see table 2) compared to the number of tokens and the other linguistic annotations afect negatively the learning process. The diference between concepts and tokens is consequence of limited coverage of the general-purpose annotator used in a highly specialized domain as the scientific. 4.5
To analyse the results of the diferent combinations of embeddings for linguistic annotations we focus on each evaluation metric. Regarding precision the top 2 classifiers are learned from combinations of sf, l and g. In addition note that the common linguistic element in the top 6 classifiers is g combined either with sf or l, and in general removing g produced least precise classifiers. Thus, precision-wise the part-of-speech information in combination with surface forms and lemmas is very relevant. Semantic information (c) also contributes to enhance precision when it is combined with lemmas and surface forms, or with lemmas and grammar information. In addition, the precision of 16 classifiers out of 22 is better than the upper bound reported in table 3, where the embeddings are optimized in the classifier learning phase, even though vecsigrafo embeddings were not learned for this specific purpose.
The recall analysis shows a diferent picture since the grammar information (g) does not seem to have a decisive role on the classifier performance. Surface forms and lemmas generates the classifier with highest recall . Nevertheless, in this analysis concepts (c) gain more relevance always in combination with either sf or l. The combination of l and c seems to benefit recall since it is presented in 3 of the top 5 classifiers. In contrast, when concepts are combined with sf the recall is lower. In general g-based embedding combinations generate classifiers with lower recall. Note that none of the classifiers reached the recall of the upper bound classifier.
The f-measure data shows more heterogeneous results since by definition it is the harmonic mean of precision and recall, and hence the embedding combinations that generate the best f-measure Linguistic Annotations
Merging Precision needs a high precision and a high recall. The combination of surface forms sf and lemmas l embeddings is at the top of the f-measure ranking, followed by their combination with c. In general, concept embeddings improves the f-measure when combined with either lemmas or surface forms. However, when used in conjunction with lemmas and surface form embeddings the performance is worse. In general, due to the low coverage of concepts in the scientific domain the classifiers that relies only on c embeddings perform worst even when combined with grammar information. Similarly surface forms ofer poor performance when combined with grammar information.
Finally note how the best classifiers were learned when the linguistic annotation embeddings are used independently which contrast to the worse results achieved when merging the embeddings. 4.6
We also test embeddings generated from word constituents. We
resorted to FastText[
FastText 5, shows that token embeddings are better than using token and character-ngram embeddings, which is in line with our assumption that using subword representations could be not convinient in the scientific domain. Note that one of the benefits of using characterngram embeddings is to avoid the out of the vocabulary words (OOV). However, in our case, the embeddings were learned from the whole scigraph corpus so we do not face the OOV problem in our experiments.
On the other hand, results in table 4 and 5 are not directly comparable since the embeddings are generated with a diferent algorithms (FastText vs Vecsigrafo). For example FastText token embeddings generate a better classifier than using Vecisgrafo token embeddings, and remarkably FastText embeddings in both cases reach the highest precision of all the tested embeddings. Nevertheless, we can see that the f-measure of the classifier that uses FastText character-ngram embeddings is lesser than the first 11 results reported in table 4, including the classifier that uses only lemmas. 5
Natural language processing has the potential to help scientists to manage and get insights out of the huge amount of scholarly communications available. Nowadays deep learning techniques based on word embeddings and language models have advanced the state of the art in diferent NLP tasks. Nevertheless, the predominant approach in NLP is to use word or subword representations as the input of deep neural architectures that requires large corpora to learn performing language models. However, in contrast to generalpurpose corpora the scientific vocabulary often contains complex terms comprising more than one word with the additional characteristic that these terms are very specific and only make sense in certain fields of knowledge (e.g., Cosmic Microwave Background Radiation). Thus models using word or subword representations could have problems to gather the necessary textual evidence to capture their meaning.
To overcome the word and subword representation limitation we propose to use embeddings based on linguistic annotations such as surface forms, lemmas, part-of-speech information, and concepts. These embeddings are jointly learned from a corpus of scientific communications using an existing approach called Vecsigrafo. We evaluate the linguistic annotation embeddings in a multilabel classification where the goal was to assign a scientific topic to each publication. Our evaluations results show that lemmas help to learn better classifiers than using space-separated words and subword representations based on character-ngrams. The best results were achieved when lemma and surface forms were used jointly. Grammar information was very useful for high precision. Concepts, on the other hand, were less helpful in general mainly due to the low coverage of concepts in the scientific domain. Since part of the analysis that identify surface forms and lemmas are based on lexical and syntactical analysis the coverage was higher.
As future work we want evaluate the linguistic annotation embeddings on other evaluation tasks diferent from text classification where understanding the the glossary can have more impact like entailment and question and answering. In addition, another line of research is to evaluate the impact of the linguistic annotations when used as input representation to learn language models.
This research has been supported by The European Language Grid project funded by the European Unions Horizon 2020 research and innovation programme undergrant agreement No 825627 (ELG).