Figurative language plays an important role in thought processes and science. Automatic detection of figurative language is gaining momentum in the open domain natural language processing research, but it is hindered in the biomedical domain by the absence of document collections for development and testing of the approaches. Reliable approaches to detection of figurative language could potentially improve automatic indexing of the literature and support clinical applications. We have developed a collection of documents annotated for literal or non-literal use of seven terms that are known to cause errors in automatic indexing of biomedical abstracts. Using the collection, we explore detection of figurative language with CNN-RNN, logistic regression and transformer models. We establish baselines for each of the seven terms, achieving the results at the level of the state-ofthe-art reported in the open domain evaluations.
Figurative language plays an important role in science, with
metaphors and idiomatic expressions viewed as foundations
for thought processes
Whereas it is relatively easy for people to discern which of
these phrases refer to physical falls, the biomedical named
entity recognition (NER) approaches often treat figurative
language as literal and link the word to inappropriate
ontology terms as a result. Specifically, in the task of
automated indexing that aims to summarize the main points of a
publication by assigning terms from a controlled vocabulary
created to index the biomedical literature: Medical Subject
Headings (MeSH) (NLM 2020
In the biomedical publications, the problem of recognizing non-literal utterances is intertwined with word sense disambiguation (WSD), and compounded by the importance of the term to the article. The WSD aspects could be illustrated by the following:
The head of each fish, including the brain and pituitary, was sampled for double-colored FISH analysis.
To many NER approaches, the first occurrence of fish is indistinguishable from FISH, which stands for fluorescent in situ hybridization. The confusion continues in:
Is being a small fish in a big pond bad for students´ psychosomatic health?
Moreover, for food products manufactured from fish, such
as fish oil, linking to Fishes also violates indexing rules.
To summarize, to label a biomedical publication with the
terms from a terminology, we need to determine if the terms
are used literally, if the sense in the context corresponds to
the sense in the terminology, and if the term is important
enough to be indexed for the article in MEDLINE/PubMed
database, which comprises more than 30 million biomedical
abstracts (NLM 2020
Whereas there continues to be a steady research in
biomedical WSD
The body of work on detection of figurative language in the
open domain is significant, and the interest to the topic is
growing, as evidenced by the workshops and shared tasks
on figurative language processing
To the best of our knowledge, our work is the first to explore the difficulties figurative language poses for automated indexing of the biomedical literature. We also provide the first publicly available biomedical literature dataset annotated for figurative language at the token and sentence level. In addition, leveraging the state-of-the-art approaches explored in the open domain, we establish baselines for detection of figurative language in biomedical abstracts using sentence or token level classification.
We analyzed 870 American English idioms
We then searched PubMed for the exact figurative expressions, and for the abstracts containing trigger terms that were either indexed or not with the corresponding MeSH headings. Abstracts with trigger terms and MeSH headings serve as examples of literal use in the training set, and abstracts without MeSH headings serve as examples of non-literal use. For the test sets, we randomly sampled files from both distributions and manually annotated the sentences containing the terms at the token level. We annotated fine-grained senses corresponding to: 1. Full MH: the literal Mesh Heading-appropriate sense, e.g., “a healthy baby at 34 weeks of gestation.” The labels assigned by the indexers were not shown to the annotators to avoid bias.
Term (MH) fall (Accidental Falls) fish (Fishes) juvenile (Adolescent) baby (Infant) bull (Cattle) cat (Cats) dog (Dogs)
Check Tag no no yes yes yes yes yes 2. Partial Literal: MH-appropriate sense, but being a part of an expression, which should not trigger mapping to MeSH, e.g., shaken baby syndrome. 3. Literal Other: Literal senses other than MH, e.g., baby hamster is still a baby, but it should not be indexed with Infant, which applies only to human babies. 4. Figurative: Non-literal use of the term, e.g., in “There’s a Baby in this Bath Water!” Each document was annotated by two annotators and the differences were reconciled.
We explored CNN-RNN (Svoboda 2020
We then approached the task as binary classification at the sentence or token level.
To train the CNN-RNN and Logistic Regression
models, sentences containing the target trigger terms were
extracted from a set of retrieved documents that were labeled
using MeSH indexing information as described above. Each
extracted sentence was assigned the label of the document
from which it was derived. Sentence embeddings were
generated using a Doc2Vec
In the CNN-RNN approach, the embeddings and associated labels served as input to a neural network containing four groups of four layers: convolutional layer, dropout, max-pooling, and dropout, followed by an LSTM layer.
The model uses a sigmoid activation function, binary crossentropy loss and the adam optimizer.
We used the SciKit Learn Logistic Regression classifier, with Doc2Vec output as inputs.
The Universal Sentence Encoder was also applied in the sentence level classification task. Unlike the Doc2Vec models, the Universal Sentence Encoder was trained on a very large corpus using a variety of sources. In our approach, each sentence vector representation was generated using the Universal Sentence Encoder during training. The vector representation and the sentence label was then passed to a twolayer neural network consisting of a RELU and a softmax layer. A categorical cross-entropy loss and the adam optimizer was used when building the model.
We used BERT encoder extended with a CRF layer
for Named Entity Recognition (Kaiyinzhou 2020
We created a collection of PubMed abstracts automatically
annotated for literal and non-literal use of seven terms that
proved to be a rich source of false positive linking to
terminologies and have sufficient amounts of training documents
in PubMed. Interestingly, one of these terms, fall was also
found to be difficult to classify as figurative in the open
domain tasks
We explored several state-of-the-art approaches, casting
the task as binary classification at the sentence and
token level. We hoped to identify one best approach for the
task and achieve state-of-the-art performance for all trigger
terms. The best results reported in the literature for the
opendomain figurative language detection and in the shared task
on metaphor detection
These preliminary results provide the basis for the further development of a non-GMO approach to modulate fish allergenicity and improve safety of aquaculture fish. (PMID: 31622806)
The indexers labeled this article with both Fishes and Seafood. When the contexts for these occurrences of fish are used in the models as positive examples, they might be too close to the contexts of the articles that present fish only in the context of food and thus serve as negative examples.
With respect to identifying one approach that would work
best for all of the trigger terms, we can see that
casting the task as sentence-level classification and using the
CNN-RNN model produces the majority of best results.
Another interesting observation is that if we want to select a method for automated indexing, we will have to decide if recall or precision are more important when suggesting the terms. For cat, dog, fish and juvenile, the differences in these two metrics achieved by different approaches are relatively large, although the F-scores are mostly close, showing a typical trade-off between the two metrics. In selecting approaches to support automated indexing, precision often plays an important role, as currently the consensus is that it is better to miss a term than to assign an inappropriate term that will mislead the search engines that rely on MeSH indexing. For that reason, we do not consider accuracy when selecting an approach for supporting automated indexing.
Our work has some limitations that we hope to address in the future. First, we addressed only seven of the hundreds of terms used figuratively in the biomedical literature. Although the seven terms provided enough information to see that no single approach is a winning strategy, additional annotations will be needed for testing approaches to figurative language detection on PubMed scale. We also found that for many remaining terms figurative use in PubMed is infrequent and additional sources of figurative language will be needed for training. For example, butterflies in my stomach is used in PubMed only two times, and butterflies AND stomach 20 times. More data will be needed to train a classifier to distinguish between these two titles:
Butterflies in My Stomach: Insects in Human Nutrition Neurotic butterflies in my stomach: the role of anxiety, anxiety sensitivity and depression in functional gastrointestinal disorders
This work presents an initial exploration of the use and detection of figurative language in biomedical publications. On the one hand, figurative language is known to play an important role in thought processes and in science, and therefore being widely used in biomedical publications, on the other hand, automated detection of figurative language in the biomedical publications has not yet attracted research. To explore feasibility of automated detection of figurative language, we created a collection of documents annotated for literal or non-literal use of seven terms that are known to cause errors in automatic indexing of biomedical abstracts with MeSH terms. We then explored sentence and tokenlevel classification approaches to detection of figurative language using CNN-RNN, logistic regression and transformer models. With the exception of one term, fish, our performance is on par with the state-of-the-art achieved in the open domain evaluations. We hope that the interesting problem of detection of figurative language in biomedical text, the dataset, and the automated approach to creation of the training sets outlined in this work will bring about further research in this area.
Data & code: https://ii.nlm.nih.gov/DataSets/index.shtml
This work was supported by the intramural research program at the U.S. National Library of Medicine, National Institutes of Health.
We thank Alan Aronson, Francois Lang, Laritza Rodriguez and Sonya Shooshan for judging parts of the collections. We thank Anna Ripple for constructing PubMed searches.