In scientific disciplines where research findings have a strong impact on society, reducing the amount of time it takes to understand, synthesize and exploit the research is invaluable. Topic modeling is an effective technique for summarizing a collection of documents to find the main themes among them and to classify other documents that have a similar mixture of co-occurring words. We show how grounding a topic model with an ontology, extracted from a glossary of important domain phrases, improves the topics generated and makes them easier to understand. We apply and evaluate this method to the climate science domain. The result improves the topics generated and supports faster research understanding, discovery of social networks among researchers, and automatic ontology generation.
The authoritative source for conveying the latest climate research findings, recommendations and mitigations steps
is the Intergovernmental Panel on Climate Change (IPCC) [
For a new climate scientist, absorbing this information in order to perform research or make policy contributions can be daunting. However, if machine understanding of these reports could be used to summarize, synthesize and model the knowledge, the researcher’s task is improved. We propose this could improve the overall scientific research contributions that could be made by making the process more efficient.
In our previous work [
We quickly discovered that the standard topic modeling approaches did not work as well as we hoped on the text
in the report domain, and were even less effective on the cited research documents from the scientific domain. One
reason is that scientific literature is written more formally and typically contains more phrases that provide the context
through which one understands the literature [
Demidova, A.J. Zaveri, E. Simperl (Eds.), ISBN: 978-3-89838-736-1, 2018, AKA Verlag Berlin. Copyright held by the authors.
Topic modeling has a long history of relevance to natural language processing, often used to model large collections
of text documents applied to problems such as document summarization [
Topic modeling is often used to find word vectors that best represent the themes in a collection of documents. Each word vector contains a set of words or word phrases, where each word/word phrase has an associated probability that represents that word or word phrase’s contribution in representing a particular theme. The approach most frequently used is to extract words from documents, removing commonly occurring words and symbols, and to generate a ‘collection vocabulary’ from the set of words. Sometimes the vector is a set of singleton words, however, word n-grams, where the ‘n’ represents the number of words that make up the phrase, are also used.
It has been shown that when word phrases are used in topic models, the topics tend to be more relevant to the
collection of documents [
Particularly for scientific domains, phrases often convey special meaning that is lost by treating them as single
words. The challenge with using word n-grams is knowing which sequences carry such meaning, making automatic
phrase or word n-gram extraction problematic. The fact that key phrase extraction is currently an active areas of
research among [
To understand this challenge further, we used tools from NLTK [
Instead we ground the topic modeling process on a domain-specific ontology seeded with predefined key word phrase concepts obtained from domain-specific sources such as domain experts, and by data mining semi-structured sources. In particular, we found the IPCC glossaries and domain experts to be good sources for defining climate-related word phrase concepts. This grounding process contextualizes the topic model such that the topics are more relevant to the domain that is being modeled. For example, given a climate change domain ontology, if a document being used to train the topic model included text unrelated to climate change, those words would potentially have a lower weight than the words which represented the ’known’ or ’seed’ concepts found in the ontology.
Table 1 shows examples from the data-mined and a bi/trigram extractor approach. Using a sample of climate
change data, all bigrams and trigrams were annotated. The documents were processed using the data mining approach,
and also using an n-gram extraction approach. The extractor approach was only able to recover 6% of the word phrases
that could be 100% represented using our data mining of glossaries approach. Similar results were found for space
science data. Though more recent research [
Glossaries typically provide key concepts that are relevant to a particular domain and consist of words and phrases whose length can be anywhere from two to ten words. For example, in the climate change community, the phrase ”soil moisture’ implies something much more meaningful than ‘soil’ and ‘moisture’ alone. Furthermore, the singleton words might be much more frequently found than the phrase. This is often an important artifact in the topic model. For example, ‘black carbon’ is a significant concept in climate change because of its impact on the research at a certain period of time. The word ‘black’ may not frequently occur with other words but among climate change literature the word ‘carbon’ occurs quite frequently with other words. An example of this is shown in Figure 1, where the phrase ‘black carbon’ has a significantly lower occurrence across the Physical Science, Impact and Synthesis books for Assessment Report 3 than the single word ‘carbon’. The single word ‘black’ not only appears within the phrase ‘black carbon’ but also within the phrase ‘black spruce’, ‘black-footed ferrets’, and within other phrases.
Our approach entails modeling the structure of the reports and their citations in the ontology. There were five IPCC assessment reports, AR1-AR5, each of which follows a similar structure consisting of four distinct books: Physical Science Basis, Impacts, Adaptations and Vulnerability, Mitigation of Climate Change, and Synthesis Reports. Each book has between 11 and 25 chapters and a chapter typically cites between 800 and 1200 external documents. The ontology consists of a similar structure as shown in Figure 2. We then obtain a list of concepts of importance from domain experts and domain glossaries. Figures 3 and 4 shows example pre-defined seed concepts represented in our IPCC ontology
Fig. 2: A Partial IPCC Ontology Used for Guiding Topic Modeling.
In addition, acronyms are mapped to the actual phrase and treated as the same concept. For example, ‘ENSO’ is
treated as the the same concept as ‘El Nino Southern Oscillation’. The ontology is then read into memory for the
preprocessing step of the topic modeling phase. As we perform preprocessing of text, we use the ontology concepts
for weighting concepts we find in the text. We do this for both the report data and the citation research papers. This
retrieval process is described in more detail in [
Since ontological word phrases are used to ground the topic modeling process, topics are also represented by an ontological structure. Topics are implicitly linked into the ontological structure, along with documents. The IPCC reports have citation information for each chapter in the report. In our work we built a topic model for the citations and another topic model for the reports. We used the ontology as a means for conveying how the two topic models were related to each other using the topics generated from the two different models as a bridge between the two models. For example, Figure 5 shows a set of common concepts from two mapped topics captured ontologically.
Since we captured the citation information ontologically, we can use this information to discover interesting social communities. For example, a class in the ontology called ‘Publication’ can have many ‘Authors’. When a number of authors are referenced together across multiple publications, this can give insight into a social relationship between these authors. Specifically, if the same authors are cited in three different chapters that relate to ‘Black Carbon’, these authors form a relationship with a common node concept ‘Black Carbon’. The ontology can also be used to observe social networks given the relationships between authors that cite authors that are also cited in the same chapter.
Figure 6 shows an example in which ‘Callaghan’ was an author in a paper cited in one chapter and author ‘Abbs’ was also cited in the same chapter.
Of the paper for which ‘Abbs’ is an author, a relationship was found between ‘Abbs’ and ‘Callaghan’ due to the fact that they were both cited in the same chapter and one cited the other in their cited paper. The ontological representation for citations can also shed light, in general, on which authors are cited across books and chapters.
We used our ontology-guided topic modeling approach to model both the IPCC reports and the citations. We first converted the reports and citations to raw text, we extract meta-information regarding the citations and we used the ontology to link this information together. When a predefined concept from our ontology was found, we weighted the phrase higher than non-ontology concepts by 5%, 10%, 25%, and 50%. To understand how ontological concept-based topic modeling differs from standard bag of words topic modeling, topic models were built using both approaches. The ontological grounded topic model was compared to a bag of words model that used the same data set, the same stop word removal, but without the ontology concepts grounding the modeling. Therefore it did not contain word phrases but could potentially contain word phrases as individual words. Perplexity was used to evaluate the two models.
In this experiment, the IPCC ‘Physical Science’ book for assessment periods one through five were used. There were 61 documents in total used in this topic model, with 11 documents in AR1, 11 documents in AR2, 14 documents in AR3, 11 documents in AR4, and 14 documents in AR5. The assessment reports were used for this experiment and each chapter was treated as a document. For perplexity evaluations, a held-out set was used for each assessment period beginning with AR2. One way to understand the differences between these two models is by simply observing the topics.
For example, in Table 2 the two topics highlighted appear to be the ‘radiative forcing’ topic for each model. The ontology guided model provides a richer set of words since the topic is composed of phrases. The same is also true for the ‘El Nino’ topic for each model. Though both models provide visually relevant topics, the ontology-guided model provides concepts that are more specific to the scientific domain.
A common metric used to evaluate topic models is perplexity. Perplexity measures how well a probability distribution predicts a held-out sample. A lower perplexity indicates the model is better at prediction. Perplexity was measured across assessment periods where t is the held-out set of documents, using a model trained on t 1 assessment documents, given t ranges from 2 to 5. Each experiment compared the ontology-grounded and non-ontology-grounded methods. In Figure 7, AR4 was used to build the topic model and AR5 was used as the held-out test set. Similar perplexity was measured for the other assessments.
Given the size of this data set, the number of topics that best represents this data set is between two and six topics. This was confirmed by visualizing the topics. When the topic size grows too large, topics tend to overlap much more. The difference in perplexity for the same held-out data set is shown, with perplexity measures (lower is better) indicating the ontologically grounded topic modeling method may improve perplexity which in turn means it may offer better predictability for scientific data. As a mean of reference, the training set is also used as a held-out set so as to show the difference in scale between the ontological method and the non-ontological method. This provides a general idea as to the performance of these two approaches.
We performed experiments to compare the ontologically-grounded word phrase approach with a standard approach, both of which use the same method for stop word removal. With each experiment the top N words are used for a given set of topics. Given a second topic model example comparing the ontology-grounded model and the nonontology grounded model, as shown in Table 3, the topics in the ontology-grounded model are more closely related to terminology found in scientific research papers when compared with the non-ontology grounded model. For example, the first four words ‘temperature’, ‘anthropogenic’, ‘carbon dioxide’, and ‘radiative forcing’ are more descriptive than ‘change’, ‘ocean’, ‘level’, and ‘global’.
Using Google to search on the combination of words, two different sets of documents (examining the top three documents) are returned. With the word phrases contained in [‘temperature’, ‘anthropogenic’, ‘carbon dioxide’, ‘radiative forcing’] the top documents included a Wikipedia page related to ‘Radiative Forcing’ and two IPCC report chapters, plus a number of Google Scholar research paper suggestions. With the four words contained in [‘change’, ‘ocean’, ‘level’, ‘global’], the top three results included pages related to ‘sea level rise’, the first hosted by NASA, a second page on the same concept hosted by NOAA, and the third hosted by EPA. There were no Google Scholar suggestions. This further supports the assertion that by grounding the topic model with concepts from the ontology, the topics created are more context-specific and hence more fine-grained than the standard approach. For scientific data, this an important point, as this level of detail provides the context needed to really understand scientific documentation.
Typically topic models use a bag-of-words approach and more recently 1-hot encoded bags of words, leaving it
to the implementer to decide what that bag of words contains. Since the early 2000s, research has focused on ways
of improving topic modeling by adding context [
Early research explored ways of using word n-grams [
Recent developments in topic modeling have started exploring its applicability to scientific concepts. Hall et al. [
More recent work [
We plan to apply this technique for other domains to get more experience and further test and evaluate the idea. We are collecting glossaries and concept lists for the cybersecurity domain and plan to develop topic models using them. We also hope to explore their use to enhance word embeddings.
This work was partially supported by a grant of computational resource services from the Microsoft AI for Earth program and a gift from the IBM AI Horizons Network.