Large amounts of written information flow in news, article databases and knowledge forums, and searching for required information often requires using proper keywords. Semantic Web ontologies describe a vocabulary of concepts and terms which are useful for Information Retrieval in their specified domain.
Ontologies can provide enhanced results in information search when multiple taxonomies of terms and keywords are used in composing a large document database. Such databases may cover for instance a multilingual, cultural or biological domain [1] where problems may be caused by diverse term variants, historical synonyms, misspellings and foreign terms.
Using automated content analysis based on machine learning, the amount of manual work in concept annotation and keyword tagging can be reduced. Automatic concept tagging makes it possible to apply ontology-based retrieval methods that combine keyword search with concept-based search [2]. This leads to a better coverage and quality compared to standard information retrieval.
I suggest here a method where a machine learning model is trained for semantic tagging. For demonstration purposes, a model is trained with a small annotated text, containing a set of examples of the terms described in the annotation ontology. In figure 1
The method intends to assist the process of adding semantic tags to individual sentences and paragraphs in new documents to be added to the database. The input text in new document is analyzed a sentence at a time, with a dependency parser. The semantic similarities between terms in new input and the reference text (training data) are estimated by the similarities in their syntactic dependencies in the new document.
A high syntactic similarity is considered as a possible semantic match. Furthermore, the method can be extended to detect a new term, without proper match. The approach also finds the closest match for an out-of-vocabulary term, that is not yet introduced in the current ontology.
By using an unsupervised machine learning method such as the Self-Organizing Map (SOM) we can even give a comprehensive, visual impression of the collection of articles available in the text database. A SOM is a Neural Network model that is different from most modern neural network architectures. It is less datahungry and it is tolerant to noise in the training data [3]. This way, it can also classify rare term occurrences that have no exact match in the training data set, by guessing the best partial match based on the syntactic features of the term. This makes it an interesting alternative model to learning features for terms, associated with a set of ontology concepts.
In the experiment, the sentences of text corpora are processed with the Stanford Parser (Penn PCFG dependency model for English). The sentences are tokenized as part of the dependency parsing process. Each token (in its actual word form) in the sentences are indexed in the training sentence bank. The dependency arcs and bi-arcs are extracted from the parse output, and each arc forms a feature descriptor on the tagged word. The arcs are bi-directional so that one dependency is tagged on the head and dependent word. The semantic features for individual word tokens are random projected indexes of the features produced by the Stanford Parser model. The syntactic context representation is very similar to the one in Dependency-Based Word Embeddings as in [4].
The experimental OntoR tool was developed in the R statistical programming environment, using the CRAN library som, based on SOM-PAK, the Self-Organizing Map Program Package (version 3.1) [5]. A screen shot from the On-toR user interface (2) demonstrates how ontology-based term structure is reflected as a SOM map containing the keywords. A modified plot of the SOM map has been developed to explore the mapping of ontology term classes and super-classes over the term model trained with the sample corpus.
The areas in the outcome SOM grid show the taxonomical hierarchy that can be seen in the mapping of ontology-terms in the unsupervised model representing the training corpus. Multiple clusters were seen with both the subterms and terms categorized in the same map cell and their neighborhood. This supports the earlier work hypothesis that a data point cluster with an internal topology, or a structure, has a strong tendency to distribute over multiple adjacent cells over the SOM lattice.
The WebSOM project [6] inspired work towards unsupervised term learning and classification with the use of Self Organizing Map, which works and learns on Internet sourced text articles, and extracts topics based on the tokens found in the text articles. Also the work by Tanev et al [7] describes the main paradigms on weakly supervised ontology population, one being the term pattern related method and the another being the context sensitive triggering. The approach described here is a contextual extension to the WebSOM model since it adds syntactic dependencies as additional information over the tokens found in the text. In this work, the suggested method for mapping concepts occurring in text into the SOM grid will analogously support automatic tagging of new term candidates in document databases. This seems applicable especially for hyponyms (terms for subclasses) and synonyms for previously categorized terms. In the following phase of the experiment, the internal weighting parameters for building numeric embeddings from syntactic analysis will be evaluated and analyzed in contrast to using plain word-based embeddings.
This method can also be seen applicable in weakly supervised ontology concept population for adding new term candidates, since the presence of some rare terms occurrences were found in distinct areas of the SOM map in the experiment. This aim to use the SOM in concept mining is also supported by work by Honkela and Pöllä [8]. The set of ontologies used with OntoR is not restricted to a medical domain, as seen with the sample experiment. The used ontologies can even cover multiple topics, for instance, history, politics, science and culture.
Acknowledgments: Research and development of the method and the OntoR tool have been supported by the MOLTO EU project and Whitelake Software Point. The suggested model and inspection of the methods described here have been developed and supported with feedback from professor Timo Honkela and the Research Seminar in Language Technology held at University of Helsinki.