Neural nets are the backbone for many innovative applications when it comes to making sense out of unstructured data. This technology, however, also has some significant downsides. Training these neural nets requires vast amounts of annotated training data and computational power and is, thus, expensive. Adding, altering, or removing concepts all trigger costly retraining of the neural net. Semantic technologies can curb these systemic disadvantages. They can be connected to the neural net and augment the detection. The ontology can be easily adapted to new situations and use cases without requiring much computational power. This paper presents a prototype application for connecting the general-purpose OpenImages detection with a semantic. Freeto-use training datasets provide a cost-efficient way to use neural nets - often, there exist even pre-trained models for the large machine learning frameworks. The semantic contextualizes the low-level knowledge and enables the realization of specific use cases. We further propose a methodology to transfer the probability of the image detection and the prominence of items in the picture to the ground truth of the ontology by introducing a new measurement Semantic Confidence (SC).
Recent advantages brought tremendous progress in image classification, object detection, and computer vision systems in general. Through the rise in computational power, neural nets are today’s state-ofthe-art for image recognition tasks and are the backbone of many artificial intelligence (ai) applications.
However, convolutional neural nets (CNN) also have downsides: They need large quantities of annotated training data to produce valid results. Due to the extensive amount of required training data, creating these neural nets is costly in terms of computational and human resources. Every change in the detection requires a retraining of the neural net, which makes the approach inflexible regarding fastchanging requirements.
That being said, the use of neural nets and deep learning for recognition tasks are (today) without a
realistic alternative. Nevertheless, a method for strengthening the reuse of neural nets and reducing the
need to collect training data and training itself could significantly mitigate the disadvantages of this
technology. The combination of semantic technologies with a general-purpose image detection (like
OpenImages [
In this paper, we present a semantic augmentation for the popular open images training set. We
implemented a prototype that builds on TensorFlow and connects an ontology that generalizes the
results of the image detection. Further, we introduce a measurement Semantic Confidence (SC) that
transfers the fuzziness of the image detection and the prominence of items in a picture to the ground
truth of the semantic. In a previous paper [
The rest of the paper is structured as follows: The following section gathers the current state-of-theart in combining ontology and image recognition. Section three then presents the prototype by describing the architecture of the application and the calculation of the SC value, instructing how to use the software, and performs a preliminary evaluation before the research is concluded.
The idea to connect deep learning recognition tasks with a semantic is not entirely new. Two recently
published literature reviews by Ding et al. [
Besides these two literature reviews, we additional consider relevant the works by Reiz et al. [
As the literature review shows, the idea of connecting a semantic to enrich the results of the image detection is not entirely new – various authors have already connected the different kinds of ai technologies. However, our approach is unique in the way that (1) we can eliminate the need for training the system entirely by building upon and adapting popular, freely available systems and (2) using a novel metric to connect the level of detection uncertainty and relative importance of an object in the picture to the semantic.
The following section gives an overview of the developed prototype. At first, the software’s architecture is presented, followed by an explanation of how to use the software. At last, we perform a preliminary evaluation of the new approach and derive the next steps.
SemOI is available under an open source license. The source code is available online in Github2. The
application is written in python and utilizes the web framework “Django”3, running inside a docker4
container. Taking the categorization framework for hybrid systems by [
The detection is based on TensorFlow with pre-trained models from the model sharing website
“TensorFlow Hub”5. The models are trained using the OpenImages training data set [
Figure 1 gives an overview on the architecture of the prototype. At first, the out-of-the-box tensorflow model detects the given objects. After a detection is completed, the application calls the semantic augmentation unit with a list of the detected items. Each object detector has an ontological twin in the semantic and can be identified using the detector label or OpenImages classifier ID. The detectors are connected with new, contextualized items (cf. Figure 2). In the semantic augmentation unit, the request is translated into a SPARQL-query that fetches results from the ontology, which returns the context items for further analysis.
After the detected contextualized items are fetched, the semantic confidence is calculated. While
the value of the first SemOI version [
= 6,108 nature + 6,374 park + 0,9801 garden = 100% park + 95.8 % park + 15.8% garden (Step 2) (Step 3)
The prototype is available online at semoi2.informatik.uni-rostock.de. At first, one needs to upload a picture. As soon as the upload is completed and the image is presented in the preview window, the application asks the user to choose the underlying model – “Faster” triggers the MobileNet, “Accurate” utilizes the ResNet. Even though on powerful machines (like the server that the application runs on), the calculation times are not expected to differ widely, this distinction allows the simulation of different application scenarios.
A click on “Analyze” triggers the recognition and augmentation tasks. After the analysis is completed, two boxes appear. The left box “Detected Objects” consists of the results from the image recognition; the box on the right “Semantic Augmentation” contains the augmentation results given by the semantic. While the software is not yet adapted to a productive environment, the evaluation shall outline the current performance and identify possible weak points. For our analysis, we manually decided for every picture if it fits to the inferred classes and whether some of them are missing. That being said, the evaluation does not consider the results or efficacy of the image recognition that is the input for the semantic. A possible error in the recognition, thus, propagates to the semantic.
The ontology is currently not targeted at a specific business use case but generalizes the results and infers contexts. Without these use cases, a precise evaluation scenario is missing. Taking the example of Figure 3, the semantic inferred the situation garden, nature, and park. One could argue that additional elements in the ontology fit the picture as well, in this case, e.g., holiday. Thus, applying binary relevant / not relevant categories is, at times, fuzzy. In these arguable cases, we evaluated in favor of the application.
The values for our analysis are presented in the table below. In total, we analyzed ten pictures. The application reached a precision value of 73% on average of all figures (67% average of the detected items) and a recall of 88% (89%). The results of the evaluation are available online7.
On the one hand, the values indicate that we need to work further on the mapping accuracy and more carefully design the ontology to prevent irrelevant items from being inferred. On the other hand, however, the significance of the evaluation is limited without a real-world application scenario. Such a use case with a specifically developed ontology would further validate the capability of the system.
Training neural nets for image recognition tasks is a tedious and costly task. While there are free-to-use models for general-purpose detection jobs available, these models will most likely not fit directly to a specific use case. The connection with a semantic, however, can provide this adaption.
In the paper, we presented a new version of SemOI, a prototype for the connection of OpenImages with a semantic augmentation. The driving idea behind this publication is the simplification of setting up AI applications using semantic adaptions to pre-trained neural nets. These quick adaptions are enabled through adjustments of the ontology. Possible changes are manifold: One could add or alter context items, change, delete and add relations between the contexts and the detected items, or switch the underlying recognition model to reuse another public accessible or private neural net. The semantic can be easily adapted to a specific use case, enabling the rapid development of new recognition applications. We believe that this connection of pre-trained models with ontologies has the potential to mitigate a lot of the downsides that come with the deployment of image recognition technologies.
The current ontology is crafted manually. We first built a script for translating the open images taxonomy into rdf, then connected the given detectors to context items. Future applications might also, depending on the given use case, consider the application automated ontology creation techniques based on machine learning or text retrieval algorithms.
However, the new methodology also has limitations. The target application needs to be concerned with at least part of the detectors/classifiers in the pre-defined model. The new approach does not work if no pre-trained data is available, e.g., in highly specific detections like material properties in a factory.
The presented prototype proves the general feasibility of the idea. Further research is concerned with implementing more sophisticated detection scenarios. We also plan to further work on the semantic confidence (SC) value, to incorporate a weighting of the relations, e.g., based on their specificity, utilizing the research of ontology matching algorithms. At last, implementing the software into a realworld application would arguably deliver the most learnings regarding the future challenges we need to tackle. Here, we invite the community to fork and adapt the prototype and share their experiences. We thank the students Henrik Bongertmann, Carl Pommerencke, and Daniel Hahn for their support in realizing the Prototype.