Semantic Interpretation of Image and Text consists in recognizing the entities and relations shown in the image and mentioned in the text and to align them with some ontological resource that contains structured knowledge about these entities and relations. A proper semantical description of entities and relations aligned in images and texts would allow for more accurate retrieval of images in the search tasks. In this research, we propose a framework for Semantic Interpretation of Image and Text, which will utilize both the low level and semantic features of image and text by using background knowledge extracted from online Knowledge base. A successful approach for the Semantic Interpretation of Image and Text should address the following challenges. First, we have to recognize entities shown in images and describe in text. Second to make links between entities, third to make links with the entities in the Knowledge base and fourth relations between entities. To solve these complex tasks, we will use state of the art methods for image object detection and textual entities recognition. Furthermore, for mapping textual, and visual entities with entities in the knowledge base, we will use supervised machine learning techniques that exploit background knowledge. In order to provide a method for training our algorithms, and to evaluate properly the results we intend to develop a dataset consisting of images, image captions, bounding box annotations, links between visual and textual entities, linked to the knowledge base and the semantic meaning of entities.
When we see a picture surrounded by text on social media, website or in a book, the rst important thing is the objects (entities) in the picture, entities in the text, the information of entities, and what relations these entities have with each other. The problem of identifying and semantically synchronizing entities and relations shown in an image and mentioned in text, with the alignment in those knowledge bases which already have knowledge about these entities and relations will enrich the information for the machine. This semantic interpretation of an image and text is the results of very complex processing which involves: 1) recognition of entities in image and text, 2) linking of entities in image and text, 3) alignment of these entities with the knowledge base and, 4) relations between entities. In my PhD, I am proposing a framework, which will recognized entities in image and text, relations between entities, links between visual and textual entities, and the links of these entities with the knowledge base which stored structural knowledge about these entities.
The source of information is the image with text, and the goal is to make links between visual and textual entities and then connect these entities with the Knowledge base. Fig. 1 shows the scenario in details. Fig. 1a shows an image with text, which describes (the girl is eating ice-cream in Piazza Duomo). The main objects that appear in the image are girl, ice-cream, hands, shirt and relation eating between girl and ice-cream. Entities that are mentioned in text are girl, ice-cream and Piazza Duomo with relation eating between girl and ice-cream. The rst step is to detect the image bounding boxes correspond to each entity in the image and textual entity, and make alignment between image and textual entities, then detect relations between entities as shown in Fig. 1b.
After aligned visual and textual elements (i.e., entities and relations), the next step is to connect these objects shown in the image and describe in the text with some ontological resource that consists of structured knowledge about these entities and relations. The result is graphically represented in Fig. 2.
In literature, there is a lot of work which involves text and online knowledge
bases [
In the rst phase of our proposed framework, we will recognize and links
the visual and textual entities to the online knowledge base and then nd the
relations between entities. We will use state of the art objects detection [
To develop my framework for the problem of semantic interpretation of image
and text, we start from building a dataset consists of images, text, and knowledge
base alignment. We start from Flickr30k [
Flickr30k (VKS Flickr30k). VKS Flickr30k consists of images, image captions, bounding box annotations, links between visual and textual entities, visual and textual entities linked to the Knowledge base and semantic meaning of entities.
For visual and textual entities recognition, we will use state of the art object
detection [
In the rst phase of the proposed framework, VKS Flickr30k dataset will be
used for training and testing algorithms to map textual and visual entities with
entities in the knowledge base. In the second phase, we will extend the work
for relations (events) detection and identi cation between entities. To evaluate
the results of our proposed framework, we will use accuracy, precision, recall
(sensitivity) and F1-score. We will compare our proposed framework with the
existing state of the art approaches [
We develop VKS Flickr30k dataset, which stored enrich details of image and textual entities and linked entities to a structural knowledge base. VKS Flickr30k dataset consists of 31,783 images with ve captions per image and 244,035 coreference chains (connect single bounding box entity to multiple instances in the image captions) to link 275,775 bounding box to 513,644 textual objects (entities). Visual and textual entities are connected to YAGO knowledge base by YAGO entity ids. Flickr30k entities are divided into eight distinct classes of people, clothing, bodyparts, animals, vehicles, instruments, scene and other.
During the development of VKS Flickr30k dataset, we used Flickr30k dataset for images, text, bounding box annotations and links between visual and textual phrases. To connect these noun phrases with the Knowledge base, we passed text through PIKES for semantic knowledge graph extraction. We connect YAGO entity Ids with textual entities extracted by PIKES. Fig. 3 shows step by step processes of VKS Flickr30k development. In my PhD, we are proposing a framework for semantic interpretation of image and text, which will link entities and relations shown in the image and mentioned in the text and to align them with reference Knowledge base that contains structural knowledge about these entities and relations. For training our algorithms to solve these complex tasks and accurately evaluate the results, we develop a dataset called VKS Flickr30 for semantic interpretation of image and text. In the future, we will use state of the art methods for image objects detection and textual entities recognition. Also, we will use supervised machine learning techniques for mapping visual, and textual entities with entities in the knowledge base. In the second phase, we will extend the work for relations (events) detection and identi cation between entities.