The aim of our research is to create appropriate models to formalize knowledge in order to support reasoning on knowledge intensive domains. Lately, our research is focused on the creation of visual knowledge models applied to imagistic domains.
Imagistic Domains are the ones where the domain expert starts the problemsolving process with a visual pattern matching over visual information input, which will further support the more abstract processes of inference. Image-based diagnosis in Medicine, and visual analysis of petroleum-reservoir rocks are common tasks executed by domain experts and are highly based on visual-information input.
Visual knowledge refers to the set of conceptualizations owned and used by individuals to recognize the relevant features in the domain and start the inference process. This kind of knowledge is built through the experience and differs from the propositional knowledge in the sense that the expert is not able to either express it verbally or in a sentential manner [Abel et al. 2005]. Since ontologies formalize knowledge by making use of propositional vocabulary, it is showing itself a challenge to formalize and represent visual knowledge using ontologies.
Visual knowledge and image are disjointed concepts. The Ullmann triangle [Ullmann 1979] describes the relation among an Object in the reality, a Concept in a conceptualization and a Symbol in a language (its extended version is presented on Figure 1, firstly published in [Lorenzatti et al. 2011]). An Object is supposed to be a real or concrete object where its existence can only be referred by the perception process of someone. Concepts, by their side, are abstractions that humans create over objects in order to deal with the external world. Symbols are the trial of individuals for representing concepts during the externalization process of communication in order to share a conceptualization among a community. A concept can have different representations and they will vary according to the purpose of the chosen language. The image concept is normally referred as a pictorial representation of a concept.
When mentioning visual knowledge we are referring to the conceptualization vertex of the Ullmann triangle. A pictorial representation is an alternative representation that can be used in the externalization process of communication. Its choice depends on the requirements for externalizing visual concepts, like in spatial or location problems that require visual representations for sharing concepts. Thus, we add the new vertex "Pictorial representation in a Visualization" to the Ullmann triangle (in the baseline of Figure 1). Thus, the same concept can be simultaneously represented by a propositional symbol in a language while being represented by a pictorial symbol in a visualization and this correspondence between them is called anchoring.
The visual knowledge that we are modeling does not reside on the image content, but into the conceptualization of the domain experts. Thus, we do not seek to model the abstract visual patterns found in the image content, but our objective is to model the set of visual concepts residing in the experts' mind.
Creating knowledge based systems requires deep systematicity in the engineering process when using ontologies to represent knowledge [Guarino and Welty 2002]. Guarino and colleagues [Guarino and Welty 2004] have proposed a systematical and domain independent methodology applied to evaluate and validate the ontological choices taken when building up an ontology. The analysis is based on ontological notions coming from philosophy and are represented by formal meta-properties.
We apply the unified foundational ontology -UFO -of Guizzardi [Guizzardi 2005] with the goal of orienting the semantic negotiation of the concepts and the consensus achievement among the interacting agents (artificial or human) within a community [Gangemi et al. 2002]. UFO was built by meta-concepts, like kind, role, phase, and mixin, whose definitions were based on formal meta-properties. The meta-properties like identity, rigidity, and uniqueness impose constraints over the taxonomical relations which prevent the creation of inconsistent knowledge models.
The meta-property identity refers to the problem of identifying a single instance based on its intrinsic characteristics that, make it unique [Guizzardi 2005]. The identity criterion concept involves the analysis of conditions and characteristics which, for example, allow the identification of a person along the time. The rigidity meta-property is related to the essentiality of an individual having a property in order to preserve its identity [Guizzardi 2005]. Being human is an essential property to all human beings, otherwise they will loose their identity. Being hard is not essential to all instances of hammer since, hammer toys should not be essentially hard. The meta-property uniqueness deals with the problem of identifying the parts and limits of objects [Guizzardi 2005]. The meta-property is used to analyze the composition of an object in order to identify if it is a whole or a composition of other objects, for example. Instances of the property being a lake have well defined boundaries, while instances from the property being water have not. Thus, the analysis of the meta-property unity prevents modeling the property being water being subsumed by the property being a lake, which sounds intuitively correct. In fact a lake is not the proper water but, it is constituted by water.
Rigid sortals are concepts where individuals have ontological rigidity. A tree is a rigid sortal, while teacher is not because an individual can become and stop being a teacher through time. A concept classified as a Kind is a concept which supplies the identity criterion to its instances, while a Sub-kind is a concept which inherits its identity criterion from the kind concept. A Quantity is a concept in which the set of its individuals refers to portions of some substance like water, for example. Quality dimension is a structure used to represent the set of values (Qualia) associated to a rigid sortal. Each of the values from a quality dimension is a Quale.
The conceptual modeling process starts by selecting objects in reality and evaluating their adequacy with representing primitives. Selecting the best primitive that fits for representing an object is a key point in conceptual modeling. Thus, meta-properties and meta-concepts analysis have turned to be a powerful tool on the evaluation and selection of primitives during the ontology construction process.
A representation is characterized by an entity assuming the role of representing another one. Furthermore, a representation is characterized by the set of relationships established between the representing entity and the represented entity [Gurr 1999]. The interpretation of a piece of knowledge represented using propositional languages is achieved by concatenating the symbols. However it is possible to explore the visual-language system intrinsic properties in order to explore the direct correspondence among the concept properties and the visual representation properties [Gurr 1999]. Atsushi Shimojima defines as inferential "free-rides" the possibility to capture semantic information through the direct correspondence among the representations' and the concepts' visual properties [Shimojima 1996]. Figure 2 depicts the use of free-rides representing two equivalents logical syllogisms. The same conclusion (iii) is achieved in a more straightforward way on Figure 2-b while exploring the correspondence between the concept and the visual representation properties.
Based on the main characteristics, Peirce apud [Burks 1949] classifies representations as symbol, index, and icon. A symbol has no direct or indirect relationship with its meaning. The meaning of a symbol is established by convention, which must be known. Indexes have associative and indicative relations with their meanings. The mercury column height of a barometer is an index of the atmospheric pressure. Icons seem with what they mean. Thus, the meaning of an icon is captured by the same perception process used to recognize the originally represented object or event.
In order to formalize visual knowledge using domain ontologies, two hybrid metaconstructs are formally defined. They are considered hybrid because the concepts classified by them are represented by a pair consisting in one propositional and one pictorial representation. While the propositional representation formalizes the domain vocabulary and it is used for communication purposes, the pictorial representation formalizes the visual knowledge that the domain expert is not able to verbally express. The representations do not fully overlap but they complement each other. These two meta-constructs are built based on the Guizzardi's UFO and the extension of the Ullmann triangle.
PictorialConcept is the meta-construct responsible by representing visual types. Based on the meta-properties proposed by Guarino this meta-construct gives or carries an identity criterion, has ontological rigidity, and unity property. According to that, and based on the UFO, this meta-construct represents the concepts classified as rigid sortals, i.e., it represents kind, sub-kind, collective, and quantity meta-concepts. The left side of Figure 3 depicts the propositional representation of a geological sedimentary structure while the right side shows the iconic representation used to express the non-verbalizable knowledge. PictorialAttribute is the meta-construct created to represent the quality dimensions' values from (visual) concept attributes. The quality dimensions classified by this meta-construct give or carry identity criterion, have ontological rigidity, but, differently from the previous meta-construct, they do not have unity criterion. Therefore, based on the UFO, this meta-construct represents concepts classified as quale, i.e., it represents the value set associated to a quality dimension. Figure 4 left side shows the propositional representation while the right side depicts the pictorial representation for the quale of the sorting attribute of sedimentary rocks.
We have built a stratigraphy domain ontology based on the proposed meta-constructs. Stratigraphy, an imagistic domain, is a sub-area from geology responsible to study the formation processes of sedimentary rocks [Press et al. 2004] where the geologist's analysis is based on the rock visual features. The formalized ontology represents the concepts both in Portuguese and English. Using the proposed meta-constructs the iconic vocabulary is defined for the visual types and for the visual attributes in association with the propositional vocabulary.
The visual knowledge based system, built upon the domain ontology, is used to visually describe petrological features (litology, textures, structures) in cores collected from exploration wells, that will be further used to understand the structural correlation of geological units in petroleum exploration. The interface of the system is fully based on the pictorial concepts and attributes which keeps the interaction closer to the way geologists use to describe cores. The pictorial features are, by their side, internally related to propositional concepts that give to the system the capability of extract geological correlation. Therefore, the hybrid model guarantees a unique ability to our system: depicts to the user core descriptions based on pictorial visualizations the user are used to, while capture real knowledge for further correlation and inference. The full model is persisted in a relational database as well the user data.
The knowledge base is built to be scalable in order to grow as the expert geologists use the system and find/create new (visual) concepts and (visual) attributes. Once the knowledge base is referenced by the user-data database, any change on the former is automatically reflected on the latter, ensuring the data integrity.
Domain novices take long time and lots of resources to be trained and accumulate sufficient knowledge to become a domain expert. When dealing with image-based problems, experts build their ability by accumulating internal abstract representations of the key visual features that allow solving problems in the domain. An intrinsic characteristic of the imagistic-domain experts is to use drawings to externalize and express their mental models. Imagistic-domain experts have difficulties to externalize their knowledge and make it accessible to share with others.
Integrating visual content into ontologies opens the possibility to formalize the visual knowledge from imagistic-domain experts. Our proposed meta-constructs give the first step in that direction. One advantage of using the proposed meta-constructs to formalize visual knowledge is to make a very specialized kind of knowledge accessible to novices. Another advantage of their usage is to open the possibility of constructing visual-knowledge based systems, since the knowledge becomes machine readable. Future directions of the studies will be exploring the relationship among properties of concepts and properties of pictorial representations in order to better understand the externalization process and to create more accurate representations.
The scholarships that supported this project were sponsored by the Government Agency of CNPq. SEBRAE and Endeeper Co. have provided financial support.