Knowledge representation is central to Artificial Intelligence, since any intelligent system should include knowledge about the world and provide representations for objects, processes, properties, and relations between objects. The cognitive shift in Terminology has affected the way lexicographers and terminologists understand and describe specialized language, promoting the inclusion of semiotic alternatives to depict knowledge and describe specialized meaning. Such representations and descriptions are then crucial in terminological knowledge bases, where different media coexist to enhance the multidimensional character of concepts. However, so far little attention has been paid in Terminology and Artificial Intelligence to graphic information, including visual resources and pictorial material, in ontology-based multimodal termbases. In this article, we explore the relationship between visual and textual information, and search for a principled way to include images that best represent the linguistic, conceptual and contextual information contained in terminological knowledge bases.
Résumé: La représentation de la connaissance s’avère essentielle pour l’Intelligence Artificielle, tout système d’intelligence devant fournir de connaissances sur le monde, ainsi que de représentations sur les objets, les procédures, les propriétés et les relations parmi les objets. Le changement cognitif dont la Terminologie a fait l’objet a influencé la façon dont les lexicographes et les terminologues comprennent et décrivent le langage spécialisé, en promouvant l’intégration d’autres possibilités sémiotiques afin de représenter la connaissance et de dépeindre le sens spécialisé. De telles représentations et descriptions s’avèrent par conséquent essentielles dans les bases de connaissances terminologiques, où de divers moyens cohabitent pour améliorer le caractère multidimensionnel des concepts. Néanmoins, les domaines de la Terminologie et de l’Intelligence Artificielle n’ont guère prêté attention aux informations graphiques, y comprises les ressources visuelles et le matériel imagé, dans les bases terminologiques multimodales fondées sur l’ontologie. Cet article analyse la relation entre les informations visuelles et textuelles, en cherchant un moyen de principe pour intégrer les images mieux représentant les informations linguistiques, conceptuelles et contextuelles comprises dans les bases de connaissances terminologiques.
Mots-clé: bases de connaissances terminologiques, traduction intersémiotique, représentation du concept, multimodalité, cartes conceptuelles. 1
The technological revolution of the last decades has led to a new digital era in specialized communication. Firstly, Terminotics has favoured innovative computerized working environments in Specialized Lexicography and Terminography. Secondly, modern-day globalized Knowledge Society has favoured new multimedia communication scenarios where images are indispensable nonlinguistic elements for the representation and transfer of knowledge. Thirdly, these phenomena have contributed to a change in the cognitive patterns of individuals, which have inspired a more cognitive-oriented approach to Artificial Intelligence (AI) research.
As a consequence of this cognitive shift, Frame-based Terminology (Faber et al. 2005, 2006, 2007) advocates a multimodal conceptual description in which the structured information in terminographic definitions meshes with visual information for a better understanding of specialized concepts, thanks to the visualization of nonlanguage-specific representations.
Nonetheless, research on the exploitation of visual resources in Specialized Lexicography and Terminology has been underestimated, despite there is a long tradition of picture dictionaries in German and French (Bildwörterbücher and dictionnaires visuels), and some authors (Hupka 1989; Kalverkämper 1993; Picht 1994; Yeo 2001; Prieto Velasco 2008, 2009) have emphasized the benefits of including images in terminographical tools. 2
According to Chandler (2003: 2), “semiotics is concerned with meaning-making and representation in many forms”, perhaps most obviously in the form of texts and media”. Hodge and Kress (1988) consider meaning-making is a social practice derived from the way people design and interpret meanings. Ogden & Richard’s (1923) triangle of meaning has been suggested since Aristotle to distinguish between objects and the words that refer to them. Do images fit in such a triangle as visual depictions of concepts? Several disciplines have tried to explain the social construction of meaning, in spite of Eco underestimating the communicative potential of a visual language, which is regarded as an inventory of peripheral semiotic artifacts (Eco 1976: 260): • The philosophical perspective accounts for the acquisition of new concepts from the visual perception of the real world in order to build a logical conceptual organization (Aristotle, Wittgestein and Peirce). 1 This research is part of the EcoSistema project “Single Space for Ontologically Structured Ecological Information and Environmental Thesaurua”, funded by the Spanish Ministry of Science and Innovation. This paper partly a short version of the article “Managing graphic information in terminological knowledge bases” by Juan A. Prieto (University Pablo de Olavide, Sevilla) and Clara I. López (University of Granada). To appear in Terminology (15:2).
The cognitive perspective encourages the visual representation of concepts by means of images depicting the linguistic, conceptual, and contextual descriptions of terms. • The (socio-)semiotic perspective explains how different sign systems convey meaning to represent real-world objects, processes, properties and relations.
Intersemiotic translation is the interpretation of verbal signs in terms of a nonverbal sign system (Jakobson 1963: 57; Eco 2009: 292), the carrying through of meaning from the source sign system to the new representation. This “migration” from words to images and vice versa is called ecphrasis or hypotyposis.
Hypotyposis consists in changing the matter of expression from the primary verbal description (term, definition or context) into a visual image (picture, animation or video) by means of the depiction of concepts, or in turning a visual text into a written text by means of the linguistic description of images. Terminographers should then develop an intersemiotic competence to build up a visual unit in terminological knowledge bases and guarantee internal coherence, since verbal language is only one of all possible semiotic languages. 3
Visual resources play an important role in the acquisition of knowledge and specialised vocabulary. The readers of specialized texts usually extract mental models in the form of image-schemas and mental images from pictures in order to understand how scientific systems actually work. In other words, visualization contributes to interrelate different types of data (both linguistic and graphic) in concept maps and terminological knowledge bases. Besides, visualization can increase the chances of non-experts having access to specialized knowledge, thus fulfilling the social function of terminology.
We understand graphic information as the visual object(s) that the terminographer associates with the concept/term in the knowledge base in order to activate previous knowledge in the user’s mind, as well as the new knowledge potentially transmitted by the lexis of the definition. Therefore, terminographers should be aware of the importance of selecting appropriate images following clear criteria (iconicity, abstraction and dynamism) (Prieto 2009), as opposed to the usual practice of including decorative images chosen at random from the Internet. 3.1
In AI terms, concept maps are knowledge representations that show individual concepts at nodes with linking words that connect two concepts and indicate the relationship between them, thus forming a proposition (Cañas and Novak in press).
Concept maps encourage meaningful learning in knowledge modelling and sharing environments (Cañas et al. 2004). So far, the existing applications for concept mapping were rather rudimentary. Nowadays, AI research has implemented much more versatile and flexible tools: the Visual Understanding Environment VUE, IHMC CMap Tools and XMIND2. They support ontology formats (RDF-S, OWL), metadata and keyword tagging, and offer overinformation control, multimodal resources (images, videos, web pages, etc.), concept sharing.
More important is the fact that they provide a way of representing relations between ideas, images and words, promoting the idea of multimodality that we explore below. They support information in a series of formats: text (plain, doc, pdf); audio (mp3, wav, wma); video (mpeg, avi, qt); images (jpeg, bmp, gif, tiff, png); presentations (ppt) and applications (executable programs). Thus, concept maps are an example of how conceptual, linguistic and graphic information converge for a better understanding of specialized concepts. In this sense, concept maps can be used as the conceptual grounding for the development of a termbase, when formal ontologies are not applicable. 3.2
A principled selection of images should contribute to depict the concepts contained in terminological entries and serve the purpose of being a kind of interlingua in multimodal databases, so as to visually convey common meaning and contribute to the sharing of knowledge (Boguslavsky et al. 2008).
Unfortunately, some of the largest terminological information systems existing nowadays do not use images systematically or do not use them at all. We have browsed several terminological databases on the environment in search of pictorial resources: Umweltforschungsdatenbank (UFORDAT), General Multilingual Environmental Thesaurus (GEMET), Environmental Application Reference Thesaurus (EARTh), Umwelt-Thesaurus (UMTHES) and SilvaTerm. These terminographical resources offer multilingual queries, hierarchical domain-like listing of topics (natural environment; social aspects, environmental policy measures; human activities and products, effects on the environment), but hardly include other information rather than terms and definitions. As a result, they appear as mere repositories of terms which do not seem to be very useful in the transfer of specialized knowledge. Traditional terminologies should then evolve towards more effective systems grounded on a consistent conceptual organization. 4
For AI, what “exists” is that which can be represented using different semiotic languages. The integration of formal concept maps in the form of ontologies has encouraged the evolution from termbases to terminological knowledge bases by means of the creation of specialized knowledge learning environments. 2 These concept mapping applications are open source free software. CmapTools (http://cmap.ihmc.us/conceptmap.html), VUE (http://vue.tufts.edu), XMIND (http://www.xmind.net).
Consequently, this transition from termbases to terminological knowledge bases should observe three main conditions: multimodality, multidimensionality and accessibility. The integration of these features into terminographical resources is a new challenge for Terminology and AI systems which would certainly promote communication by linking images and terms in terms of ecphrasis and hypotyposis. For this purpose, termbases should mesh the information contained in basic concept maps with terminological information thanks to an ontology exchange language/format (like XOL).
The EcoLexicon Environmental Visual Thesaurus is an example of how a terminological knowledge base should merge multimodal information and highlight the multidimensional character of knowledge representations. 4.1
Multimodal knowledge bases are important because they show the systematic role played by different data categories (ISO 12620), that is, multiple semiotic languages which are co-dependent. According to the World Wide Web Consortium (W3C), multimodality is a process through which a series of devices and users carry out an interaction (audiovisual or gestural) in order to provide complete access to information (W3C 2005).
Multimodal language resources have widened the ways we receive information and understand concepts. Images, videos, sounds are part of our everyday interaction with the media. Multimodality is then essential to reflect the duality between texts which help to understand images and images which depict specialized texts. 4.2
Faber et al. (2007: 41) understand multidimensionality as the different ways of describing and representing a given concept or specialized domain by means of a set of hierarchical and non-hierarchical relations; in other words, multidimensional representations include all possible conceptual organizations of concepts alluding to the same physical entity or process within one integrated system (Rogers 2004: 218).
This conception has changed the way we describe concepts, because images encourage multidimensional representations by showing the different facets of concepts simultaneously. Indeed, it is the context that determines which conceptual dimensions are relevant and which not, clarifying meaning. As León Araúz (2007: 244-245) points out, multidimensionality thus involves (a) the representation of the different values of specialized concepts which activate context-relevant characteristics and (b) the representation of classification criteria indicating how specific concepts inherit attributes of more general concepts. 4.3
Accessibility must be prioritized so as to improve both the intersemiotic competence of terminographers and the terminological competence of the final users of terminographical resources. Accessibility to specialized language resources arises from the fact that not all users can access specialized knowledge or easily interact with multimodal information, especially in multimedia documents, either due to legal, economic, technical or methodological constraints (Budin and Melby 2000).
Accessible resources should guarantee the interaction of a heterogeneous audience with terminographical tools by providing equal opportunities, especially, to those with disabilities, poor reading abilities or a less developed cognitive competence on the subject field (Prieto, Tercedor and López 2007). It is a must for AI to implement accessible terminological systems and help terminographers develop some technical skills in order to adapt pictorial materials and make them accessible for all types of users: managing the ‘alt’ and ‘longdesc’ attributes, providing graphical depictions for definitions and textual equivalents for pictures, etc. (Tercedor et al. in press). 5
EcoLexicon is an ontology-based visual thesaurus on environmental terminology founded on the domain structure derived from the categories and conceptual relations represented in a frame-based conceptual organization called Environmental Event (EE). EcoLexicon is accessible through the Internet (http://manila.ugr.es/visual/index.html) in the form of a dynamic visual thesaurus generated thanks to the ThinkMap Technology. In EcoLexicon, every terminological entry contains meaningful data categories (ISO 12620:1999) providing (a) linguistic information (definitions, synonyms, equivalents in other languages, syntactic and collocational information); (b) conceptual information (conceptual relations and domain structure); (c) contextual information (concordances and contexts), and (d) graphic information (URLs, images, videos, etc.). As a result, the categories (AGENT, PROCESS, PATIENT/RESULT and DESCRIPTION), hierarchical relations (IS-A, PART-OF), and domain-specific non-hierarchical relations (HAS-FUNCTION, TAKES-PLACE-IN, DELIMITED-BY) are represented in the knowledge base ontological structure.
As stated before, terminological knowledge bases should be multimodal, multidimensional and accessible. Nevertheless, images alone may not be very helpful; it is necessary to study the relationship between visual resources and linguistic, conceptual, and contextual information. In our opinion, images ought to satisfy users’ needs as regards the reception, production and translation of specialized texts in their native/foreign language. Therefore, images should: • be coherently linked with other data categories including multimodal information; • • focus on significant aspects of definitions; highlight the most relevant conceptual relations; • reflect the ontological organization of the domain structure and add concreteness to vague contexts.
In order to illustrate the functional role of images in ontology-based terminographical resources and exemplify their relation to different types of information (linguistic, conceptual and contextual), we propose the following explanatory image from the “Fight desertification” campaign by the Argentinean Ministry of Environment and Sustainable Development. Figure 2 is a good example of how an illustration can depict the whole conceptual network around a given concept, DESERTIFICATION. From a Frame Semantics perspective, this image is a graphic representation of knowledge, in the same way an event/frame is a mental representation in which one concept can evoke a series of associated concepts and relations.
In this respect, Figure 2 explains the desertification process by specifying its causes, consequences and what can be done to prevent it, as summarized in Table 1.
Concept desertification
Conceptual
relations Æ
has_agent causes Å has_result Æ result_of Å prevented_by Æ
Related concepts global warming anthropogenic damage to the environment biodiversity losses worsening in global warming poverty migrations, wealth redistribution restoration of damaged ecosystems protection of biodiversity According to the EE category structure, desertification can be defined as a PROCESS [PROTOTYPICAL SEMANTIC ROLE] by which soils [PATIENT] in arid, semi-arid and dry sub-humid zones [LOCATION] are degraded as a result of human activities and natural agents [AGENT] resulting in poor quality infertile land [RESULT]. The concepts and semantic roles evoked by the term desertification should be made explicit both in the definitions of the term and in the images illustrating them. In relation to definitions, images can be used to visually encode necessary parts of the definition, to provide supplementary data corresponding to the definition, or to depict the different facets of a term belonging to more than one domain or subject field.
In relation to the ontological domain structure of knowledge bases, images can represent similar terms that non-experts may understand as synonyms because they are conceptually related, and indicate the level of expertise of the intended user by showing how deep in the domain structure a concept is located. Domain structures facilitate the modelling of the conceptual organization of a subject field in an ontology-like hierarchy; images should then reflect the concept’s location within the EcoLexicon domain structure and evoke its semantic role, so that it can be assigned to one of the categories in the EE, as shown in Figure 2.
In relation to contexts, images can restrict or disambiguate the meaning of a term in the same way as the linguistic context does. Contextual and graphic information have a reciprocal relationship based on the notion of disambiguation, which evokes the duality between images and textual information (Table 2). When presenting the previous image along with either of the following contexts we can obtain different benefits, depending on how meaningful/meaningless the context is about a given term.
While desertification has received tremendous publicity by the political and news media, there are still many things that we don't know about the degradation of productive lands and the expansion of deserts.
Desertification is induced by several factors, primarily anthropogenic beginning in the Holocene era. The primary reasons are overgrazing, overcultivation, increased fire frequency, water impoundment, deforestation, overdrafting of groundwater, increased soil salinity, and global climate change.
On the one hand, images add concreteness to what contexts say about a particular term, especially in the case of meaningless contexts (context A). On the other hand, contexts enrich the semantic content of images by adding information to what is depicted in images (context B). In this regard, it is possible to identify three main types of contexts as far as their relationship with images is concerned: • explanatory contexts, contributing to concretize images which are not meaningful enough to make their meaning clear on their own; • •
disambiguating contexts, which are used to concrete the meaning of vague or polysemous images;
meaning-delimiting contexts, aimed at specifying the semantic content of images by highlighting their connotative or denotative meaning. 6
In this paper we have tried to present the theoretical foundations for a more systematic use of images in AI systems: concept maps and terminological knowledge bases. So far, Terminology and AI have underestimated the importance of images. However, recent cognitive approaches to Specialized Lexicography, like Frame-based Terminology, have emphasized the potential benefits of pictures in the understanding of specialized concepts. The EcoLexicon knowledge base is a good example of how images can be integrated in terminographical tools; despite this, our experience tells us that a deeper research may be still required in order to confirm our arguments.
This paper has shown how graphic information contributes to the representation of specialized knowledge by integrating different semiotic channels in multimodal term entries. For that reason, it is essential that new trends in Terminology and AI pay due attention to graphic information, encourage the study of visual learning towards terminological problem-solving activities, and foster criticism and reflection on the production, manipulation, adaptation and translation of multimedia materials.
BOGULAVSKY, I., CARDEÑOSA, J. & GALLARDO, C. (2008). “A novel approach to creating disambiguated multilingual dictionaries”. Applied Linguistics (to appear), p. 1-23. BUDIN, G. & MELBY, A. (2000). “Accessibility of multilingual terminological resources: current problems and prospects for the future”. In Proceedings of the 2nd International Conference on Language Resources & Evaluation. Atenes National Technical University of Athens II, 31 May-2 June 2000. Athens.
CAÑAS, A. & NOVAK, J. (in press). In A. Okada, S. J. Buckingham Shum & T. Sherborne (eds.), Knowledge Cartography, Software Tools and Mapping Techniques. Springer. CHANDLER (2003: 2) Chandler (2003) Semiotics for beginners. http://www.aber.ac.uk/media/Documents/S4B.
ECO, U. (1976). Tratado de semiótica general. Lumen: Barcelona.
Available online at: ECO, U. (2009). Decir casi lo mismo: la traducción como experiencia. De Bolsillo. FABER, P., MÁRQUEZ LINARES, C. & VEGA EXPÓSITO, M. (2005). “Framing Terminology: A Process-Oriented Approach”. META 50 (4). Available at: http://www.erudit.org/livre/meta/2005/000255co.pdf.
FABER, P., LEÓN P., PRIETO, J. A. & REIMERINK, A. (2007). “Linking images and words: the description of specialized concepts”. International Journal of Lexicography 20(1), p. 39-65. FABER, P., MONTERO MARTÍNEZ, S., CASTRO PRIETO, M. R., SENSO RUIZ, J., PRIETO VELASCO, J. A., LEÓN ARAÚZ, P., MÁRQUEZ LINARES, C. & VEGA EXPÓSITO, M. (2006). “Process-oriented terminology management in the domain of Coastal Engineering”. Terminology 12(2), 189-213. HODGE, R. AND G. KRESS. (1988). Social Semiotics. Cambridge: Polity.
HUPKA, W. 1989. Wort und Bild. Die Illustrationen in Wörterbüchern und Enzyklopädien. Tübingen.
ISO 12620. (1999). Computer applications in terminology-data categories. Geneve: International Organization for Standardization. JAKOBSON, R. (1963). Essais de linguistique générale. Spanish trans. J. M. Pujol and J. Cabanes: Ensayos de lingüística general. Barcelona: Seix Barral.
JAKOBSON, R. (1990). On Language. Cambridge, Mass: Harvard University Russian Language Project.
KALVERKÄMPER, H. (1993). “Das fachliche Bild: Zeichenprozessse in der Darstellung wissenschaftlicher Ergebnisse”. In H. Schröder (ed.) Fachtextpragmatik (FFF 19). Tübingen: Gunter Narr Verlag, 215-238.
LEÓN ARAÚZ, P. (2007). Representación del conocimiento especializado: El uso de marcos desde la macroestructura hasta la microestructura. Unpublished predoctoral dissertation. University of Granada, Spain.
OGDEN C. K. and RICHARDS I. (1923). The Meaning of Meaning: A Study of the Influence of Language Upon Thought and of the Science of Symbolism. London: Routledge and Kegan Paul.
PICHT, H. (1994). “Die Anwendung graphisch-figurativer Mittel bei der Begriffsrepräsentation”. In M. Brekke, Q. Andersen, T, Dahl, & J. Myking (eds.) Proceedings of the 9th European Symposium on Language for Special Purposes: Applications and Implications of Current LSP Research. Bergen: Universitet i Bergen, Norges Handelshøyskole, vol. II, 932-941.
PRIETO VELASCO, J. A. (2008). Información gráfica y grados de especialidad en el discurso científico-técnico: un estudio de corpus. Doctoral dissertation. University of Granada, Spain. ISBN 978-84-691-3940-0. Available at: hera.ugr.es/tesisugr/17491332.pdf.
PRIETO VELASCO, J. A. (2009). Traducción e imagen: la información visual en textos especializados. Ediciones Tragacanto: Granada.
PRIETO VELASCO, J. A., M. I. TERCEDOR SÁNCHEZ & C. I. LÓPEZ RODRÍGUEZ. (2008). “Using Multimedia Materials in the Teaching of Scientific and Technical Translation”. Linguistica Antverpiensia 6, 115-134.
ROGERS, M. (2004). “Multidimensionality in concepts systems: a bilingual textual perspective”. Terminology 10(2), 215-240.
TERCEDOR SÁNCHEZ, M. I., ALARCÓN NAVÍO, E. PRIETO VELASCO, J. A. & LÓPEZ RODRÍGUEZ, C. I. (2009). “Images as part of technical translation courses: implications and applications”. JoSTrans, Journal of Specialised Translation 11, p. 143-168. Available at: http://www.jostrans.org/issue11/art_tercedoretal.pdf.
YEO, R. (2001). Encyclopaedic Visions. Scientific Dictionaries and Enlightenment Culture. Cambridge: Cambridge University Press.