Supporting User Interface Design with Image Schemas: The ISCAT Database as a Research Tool Jörn Hurtienne 1, Stephan Huber 1 and Cordula Baur 1 1 Julius-Maximilians-Universität, Chair of Psychological Ergonomics, Oswald-Külpe-Weg 82, 97074 Würzburg, Germany Abstract This paper introduces ISCAT – image schema catalogue – that has been developed for supporting image schema research in human-computer interaction. It contains image schema definitions collected from the literature and image schema analyses of a wide range of user interfaces. It also contains metaphorical extensions of image schemas with linguistic examples and associated studies into the psychological reality of metaphors. Keywords 1 Image schemas, image-schematic metaphors, research database 1. Introduction: Image Schemas for User Interface Design In the field of human-computer interaction (HCI), image schemas and image-schematic metaphors have proven to be valuable tools for the design of user interfaces. As mental patterns of thought, it is believed that mirroring image-schematic metaphors at the user interface makes the user interface more intuitive to use. Also, because of their relatively frequent occurrence in daily sensorimotor experience and domains of human judgment they have become firmly anchored in cognition and are activated automatically. Their early acquisition and multimodality make them suitable also for designing inclusive user interfaces, e.g. when designing user interfaces for the elderly. Their abstractness leaves space for the creativity of user interface designers because they can see what is essential while keeping the concrete instantiation of an image schema open to be designed (e.g. UP-DOWN can be instantiated in a vertical slider, a string to be pulled from above, or a vertical sliding gesture on a touchscreen). Across several studies, all three criteria, intuitive use, inclusiveness and innovativeness, have been shown to be positively influenced when applying image-schematic metaphors to user interface design (e.g. [4, 7]; see [3] for a review of the HCI literature). To better support user interface design with image schemas we created the ISCAT database as a central image schema repository. In the following we report on the requirements, the structure of the database and more recent spin-offs from this work. 2. Requirements for a Central Image Schema Repository From the results of previous studies applying image schemas in the design of user interfaces, several requirements were formulated that could make an image-schema database useful in different phases of a user-centred design process [2] (p. 191): • Supporting the phase of context-of-use analysis. To enhance the reliability of image-schema extractions, analysts need a good command of image-schema definitions and their interrelations. As image-schema definitions sometimes make subtle distinctions (e.g. the distinction between active and passive ENABLEMENT or the object and observer view of NEAR-FAR), analysts need a The Sixth Image Schema Day (ISD6), March 24–25, 2022, Jönköping, Sweden EMAIL: joern.hurtienne@uni-wuerzburg.de (J. Hurtienne); stephan.huber@uni-wuerzburg.de (S. Huber); cordula.baur@uni-wuerzburg.de (C. Baur) ORCID: 0000-0001-5295-9772 (J. Hurtienne); 0000-0002-5619-3222 (S. Huber); 0000-0002-0228-9747 (C. Baur) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Wor Pr ks hop oceedi ngs ht I tp: // ceur - SSN1613- ws .or 0073 g CEUR Workshop Proceedings (CEUR-WS.org) possibility to consult definitions of image schemas. Hence, the database should provide a central repository of definitions and descriptions of image schemas, their relations to other image schemas and typical user tasks. • Supporting the phases of producing design solutions and evaluation with inspirational and evaluated examples. If using requirements lists including image schemas, designers may wish to get inspired by image-schema instantiations in current user interfaces. To support this, the database needs to hold (and make available) a range of user interface examples from a variety of interactive products, along with evaluations of the effects these image-schema instances have and possible constraints of their use. • Supporting the phase of producing design solutions with image-schematic metaphors. Even without employing image schemas during the whole design process, designers might want to look up how user interface elements can be designed to convey certain abstract meanings. A question might be: “How can I convey the idea of importance to the user?” As this relates to metaphorical extensions (e.g. IMPORTANT IS CENTRAL, IMPORTANT IS UP, IMPORTANT IS BIG), designers would benefit from a centralised collection of metaphorical extensions already known that are currently scattered throughout the literature. • Supporting the extension and customisation of the database. The research community should be able to add to the database other and new instances of image schemas in user interfaces. Taking these requirements into account, a database was built that provides a rich set of information on image schemas and their applications in user interface design. The database is realised as a Zope / Plone content management system with a custom add-on written in Python. It is currently available at https://iscat.psyergo.uni-wuerzburg.de/ and new users can be registered when they contact the first author. 3. ISCAT Overview The user interface of ISCAT (Image-Schema CATalogue) is structured in two main parts. The first part contains image-schema descriptions and metaphorical extensions found in language (Figure 1). Each image-schema description contains a definition, its entailments, and synonyms. It contains notes on the experiential grounding of the image schema and lists relations to other image schemas. Attached to image-schema entries are their metaphorical extensions as found in the linguistic and psychological literature. Metaphorical extensions are documented with their target domains, experiential grounding, and linguistic examples from different languages (not shown in Figure 1). As of today, the database contains 49 image schemas and 349 metaphorical extensions. Each description of an image schema is linked to image-schema instances from user interface studies (right hand column in Figure 1). The second part, user interface studies, contains a number of user interface examples. A user interface study can be divided into sub-studies and user interfaces. For instance, Figure 2, shows a flap lever (user interface) in the sub-study Centre Pedestal of the study Cockpit A320. Each user interface contains one or several images depicting the user interface element and a short description of what the UI element is for and what it does. All image-schema instances that were extracted from the UI element are shown below this information. They include a description of why the analyst thought that the image- schema category applies to the interface, the effect that is achieved by the image-schema instance, how strong the effect is, whether it supports, hinders or is neutral to usability (indicated by a smiley icon), and whether the analyst thought this is a prototypical instantiation of the image schema (indicated by the light-bulb icon). Further notes can be given regarding the decisions the analyst had made in analysing an instance of the image schema. User interface examples were collected by extracting image-schema instances from products as far ranging like cash and ticket machines, enterprise resource planning systems, websites, CAD systems, Virtual Reality interfaces, tangible interfaces, data visualisations and an Airbus A320 cockpit. As of today, the database contains 116 studies and sub-studies with 754 user interfaces, and 2693 instances of image schemas in user interface elements. Figure 1: Example page from the ISCAT database – information on the image schema PATH Figure 2: Example page from the ISCAT database – the user interface study of an Airbus A320 Figure 3: Example from ISCAT – empirical studies of the image-schematic metaphor SIMILAR IS NEAR – DIFFERENT IS FAR In addition, users have the ability to search the database via the search box provided in the upper right corner (Figures 1, 2). Also, specific searches for image-schema instances are possible, the user can set more specific filters, e.g. on specific user interfaces, effect strength, and so on. Further, the advanced search offers possibilities to search for specific items like pictures, metaphorical extensions, etc. The database has an added user management system, so that new users can become members of the ISCAT community and contribute to the contents of the database. 4. ISCAT Spin-offs Over time the image schema catalogue has become a central repository of all things image- schematic. It has been well accepted by researchers, students and practitioners. Currently there are 725 registered users of which 40 appear as contributors and study authors in the database. Its comprehensiveness, however, stands against the requirement of an easy-to-use inspirational tool for designers. Thus, spin-off-projects have been developed to promote a better ease-of-use of image- schemas and their metaphors in design: 1. Supporting researchers. Since the inception of the database in 2007, a great number of studies have been conducted into the psychological reality of a number of metaphorical extensions. These were mainly conducted in cognitive social psychology, consumer psychology, cognitive linguistics and HCI. Since these are informative for researchers, we added the possibility in ISCAT to collect empirical studies of each metaphorical extension (Figure 3). At the moment this feature is experimental and only available to a registered subset of users. 2. Supporting designers. In a recent book project, 65 of the most salient metaphorical extensions were selected and illustrated, along with linguistic examples, empirical evidence for their psychological reality and examples of their applications in design [6]. The book can be easily browsed due to its visual attractiveness and short texts. It thus makes for an attractive coffee table book and inspiration for designers. 3. Supporting design processes. We are currently working on interactive toolkits that support designers experiencing (and thus understanding) the features and entailments of single image schemas for design. A first attempt in this direction was making force-dynamic image schemas mentally and physically graspable through interactive tangible prototypes [5]. Our ongoing work introduces a physical toolkit for image schemas aimed at people working on designing physical data representations (aka data physicalisations) [1]. 5. Conclusion and Outlook The image-schema database ISCAT seems a good starting point for providing the HCI and design communities with an information repository on image schemas and image-schematic metaphors. But also psychologists and cognitive linguists have found the database useful for their research. Image- schema definitions, for example, can help to enhance the agreement between coders and reduce subjectivity when extracting image schemas during a context-of-use analysis. Examples of image- schema instantiations could inspire user interface designers in designing future solutions. Both, linguistic data and user interface examples, could contribute to new insights in user interface design. Although such a database is practical, its users need to be reminded that the image schemas extracted from user interfaces (as well as the metaphorical extensions from linguistics) reflect the subjective view of the respective analyst. Thus, the examples in the database should be seen as a repository of hypotheses for researchers and heuristics for designers. The accumulation of a large number of image-schema instances leads to the emergence of new research questions. The database could be used for mining general rules of image-schema application. Thus, implicitly applied design rules can be detected across different user-interface domains and can be subjected to further research. Early analyses of the current set of over 2600 image-schema instances reveal • rules of image-schema co-occurrences (e.g. BLOCKAGE needs to be followed by RESTRAINT REMOVAL, ATTRACTION is resulting in DIVERSION); • image-schema transformation rules (e.g. UP-DOWN is readily substituted for by FRONT-BACK relations); • typical problems (e.g. UI elements that belong to the same task are often far away from each other without communicating their relation via a LINK or common CONTAINER image schema). The database – tool and content – is open for further improvements. A future version could better meet the specific requirements of design practitioners. Such a version would feature better interconnections between the two parts of the database (linguistic and UI data). It would also provide easier community support. The database could profit from including more kinds of data, for example data on typical user tasks and their associated image schemas. Other information could include typical constraints in using specific instantiations of image schemas that are determined by differing contexts of use. A perennial topic is maintaining the quality and completeness of the data with a range of contributors to the database. Finally, a formal evaluation of ISCAT will be necessary to validate its usability and general value for its users. 6. References [1] C. Baur, C. Wienrich, J. Hurtienne, Form follows mental models: Finding instantiations of image schemas using a design research approach, in: Proceedings DIS’22 Designing Interactive Systems Conference, ACM, New York, NY, 2022. [2] J. Hurtienne, Image schemas and design for intuitive use: Exploring new guidance for user interface design, Ph.D. thesis, Technische Universität Berlin, 2011. doi: 10.14279/depositonce- 2753. [3] J. Hurtienne, How cognitive linguistics inspires HCI: Image schemas and image-schematic metaphors, International Journal of Human-Computer Interaction, 33 (2017) 1-20. doi:10.1080/10447318.2016.1232227. [4] J. Hurtienne, K. Klöckner, S. Diefenbach, C. Nass, A. Maier, Designing with Image Schemas: Resolving the Tension Between Innovation, Inclusion and Intuitive Use, Interacting with Computers, 27 (2015) 235–255. doi:10.1093/iwc/iwu049 [5] J. Hurtienne, D. Löffler, P. Gadegast, S. Hußlein, Comparing Pictorial and Tangible Notations of Force Image Schemas, in: Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction, ACM, New York, NY, 2015, pp. 249–256. doi:10.1145/2677199.2680553. [6] J. Hurtienne, D. Löffler, C. Hüsch, D. Reinhardt, R. Tscharn, S. Huber, S., Happy is up, sad is down. 65 metaphors for design, BIS Publishers, Amsterdam, 2020. [7] R. Tscharn, Innovative and age-inclusive interaction design with image-schematic metaphors, Ph.D. thesis, Julius-Maximilians-Universität Würzburg, 2019. doi:10.25972/OPUS-17576.