Multimodal meets Intuitive? Comparing Visual and Tangible Image Schema Representations Cordula Baur 1, Fredrik Stamm1, Carolin Wienrich 2 and Jörn Hurtienne 1 1 Julius-Maximilians-Universität Würzburg, Chair of Psychological Ergonomics, Oswald-Külpe-Weg 82, 97074 Würzburg, Germany 2 Julius-Maximilians-Universität Würzburg, Human-Technology-Systems, Oswald-Külpe-Weg 82, 97074 Würzburg, Germany Abstract Image schemas are abstract representations of recurring multimodal experiences in the world. Together with image-schematic metaphors, which connect image schemas with abstract domains, they support the design process and foster more inclusive, intuitive, and innovative designs. However, using image schemas in the design process requires extra effort and actual image schema repositories do not meet designers’ requirements. Alternative forms of representation like visualisations or physicalisations of image schemas can increase their accessibility. This work presents an empirical study that evaluates Image Schema Icons and Image Schema Objects in terms of their intuitive use, comprehensibility, and participants’ preference. Correct matches of representations to image-schematic metaphors were recorded, interactions were observed, and the representations were evaluated by questionnaires. The results showed that visual representations are more intuitive and achieved more correct matches, but tangible representations were preferred. This directs further investigation and the further development of image-schema-based design tools. Keywords Image Schemas, Design, Design Research, Evaluation, Intuitive Use 1 1. Introduction Image schemas are representations of repeated, multimodal experiences aiding our understanding of the environment [4, 35, 45]. Image-schematic metaphors emerge when image schemas are connected with subjective experiences or judgments [36]. These metaphors assist in organising and structuring the comprehension of abstract concepts [10, 17, 27, 32, 33, 35, 39, 40]. In Human-Computer Interaction image schemas and their metaphors have been used for interface design and showed to foster more inclusive, intuitive, and innovative designs [21, 26]. However, utilising image schemas for design demands extra effort and time [19, 38, 47]. To tackle this, previous work recommended to use existing image schema lists [21, 24, 47]. However, actual repositories are extensive databases [26] that lack accessibility and applicability in the design process. Researchers in cognitive linguistics and Human- Computer Interaction proposed visual representations of image schemas [4, 11–14, 32, 41, 44, 46] to enhance the understanding of image schema theory. Additionally, tangible and visual representations of FORCE image schemas have been suggested to support the design process [17]. In previous work we described an iterative Research through Design process to create tangible and visual image schema representations which aim at fostering the design of data physicalisations [2]. While the feedback during the design process was positive, further evaluation was required. In this paper we present an empiric Proceedings of The Eight Image Schema Day (ISD8), November 27-28, 2024, Bozen-Bolzano, Italy EMAIL: cordula.baur@uni-wuerzburg.de (C. Baur); fredrikstamm@gmail.com (F. Stramm); carolin.wienrich@uni-wuerzburg.de (C. Wienrich); joern.hurtienne@uni-wuerzburg.de (J. Hurtienne) ORCID: 0000-0002-0228-9747 (C. Baur); 0009-0009-3194-9925 (F.Stamm); 0000-0003-3052-7172 (C. Wienrich); 0000-0001-5295-9772 (J. Hurtienne) ©️ 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings evaluation study where participants matched the representations to image-schematic metaphors, rated intuitive use and comprehensibility, and stated preference. 2. Background 2.1. Image Schemas Initially rooted in cognitive linguistics [21] image schemas were introduced by Johnson [32] and Lakoff [34] as “recurring, dynamic pattern[s] of perceptual interactions and motor programs that give coherence and structure to our experience” [32] (p. xiv). Image schemas link embodied experiences and mental representations [32, 34] to provide structure to human perception and experiences, foster representation in mind, and aid in understanding our surrounding world [4, 6, 9, 35, 45]. For instance, when a baby’s beloved stuffed animal drops to the ground, it experiences gravity. The baby being repeatedly lifted or placed in a pushchair or crib reinforces the experience of up and down movements. The repetition of such experiences leads to the formulation of the UP-DOWN image schema. Image schemas as abstract concepts [21, 23, 27, 32] do not refer to specific objects [21]. Image schemas are multimodal [4, 18, 21, 22, 32], integrating experiences from multiple modalities [17, 18, 27, 28] and can be represented visually, haptically, kinaesthetically or acoustically [17, 27, 28]. They are analogue [21, 23] and function subconsciously [21, 23, 27, 28], encoding and retrieving information from memory repeatedly [21]. Additionally, image schemas proved to be largely cultural- and language-independent [39]. 2.2. Image-schematic Metaphors When an abstract concept that lacks sensory-motor experiences is assigned to a particular image schema, an image-schematic metaphor emerges [18, 22, 25, 45]. This helps to organise and structure the understanding of abstract concepts [10, 17, 27, 32, 33, 35, 39, 40] and supports the transfer of information between different domains [4]. Projecting image schemas onto various abstract domains enables reasoning about these domains [32]. The UP-DOWN image schema is associated with the judgement of good and bad, forming the image-schematic metaphor UP IS GOOD – BAD IS DOWN. Additionally, the UP-DOWN image schema is also linked to quantity (MORE IS UP – LESS IS DOWN) and emotions (HAPPY IS UP-SAD IS DOWN). Linguistic analyses have identified over 250 metaphorical extensions [24, 25]. These image-schematic metaphors are universal, are shared by a wide range of people [19, 25], and were found to overlap across various languages and cultures [5, 8, 39, 42]. Furthermore they are automatically and intuitively understood [18]. 2.3. Image Schemas for Design Image schemas and their accompanying metaphors foster inclusive, intuitive, and innovative designs. Inclusive design is fostered as image schemas are promising to work universally across user groups with varying levels of technical proficiency and cultural backgrounds, because of their connection to fundamental multimodal experiences [19, 21]. Furthermore, metaphor processing should not be affected by a decline in conscious cognitive abilities of elderly, because it relies on automatic and unconscious memory recall [21, 24, 26]. This makes image schemas universally applicable across age groups [24]. Their multimodal nature enables also more inclusive design for people with sensorimotor deficiencies [21, 26]. Image schemas promise to support the intuitive use of interfaces due to their relation to fundamental human mental models and their subconscious appliance [23]. When designs are informed by image schemas and their metaphoric extensions, they reflect the user’s mental models [38]. Furthermore, image schemas and metaphors are readily available for human information processing due to their frequent and continual repetition [16, 27]. Additionally, image schemas can help to identify essential aspects in design while keeping the concrete instantiation open [26]. Image schemas do not propose a specific design solution, instead they leave room for the designer to decide the implementation and create innovative solutions that go beyond current standards [19, 21], therefore fostering more innovative designs. Image schemas and their accompanying metaphors were successfully used to provide inspiration and to generate novel design ideas [18, 19, 23, 28, 38, 39, 45]. They can also structure the design process [45] and be used to describe affordances and design solutions [19, 23, 27]. Additionally, they can support deeper thought about design decisions [39] and help to justify them [19]. However, it needs to be considered that using image schemas and metaphors in the design process requires extra effort [19, 38, 47]. To address this, utilising established image schema lists is most promising [21, 24, 47]. Such a list is provided by the Image Schema Catalogue (ISCAT) [26], but this database does not serve as design tool, as it lacks easy accessibility and intuitive use, due to its large volume and complex structure. 2.4. Visual Representations of Image Schemas In cognitive linguistics illustrations were used to explain image schemas by highlighting their salient characteristics [14]. Johnson suggested using diagrams to intuitively demonstrate how image schemas operate periconceptually and has developed a notational system [32]. Talmy [44] depicted FORCE image schemas by a system which consists of Agonist and Antagonist. Mandler [41] created a series of pictorial representations to depict nonverbal concepts instead of exact interpretations. In Human- Computer Interaction, Wilkie et al. [46] proposed visual representations of image schemas Besold et al. [4], Hedblom et al. [11], and Hedblom [12], provided sequences of visualisations to show a process. Hedblom and Neuhaus [14] later proposed a Diagrammatic Image Schema Language, a holistic system to visually represent image schemas. This language provides organised and systematic representations of abstract concepts. Furthermore, Hedblom [12] and Hedblom and Kutz [13] examined the relationship between everyday objects and image schemas, using illustrations and names of image schemas. In this work the authors stated the challenge of creating visuals that capture all characteristics of an image schema. 2.5. Image Schema Representations to Support Design Previous approaches applying image schemas to the design process required too much time and effort [19, 38, 47]. In contrast, Hurtienne et al. [17] proposed visual as well as tangible representations of FORCE image schemas. The characteristics of FORCE image schemas informed icons, while the notion that a tangible representation might convey FORCE image schemas more effectively encouraged the design of tangible representations. Image Schemas were instantiated as interactive physical rotatory dials. Both sets were tested for their effectiveness in identifying and distinguishing the represented image schemas as well as their usefulness in the brainstorming process. The icons were identified more frequently correctly than the tangible representations. Additionally, the visual representations were mentioned to foster the generation of more ideas and in this condition, participants appreciated FORCE image schemas to be more crucial and beneficial for design. Design ideas created using tangible representations were perceived as more qualitative: ideas were considered to be more interactive, haptic and visual [17]. In previous work [2] we used an iterative research-oriented Design process [7] to develop icons (called Image Schema Icons) and clay objects (called Image Schema Objects) that represent image schemas. We propose the use of tangible representations to facilitate data physicalisation design, as these representations are more similar to the desired design outcome, which represents abstract data through shape or material properties [30]. Designers no longer need to handle descriptions and textual definitions of image schemas. The representations make image schemas easier to examine, contrast and compare, to figure out which one works best for the actual design task. Additionally, specific examples for including image schemas in a data physicalisation are provided by the tangible representations. This might address the identified challenges of extra time and effort when using image schemas in the design process. The process of designing the image schema representations provided promising feedback and the tangible representations were already tested in a workshop setting [1] but a comprehensive evaluation of their effectiveness is required. Before testing image schema representations in the design process, it is necessary to assess their comprehensiveness and intuitive use and choose one of the instantiation types. Therefore, we investigate in this work the research question, whether Image Schema Icons or Image Schema Objects depict image schemas in a more intuitive and comprehensive manner. Additionally, we explore user preferences. Tangible representations may be appropriate for image schemas, because they are able to represent the multimodality of experiences incorporated in image schemas [27, 28, 18, 17]. Hurtienne et al. [17] assessed visual and tangible representations of FORCE image schemas and found tangible representations to encourage the formation of more interactive, visual, and haptic ideas, while visual instantiations were more precisely identified and fostered a greater quantity of ideas. It needs to be considered that FORCE image schemas are a special subset of image schemas. Because of their temporary, abstract, and dynamic nature, they can be hard to recognise and categorise [17]. This work focuses on different image schemas. When creating icons and clay objects to represent image schemas we identified some image schemas being easier to recognise and represent in visual form, other image schemas in tangible form [2]. Therefore, Hurtienne et al.’s [17] findings might not be generalisable for all image schemas. In some cases, the tangible representation may be identified as well or even better. It is necessary to evaluate the intuitive use and comprehensiveness of different representation modalities. Our explorative hypothesis is that visual and tangible representations differ in terms of intuitive use, comprehensiveness, and accurate matches of representations to image-schematic metaphors, as well as in preference ratings. 3. Method To evaluate the intuitive use and the comprehensibility of the image schema representations we conducted a within-subject design study. Randomly assigned to groups, participants of group one began with visual representations and continued with the objects, while group two followed the reverse order. This setup was intended to avoid cross-over effects. Participants matched image-schematic metaphors to the presented Image Schema Icons or Image Schema Objects and rated Intuitive Use and Comprehensibility in questionnaires. In the end, they were asked for their preference. Interaction with the representations was observed and correct matches were counted. 3.1. Participants Recruited from the universities’ participant pool, participants received 0.5 credit points as compensation. No exclusion criteria were applied, as image schemas claim being universal across cultural backgrounds and age [18, 20, 21, 23, 28, 38, 39, 45]. The study was conducted in German but to avoid altering their meaning through translation, we presented the image schematic metaphors in their original language (English). To avoid confusion, we provided a list of English-German translations for the terms used. Additionally, participants were asked about their English proficiency level and prior experience with image schemas. A total of fifty participants (n = 50), with an average age of 21.22 years (Standard Deviation (SD) = 1.36) participated. None of them had any prior experience with image schemas. Ten participants (20 %) rated their English at C1 level, 29 (58 %) at B2 level, eight (16 %) at B1 level, three participants (6 %) at A2 level, and no participants rated their English level A1. In the following the participants are identified as P4 to P54; P1 to P4 were not included in the data analysis but took part in pilot testing to improve the research design. 3.2. Procedure The study lasted for approximately 30 minutes. After welcoming and conducting informed consent, a demographic questionnaire was completed and participants were given written instructions (Supplementary Material 1). Participants were asked to read statements (image-schematic metaphors) presented on A5 printouts and to select the icon or icon pair or object or object-pair best fitting the metaphor. Fourteen image-schematic metaphors were conducted in total. After completing the task, participants filled in questionnaires (Supplementary Material 2) to rate the intuitive use and comprehensibility of the stimuli. This procedure was repeated with 14 new metaphors and the other representation modality. Participants who first used objects, now used icons, and vice versa. Intuitive use and comprehensiveness were rated again using the same questionnaires. Additionally, participants were asked which stimuli they preferred and why. During the matching task, the researchers observed whether the participants interacted with the objects physically and recorded correct matches. 3.3. Material and Setup In a previous phase of this project, we selected a subset of image schemas to be represented in visual and tangible way, regarding their potential to support data physicalisation design. This decision was informed by analyses of existing data physicalisations regarding incorporated image schemas [3] and the potential for improvement through additional image schemas [under review]. Furthermore, recommendations made in literature, which image schemas serve to foster the design of tangible user interfaces [25, 28] informed our selection. For more details regarding our selection criteria for image schemas see [under review]. A6 cards were used to display the Image Schema Icons (Figure 1), while the objects (Figure 2) were already crafted in a Research-oriented Design process [7]. For a detailed description of the design process of the Image Schema Icons and Image Schema Objects see [2]. Figure 1: Visual representations of image schemas, printed on A6 cards. From left to right, first row: STRAIGHT-CROOKED, SMOOTH-ROUGH, UP-DOWN, PAINFUL; second row: CONTENT-CONTAINER, OBJECT, HEAVY-LIGHT, HARD-SOFT; third row: CENTRE-PERIPHERY, LEFT-RIGHT, PART-WHOLE, NEAR- FAR; fourth row: LINKAGE, STRONG-WEAK. Figure 2: Handcrafted tangible representations of image schemas made of clay. From left to right, last row: STRAIGHT-CROOKED, SMOOTH-ROUGH, UP-DOWN, PAINFUL; second last row: CONTENT- CONTAINER, OBJECT, HEAVY-LIGHT, HARD-SOFT; second front row: CENTRE-PERIPHERY, LEFT-RIGHT, PART-WHOLE, NEAR-FAR; front row: LINKAGE, STRONG-WEAK. For the matching task, well-established image-schematic metaphors were selected based on high confirmation rates [24, 25, 29, 40] or their well-documented linguistic findings. For image schemas where this was not possible, a metaphor was chosen from the ISCAT database [26]. The metaphors, accompanied with selection criteria, and alternative image schemas are provided as supplementary material 3. For each metaphor, we presented the correct image schema representation and two incorrect options. To be able to show all three choices simultaneously and to avoid presenting the different choices for different duration, we used a cardboard cover while arranging the stimuli. We varied the position of the correct choice for each metaphor. The study setup is depicted in Figure 3. Representations that are easily confused, such as HARD-SOFT, SMOOTH-ROUGH, or STRAIGHT-CROOKED, or those with similar characteristics, like STRONG-WEAK, or HEAVY-LIGHT were presented together. OBJECT, LINKAGE, and PAINFUL, each consisting of only one term, were presented as alternatives to avoid the lack of bi-dimensional structure being used as exclusion criteria. To ensure clarity for the researcher who conducted the data collection, the metaphors were presented in the same order for each participant. Figure 3: Study setup with cardboard coverage and cardboard area to present stimuli. 3.4. Collection and Analysis of Data The representations’ intuitive use and comprehensibility were evaluated using the Modular Extension of the User Experience Questionnaire (UEQ+) [43]. The 7-point subscale intuitive use measures the ease of use with the items difficult-easy, illogical-logical, not plausible-plausible, and inconclusive-conclusive. Comprehensibility is measured with the items complicated-simple, unambiguous-ambiguous, inaccurate-accurate, and enigmatic-explainable. The UEQ+ is a well- established questionnaire, frequently used to evaluate products’ user experience and therefor it was deemed appropriate to use it for assessing the experience with prototypical design tools. To evaluate how well the presented image schemas can be identified, we recorded correct matches of image- schematic metaphors to image schema representations. To determine whether the choice was solely informed by the visual appearance of the objects, we observed whether participants physically interacted with the tangible image schema representations. Furthermore, participants indicated their preference for icons or clay objects. Data was collected using LimeSurvey [37] and analysed using the statistics software JASP [31], which was also used to provide values for Mean (M) and Standard Deviation (SD). The qualitative data was analysed by creating an Affinity Diagram, loosely applying the Contextual Design Approach [15] for data evaluation. From the participants’ answers we created Affinity Notes and organised them into groups based on inductive reasoning. 4. Results To compare visual and tangible representations, we conducted dependant t-tests. We chose this test, as it is a often used und reliable test for within-design study setups. No outliers were excluded, no data values were missing. Even when the data showed no normal distribution, we proceeded with the analysis, because our sample (n) is bigger than 30 and therefore the data is robust against violation of the normal distribution. The significance level α describes the maximum probability that a null hypothesis (no difference) is incorrectly rejected. It was set at alpha = .05. In terms of intuitive use the icons (M = 6.11, SD = 0.67) and objects (M = 5.71, SD = 0.89) showed significant difference (t(49) = 3.239, p = .002, d = .162). Here t describes the t-value which is used to define the p-value; p shows the significance; d describes Cohens’d and shows the effect size, which can be used to compare the results with studies measuring the same dependent variable. The rating of comprehensibility showed no significant difference (t(49) = .509, p = .613, d = .072) between icons (M = 5.56, SD = 0.94) and objects (M = 5.49, SD = 1.01). Counting the number of correct matches showed that for 630 times (90.00 %) the correct icons were selected, but only for 571 times (81.57 %) the correct objects were selected. This is a significant difference t(699) = 4.982, p < .001, d = .188. However, Image Schema Icons and Image Schema Objects both showed a high number of correct matches. The visual representations of STRONG-WEAK and CONTENT-CONTAINER, as well as the tangible representations of HEAVY-LIGHT and STRONG-WEAK showed the lowest number of correct matches. Figure 4 shows the correct matches per image schema. The full data is provided as supplementary material 4. 60 49 50 49 50 49 50 48 45 46 46 47 47 49 47 47 49 46 46 45 44 44 41 44 40 39 40 40 31 30 20 15 8 10 0 Icons Objects Figure 4: Number of correct matches of visual and tangible representations to image-schematic metaphors. Sixteen participants (32.00 %) preferred icons, while 34 participants (68.00 %) preferred objects. The participants stated that the icons are more intuitive (P8, P25, P34, P46, P49, P52) and less difficult to match (P16, P20, P25, P33, P37, P45, P52). Some appreciated the icons for their details (P35, P45), others the room for interpretation they provide (P19, P40). However, the majority preferred the Image Schema Objects which were experienced as easier to comprehend (P12, P21, P23, P26, P28, P32, P38, P42, P43, P48) and better suited for matching metaphors due to their three-dimensional shape (P10, P14, P24, P32, P48, P54). Participants stated the objects show a higher aesthetic quality (P9, P18, P45, P50, P51). Furthermore, they highlighted the objects as being more graspable (P9, P12, P22, P29, P38, P39, P54) and liked the opportunity of touching and interacting with them (P4, P27, P29, P31, P32, P36, P41, P42, P51). Our observation revealed that most participants made their choice and expressed their preference solely based on the visual appearance of the stimuli. Only 14 participants (28.00 %) showed physical interaction. Of 48 interactions (excluding 10 interactions with the wrong objects), 41 interactions (85.00 %) resulted in a correct match. The most frequently interacted objects were, HEAVY-LIGHT and STRONG- WEAK, followed by CONTENT-CONTAINER. Conversely, the objects that showed least correct matches were most frequently interacted with. Figure 5 shows the interactions per Image Schema Object. 12 9 9 10 8 8 6 5 6 4 2 2 2 2 1 1 1 2 0 0 0 Figure 5: Number of physical interactions per Image Schema Object. 5. Discussion Utilising image schemas showed to foster more inclusive, intuitive, and innovative designs and to aid the design process. However, the use of image schemas demands extra effort and time. Currently available image schema repositories do not provide an easily applicable design-tool. To address this issue, we developed visual and tangible representations to make image schemas accessible and incorporable in the data physicalisation design process. In this study we evaluated these representations to determine if they convey image schemas in an intuitive and comprehensive way and which modality of representation works best. Participants matched image-schematic metaphors to visual or tangible image schema representations and rated intuitive use, comprehensiveness, and their preference for one representation modality (visual or tangible). The study utilised questionnaires, recorded correct matches, and observed interactions with the tangible representations. In previous research [17], visual representations of FORCE image schemas were more often identified correctly. However, due to the special character of FORCE image schemas, these findings may not be generalisable for all image schemas. Investigating different representation modalities of other image schemas in an explorative way, this work provides evidence for the Image Schema Icons being more intuitive and resulting in more correct matches, than the Image Schema Objects. However, participants preferred the tangible representations more often. Previous research already highlighted that the way how image schemas are instantiated is important for their comprehensiveness [25, 29, 40]. Consistent with previous work, which demonstrated that visual representations were more accurately identified [17], this study also found that the Image Schema Icons resulted in more correct matches and were perceived as more intuitive (qualitative and quantitative). They were also rated as more comprehensive, but without significant evidence. Previous findings were confirmed and showed to be applicable also to other image schemas. However, it should be noted that participants showed limited interaction with the tangible instantiations and their ratings were primarily based on the objects’ visual appearance rather than a tangible experience. Therefore a reason could be that since childhood people are trained in educational but also exhibition settings, not to touch physical artifacts. The tangible characteristics of the Image Schema Objects may not have been experienced and the objects didn’t realise their full potential. Therefore, they might have influenced the participants’ rating only to a small extent. Both visual and tangible representations achieved high numbers of correct matches. Only HEAVY- LIGHT and STRONG-WEAK showed a major difference between conditions. For both image schemas the correct matches of tangible representations were much lower than for visual representations. Most Image Schema Objects’ visual appearance is similar to the Image Schema Icons, but not for HEAVY- LIGHT and STRONG-WEAK. The design process [2] showed that finding appropriate visual and tangible representations for HEAVY-LIGHT and STRONG-WEAK was difficult and participants struggled with their recognition. The final tangible representations require tangible interaction and exploration to fully convey the image schemas’ characteristics and to be identified correctly. In fact, the tangible representations of these image schemas showed the highest interaction. However, in total only a minority of participants interacted with the objects. Therefore, for most participants the tangible representations of HEAVY-LIGHT and STRONG-WEAK remained concealed which impeded a correct choice. In the qualitative data, the icons were stated to be more intuitive to understand (P8, P25, P34, P46, P49, P52) and easier to match to metaphors (P16, P20, P25, P33, P37, P45, P52). Although visual representations lead to more correct matches and were rated to be more intuitive, participants preferred more often the Image Schema Objects. Even they showed only a limited number of tangible interactions, they stated to appreciate the opportunity of touching and interacting with the objects (P4, P27, P29, P31, P32, P36, P41, P42, P51). Furthermore, the tangible representations were preferred because of their three-dimensionality, which supports matching to metaphors (P10, P14, P24, P32, P48, P54), and were experienced as more graspable (P9, P12, P22, P29, P38, P39, P54). Additionally, some participants stated they are easy to understand (P12, P21, P23, P26, P28, P32, P38, P42, P43, P48), while others found the objects aesthetically pleasing (P9, P18, P45, P51). 5.1. Limitations Participants may have recognised similar visual appearances of Image Schema Objects and Image Schema Icons among conditions, which could have caused a learning effect. However, the crossover- design was implemented to prevent this from confounding the results. Another potential limitation of this work is that for STRAIGHT-CROOKED the same metaphor was used in both conditions. However, as this was one of 14 metaphors presented per condition, it is unlikely that participants noticed this and referred to their choice made in the previous condition. A more crucial aspect is participants’ English proficiency. The majority stated their English level higher than A1 and only one participant used the provided translation sheet. However, some participants appeared to be confused or uncertain regarding the meaning of some metaphors. It is possible that they felt embarrassed to admit a lack of English knowledge and therefore didn’t use the translation sheet. This may have led to misunderstandings of the image-schematic metaphors and affected the accuracy of the matches and ratings. Furthermore, instructing participants to make intuitive decisions may have influenced their choices. Some participants stated in retrospect that if they had invested more time, they would have chosen different icons or objects. The instructions aimed to encourage intuitive decision-making and prevent participants from overthinking their choices. This raises the question of whether a more deliberate decision would increase or decrease the number of correct matches. Furthermore, the instructions prevented participants from taking the time to explore and interact with the objects more intensely. Allowing more time could promote more intense interaction and with this a more multimodal experience of the objects. These aspects, both worth further research. 6. Conclusion Image schemas enhance both, design outcome and the design process. To overcome the additional effort and time for using image schemas in design, a more accessible way to represent and utilise them is required. This work compared and evaluated visual and tangible representations of image schemas to determine which modality conveys image schemas best. Therefore, an empiric study was conducted, where participants matched image-schematic metaphors to visual and tangible representations, rated intuitive use and comprehensibility and indicated their preference. The Image Schema Icons showed higher ratings for intuitive use and a higher number of correct matches. The Image Schema Objects also showed high numbers of correct matches and were preferred more often due to their opportunity for physical interaction. 6.1. Outlook In the next step, we are going to evaluate image schema representations’ effectiveness for designing data physicalisations. Further work could explore the transferability of Image Schema Icons and Image Schema Objects and their usefulness for other design tasks, such as tangible interfaces. Previous research has already highlighted image schemas’ potential for tangible user interface design [25, 28], which could be further reinforced by our proposed image schema representations. 7. References [1] Baur, C. et al. 2022. Designing Data Physicalisations with Physical Image Schema Instantiations. Short Paper Proceedings of the 5th European Tangible Interaction Studio (Toulouse France, Nov. 2022). [2] Baur, C. et al. 2022. Form Follows Mental Models: Finding Instantiations of Image Schemas Using a Design Research Approach. DIS ’22: Proceedings of the 2022 ACM Designing Interactive Systems Conference (Virtual Event Australia, Jun. 2022), 586–598. [3] Baur, C. et al. 2023. Image Schemas as Tool for Exploring the Design Space of Data Physicalisations. Proceedings of The Seventh Image Schema Day (Rhodes Greece, Sep. 2023). [4] Besold, T.R. et al. 2017. A narrative in three acts: Using combinations of image schemas to model events. Biologically Inspired Cognitive Architectures. 19, (2017), 10–20. DOI:https://doi.org/10.1016/j.bica.2016.11.001. [5] Cienki, A. 1998. STRAIGHT: An image schema and its metaphorical extensions. (1998). [6] Dimitra Bourou et al. 2021. Image Schemas and Conceptual Blending in Diagrammatic Reasoning: The Case of Hasse Diagrams. Diagrammatic Representation and Inference. Amrita Basu et al., eds. Springer. 297–314. [7] Fallman, D. 2007. Why Research-Oriented Design Isn’t Design-Oriented Research: On the Tensions Between Design and Research in an Implicit Design Discipline. Knowledge, Technology & Policy. 20, 3 (Oct. 2007), 193–200. DOI:https://doi.org/10.1007/s12130-007-9022-8. [8] Forceville, C. 2006. Metaphor in Culture: Universality and Variation, Zoltán Kövecses. Cambridge University Press, Cambridge (2005), 314 pp., ISBN 0 521 84447 9 (hardback). Journal of Pragmatics. 38, (Sep. 2006), 1528–1531. DOI:https://doi.org/10.1016/j.pragma.2006.03.003. [9] Gibbs, R.W. and Colston, H.L. 1995. The cognitive psychological reality of image schemas and their transformations. Cognitive Linguistics. 6, 4 (1995), 347–378. DOI:https://doi.org/10.1515/cogl.1995.6.4.347. [10] Grady, J.E. 1997. Foundations of meaning: Primary metaphors and primary scenes. Department of Linguistics, University of California at Berkeley. [11] Hedblom, M.M. et al. 2017. Between Contact and Support: Introducing a Logic for Image Schemas and Directed Movement. AI*IA 2017 Advances in Artificial Intelligence, Proceedings of the XVIth International Conference of the Italian Association for Artificial Intelligence (Nov. 2017), 256–268. [12] Hedblom, M.M. 2020. Image Schemas and Concept Invention: Cognitive, Logical, and Linguistic Investigations. Springer. [13] Hedblom, M.M. and Kutz, O. 2019. Conceptual Puzzle Pieces. Modeling and Using Context. CONTEXT 2019 (Cham, 2019), 98–111. [14] Hedblom, M.M. and Neuhaus, F. 2022. Visualising Image Schemas: A Preliminary Look at the Diagrammatic Image Schema Language (DISL). Proceedings of the Sixth Image Schema Day (Jönköping Sweden, Mar. 2022). [15] Holtzblatt, K. and Beyer, H. 2014. Contextual Design: Evolved. Morgan & Claypool Publishers. [16] Hurtienne, J. 2009. Cognition in HCI: An Ongoing Story. Human Technology. 5, 1 (May 2009), 12–28. DOI:https://doi.org/10.17011/ht/urn.20094141408. [17] Hurtienne, J. et al. 2015. Comparing Pictorial and Tangible Notations of Force Image Schemas. TEI ’15: Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (Stanford California USA, Jan. 2015), 249–256. [18] Hurtienne, J. et al. 2008. Cooking up real world business applications combining physicality, digitality, and image schemas. TEI ’08: Proceedings of the 2nd international conference on Tangible and embedded interaction (Bonn Germany, Feb. 2008), 239–246. [19] Hurtienne, J. et al. 2015. Designing with Image Schemas: Resolving the Tension Between Innovation, Inclusion and Intuitive Use. Interacting with Computers. 27, (Apr. 2015). DOI:https://doi.org/10.1093/iwc/iwu049. [20] Hurtienne, J. et al. 2015. Designing with Image Schemas: Resolving the Tension Between Innovation, Inclusion and Intuitive Use. Interacting with Computers. 27, 3 (May 2015), 235–255. DOI:https://doi.org/10.1093/iwc/iwu049. [21] Hurtienne, J. 2016. How Cognitive Linguistics Inspires HCI: Image Schemas and Image- Schematic Metaphors. International Journal of Human-Computer Interaction. 33, 1 (Sep. 2016), 1–20. DOI:https://doi.org/10.1080/10447318.2016.1232227. [22] Hurtienne, J. et al. 2007. Image schemas: a new language for user interface design? Prospektive Gestaltung von Mensch-Technik-Interaktion. M. Rötting et al., eds. VDI Verlag. 167–172. [23] Hurtienne, J. 2011. Image Schemas and Design for Intuitive Use. Technische Universität Berlin. [24] Hurtienne, J. et al. 2010. Physical gestures for abstract concepts: Inclusive design with primary metaphors. Interacting with Computers. 22, 6 (Nov. 2010), 475–484. DOI:https://doi.org/10.1016/j.intcom.2010.08.009. [25] Hurtienne, J. et al. 2009. Sad is heavy and happy is light: population stereotypes of tangible object attributes. TEI ’09: Proceedings of the 3rd International Conference on Tangible and Embedded Interaction (Cambridge United Kingdom, Feb. 2009), 61–68. [26] Hurtienne, J. et al. 2022. Supporting User Interface Design with Image Schemas: The ISCAT Database as a Research Tool. Proceedings of the Sixth Image Schema Day (Jönköping Sweden, Mar. 2022). [27] Hurtienne, J. and Blessing, L. 2007. Design for intuitive use - Testing Image Schema Theory for User Interface Design. DS 42: Proceedings of ICED 2007, the 16th International Conference on Engineering Design (Paris France, Jul. 2007), 829–830. [28] Hurtienne, J. and Israel, J.H. 2007. Image schemas and their metaphorical extensions: intuitive patterns for tangible interaction. TEI ’07: Proceedings of the 1st international conference on Tangible and embedded interaction (Baton Rouge Lousiana, Feb. 2007), 127–134. [29] Hurtienne, J. and Meschke, O. 2016. Soft Pillows and the Near and Dear: Physical-to-Abstract Mappings with Image-Schematic Metaphors. TEI ’16: Proceedings of the TEI ’16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction (Eindhoven Netherlands, Feb. 2016), 324–331. [30] Jansen, Y. et al. 2015. Opportunities and Challenges for Data Physicalization. CHI ’15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul Republic of Korea, Apr. 2015), 3227–3236. [31] JASP Team 2024. JASP. [32] Johnson, M. 1987. The body in the mind: The bodily basis of meaning, imagination, and reason. University of Chicago Press. [33] Johnson, M. 2005. The philosophical significance of image schemas. From Perception to Meaning: Image Schemas in Cognitive Linguistics. B. Hampe, ed. De Gruyter Mouton. 15–34. [34] Lakoff, G. 1987. Women, Fire, and Dangerous Things: What Categories Reveal about the Mind. University of Chicago Press. [35] Lakoff, G. and Johnson, M. 1980. Metaphors we live by. University of Chicago Press. [36] Lakoff, G. and Johnson, M. 1999. Philosophy in The Flesh: The Embodied Mind And Its Challenge To Western Thought. Basic Books. [37] Limesurvey GmbH 2023. LimeSurvey: An Open Source survey Tool. [38] Löffler, D. et al. 2013. Developing Intuitive User Interfaces by Integrating Users’ Mental Models into Requirements Engineering. BCS-HCI ’13: Proceedings of the 27th International BCS Human Computer Interaction Conference (London, UK, Sep. 2013), 1–10. [39] Löffler, D. et al. 2014. Mixing Languages’: image schema inspired designs for rural Africa. CHI EA ’14: CHI ’14 Extended Abstracts on Human Factors in Computing Systems (Toronto Ontario Canada, May 2014), 1999–2004. [40] Macaranas, A. et al. 2012. Bridging the gap: attribute and spatial metaphors for tangible interface design. TEI ’12: Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction (Kingston Ontario Canada, Feb. 2012), 161–168. [41] Mandler, J.M. 1992. How to build a baby: II. Conceptual primitives. Psychological Review. 99, 4 (Nov. 1992), 587–604. DOI:https://doi.org/10.1037/0033-295X.99.4.587. [42] Neumann, C. 2001. Is Metaphor Universal? Cross-Language Evidence From German and Japanese. Metaphor and Symbol - METAPHOR SYMB. 16, (Apr. 2001), 123–142. DOI:https://doi.org/10.1207/S15327868MS1601&2_9. [43] Schrepp, M. and Thomaschewski, J. 2019. Eine modulare Erweiterung des User Experience Questionnaire. Usability Professionals (UP19) (2019), 148–156. [44] Talmy, L. 1988. Force Dynamics in Language and Cognition. Cognitive Science. 12, 1 (Jan. 1988), 49–100. DOI:https://doi.org/10.1207/s15516709cog1201_2. [45] Tscharn, R. 2017. Design of Age-Inclusive Tangible User Interfaces Using Image-Schematic Metaphors. TEI ’17: Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction (Yokohama Japan, Mar. 2017), 693–696. [46] Wilkie, K. et al. 2009. Evaluating Musical Software Using Conceptual Metaphors. BCS-HCI ’09: Proceedings of the 23rd British HCI Group Annual Conference on People and Computers: Celebrating People and Technology (Cambridge Great Britain, Sep. 2009), 232–237. [47] Winkler, A. et al. 2016. Evaluation of an Application Based on Conceptual Metaphors for Social Interaction Between Vehicles. DIS ’16: Proceedings of the 2016 ACM Conference on Designing Interactive Systems (Brisbane QLD Australia, Jun. 2016), 1148–1159. Supplementary Material 1 Instructions Sie bekommen insgesamt 14 nummerierte Zettel. Auf der Rückseite der nummerierten Zettel steht jeweils ein kurzer Satz. Sie lesen immer den jeweiligen Satz. Anschließend zeigt Ihnen die Versuchsleitung drei Icons bzw. Icon-Paare. Wählen Sie das Icon bzw. Icon-Paar aus, dass Ihrer Meinung nach im Satz enthalten ist. Denken Sie nicht zu lange nach, entscheiden Sie intuitiv aus dem Bauch heraus. Hinweis: Die Sätze sind auf Englisch. Wenn Sie eine Übersetzungsliste brauchen, können Sie im Fragebogen einmal auf „Weiter“ klicken. You will be given a total of 14 numbered sheets of paper. On the back of each sheet there is a short sentence. You will read each sentence. The experimenter will then show you three icons or pairs of icons. Choose the icon or pair of icons that you think is in the sentence. Don't think too long, make an intuitive decision. Note: The sentences are in English. If you need a translation list, you can click 'Next' once in the questionnaire. Sie bekommen insgesamt 14 nummerierte Zettel. Auf der Rückseite der nummerierten Zettel steht jeweils ein kurzer Satz. Sie lesen immer den jeweiligen Satz. Anschließend zeigt Ihnen die Versuchsleitung drei Objekte bzw. Objekt-Paare. Wählen Sie das Objekt bzw. Objekt-Paar aus, dass Ihrer Meinung nach im Satz enthalten ist. Denken Sie nicht zu lange nach, entscheiden Sie intuitiv aus dem Bauch heraus. Hinweis: Die Sätze sind auf Englisch. Wenn Sie eine Übersetzungsliste brauchen, können Sie im Fragebogen einmal auf „Weiter“ klicken. You will be given a total of 14 numbered sheets of paper. On the back of each sheet there is a short sentence. You will read each sentence. The experimenter will then show you three objects or pairs of objects. Choose the object or pair of objects that you think is in the sentence. Don't think too long, make an intuitive decision. Note: The sentences are in English. If you need a translation list, you can click 'Next' once in the questionnaire. 2 Questionnaires Demographic Data • Welches ist Ihr bisher höchster Bildungsabschluss? What is your highest educational qualification to date? • Welches ist Ihr Geschlecht? What is your gender? • Wie alt sind Sie gemessen in Jahren? How old are you in years? • Welches ist Ihre Muttersprache? What is your mother tongue? • Wie würden Sie Ihre Englischkenntnisse einordnen? How would you categorise your English language skills? A1, A2, B1, B2, C1, C2 • Welcher beruflichen oder berufsqualifizierenden Tätigkeit gehen Sie derzeit hauptsächlich nach? What is your main professional or vocational activity at present? • Haben Sie bereits Vorerfahrung im Themengebiet Image Schemas? Do you have any previous experience in the field of image schemas? • Welche Erfahrungen im Themengebiet Image Schemas haben Sie? What experience do you have in the field of image schemas? UEQ+: Intuitive Bedienung UEQ+: Intuitive Use • Die Zuordnung der Icons/Objekte war für mich … The assignment of the icons/objects was … o mühevoll – mühelos difficult – easy o unlogisch – logisch illogical – logical o nicht einleuchtend – einleuchtend not plausible – plausible o nicht schlüssig – schlüssig inconclusive – conclusive UEQ+: Verständnis UEQ+: Comprehensibility • Die Icons/Objekte sind für mich … The icons/objects are … o kompliziert – einfach complicated – simple o ungenau – genau unambiguous – ambiguous o nicht eindeutig – eindeutig inaccurate – accurate o rätselhaft – erklärbar enigmatic – explainable Präferenz Preference • Welche Darstellungsform hat Ihnen insgesamt besser gefallen und warum? Which form of representation did you like better? Why? • Gibt es zum Schluss noch etwas, dass Sie uns mitteilen möchten? (optional) Finally, is there anything else you would like to tell us? (optional) 3 Image Schematic Metaphors and Selection Criteria Presented image schemas, metaphors, selection criteria and presented alternatives for task one (group one: icons, group two: objects). Image Metaphor Selection criteria Presented schema alternatives 1 UP-DOWN POWERFUL IS UP – [29] CENTRE-PERIPHERY POWERLESS IS DOWN STRAIGHT- CROOKED 2 CONTENT- THE BODY/MIND/A ISCAT: metaphor which refers LEFT-RIGHT CONTAINER PERSON IS A to both, content, and container PART-WHOLE CONTAINER FOR THE SELF ABILITIES ARE THE CONTENT OF A PERSON-CONTAINER 3 NEAR-FAR THE PRESENT IS NEAR [29] HEAVY-LIGHT – THE PAST IS FAR CENTRE-PERIPHERY 4 CENTRE- IMPORTANCE IS ISCAT: metaphor which refers PART-WHOLE PERIPHERY CENTRALITY to both, centre and periphery LEFT-RIGHT UNIMPORTANT ISSUES ARE GIVEN PERIPHERAL POSITIONS 5 STRONG- MUCH IS STRONG – [29] UP-DOWN WEAK LITTLE IS WEAK >> HARD-SOFT MORE IS STRONG – LESS IS WEAK 6 PAINFUL FEAR/BEING AFRAID IS ISCAT: only two metaphors in LINKAGE PAIN English available OBJECT 7 STRAIGHT- MORAL IS STRAIGHT – [29] SMOOTH-ROUGH CROOKED CORRUPT IS CROOKED HARD-SOFT 8 HARD-SOFT INTENSIVE IS HARD – [29] SMOOTH-ROUGH SENSITIVE IS SOFT STRAIGHT- CROOKED 9 SMOOTH- POLITE IS SMOOTH – [25] CONTENT- ROUGH IMPOLITE IS ROUGH CONTAINER STRONG-WEAK 10 LEFT- CONSERVATIVE IS ISCAT: metaphor which NEAR-FAR RIGHT RIGHT – SOCIAL clearly maps left STRONG-WEAK DEMOCRATIC IS LEFT 11 LINKAGE LOVE IS A BOND ISCAT: most striking/easy to PAINFUL understand OBJECT 12 PART- CREATIVITY IS ISCAT: most striking/easy to NEAR-FAR WHOLE PUTTING PARTS understand HEAVY-LIGHT TOGETHER 13 HEAVY- IMPORTANT IS HEAVY [25] UP-DOWN LIGHT – UNIMPORTANT IS CONTENT- LIGHT CONTAINER 14 OBJECT IDEAS ARE OBJECTS ISCAT: metaphor which only LINKAGE refers to object, not further PAINFUL attributes or context Presented image schemas, metaphors, selection criteria and presented alternatives for task two (group one: icons, group two: objects). Image Metaphor Selection criteria Presented schema alternatives 1 UP-DOWN happy is up – sad is [29] CENTRE- down PERIPHERY STRAIGHT- CROOKED 2 CONTENT- the mind ISCAT: metaphor which LEFT-RIGHT CONTAINER (consciousness) is a refers to both, content, and PART-WHOLE container (for idea container objects) 3 NEAR-FAR emotional is near – [29] HEAVY-LIGHT unemotional is far CENTRE- PERIPHERY 4 CENTRE- identity is central ISCAT: most striking/easy PART-WHOLE PERIPHERY to understand LEFT-RIGHT 5 STRONG- powerful is strong – [29] UP-DOWN WEAK powerless is weak HARD-SOFT 6 PAINFUL disgust/being ISCAT: only two metaphors LINKAGE disgusted is pain in English available OBJECT 7 STRAIGHT- moral is straight – [29] SMOOTH-ROUGH CROOKED corrupt is crooked HARD-SOFT 8 HARD-SOFT stressful is hard – [29] SMOOTH-ROUGH relaxing is soft STRAIGHT- CROOKED 9 SMOOTH- boring is smooth – [25] CONTENT- ROUGH dangerous is rough CONTAINER STRONG-WEAK 10 LEFT-RIGHT moral is right – ISCAT: metaphor which NEAR-FAR immoral is left clearly maps left/right STRONG-WEAK 11 LINKAGE social relationships ISCAT: most striking/easy PAINFUL are links to understand OBJECT 12 PART-WHOLE coherent is whole ISCAT: metaphor which NEAR-FAR contains at least the term HEAVY-LIGHT whole 13 HEAVY-LIGHT more is heavy–less is [25] UP-DOWN light CONTENT- CONTAINER 14 OBJECT opportunities are ISCAT: metaphor which LINKAGE objects only refers to object, not PAINFUL further attributes or context 4 Correct Match Correct match of Image Schema Icons and Image Schema Objects in total numbers and percentage. Image Schema Icons Icons % Objects Objects % TOTAL 630 90 571 81.57 UP-DOWN 48 96 49 98 CONTENT-CONTAINER 40 80 45 90 NEAR-FAR 44 88 39 78 CENTRE-PERIPHERY 46 92 46 92 STRONG-WEAK 31 62 15 30 PAINFUL 50 100 49 98 STRAIGHT-CROOKED 44 88 40 80 HARD-SOFT 47 94 47 94 SMOOTH-ROUGH 49 98 50 100 LEFT-RIGHT 47 94 47 94 LINKAGE 49 98 46 92 PART-WHOLE 46 92 41 82 HEAVY-LIGHT 45 90 8 16