=Paper=
{{Paper
|id=Vol-3888/paper1
|storemode=property
|title=Multimodal meets Intuitive? Comparing Visual and Tangible
Image Schema Representations
|pdfUrl=https://ceur-ws.org/Vol-3888/Paper_1.pdf
|volume=Vol-3888
|authors=Cordola Baur,Fredrik Stamm,Carolin Wienrich,Jörn Hurtienne
|dblpUrl=https://dblp.org/rec/conf/isd2/BaurSWH24
}}
==Multimodal meets Intuitive? Comparing Visual and Tangible
Image Schema Representations==
https://ceur-ws.org/Vol-3888/Paper_1.pdf
Multimodal meets Intuitive? Comparing Visual and Tangible
Image Schema Representations
Cordula Baur 1, Fredrik Stamm1, Carolin Wienrich 2 and Jörn Hurtienne 1
1
Julius-Maximilians-Universität Würzburg, Chair of Psychological Ergonomics, Oswald-Külpe-Weg 82, 97074
Würzburg, Germany
2
Julius-Maximilians-Universität Würzburg, Human-Technology-Systems, Oswald-Külpe-Weg 82, 97074
Würzburg, Germany
Abstract
Image schemas are abstract representations of recurring multimodal experiences in the world.
Together with image-schematic metaphors, which connect image schemas with abstract
domains, they support the design process and foster more inclusive, intuitive, and innovative
designs. However, using image schemas in the design process requires extra effort and actual
image schema repositories do not meet designers’ requirements. Alternative forms of
representation like visualisations or physicalisations of image schemas can increase their
accessibility. This work presents an empirical study that evaluates Image Schema Icons and
Image Schema Objects in terms of their intuitive use, comprehensibility, and participants’
preference. Correct matches of representations to image-schematic metaphors were recorded,
interactions were observed, and the representations were evaluated by questionnaires. The
results showed that visual representations are more intuitive and achieved more correct
matches, but tangible representations were preferred. This directs further investigation and the
further development of image-schema-based design tools.
Keywords
Image Schemas, Design, Design Research, Evaluation, Intuitive Use
1
1. Introduction
Image schemas are representations of repeated, multimodal experiences aiding our understanding of
the environment [4, 35, 45]. Image-schematic metaphors emerge when image schemas are connected
with subjective experiences or judgments [36]. These metaphors assist in organising and structuring the
comprehension of abstract concepts [10, 17, 27, 32, 33, 35, 39, 40]. In Human-Computer Interaction
image schemas and their metaphors have been used for interface design and showed to foster more
inclusive, intuitive, and innovative designs [21, 26]. However, utilising image schemas for design
demands extra effort and time [19, 38, 47]. To tackle this, previous work recommended to use existing
image schema lists [21, 24, 47]. However, actual repositories are extensive databases [26] that lack
accessibility and applicability in the design process. Researchers in cognitive linguistics and Human-
Computer Interaction proposed visual representations of image schemas [4, 11–14, 32, 41, 44, 46] to
enhance the understanding of image schema theory. Additionally, tangible and visual representations
of FORCE image schemas have been suggested to support the design process [17]. In previous work we
described an iterative Research through Design process to create tangible and visual image schema
representations which aim at fostering the design of data physicalisations [2]. While the feedback during
the design process was positive, further evaluation was required. In this paper we present an empiric
Proceedings of The Eight Image Schema Day (ISD8), November 27-28, 2024, Bozen-Bolzano, Italy
EMAIL: cordula.baur@uni-wuerzburg.de (C. Baur); fredrikstamm@gmail.com (F. Stramm); carolin.wienrich@uni-wuerzburg.de (C.
Wienrich); joern.hurtienne@uni-wuerzburg.de (J. Hurtienne) ORCID: 0000-0002-0228-9747 (C. Baur); 0009-0009-3194-9925
(F.Stamm); 0000-0003-3052-7172 (C. Wienrich); 0000-0001-5295-9772 (J. Hurtienne)
©️ 2024 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
evaluation study where participants matched the representations to image-schematic metaphors, rated
intuitive use and comprehensibility, and stated preference.
2. Background
2.1. Image Schemas
Initially rooted in cognitive linguistics [21] image schemas were introduced by Johnson [32] and Lakoff [34]
as “recurring, dynamic pattern[s] of perceptual interactions and motor programs that give coherence and
structure to our experience” [32] (p. xiv). Image schemas link embodied experiences and mental
representations [32, 34] to provide structure to human perception and experiences, foster representation in
mind, and aid in understanding our surrounding world [4, 6, 9, 35, 45]. For instance, when a baby’s beloved
stuffed animal drops to the ground, it experiences gravity. The baby being repeatedly lifted or placed in a
pushchair or crib reinforces the experience of up and down movements. The repetition of such experiences
leads to the formulation of the UP-DOWN image schema. Image schemas as abstract concepts [21, 23, 27, 32]
do not refer to specific objects [21]. Image schemas are multimodal [4, 18, 21, 22, 32], integrating
experiences from multiple modalities [17, 18, 27, 28] and can be represented visually, haptically,
kinaesthetically or acoustically [17, 27, 28]. They are analogue [21, 23] and function subconsciously [21,
23, 27, 28], encoding and retrieving information from memory repeatedly [21]. Additionally, image schemas
proved to be largely cultural- and language-independent [39].
2.2. Image-schematic Metaphors
When an abstract concept that lacks sensory-motor experiences is assigned to a particular image
schema, an image-schematic metaphor emerges [18, 22, 25, 45]. This helps to organise and structure
the understanding of abstract concepts [10, 17, 27, 32, 33, 35, 39, 40] and supports the transfer of
information between different domains [4]. Projecting image schemas onto various abstract domains
enables reasoning about these domains [32]. The UP-DOWN image schema is associated with the
judgement of good and bad, forming the image-schematic metaphor UP IS GOOD – BAD IS DOWN.
Additionally, the UP-DOWN image schema is also linked to quantity (MORE IS UP – LESS IS DOWN) and
emotions (HAPPY IS UP-SAD IS DOWN). Linguistic analyses have identified over 250 metaphorical
extensions [24, 25]. These image-schematic metaphors are universal, are shared by a wide range of
people [19, 25], and were found to overlap across various languages and cultures [5, 8, 39, 42].
Furthermore they are automatically and intuitively understood [18].
2.3. Image Schemas for Design
Image schemas and their accompanying metaphors foster inclusive, intuitive, and innovative
designs. Inclusive design is fostered as image schemas are promising to work universally across user
groups with varying levels of technical proficiency and cultural backgrounds, because of their
connection to fundamental multimodal experiences [19, 21]. Furthermore, metaphor processing should
not be affected by a decline in conscious cognitive abilities of elderly, because it relies on automatic
and unconscious memory recall [21, 24, 26]. This makes image schemas universally applicable across
age groups [24]. Their multimodal nature enables also more inclusive design for people with
sensorimotor deficiencies [21, 26].
Image schemas promise to support the intuitive use of interfaces due to their relation to fundamental
human mental models and their subconscious appliance [23]. When designs are informed by image
schemas and their metaphoric extensions, they reflect the user’s mental models [38]. Furthermore,
image schemas and metaphors are readily available for human information processing due to their
frequent and continual repetition [16, 27].
Additionally, image schemas can help to identify essential aspects in design while keeping the
concrete instantiation open [26]. Image schemas do not propose a specific design solution, instead they
leave room for the designer to decide the implementation and create innovative solutions that go beyond
current standards [19, 21], therefore fostering more innovative designs.
Image schemas and their accompanying metaphors were successfully used to provide inspiration
and to generate novel design ideas [18, 19, 23, 28, 38, 39, 45]. They can also structure the design process
[45] and be used to describe affordances and design solutions [19, 23, 27]. Additionally, they can
support deeper thought about design decisions [39] and help to justify them [19].
However, it needs to be considered that using image schemas and metaphors in the design process
requires extra effort [19, 38, 47]. To address this, utilising established image schema lists is most
promising [21, 24, 47]. Such a list is provided by the Image Schema Catalogue (ISCAT) [26], but this
database does not serve as design tool, as it lacks easy accessibility and intuitive use, due to its large
volume and complex structure.
2.4. Visual Representations of Image Schemas
In cognitive linguistics illustrations were used to explain image schemas by highlighting their salient
characteristics [14]. Johnson suggested using diagrams to intuitively demonstrate how image schemas
operate periconceptually and has developed a notational system [32]. Talmy [44] depicted FORCE image
schemas by a system which consists of Agonist and Antagonist. Mandler [41] created a series of
pictorial representations to depict nonverbal concepts instead of exact interpretations. In Human-
Computer Interaction, Wilkie et al. [46] proposed visual representations of image schemas Besold et al.
[4], Hedblom et al. [11], and Hedblom [12], provided sequences of visualisations to show a process.
Hedblom and Neuhaus [14] later proposed a Diagrammatic Image Schema Language, a holistic system
to visually represent image schemas. This language provides organised and systematic representations
of abstract concepts. Furthermore, Hedblom [12] and Hedblom and Kutz [13] examined the relationship
between everyday objects and image schemas, using illustrations and names of image schemas. In this
work the authors stated the challenge of creating visuals that capture all characteristics of an image
schema.
2.5. Image Schema Representations to Support Design
Previous approaches applying image schemas to the design process required too much time and
effort [19, 38, 47]. In contrast, Hurtienne et al. [17] proposed visual as well as tangible representations
of FORCE image schemas. The characteristics of FORCE image schemas informed icons, while the notion
that a tangible representation might convey FORCE image schemas more effectively encouraged the
design of tangible representations. Image Schemas were instantiated as interactive physical rotatory
dials.
Both sets were tested for their effectiveness in identifying and distinguishing the represented image
schemas as well as their usefulness in the brainstorming process. The icons were identified more
frequently correctly than the tangible representations. Additionally, the visual representations were
mentioned to foster the generation of more ideas and in this condition, participants appreciated FORCE
image schemas to be more crucial and beneficial for design. Design ideas created using tangible
representations were perceived as more qualitative: ideas were considered to be more interactive, haptic
and visual [17].
In previous work [2] we used an iterative research-oriented Design process [7] to develop icons
(called Image Schema Icons) and clay objects (called Image Schema Objects) that represent image
schemas. We propose the use of tangible representations to facilitate data physicalisation design, as
these representations are more similar to the desired design outcome, which represents abstract data
through shape or material properties [30]. Designers no longer need to handle descriptions and textual
definitions of image schemas. The representations make image schemas easier to examine, contrast and
compare, to figure out which one works best for the actual design task. Additionally, specific examples
for including image schemas in a data physicalisation are provided by the tangible representations. This
might address the identified challenges of extra time and effort when using image schemas in the design
process. The process of designing the image schema representations provided promising feedback and
the tangible representations were already tested in a workshop setting [1] but a comprehensive
evaluation of their effectiveness is required. Before testing image schema representations in the design
process, it is necessary to assess their comprehensiveness and intuitive use and choose one of the
instantiation types. Therefore, we investigate in this work the research question, whether Image Schema
Icons or Image Schema Objects depict image schemas in a more intuitive and comprehensive manner.
Additionally, we explore user preferences.
Tangible representations may be appropriate for image schemas, because they are able to represent
the multimodality of experiences incorporated in image schemas [27, 28, 18, 17]. Hurtienne et al. [17]
assessed visual and tangible representations of FORCE image schemas and found tangible
representations to encourage the formation of more interactive, visual, and haptic ideas, while visual
instantiations were more precisely identified and fostered a greater quantity of ideas. It needs to be
considered that FORCE image schemas are a special subset of image schemas. Because of their
temporary, abstract, and dynamic nature, they can be hard to recognise and categorise [17]. This work
focuses on different image schemas. When creating icons and clay objects to represent image schemas
we identified some image schemas being easier to recognise and represent in visual form, other image
schemas in tangible form [2]. Therefore, Hurtienne et al.’s [17] findings might not be generalisable for
all image schemas. In some cases, the tangible representation may be identified as well or even better.
It is necessary to evaluate the intuitive use and comprehensiveness of different representation
modalities. Our explorative hypothesis is that visual and tangible representations differ in terms of
intuitive use, comprehensiveness, and accurate matches of representations to image-schematic
metaphors, as well as in preference ratings.
3. Method
To evaluate the intuitive use and the comprehensibility of the image schema representations we
conducted a within-subject design study. Randomly assigned to groups, participants of group one began
with visual representations and continued with the objects, while group two followed the reverse order.
This setup was intended to avoid cross-over effects. Participants matched image-schematic metaphors
to the presented Image Schema Icons or Image Schema Objects and rated Intuitive Use and
Comprehensibility in questionnaires. In the end, they were asked for their preference. Interaction with
the representations was observed and correct matches were counted.
3.1. Participants
Recruited from the universities’ participant pool, participants received 0.5 credit points as
compensation. No exclusion criteria were applied, as image schemas claim being universal across
cultural backgrounds and age [18, 20, 21, 23, 28, 38, 39, 45]. The study was conducted in German but
to avoid altering their meaning through translation, we presented the image schematic metaphors in
their original language (English). To avoid confusion, we provided a list of English-German translations
for the terms used. Additionally, participants were asked about their English proficiency level and prior
experience with image schemas.
A total of fifty participants (n = 50), with an average age of 21.22 years (Standard Deviation (SD) =
1.36) participated. None of them had any prior experience with image schemas. Ten participants (20
%) rated their English at C1 level, 29 (58 %) at B2 level, eight (16 %) at B1 level, three participants (6
%) at A2 level, and no participants rated their English level A1. In the following the participants are
identified as P4 to P54; P1 to P4 were not included in the data analysis but took part in pilot testing to
improve the research design.
3.2. Procedure
The study lasted for approximately 30 minutes. After welcoming and conducting informed consent,
a demographic questionnaire was completed and participants were given written instructions
(Supplementary Material 1). Participants were asked to read statements (image-schematic metaphors)
presented on A5 printouts and to select the icon or icon pair or object or object-pair best fitting the
metaphor. Fourteen image-schematic metaphors were conducted in total. After completing the task,
participants filled in questionnaires (Supplementary Material 2) to rate the intuitive use and
comprehensibility of the stimuli. This procedure was repeated with 14 new metaphors and the other
representation modality. Participants who first used objects, now used icons, and vice versa. Intuitive
use and comprehensiveness were rated again using the same questionnaires. Additionally, participants
were asked which stimuli they preferred and why. During the matching task, the researchers observed
whether the participants interacted with the objects physically and recorded correct matches.
3.3. Material and Setup
In a previous phase of this project, we selected a subset of image schemas to be represented in visual
and tangible way, regarding their potential to support data physicalisation design. This decision was
informed by analyses of existing data physicalisations regarding incorporated image schemas [3] and
the potential for improvement through additional image schemas [under review]. Furthermore,
recommendations made in literature, which image schemas serve to foster the design of tangible user
interfaces [25, 28] informed our selection. For more details regarding our selection criteria for image
schemas see [under review].
A6 cards were used to display the Image Schema Icons (Figure 1), while the objects (Figure 2) were
already crafted in a Research-oriented Design process [7]. For a detailed description of the design
process of the Image Schema Icons and Image Schema Objects see [2].
Figure 1: Visual representations of image schemas, printed on A6 cards. From left to right, first row:
STRAIGHT-CROOKED, SMOOTH-ROUGH, UP-DOWN, PAINFUL; second row: CONTENT-CONTAINER,
OBJECT, HEAVY-LIGHT, HARD-SOFT; third row: CENTRE-PERIPHERY, LEFT-RIGHT, PART-WHOLE, NEAR-
FAR; fourth row: LINKAGE, STRONG-WEAK.
Figure 2: Handcrafted tangible representations of image schemas made of clay. From left to right, last
row: STRAIGHT-CROOKED, SMOOTH-ROUGH, UP-DOWN, PAINFUL; second last row: CONTENT-
CONTAINER, OBJECT, HEAVY-LIGHT, HARD-SOFT; second front row: CENTRE-PERIPHERY, LEFT-RIGHT,
PART-WHOLE, NEAR-FAR; front row: LINKAGE, STRONG-WEAK.
For the matching task, well-established image-schematic metaphors were selected based on high
confirmation rates [24, 25, 29, 40] or their well-documented linguistic findings. For image schemas
where this was not possible, a metaphor was chosen from the ISCAT database [26]. The metaphors,
accompanied with selection criteria, and alternative image schemas are provided as supplementary
material 3. For each metaphor, we presented the correct image schema representation and two incorrect
options. To be able to show all three choices simultaneously and to avoid presenting the different
choices for different duration, we used a cardboard cover while arranging the stimuli. We varied the
position of the correct choice for each metaphor. The study setup is depicted in Figure 3.
Representations that are easily confused, such as HARD-SOFT, SMOOTH-ROUGH, or STRAIGHT-CROOKED,
or those with similar characteristics, like STRONG-WEAK, or HEAVY-LIGHT were presented together.
OBJECT, LINKAGE, and PAINFUL, each consisting of only one term, were presented as alternatives to
avoid the lack of bi-dimensional structure being used as exclusion criteria. To ensure clarity for the
researcher who conducted the data collection, the metaphors were presented in the same order for each
participant.
Figure 3: Study setup with cardboard coverage and cardboard area to present stimuli.
3.4. Collection and Analysis of Data
The representations’ intuitive use and comprehensibility were evaluated using the Modular
Extension of the User Experience Questionnaire (UEQ+) [43]. The 7-point subscale intuitive use
measures the ease of use with the items difficult-easy, illogical-logical, not plausible-plausible, and
inconclusive-conclusive. Comprehensibility is measured with the items complicated-simple,
unambiguous-ambiguous, inaccurate-accurate, and enigmatic-explainable. The UEQ+ is a well-
established questionnaire, frequently used to evaluate products’ user experience and therefor it was
deemed appropriate to use it for assessing the experience with prototypical design tools. To evaluate
how well the presented image schemas can be identified, we recorded correct matches of image-
schematic metaphors to image schema representations. To determine whether the choice was solely
informed by the visual appearance of the objects, we observed whether participants physically
interacted with the tangible image schema representations. Furthermore, participants indicated their
preference for icons or clay objects. Data was collected using LimeSurvey [37] and analysed using the
statistics software JASP [31], which was also used to provide values for Mean (M) and Standard
Deviation (SD). The qualitative data was analysed by creating an Affinity Diagram, loosely applying
the Contextual Design Approach [15] for data evaluation. From the participants’ answers we created
Affinity Notes and organised them into groups based on inductive reasoning.
4. Results
To compare visual and tangible representations, we conducted dependant t-tests. We chose this test,
as it is a often used und reliable test for within-design study setups. No outliers were excluded, no data
values were missing. Even when the data showed no normal distribution, we proceeded with the
analysis, because our sample (n) is bigger than 30 and therefore the data is robust against violation of
the normal distribution. The significance level α describes the maximum probability that a null
hypothesis (no difference) is incorrectly rejected. It was set at alpha = .05.
In terms of intuitive use the icons (M = 6.11, SD = 0.67) and objects (M = 5.71, SD = 0.89) showed
significant difference (t(49) = 3.239, p = .002, d = .162). Here t describes the t-value which is used to
define the p-value; p shows the significance; d describes Cohens’d and shows the effect size, which can
be used to compare the results with studies measuring the same dependent variable. The rating of
comprehensibility showed no significant difference (t(49) = .509, p = .613, d = .072) between icons (M
= 5.56, SD = 0.94) and objects (M = 5.49, SD = 1.01). Counting the number of correct matches showed
that for 630 times (90.00 %) the correct icons were selected, but only for 571 times (81.57 %) the correct
objects were selected. This is a significant difference t(699) = 4.982, p < .001, d = .188. However,
Image Schema Icons and Image Schema Objects both showed a high number of correct matches. The
visual representations of STRONG-WEAK and CONTENT-CONTAINER, as well as the tangible
representations of HEAVY-LIGHT and STRONG-WEAK showed the lowest number of correct matches.
Figure 4 shows the correct matches per image schema. The full data is provided as supplementary
material 4.
60
49 50 49 50 49
50
48 45 46 46 47 47 49 47 47 49 46 46 45
44 44 41 44
40 39 40
40
31
30
20 15
8
10
0
Icons Objects
Figure 4: Number of correct matches of visual and tangible representations to image-schematic
metaphors.
Sixteen participants (32.00 %) preferred icons, while 34 participants (68.00 %) preferred objects.
The participants stated that the icons are more intuitive (P8, P25, P34, P46, P49, P52) and less difficult
to match (P16, P20, P25, P33, P37, P45, P52). Some appreciated the icons for their details (P35, P45),
others the room for interpretation they provide (P19, P40). However, the majority preferred the Image
Schema Objects which were experienced as easier to comprehend (P12, P21, P23, P26, P28, P32, P38,
P42, P43, P48) and better suited for matching metaphors due to their three-dimensional shape (P10,
P14, P24, P32, P48, P54). Participants stated the objects show a higher aesthetic quality (P9, P18, P45,
P50, P51). Furthermore, they highlighted the objects as being more graspable (P9, P12, P22, P29, P38,
P39, P54) and liked the opportunity of touching and interacting with them (P4, P27, P29, P31, P32,
P36, P41, P42, P51).
Our observation revealed that most participants made their choice and expressed their preference
solely based on the visual appearance of the stimuli. Only 14 participants (28.00 %) showed physical
interaction. Of 48 interactions (excluding 10 interactions with the wrong objects), 41 interactions (85.00
%) resulted in a correct match. The most frequently interacted objects were, HEAVY-LIGHT and STRONG-
WEAK, followed by CONTENT-CONTAINER. Conversely, the objects that showed least correct matches
were most frequently interacted with. Figure 5 shows the interactions per Image Schema Object.
12
9 9
10 8
8 6
5
6
4 2 2 2 2
1 1 1
2 0 0
0
Figure 5: Number of physical interactions per Image Schema Object.
5. Discussion
Utilising image schemas showed to foster more inclusive, intuitive, and innovative designs and to
aid the design process. However, the use of image schemas demands extra effort and time. Currently
available image schema repositories do not provide an easily applicable design-tool. To address this
issue, we developed visual and tangible representations to make image schemas accessible and
incorporable in the data physicalisation design process. In this study we evaluated these representations
to determine if they convey image schemas in an intuitive and comprehensive way and which modality
of representation works best. Participants matched image-schematic metaphors to visual or tangible
image schema representations and rated intuitive use, comprehensiveness, and their preference for one
representation modality (visual or tangible). The study utilised questionnaires, recorded correct
matches, and observed interactions with the tangible representations. In previous research [17], visual
representations of FORCE image schemas were more often identified correctly. However, due to the
special character of FORCE image schemas, these findings may not be generalisable for all image
schemas. Investigating different representation modalities of other image schemas in an explorative
way, this work provides evidence for the Image Schema Icons being more intuitive and resulting in
more correct matches, than the Image Schema Objects. However, participants preferred the tangible
representations more often.
Previous research already highlighted that the way how image schemas are instantiated is important
for their comprehensiveness [25, 29, 40]. Consistent with previous work, which demonstrated that
visual representations were more accurately identified [17], this study also found that the Image Schema
Icons resulted in more correct matches and were perceived as more intuitive (qualitative and
quantitative). They were also rated as more comprehensive, but without significant evidence. Previous
findings were confirmed and showed to be applicable also to other image schemas. However, it should
be noted that participants showed limited interaction with the tangible instantiations and their ratings
were primarily based on the objects’ visual appearance rather than a tangible experience. Therefore a
reason could be that since childhood people are trained in educational but also exhibition settings, not
to touch physical artifacts. The tangible characteristics of the Image Schema Objects may not have been
experienced and the objects didn’t realise their full potential. Therefore, they might have influenced the
participants’ rating only to a small extent.
Both visual and tangible representations achieved high numbers of correct matches. Only HEAVY-
LIGHT and STRONG-WEAK showed a major difference between conditions. For both image schemas the
correct matches of tangible representations were much lower than for visual representations. Most
Image Schema Objects’ visual appearance is similar to the Image Schema Icons, but not for HEAVY-
LIGHT and STRONG-WEAK. The design process [2] showed that finding appropriate visual and tangible
representations for HEAVY-LIGHT and STRONG-WEAK was difficult and participants struggled with their
recognition. The final tangible representations require tangible interaction and exploration to fully
convey the image schemas’ characteristics and to be identified correctly. In fact, the tangible
representations of these image schemas showed the highest interaction. However, in total only a
minority of participants interacted with the objects. Therefore, for most participants the tangible
representations of HEAVY-LIGHT and STRONG-WEAK remained concealed which impeded a correct
choice.
In the qualitative data, the icons were stated to be more intuitive to understand (P8, P25, P34, P46, P49,
P52) and easier to match to metaphors (P16, P20, P25, P33, P37, P45, P52). Although visual
representations lead to more correct matches and were rated to be more intuitive, participants preferred
more often the Image Schema Objects. Even they showed only a limited number of tangible interactions,
they stated to appreciate the opportunity of touching and interacting with the objects (P4, P27, P29,
P31, P32, P36, P41, P42, P51). Furthermore, the tangible representations were preferred because of
their three-dimensionality, which supports matching to metaphors (P10, P14, P24, P32, P48, P54), and
were experienced as more graspable (P9, P12, P22, P29, P38, P39, P54). Additionally, some
participants stated they are easy to understand (P12, P21, P23, P26, P28, P32, P38, P42, P43, P48),
while others found the objects aesthetically pleasing (P9, P18, P45, P51).
5.1. Limitations
Participants may have recognised similar visual appearances of Image Schema Objects and Image
Schema Icons among conditions, which could have caused a learning effect. However, the crossover-
design was implemented to prevent this from confounding the results.
Another potential limitation of this work is that for STRAIGHT-CROOKED the same metaphor was
used in both conditions. However, as this was one of 14 metaphors presented per condition, it is unlikely
that participants noticed this and referred to their choice made in the previous condition.
A more crucial aspect is participants’ English proficiency. The majority stated their English level
higher than A1 and only one participant used the provided translation sheet. However, some participants
appeared to be confused or uncertain regarding the meaning of some metaphors. It is possible that they
felt embarrassed to admit a lack of English knowledge and therefore didn’t use the translation sheet.
This may have led to misunderstandings of the image-schematic metaphors and affected the accuracy
of the matches and ratings.
Furthermore, instructing participants to make intuitive decisions may have influenced their choices.
Some participants stated in retrospect that if they had invested more time, they would have chosen
different icons or objects. The instructions aimed to encourage intuitive decision-making and prevent
participants from overthinking their choices. This raises the question of whether a more deliberate
decision would increase or decrease the number of correct matches. Furthermore, the instructions
prevented participants from taking the time to explore and interact with the objects more intensely.
Allowing more time could promote more intense interaction and with this a more multimodal
experience of the objects. These aspects, both worth further research.
6. Conclusion
Image schemas enhance both, design outcome and the design process. To overcome the additional
effort and time for using image schemas in design, a more accessible way to represent and utilise them
is required. This work compared and evaluated visual and tangible representations of image schemas to
determine which modality conveys image schemas best. Therefore, an empiric study was conducted,
where participants matched image-schematic metaphors to visual and tangible representations, rated
intuitive use and comprehensibility and indicated their preference. The Image Schema Icons showed
higher ratings for intuitive use and a higher number of correct matches. The Image Schema Objects also
showed high numbers of correct matches and were preferred more often due to their opportunity for
physical interaction.
6.1. Outlook
In the next step, we are going to evaluate image schema representations’ effectiveness for designing
data physicalisations. Further work could explore the transferability of Image Schema Icons and Image
Schema Objects and their usefulness for other design tasks, such as tangible interfaces. Previous
research has already highlighted image schemas’ potential for tangible user interface design [25, 28],
which could be further reinforced by our proposed image schema representations.
7. References
[1] Baur, C. et al. 2022. Designing Data Physicalisations with Physical Image Schema Instantiations.
Short Paper Proceedings of the 5th European Tangible Interaction Studio (Toulouse France, Nov.
2022).
[2] Baur, C. et al. 2022. Form Follows Mental Models: Finding Instantiations of Image Schemas
Using a Design Research Approach. DIS ’22: Proceedings of the 2022 ACM Designing
Interactive Systems Conference (Virtual Event Australia, Jun. 2022), 586–598.
[3] Baur, C. et al. 2023. Image Schemas as Tool for Exploring the Design Space of Data
Physicalisations. Proceedings of The Seventh Image Schema Day (Rhodes Greece, Sep. 2023).
[4] Besold, T.R. et al. 2017. A narrative in three acts: Using combinations of image schemas to model
events. Biologically Inspired Cognitive Architectures. 19, (2017), 10–20.
DOI:https://doi.org/10.1016/j.bica.2016.11.001.
[5] Cienki, A. 1998. STRAIGHT: An image schema and its metaphorical extensions. (1998).
[6] Dimitra Bourou et al. 2021. Image Schemas and Conceptual Blending in Diagrammatic
Reasoning: The Case of Hasse Diagrams. Diagrammatic Representation and Inference. Amrita
Basu et al., eds. Springer. 297–314.
[7] Fallman, D. 2007. Why Research-Oriented Design Isn’t Design-Oriented Research: On the
Tensions Between Design and Research in an Implicit Design Discipline. Knowledge, Technology
& Policy. 20, 3 (Oct. 2007), 193–200. DOI:https://doi.org/10.1007/s12130-007-9022-8.
[8] Forceville, C. 2006. Metaphor in Culture: Universality and Variation, Zoltán Kövecses.
Cambridge University Press, Cambridge (2005), 314 pp., ISBN 0 521 84447 9 (hardback).
Journal of Pragmatics. 38, (Sep. 2006), 1528–1531.
DOI:https://doi.org/10.1016/j.pragma.2006.03.003.
[9] Gibbs, R.W. and Colston, H.L. 1995. The cognitive psychological reality of image schemas and
their transformations. Cognitive Linguistics. 6, 4 (1995), 347–378.
DOI:https://doi.org/10.1515/cogl.1995.6.4.347.
[10] Grady, J.E. 1997. Foundations of meaning: Primary metaphors and primary scenes. Department
of Linguistics, University of California at Berkeley.
[11] Hedblom, M.M. et al. 2017. Between Contact and Support: Introducing a Logic for Image
Schemas and Directed Movement. AI*IA 2017 Advances in Artificial Intelligence, Proceedings
of the XVIth International Conference of the Italian Association for Artificial Intelligence (Nov.
2017), 256–268.
[12] Hedblom, M.M. 2020. Image Schemas and Concept Invention: Cognitive, Logical, and Linguistic
Investigations. Springer.
[13] Hedblom, M.M. and Kutz, O. 2019. Conceptual Puzzle Pieces. Modeling and Using Context.
CONTEXT 2019 (Cham, 2019), 98–111.
[14] Hedblom, M.M. and Neuhaus, F. 2022. Visualising Image Schemas: A Preliminary Look at the
Diagrammatic Image Schema Language (DISL). Proceedings of the Sixth Image Schema Day
(Jönköping Sweden, Mar. 2022).
[15] Holtzblatt, K. and Beyer, H. 2014. Contextual Design: Evolved. Morgan & Claypool Publishers.
[16] Hurtienne, J. 2009. Cognition in HCI: An Ongoing Story. Human Technology. 5, 1 (May 2009),
12–28. DOI:https://doi.org/10.17011/ht/urn.20094141408.
[17] Hurtienne, J. et al. 2015. Comparing Pictorial and Tangible Notations of Force Image Schemas.
TEI ’15: Proceedings of the Ninth International Conference on Tangible, Embedded, and
Embodied Interaction (Stanford California USA, Jan. 2015), 249–256.
[18] Hurtienne, J. et al. 2008. Cooking up real world business applications combining physicality,
digitality, and image schemas. TEI ’08: Proceedings of the 2nd international conference on
Tangible and embedded interaction (Bonn Germany, Feb. 2008), 239–246.
[19] Hurtienne, J. et al. 2015. Designing with Image Schemas: Resolving the Tension Between
Innovation, Inclusion and Intuitive Use. Interacting with Computers. 27, (Apr. 2015).
DOI:https://doi.org/10.1093/iwc/iwu049.
[20] Hurtienne, J. et al. 2015. Designing with Image Schemas: Resolving the Tension Between
Innovation, Inclusion and Intuitive Use. Interacting with Computers. 27, 3 (May 2015), 235–255.
DOI:https://doi.org/10.1093/iwc/iwu049.
[21] Hurtienne, J. 2016. How Cognitive Linguistics Inspires HCI: Image Schemas and Image-
Schematic Metaphors. International Journal of Human-Computer Interaction. 33, 1 (Sep. 2016),
1–20. DOI:https://doi.org/10.1080/10447318.2016.1232227.
[22] Hurtienne, J. et al. 2007. Image schemas: a new language for user interface design? Prospektive
Gestaltung von Mensch-Technik-Interaktion. M. Rötting et al., eds. VDI Verlag. 167–172.
[23] Hurtienne, J. 2011. Image Schemas and Design for Intuitive Use. Technische Universität Berlin.
[24] Hurtienne, J. et al. 2010. Physical gestures for abstract concepts: Inclusive design with primary
metaphors. Interacting with Computers. 22, 6 (Nov. 2010), 475–484.
DOI:https://doi.org/10.1016/j.intcom.2010.08.009.
[25] Hurtienne, J. et al. 2009. Sad is heavy and happy is light: population stereotypes of tangible object
attributes. TEI ’09: Proceedings of the 3rd International Conference on Tangible and Embedded
Interaction (Cambridge United Kingdom, Feb. 2009), 61–68.
[26] Hurtienne, J. et al. 2022. Supporting User Interface Design with Image Schemas: The ISCAT
Database as a Research Tool. Proceedings of the Sixth Image Schema Day (Jönköping Sweden,
Mar. 2022).
[27] Hurtienne, J. and Blessing, L. 2007. Design for intuitive use - Testing Image Schema Theory for
User Interface Design. DS 42: Proceedings of ICED 2007, the 16th International Conference on
Engineering Design (Paris France, Jul. 2007), 829–830.
[28] Hurtienne, J. and Israel, J.H. 2007. Image schemas and their metaphorical extensions: intuitive
patterns for tangible interaction. TEI ’07: Proceedings of the 1st international conference on
Tangible and embedded interaction (Baton Rouge Lousiana, Feb. 2007), 127–134.
[29] Hurtienne, J. and Meschke, O. 2016. Soft Pillows and the Near and Dear: Physical-to-Abstract
Mappings with Image-Schematic Metaphors. TEI ’16: Proceedings of the TEI ’16: Tenth
International Conference on Tangible, Embedded, and Embodied Interaction (Eindhoven
Netherlands, Feb. 2016), 324–331.
[30] Jansen, Y. et al. 2015. Opportunities and Challenges for Data Physicalization. CHI ’15:
Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems
(Seoul Republic of Korea, Apr. 2015), 3227–3236.
[31] JASP Team 2024. JASP.
[32] Johnson, M. 1987. The body in the mind: The bodily basis of meaning, imagination, and reason.
University of Chicago Press.
[33] Johnson, M. 2005. The philosophical significance of image schemas. From Perception to
Meaning: Image Schemas in Cognitive Linguistics. B. Hampe, ed. De Gruyter Mouton. 15–34.
[34] Lakoff, G. 1987. Women, Fire, and Dangerous Things: What Categories Reveal about the Mind.
University of Chicago Press.
[35] Lakoff, G. and Johnson, M. 1980. Metaphors we live by. University of Chicago Press.
[36] Lakoff, G. and Johnson, M. 1999. Philosophy in The Flesh: The Embodied Mind And Its
Challenge To Western Thought. Basic Books.
[37] Limesurvey GmbH 2023. LimeSurvey: An Open Source survey Tool.
[38] Löffler, D. et al. 2013. Developing Intuitive User Interfaces by Integrating Users’ Mental Models
into Requirements Engineering. BCS-HCI ’13: Proceedings of the 27th International BCS Human
Computer Interaction Conference (London, UK, Sep. 2013), 1–10.
[39] Löffler, D. et al. 2014. Mixing Languages’: image schema inspired designs for rural Africa. CHI
EA ’14: CHI ’14 Extended Abstracts on Human Factors in Computing Systems (Toronto Ontario
Canada, May 2014), 1999–2004.
[40] Macaranas, A. et al. 2012. Bridging the gap: attribute and spatial metaphors for tangible interface
design. TEI ’12: Proceedings of the Sixth International Conference on Tangible, Embedded and
Embodied Interaction (Kingston Ontario Canada, Feb. 2012), 161–168.
[41] Mandler, J.M. 1992. How to build a baby: II. Conceptual primitives. Psychological Review. 99, 4
(Nov. 1992), 587–604. DOI:https://doi.org/10.1037/0033-295X.99.4.587.
[42] Neumann, C. 2001. Is Metaphor Universal? Cross-Language Evidence From German and
Japanese. Metaphor and Symbol - METAPHOR SYMB. 16, (Apr. 2001), 123–142.
DOI:https://doi.org/10.1207/S15327868MS1601&2_9.
[43] Schrepp, M. and Thomaschewski, J. 2019. Eine modulare Erweiterung des User Experience
Questionnaire. Usability Professionals (UP19) (2019), 148–156.
[44] Talmy, L. 1988. Force Dynamics in Language and Cognition. Cognitive Science. 12, 1 (Jan.
1988), 49–100. DOI:https://doi.org/10.1207/s15516709cog1201_2.
[45] Tscharn, R. 2017. Design of Age-Inclusive Tangible User Interfaces Using Image-Schematic
Metaphors. TEI ’17: Proceedings of the Eleventh International Conference on Tangible,
Embedded, and Embodied Interaction (Yokohama Japan, Mar. 2017), 693–696.
[46] Wilkie, K. et al. 2009. Evaluating Musical Software Using Conceptual Metaphors. BCS-HCI ’09:
Proceedings of the 23rd British HCI Group Annual Conference on People and Computers:
Celebrating People and Technology (Cambridge Great Britain, Sep. 2009), 232–237.
[47] Winkler, A. et al. 2016. Evaluation of an Application Based on Conceptual Metaphors for Social
Interaction Between Vehicles. DIS ’16: Proceedings of the 2016 ACM Conference on Designing
Interactive Systems (Brisbane QLD Australia, Jun. 2016), 1148–1159.
Supplementary Material
1 Instructions
Sie bekommen insgesamt 14 nummerierte Zettel. Auf der Rückseite der nummerierten Zettel steht
jeweils ein kurzer Satz. Sie lesen immer den jeweiligen Satz. Anschließend zeigt Ihnen die
Versuchsleitung drei Icons bzw. Icon-Paare. Wählen Sie das Icon bzw. Icon-Paar aus, dass Ihrer
Meinung nach im Satz enthalten ist. Denken Sie nicht zu lange nach, entscheiden Sie intuitiv aus dem
Bauch heraus.
Hinweis: Die Sätze sind auf Englisch. Wenn Sie eine Übersetzungsliste brauchen, können Sie im
Fragebogen einmal auf „Weiter“ klicken.
You will be given a total of 14 numbered sheets of paper. On the back of each sheet there is a short
sentence. You will read each sentence. The experimenter will then show you three icons or pairs of
icons. Choose the icon or pair of icons that you think is in the sentence. Don't think too long, make an
intuitive decision.
Note: The sentences are in English. If you need a translation list, you can click 'Next' once in the
questionnaire.
Sie bekommen insgesamt 14 nummerierte Zettel. Auf der Rückseite der nummerierten Zettel steht
jeweils ein kurzer Satz. Sie lesen immer den jeweiligen Satz. Anschließend zeigt Ihnen die
Versuchsleitung drei Objekte bzw. Objekt-Paare. Wählen Sie das Objekt bzw. Objekt-Paar aus, dass
Ihrer Meinung nach im Satz enthalten ist. Denken Sie nicht zu lange nach, entscheiden Sie intuitiv aus
dem Bauch heraus.
Hinweis: Die Sätze sind auf Englisch. Wenn Sie eine Übersetzungsliste brauchen, können Sie im
Fragebogen einmal auf „Weiter“ klicken.
You will be given a total of 14 numbered sheets of paper. On the back of each sheet there is a short
sentence. You will read each sentence. The experimenter will then show you three objects or pairs of
objects. Choose the object or pair of objects that you think is in the sentence. Don't think too long, make
an intuitive decision.
Note: The sentences are in English. If you need a translation list, you can click 'Next' once in the
questionnaire.
2 Questionnaires
Demographic Data
• Welches ist Ihr bisher höchster Bildungsabschluss?
What is your highest educational qualification to date?
• Welches ist Ihr Geschlecht?
What is your gender?
• Wie alt sind Sie gemessen in Jahren?
How old are you in years?
• Welches ist Ihre Muttersprache?
What is your mother tongue?
• Wie würden Sie Ihre Englischkenntnisse einordnen?
How would you categorise your English language skills?
A1, A2, B1, B2, C1, C2
• Welcher beruflichen oder berufsqualifizierenden Tätigkeit gehen Sie derzeit hauptsächlich
nach?
What is your main professional or vocational activity at present?
• Haben Sie bereits Vorerfahrung im Themengebiet Image Schemas?
Do you have any previous experience in the field of image schemas?
• Welche Erfahrungen im Themengebiet Image Schemas haben Sie?
What experience do you have in the field of image schemas?
UEQ+: Intuitive Bedienung
UEQ+: Intuitive Use
• Die Zuordnung der Icons/Objekte war für mich …
The assignment of the icons/objects was …
o mühevoll – mühelos
difficult – easy
o unlogisch – logisch
illogical – logical
o nicht einleuchtend – einleuchtend
not plausible – plausible
o nicht schlüssig – schlüssig
inconclusive – conclusive
UEQ+: Verständnis
UEQ+: Comprehensibility
• Die Icons/Objekte sind für mich …
The icons/objects are …
o kompliziert – einfach
complicated – simple
o ungenau – genau
unambiguous – ambiguous
o nicht eindeutig – eindeutig
inaccurate – accurate
o rätselhaft – erklärbar
enigmatic – explainable
Präferenz
Preference
• Welche Darstellungsform hat Ihnen insgesamt besser gefallen und warum?
Which form of representation did you like better? Why?
• Gibt es zum Schluss noch etwas, dass Sie uns mitteilen möchten? (optional)
Finally, is there anything else you would like to tell us? (optional)
3 Image Schematic Metaphors and Selection Criteria
Presented image schemas, metaphors, selection criteria and presented alternatives for task one
(group one: icons, group two: objects).
Image Metaphor Selection criteria Presented
schema alternatives
1 UP-DOWN POWERFUL IS UP – [29] CENTRE-PERIPHERY
POWERLESS IS DOWN STRAIGHT-
CROOKED
2 CONTENT- THE BODY/MIND/A ISCAT: metaphor which refers LEFT-RIGHT
CONTAINER PERSON IS A to both, content, and container PART-WHOLE
CONTAINER FOR THE
SELF
ABILITIES ARE THE
CONTENT OF A
PERSON-CONTAINER
3 NEAR-FAR THE PRESENT IS NEAR [29] HEAVY-LIGHT
– THE PAST IS FAR CENTRE-PERIPHERY
4 CENTRE- IMPORTANCE IS ISCAT: metaphor which refers PART-WHOLE
PERIPHERY CENTRALITY to both, centre and periphery LEFT-RIGHT
UNIMPORTANT ISSUES
ARE GIVEN
PERIPHERAL
POSITIONS
5 STRONG- MUCH IS STRONG – [29] UP-DOWN
WEAK LITTLE IS WEAK >> HARD-SOFT
MORE IS STRONG –
LESS IS WEAK
6 PAINFUL FEAR/BEING AFRAID IS ISCAT: only two metaphors in LINKAGE
PAIN English available OBJECT
7 STRAIGHT- MORAL IS STRAIGHT – [29] SMOOTH-ROUGH
CROOKED CORRUPT IS CROOKED HARD-SOFT
8 HARD-SOFT INTENSIVE IS HARD – [29] SMOOTH-ROUGH
SENSITIVE IS SOFT STRAIGHT-
CROOKED
9 SMOOTH- POLITE IS SMOOTH – [25] CONTENT-
ROUGH IMPOLITE IS ROUGH CONTAINER
STRONG-WEAK
10 LEFT- CONSERVATIVE IS ISCAT: metaphor which NEAR-FAR
RIGHT RIGHT – SOCIAL clearly maps left STRONG-WEAK
DEMOCRATIC IS LEFT
11 LINKAGE LOVE IS A BOND ISCAT: most striking/easy to PAINFUL
understand OBJECT
12 PART- CREATIVITY IS ISCAT: most striking/easy to NEAR-FAR
WHOLE PUTTING PARTS understand HEAVY-LIGHT
TOGETHER
13 HEAVY- IMPORTANT IS HEAVY [25] UP-DOWN
LIGHT – UNIMPORTANT IS CONTENT-
LIGHT CONTAINER
14 OBJECT IDEAS ARE OBJECTS ISCAT: metaphor which only LINKAGE
refers to object, not further PAINFUL
attributes or context
Presented image schemas, metaphors, selection criteria and presented alternatives for task two
(group one: icons, group two: objects).
Image Metaphor Selection criteria Presented
schema alternatives
1 UP-DOWN happy is up – sad is [29] CENTRE-
down PERIPHERY
STRAIGHT-
CROOKED
2 CONTENT- the mind ISCAT: metaphor which LEFT-RIGHT
CONTAINER (consciousness) is a refers to both, content, and PART-WHOLE
container (for idea container
objects)
3 NEAR-FAR emotional is near – [29] HEAVY-LIGHT
unemotional is far CENTRE-
PERIPHERY
4 CENTRE- identity is central ISCAT: most striking/easy PART-WHOLE
PERIPHERY to understand LEFT-RIGHT
5 STRONG- powerful is strong – [29] UP-DOWN
WEAK powerless is weak HARD-SOFT
6 PAINFUL disgust/being ISCAT: only two metaphors LINKAGE
disgusted is pain in English available OBJECT
7 STRAIGHT- moral is straight – [29] SMOOTH-ROUGH
CROOKED corrupt is crooked HARD-SOFT
8 HARD-SOFT stressful is hard – [29] SMOOTH-ROUGH
relaxing is soft STRAIGHT-
CROOKED
9 SMOOTH- boring is smooth – [25] CONTENT-
ROUGH dangerous is rough CONTAINER
STRONG-WEAK
10 LEFT-RIGHT moral is right – ISCAT: metaphor which NEAR-FAR
immoral is left clearly maps left/right STRONG-WEAK
11 LINKAGE social relationships ISCAT: most striking/easy PAINFUL
are links to understand OBJECT
12 PART-WHOLE coherent is whole ISCAT: metaphor which NEAR-FAR
contains at least the term HEAVY-LIGHT
whole
13 HEAVY-LIGHT more is heavy–less is [25] UP-DOWN
light CONTENT-
CONTAINER
14 OBJECT opportunities are ISCAT: metaphor which LINKAGE
objects only refers to object, not PAINFUL
further attributes or context
4 Correct Match
Correct match of Image Schema Icons and Image Schema Objects in total numbers and percentage.
Image Schema Icons Icons % Objects Objects %
TOTAL 630 90 571 81.57
UP-DOWN 48 96 49 98
CONTENT-CONTAINER 40 80 45 90
NEAR-FAR 44 88 39 78
CENTRE-PERIPHERY 46 92 46 92
STRONG-WEAK 31 62 15 30
PAINFUL 50 100 49 98
STRAIGHT-CROOKED 44 88 40 80
HARD-SOFT 47 94 47 94
SMOOTH-ROUGH 49 98 50 100
LEFT-RIGHT 47 94 47 94
LINKAGE 49 98 46 92
PART-WHOLE 46 92 41 82
HEAVY-LIGHT 45 90 8 16