Which factors influence the human assessment of creativity exhibited by a computational system is a core question of computational creativity (CC) research. Recently, the system's embodiment has been put forward as such a factor, but empirical studies of its effect are lacking. To this end, we propose an experimental framework which isolates the effect of embodiment on the perception of creativity from its effect on creativity per se. We manipulate not only the system's embodiment but also the human perception of creativity, which we factorise into the assessment of creativity, and the perceptual evidence that feeds into that assessment. We motivate the core framework with embodiment and perceptual evidence as independent and the creative process as a controlled variable, and we provide recommendations on measuring the assessment of creativity as a dependent variable. We propose three types of perceptual evidence with respect to the creative system, the creative process and the creative artefact, borrowing from the popular four perspectives on creativity. We hope the framework will inspire and guide others to study the human perception of embodied CC in a principled manner.
Which factors contribute to people's perception of creativity in artificial systems has been recognised as a central question in computational creativity (CC) research [1]. Insights into this question can not only support the fairer comparison of existing and the design of new systems but also inform strategies to foster people's acceptance of AI systems as creative contributors.
By "the human perception of creativity", we denote a person's subjective assessment of an AI system's creativity based on available evidence. Some of the factors identified or hypothesised to affect this perception -assuming that the creative process and product remain constant -are the human assessor's competence [2] and technological optimism [3], the system's motivational mechanism [4] and the framing of its creative process [5], including the description of "life experiences" [6]. In this paper, we focus on a factor that has only recently been hypothesised to affect the human perception of creativity [7]: the system's embodiment as -roughly speakingthe shape of its body and its means to interact with the environment [cf. 8].
A recent survey of publications [7] at the International Conference on Computational Creativity (ICCC) attests a long-lasting, steady and recently increasing interest in engineering autonomous and co-creative, embodied CC systems in various domains. Crucially though, the survey has not found a single study offering generalising, empirical insights about the effect of a system's embodiment on the perception of creativity. This is highly problematic in that such an effect could be substantial; some even hypothesise that, given a large "embodiment gap" between a system and its observer, the system's creativity might remain entirely unnoticed [4].
We pave the way towards gaining such insights by proposing a framework for conducting empirical studies on how a system's embodiment impacts the human perception of its creativity. The framework is intended for and relies on comparing this facet of human perception between two or more systems. Within it, we highlight the importance and discuss the potential values and relationship of two independent variables: the participant's perceptual evidence, and the system's embodiment. By manipulating these variables, we can study what kind of observations about an artificial system lead to the attribution of creativity, and how. We support our discussion with examples but focus on the domain of visual art.
Crucially, we target the effect of embodiment on the perception of creativity, rather than on creativity per se, which would alter the creative process and product [9]. To this end, we promote that it is essential to encapsulate the system and its perceptible qualities from the participants while keeping the creative behaviour constant between the compared systems. Isolating these effects may offer us insight into how a system's embodiment can be changed for it to appear more creative, even though the underlying creative processes remain the same.
We next introduce our experimental study framework, justify the controlled and independent variables, discuss their possible values and provide recommendations for the measurement of creativity as the dependent variable. We finish with conclusions and future work.
At the basis of our experimental framework, we factorise what is commonly denoted the "perception of creativity" [10] into (1) perceptual evidence, constraining the observable aspects of the system and its interaction, and (2) the assessment of the embodied system's creativity based on the available perceptual evidence. This perceptual evidence, in turn, is shaped by (3) the system's embodiment, and (4) the creative process which the system executes. Some aspects of this process typically remain hidden from the observer. We consequently model two independent, one dependent and one controlled variable. We manipulate the compared system's embodiment (independent variable, IV) and which aspects of the systems' behaviour the participants can perceive, i.e. their perceptual evidence (IV). We measure the participants' perceived creativity assessment (dependent variable, DV), and hold the underlying creative process, which is reflected in the perceptual evidence, constant between conditions (controlled variable, CV). Figure 1 illustrates the general composition of the experiment setting.
We distinguish types of perceptual evidence based on three key concepts: the system, the process and the artefact. These correspond to three of the "four perspectives on creativity",
The perceptual evidence type may block some observation targets
Creative Process
The general composition of the experiment setting. Depending on the perceptual evidence type designated for the participant, they may observe the artefact, the production of the artefact and/or the system producing the artefact. Moreover, the creative process may be partially hidden from the study participant regardless of the designated perceptual evidence type.
identified by Rhodes [11], Mooney [12] and potentially others [cf. 9]. The system corresponds to the creative Person [11,12] or Producer [9], e.g. a plotter or a humanoid robot. Moreover, we can consider the creative Process as the steps the Producer engages in when being creative, such as producing movements with a pen on paper. Finally, we can consider the creative Product, or artefact, as the outcome of this process, e.g. the finished drawing. We omit in our analysis the perspective of the Press, which speaks to the socio-cultural environment or context of creativity. We consider this an important view as well, but it requires a further division to provide meaningful, manipulable variables for testing, which will be subject of future work. The experimental framework imposes some restrictions on the studied systems. Namely, the creative process of the system must be kept the same while altering the embodiment or the perceptual evidence. Moreover, altering the perceptual evidence may be impossible in domains where the embodiment itself is part of the observed artefact, such as in dance.
Next, we discuss each variable and its possible values in detail, and conclude the framework description with instructions for instantiating and executing a specific experiment.
Our framework allows for comparing people's perceptions of multiple systems, each with a different embodiment. This independent variable defines the embodiments to be compared, with each value corresponding to one system and embodiment. In practice, several values could be represented by the same system through modifications to its embodiment. However, some embodiments are too different to be expressed on the same hardware and implementing very varied embodiments on different systems can enable comparisons with stronger effects.
While "embodied" is often considered synonymous with "robotic" systems, cognitive science has distinguished several more types of embodiment, which lend themselves naturally as values of our independent variable. We adopt the typology by Ziemke [8], which has been slightly extended by Guckelsberger et al. [7, highlighted in italics]: structural coupling, characterising systems that can perturbate, and, vice versa, be perturbated by their surrounding environment [13].
historical, characterising systems whose present state is the result of a history of structural couplings, developed through interactions with the environment over time [13,14].
virtual, characterising simulated systems embedded in and distinguished from a simulated environment. The virtual body can affect the environment and vice versa.
physical, characterising systems with a physical body [15,16] that can interact with the environment by being subjected to and exercising physical force. Includes most robots.
organismoid, characterising virtually or physically embodied systems with the same or similar form and sensorimotor equipment as living organisms. Approximations of a humanoid embodiment can be considered a subset.
organismic, applying to living and artificial systems capable of autopoiesis, i.e. of maintaining their internal organisation and surrounding boundary against internal and external perturbations by means of self-producing processes [17,18].
While structural coupling establishes the baseline for what can be considered an embodied system, historical embodiment practically characterises any non-theoretical system. Consequently, we suggest only considering the latter four types and their sub-types for experimentation. A specific system can instantiate several types, which can be combined as values of the independent variable. For instance, one might be interested in comparing a system with virtual+humanoid (a subset of organismoid) with a system implementing physical+humanoid embodiment. While these types provide some initial orientation, they would typically be differentiated further. For instance, robots in the shape and with similar sensorimotor equipment to an ant, a dolphin, or a bird all share the same type of physical, organismoid but non-humanoid embodiment. While such diverse comparisons are possible, experimenters might also be interested in measuring the effect of subtle changes to the same embodiment, e.g. equipping a humanoid, physically embodied system with a different hand.
The choice of values in a specific experiment will likely be informed by the overarching research question and the systems at hand. If no requirements are imposed, we argue that the most value to the community at present can be created by comparing those types of embodiment that most strongly differentiate existing CC studies: virtual vs physical embodiment, followed by humanoid vs non-humanoid, organismoid embodiment [7].
We hold that how a person observes an embodied system, yielding different kinds of perceptual evidence (PE), plays a key role in their attribution of creativity. This is why our framework factorises the perception of creativity into the creativity assessment and the perceptual evidence (PE), and manipulates the latter as independent variable. We propose three types of perceptual evidence: the finished artefact (PE1), the artefact as it is being produced (PE2), and the embodied system as it is producing the artefact (PE3). In all cases, the participant is assumed to remain in a fixed location. These types as values of the independent variable can be further differentiated, e.g. by distinguishing different observation angles, or allowing the participant to observe freely. We next discuss how each of the proposed values might affect the attribution of creativity as dependent variable, using the four perspectives on creativity framework [11,12,9].
The artefact The participant only observes the finished, produced artefact, but not the creation process or the creating, embodied system. While the artefact evidences an embodied system's historical, sensorimotor interaction with its environment, the participant can only speculate on the details of its embodiment. This can serve as the baseline to which other types of perceptual evidence are compared. Creativity: Observing only the artefact allows assessing creativity from the Product perspective. While not observing the production makes it challenging to assess creativity from the Process perspective, some hints of the Process or Producer qualities (e.g. skill, imagination) may be evidenced in the artefact, becoming available for observation.
The participant observes the creation process of the artefact, e.g. lines drawn on the screen, and the finished artefact (PE1). The embodiment of the system, however, is not explicitly revealed. This is straightforward when dealing with modifications of the same embodiment type, but comparisons between different types, e.g. virtual and physical embodiment, require more sophisticated setups. While the creative process is controlled and should thus not reveal a particular embodiment, we yet hold that observing it might allow the participant to infer a particular kind of embodiment. Including PE2 accounts for such effects. Creativity: Allowing the participant to observe the artefact as it is being produced enables (in suitable domains) assessing creativity from the Process perspective. However, the full (creative) process of the system is not observable as the embodiment is left out. Using PE2 as an intermediate step between PE1 and PE3 allows comparing, e.g. how observing the artefact creation process itself affects the assessment of the produced artefacts (comparing PE1 and PE2).
The participant observes the embodied system while it is producing the artefact. In contrast to PE2, the participant can also observe how the embodied system actually operates, e.g. drawing lines on a canvas. This allows for including all relevant embodiment types in the participant's observations, although organismic embodiment might be hard to realise within a sensible experimental time frame. Creativity: This type allows the participant to assess a system's externally perceivable Process, and observing the embodiment may give hints about the system's properties from the Producer perspective. Comparing the same system on PE2 and PE3 enables studying the effect of observing the embodiment itself on the perceived creativity of the Products and their production Process.
Depending on the goals of the experiment, a participant's perceived creativity assessment can be measured either qualitatively or quantitatively. While a comprehensive overview of creativity measurement practices is out of scope for this paper, we make the general recommendation to use standardised and CC-specific assessment practices such as SPECS [19]. Here, the study designer first specifies what is meant by creativity and then defines the criteria for how it can be assessed based on perceptual evidence. Moreover, combining observations (by the experimenter) and self-reporting of the participants has been known to produce apt results in studies assessing human experience while using creative systems [20].
If conducting an interview, close attention must be paid to the formulation of the questions in order to foster insights on a particular aspect of the experiment, e.g. the observed artefact, its creation process, or the system. Moreover, any question must explicitly distinguish between the observed evidence and unobserved variables: a participant exposed to PE1 can be asked about the embodied system's properties, but it should be clearly communicated that answering the question requires them to imagine system properties which they have not directly observed.
In order to measure the effect of embodiment on the perception of creativity rather than on creativity per se, we must keep the creative process constant between conditions. We distinguish two means to accomplish this, with individual benefits and disadvantages:
Online The system must be modified ahead of the experiment to produce equal action sequences between runs, e.g. by fixing random seeds, setting learning rates to zero, etc. This comes with the advantage that the system can still condition its actions on previous sensor states in a closed loop fashion, potentially producing a process which incorporates its present environment. However, it also risks variations between runs through e.g. different participants prompting different reactions from the system. Moreover, it is easy to overlook any necessary modifications to keep the process constant.
Offline All process-constituting actions are pre-recorded from a normal system run, and replayed for all conditions. This guarantees that the process is held constant between runs. However, it also implies that the system will stop reacting to any sensor inputs such as changes in its surroundings, including the participant. We hold that this change to open-loop control might affect the human creativity assessment unnaturally.
Even when keeping the system's internal action signals constant, maintaining the outside process as perceived by the participant the same may not always be possible. In particular, in physically embodied systems, actuator and sensor noise and/or fidelity will yield slight variations in the observable process, and, thus, variations in the resulting artefact. How sensitive the measurement of the dependent variable is to such fluctuations is an open research question.
We recommend accounting for this through repeated measurements of the same condition and ensuring that the variations only produce minuscule differences in the produced artefacts.
Lastly, we anticipate that the human assessment of creativity might be affected by many latent properties of the creative process, e.g. its length, execution speed, or complexity. To account for such biases, we can evaluate multiple creative processes with varying properties, effectively turning the creative process into an independent variable.
To instantiate an experiment with this framework for a specific research question, one must resolve the following action points:
• Select the independent variables to manipulate (system embodiment, perceptual evidence, both). If both independent variables are manipulated, we have a factorial design. • Select the values to assume for each of the independent variables. To yield more than one condition for a comparative measurement with only one independent variable, it is crucial to select at least two or more different values of perceptual evidence, or embodiments. • Select the creative process to perform and control for.
• Plan how to measure the assessment of creativity as dependent variable.
• Decide whether to expose each participant to one, all, or only some conditions, yielding a between/within/mixed experimental design, respectively. • Plan how to analyse the collected data (e.g. exploratory statistics and statistical significance testing for quantitative, thematic analysis for qualitative data).
By controlling the perceptual evidence, we can assess how strongly the subjective assessment of embodied CC depends on what data on the artefact-process-system whole has been gathered. This can also be interpreted as manipulating not the system's, but the participant's sensorimotor embodiment, which serves as additional justification for including perceptual evidence in our framework which focuses on embodiment effects. The experiment also allows studying how embodiment affects the perceived creativity by altering embodiment (e.g. virtual and physical) while keeping the process and the perceptual evidence the same. Both effects can be investigated together in a factorial design with appropriate analysis methods. The choice of experimental design (between/within/mixed) can for instance be informed by the number of conditions, e.g. to prevent participant fatigue, and the available hardware (e.g. only one robot which must be repeatedly modified to create various embodiment study conditions). Crucially, if the same participant is exposed to multiple conditions, potential order effects must be mitigated by randomising the exposition order for each participant.
The experiment should be executed for individuals, and not groups of participants, in a lab setting and without distractions. After (1) receiving an introduction to the experiment, the participant must (2) provide consent to the use of their data and (3) provide demographic information. The experimenter should also (4) rule out any risk of bias in conversation with the participant, e.g. due to existing knowledge of the system or its designer(s). The participant is then (5) exposed to perceptual evidence of a single embodied system (i.e. process and embodiment). Data is recorded during or after this exposure, depending on the selected measurement. The experiment can stop (between), commence with all remaining conditions (within) or cover a subset (mixed), depending on the choice of experimental design. The experiment can be concluded with a (6) post-study questionnaire and (7) debriefing session.
We have presented an experimental framework for studying how a computational creativity (CC) system's embodiment affects the human perception of its creativity. To this end, we have factorised the perception of creativity into a person's creativity assessment (dependent variable), and perceptual evidence (independent variable) of the system and its embodiment (independent variable) as a mediator. Moreover, we isolated the impact of embodiment on the perception of creativity from its impact on creativity per se by keeping the artefact production process (controlled variable) constant between conditions. We distinguished six types of embodiment based on existing typologies [8,7], discussed two distinct means to control the creative process, and put forward three types of perceptual evidence, corresponding to the Product, Process and Producer in the four perspectives on creativity framework [11,12,9]. Using the experimental framework, we can answer many interesting questions about the perceived creativity of an embodied system, e.g. "How does observing the artefact production process affect the creativity assessment when compared to observing only the finished artefact?", or "How does a more human-like embodiment impact the perceived creativity assessment?".
The core framework presented in this paper can be further extended. For example, we have focused on visual observations, but allowing other or multi-modal observations is feasible and worthwhile. Moreover, we could treat other types of evidence, such as framing [5] -e.g. a piece of text covering the motivation behind the artefact and other not directly observable information about the system -as a controlled or independent variable. Additionally, further aspects from the socio-cultural environment of the creative phenomenon -i.e. the Press in the four perspectives on creativity framework [9,11] -could be brought in for analysis.
Our immediate next step is to instantiate this framework and perform a first principled study on the effect of embodiment and perceptual evidence on the human perception of computational creativity. Our focus will be on comparing the effect of different virtual and physical embodiments on the assessment of creativity, and on how different kinds of perceptual evidence moderate this effect. Another goal for future work is to use the framework on an existing, established embodied CC system to allow assessing its perceived creativity either with different embodiment modifications or different perceptual evidence. Finally, we would like to test the flexibility of this framework by conducting an experiment outside the lab, e.g. in an exhibition space, in order to improve the ecological validity of our findings. We consider this framework an important step toward better embodied CC research [7], enabling empirical studies to inform the design and comparison of CC systems that interact with people.
SL and AK were funded by the Academy of Finland (project #328729), and CG at an early stage by the Academy of Finland programme "Finnish Center for Artificial Intelligence" (FCAI).