Towards Cognitive Social Machines for Bridging the Cognitive-Computational Gap in Creativity and Creative Reasoning

Abstract

This position paper presents a view on bridging the cognitivecomputational gap in the field of creativity, creative reasoning and problem solving. Starting from the levels of Cognition and Computation, their potential bootstrap is discussed in relation to the concept of Cognitive Social Machines. Five distinct aspects of bridging the cognitivecomputational gap in creative reasoning using cognitive social machines are described and discussed in the creativity domain. These aspects refer to (i) building systems and models which solve creative reasoning tasks; (ii) computationally generated tools for a deeper understanding of cognition; (iii) cognitively-inspired processes and knowledge representation types; (iv) computational cognitive assistance, support and training and (v) evaluative informativity metrics for cognitive social machines.

-

In an experiment room, a human participant is given an image to look at, like the one in Figure 1a, and asked what they can see. Presuming that she sees a snail, she is asked whether she can also see an elephant (or the other way around). Presuming she can see both, she is asked to try to switch between seeing one or the other, while pressing a key for each successful switch she manages. The response times will be measured to see how often she can do this switch. This ability to re-encode features will be considered as a potential correlate for her creativity levels.

Another participant may be prompted to focus on the image in Figure 1b (stimulus provided by [ 46 ]). He will then be asked what this image represents. He might provide answers like: a boardroom meeting, around a triangular table, viewed from above; a pendant; three bottles of wine arranged around a triangle of cheese on a shelf, etc. The number of his answers, the semantic domains they span, their novelty as rated by other human participants or their originality in comparison to answers from other participants, will be rated to provide a creativity score for his answers.

Meanwhile, a computational system may analyse newspaper articles or online news to construct their mood for the day. Subsequently, it may determine what article to base a poem on, and what template to use for this poem. After writing a poem, the system might computationally generate a framing for this poem, like the following [ 7 ]:

It was generally a bad news day. I read an article in the Guardian entitled: “Police investigate alleged race hate crime in Rochdale”. Apparently, “StringerPrince, 17, has undergone surgery following the attack on Saturday in which his skull, eye sockets and cheekbone were fractured” and “This was a completely unprovoked and relentless attack that has left both victims shoked by their ordeal”. I decided to focus on mood and lyricism, with an emphasis on syllables and matching line lengths, with very occasional rhyming. I like how words like attack and snake sound together. I wrote this poem.

Framing, thus the ability of the system to provide a commentary on the whole process, would be evaluated as part of its creativity too, besides the poem.

These example reflect how the study of creativity is approached from the cognitive versus computational realm. Both are supposed to measure creativity, however, the agendas of the fields seem quite di↵ erent. The di↵ erences do not involve just tasks, but also of goals and methods. The fields have di↵ erent identities, communities, and provide di↵ erent interesting answers to di↵ erent research question contexts.

Given these di↵ erences, can a cognitive-computational bridge can be built? Could such a bridge serve both the cognitive and computational communities? This paper explores how cognitive social machines can be used for bridging the cognitive-computational gap in the creativity domain.

The rest of this paper is organized as follows. The background of and di↵ erences between creative cognition and computational creativity are briefly summarized, and a direction for bridging the gap is proposed in section 2. Sections 3 to 7 elaborate on this direction, by exploring five di↵ erent aspects of a cognitivecomputational bootstrap using cognitive social machines. A summary discussion of the approach’s key points is provided in section 8.

Background

Creativity, creative reasoning and creative problem solving are fields studied across cognitive and computational disciplines with fairly di↵ erent goals and methods.

Human creativity is studied in cognitive psychology, with the purpose of understanding the human creative process. Various creativity tests [ 26, 17, 20, 46, 25, 10 ] are deployed in evaluating creative performance, and for studying hypotheses of how various conditions impact creativity and creative problem solving.

The computational creativity community studies the question what does it take for a machine to be creative. It builds computationally creative systems in a wide variety of fields, including mathematics [ 24, 6, 4 ], music [ 39, 38, 44, 11 ], art [ 3, 5 ], poetry and text composition [ 2, 15, 16, 7 ], architecture and design [ 42 ], discovery of physical laws [ 21–23 ], magic trick making [ 48 ]; and video games [ 9 ]. Computational creativity also devises ways of evaluating computational creativity [ 47, 41, 40, 8, 45, 19 ]

In the middle ground of the creative cognition and computational creativity fields, a few research projects (i) computationally study processes that are also presupposed to play a role in the human cognition literature or (ii) computationally implement cognitive creativity theories. Some examples of such projects have produced work on concept blending [ 13 ], analogy [ 14, 12 ], re-interpretation and re-representation [ 27 ].

However, the authors believe that a stronger bootstrap between the cognitive and computational sides of the creativity coin is possible if comparability of cognitive and computing approaches is allowed for, and if the bootstrap is situated in the domain of cognitive social machines. While previous work has aimed at providing an initial point of reference for comparability [ 33 ], this work focuses on situating such a bootstrap in the context of cognitive social machines. For this, a working definition of cognitive social machines would be useful.

Social machines are defined as an environment comprising humans and technology interacting and producing outputs or action which would not be possible without both parties present.1. One of the primary characteristics of social machines is that, having both human and computational participants, the line between computational process and human process becomes blurred [ 43 ]. While social machines are generally imagined around the web [ 18 ], this is just a tool that makes the blending of human and computational work more likely.

Cognitive systems, on the other hand, are considered to be systems which take inspiration, simulate or aim to replicate cognitive process, type of knowledge and performance. Depending on one’s definition, these can range from cognitive computational models which aim to predict and replicate human performace, to systems inspired by a cognitive metaphor. Sometimes the term cognitive systems is also used in a way which is similar to the concept of human-computer interaction (HCI), to describe the fact that particular systems interact well with 1 From https://en.wikipedia.org/wiki/Social_machine - retrieved 17.06.2017. their user, taking their user’s cognitive limitations into account, or taking into account other cognitive phenomena – like user attention span.

We define cognitive social machines as sets of agents (humans and cognitive systems), their processes (artificially or naturally cognitive) and data, which act productively, informing and enabling eachother’s progress.

We believe cognitive and computational input and process can be bootstrapped in cognitive social machines in such a way that both cognitive and computational fields gain and advance from it. The main research question of this paper is thus:

How can we bootstrap Cognition and Computation to yield Cognitive Social Machines that are: – (a) greater than the sum of their parts – (b) which help improve both Cognition and Computation?

In this paper, this question will be deployed on the specific domain of creativity and creative problem solving. This research can be perceived as operating on three levels: – Cognitive level (Cog) - human reasoning and process as examined via cognitive science tools – Computational level (Comp) - artificial creative cognitive systems – Coupling (1 ) - cognitive social machines The premise of what follows is that, if ways to reliably boost the computational level via the cognitive level (Cog ! Comp) and the cognitive level via the computational level (Cog Comp) can be found, then a successful (Cog 1 Comp) coupling level can be achieved.

In the following, five aspects of cognition, computation, and their cognitive social machines coupling are described, in the context of the domain of creativity and creative problem solving. These five aspects are: - SYSTEMS – Systems and models which solve reasoning and creativity tasks (section 3); - TOOLS – Computationally generated tools for cognitive science (section 4); - PROCESSES & KR – Cognitively-inspired processes and knowledge representation types (section 5); - ASSISTANCE – Computational cognitive assistance, support and training (section 6) and - METRICS – Evaluative informativity metrics of social machines (section 7). 3

Systems and models which solve reasoning and creativity tasks (SYSTEMS)

The first aspect refers to computational work done to enable models and prototype systems which are capable of creative problem solving feats, similar and comparable to humans. The process of realizing this aspect starts by choosing a creative reasoning ability, then finding a creativity test which evaluates this ability in humans. If such a test does not exist, a cognitive form of evaluation can be built. The next steps are to search for a source of cognitive knowledge acquisition, understand the types of cognitive process involved in the ability (or have a good cognitively-inspired process hypothesis) and to attempt to implement this ability in a computational solver. Afterwards, comparative evaluation can be performed between the human and the computational solvers. This process is shown in Figure 2.

Besides providing systems which are capable of solving tasks similar to the tasks humans can solve, this aspect provides other benefits. Systems which are implemented using cognitive processes, and which can be evaluated using comparable tasks can (i) later be used by cognitive psychologists as tools to understand and base more refined cognitive models on and (ii) can shed light on possibilities of cognitive process which remain ambiguous while only theorized about, without implementation.

For example, the Remote Associates Test [ 26 ] is a test used to measure creativity as a function the cognitive ability for association. The format of the test is that three words are given to a participant, like Dew, Comb and Bee. The participant is asked to come up with an answer that relates to all these three words – a possible answer in this case is Honey. In an attempt to cognitively solve this test computationally, [ 31 ] applied the principles of a cognitive framework of creative problem solving [ 36, 28 ] and built a system (comRAT-C) which solves the Remote Associates Test via a cognitively inspired process of association, divergence, and convergence on association overlap. Not only was the computational solver correct in a great proportion of cases, but it brought to light the issue that sometimes multiple answers might be plausible, something which was not examined in human normative data, where only one correct answer is given. The system also correlated with human data: the harder the query for humans, the lower the probability metric provided by the system. Thus the system can be used as a tool by cognitive psychologists to build more refined models starting from the initial coarse mechanism, which is now known to solve the task in a way that correlates with human performance.

Cognitive Social Machine The cognitive social machine at the SYSTEMS level can be described as follows. Cognitive science and the human part of the social machine provide the cognitive knowledge, cognitive process and cognitive evaluation. These are used to construct and assess computational systems (Cog ! Comp). The computational systems in turn become models and tools, which can then be used to better understand cognitive functioning (Comp ! Cog).

This approach can be generalized for a variety of tasks, as shown in Table 1. For example, in the Wallach Kogan similarity test, human participants are asked to provide ways in which various concepts, like fruits or objects are alike. A system generating ad-hoc similarities on an equivalent dataset of objects as the ones given to humans could be implemented and used for comparability. Various types of object similarity algorithms exist which could be cognitively evaluated. More such algorithms could be created with inspiration from the cognitive processes. Systems which implement them could further be used by cognitive modelers, and in tasks in which cognitive similarity between computational-human partners is important (as will be shown in section 6). 4

Computationally generated tools for cognitive science (TOOLS)

The second aspect refers to using cognitive and computational principles and variants from the existing systems in the previous section, together with cognitive data, in order to construct creativity and creative reasoning task generators. Systems which allow for the generation of large datasets of creativity and creative reasoning queries can be used to control for various parameters of such queries. Sets of queries can then be designed to investigate specific empirical questions, at a depth which is impossible without computational intervention in crafting the stimuli. This process is shown in Figure 3. Test

Example task

System/ability Remote Associates Test

COTTAGE SWISS CAKE comRAT – RAT solver [ 31 ] The Alternative Uses test What can you use a brick for? Similarity test Wallach Kogan

Tell me all the ways in which an apple and an orange are alike

Creative object replacement system [ 32 ] Generating ad-hoc similarities Ambiguous figures Feature grouping system Pattern Meanings Test Wallach Kogan Multiple memory search based on features Insight tests Practical object problem solver

For example, here is an application of this aspect involving the Remote Associates Test (RAT). Cognitive psychologists administering this test do not normally have control over variables like the frequency of each of the query words, the frequency of the answer, the probability (based of frequency) that a particular answer would be found. There is a small normative dataset of compound RAT queries often used in the literature, comprising 144 items [ 1 ]. The principles of knowledge organization from the comRAT-C solver [ 31 ] were reverse engineered to create a RAT-query generator (comRAT-G). With this generator, a set of test items that spans the entirety of English language nouns was created [ 35 ]. comRAT-G provided 17 million items which can be used by cognitive psychologists in their work to understand the human creative process, by controlling frequency and probability variables of the query words and the answer words. This can allow for more complex experimental designs. From both a computational and cognitive perspective, this system opens the door to the exploration of interesting questions, like for example what is a good Remote Associates Test query, which requires creativity to answer?

Cognitive Social Machine

The cognitive social machine at the TOOLS level can be described as follows. Humans help evaluate the quality of the computationally created test items (Cog ! Comp). The computationally created items are then used to deeper understand the human creative reasoning process (Comp ! Cog).

Various tasks could be generated based on various types of cognitive data, a few examples of which are shown in 2. For example, cognitive data on words associates (rather than compound words) can be used to generate the functional version of the Remote Associates Test. Data on cognitive visual similarity can be used to generate some of the Wallach Kogan visual tests, etc. Some initial work has been done in this direction [ 34, 37, 30 ].

Cognitively-inspired processes and knowledge representation types (PROCESSES & KR)

The third aspect – PROCESSES & Knowledge Representation – refers to (i) learning from cognitive processes to get inspiration for computational processes and (ii) learning from cognitive knowledge representation to support new types of computational knowledge representation. The aim of this, as shown in Figure 4, is to obtain cognitively friendly and innovative types of processes and knowledge representation.

Cognitive Social Machine

The cognitive social machine at the level of PROCESSES & KR can be described as follows. Humans provide inspiration of new processes and new types of knowledge organization (Cog ! Comp). Computational systems using cognitive processes and knowledge representation may be capable of new tasks, of tackling old tasks in new ways and may be more cognitively friendly (Comp ! Cog).

This aspect of the bootstrapping can lead to innovations of process and knowledge representation, and to cognitive adaptations of already existing processes and types of knowledge representation, as shown in Table 3. For example, the cognitive process of restructuring and re-representation can inspire new types of multiple pattern matching by framing sets of initial features from multiple representation perspectives.

Systems which process, encode knowledge and communicate information in cognitively inspired manner might be perceived as more cognitively friendly, and thus be able to o↵ er better support and assistance to natural cognitive agents. 6

Computational cognitive assistance, support and training (ASSISTANCE)

Aspect four refers to using systems capable of tackling similar tasks as humans, and using cognitive or cognitively inspired processed, types of knowledge organization in order to provide computational cognitive assistance. Systems under this descriptions can be roughly split into supportive systems (S) and training systems (T). Supportive systems would aim to assist their user in performing creative and creative reasoning tasks, as a partner, co-creator, co-reasoner or “muse”. Training systems would aim to help maintain or enhance creative and creative reasoning abilities of their human partner. Input from the partner will also be used to improve the assistive system.

Assistive systems could for example propose ideas for cooking new recipes, reusing objects, furniture and room redesign, as shown in Table 4. However, they could also be used in a deeper form of cognitive support, to provide the kind of information which would lead to productive association of ideas and restructuring. Training systems could be used to target and improve precise creativity related skills, like the ability to exit functional fixedness. (S) Providing the right information for – promotes productive association of ideas CR in cognitive tasks and search – leads to restructuring (S) Recycling - recommenders and crowdsourced human data – object recycling in households – support for recycling of wind turbines

Evaluative informativity metrics of cognitive social machines (METRICS)

The metrics level aims to evaluate the successful functioning of the cognitive social machines set in place, and to optimize the distribution of processes, and information flow. This level thus deals with questions as the following: – Which part of processing and knowledge representation should each of the cognitive and computational partners do? – What are the information gains of various processing and knowledge representation set-ups? (of various types of organizining and distributing the parts of the machine) – How does working together increase the generativity of both natural and artificial cognitive systems? – What measures protect working together? Information coherence? Information structure?

While the first two questions pertain to organizing and optimizing the machine, the third question focuses on generativity as an evaluative metric – thus the increase in productive capacity of both natural and artificial systems. This productive capacity can be seen both as the ability to solve more problems, and the ability to come up with new solutions – it is thus a creativity and creative reasoning type of metric. Other cognitive metrics could also be devised to assess various cognitive social machine set-ups.

The fourth question addresses measures which protect working together from the perspective of cognitive adaptation of computational and cognitive parts to each other. Thus systems which possess non-contradictory information, knowledge organized in similar ways or similar ways of structuring information [ 29 ] might be more productive than other systems, through being better adapted to working together. Complementarity of such measures could also be more formally defined, in ways in which it is usually defined for social interactions between natural agents. While this has some connections to the fields of HCI and adaptive robotics. It focuses on cognitive informational measures which have as e↵ ect the protection of cognitive resources and the protection of cognitive social work done in partnership. 8

Conclusion and Future Work

A coherent view of bridging the cognitive-computational gap in the domain of creativity and creative problem solving was proposed, using cognitive social machines. Five aspects where described to showcase the possible uses of this view.

In the SYSTEMS aspect, the human part of the machine provides cognitive knowledge, processes and access to cognitive evaluation. These are used by the computational part to build systems which can perform tasks and have abilities which are similar or comparable to those of humans. This allows for comparative evaluation between the cognitive and computational counterparts. The systems can also be used as tools for futher cognitive models.

In the TOOLS aspect, the computational systems previously constructed are used to generate ample creativity and creative reasoning tasks. Such tasks are evaluated by humans in comparison to classical datasets of those tasks, or via other types of qualitative and quantitative assessment. The validated tasks can be used to control for variables and allow for more complex empirical designs, and thus provide more precise tools to explore human cognition with. Methodological and computational questions about what does it constitute a good creativity or creative reasoning task can lead to developments in the theoretical and philosophical foundations of the concept of creativity.

In the PROCESSES & KR aspect, cognitive processes and cognitive types of knowledge representation are used to inform the computational process and the computational types of knowledge representation. This cognitive inspiration can lead to new types of processes and knowledge representation, but also to more cognitively friendly systems, which process and communicate information in a way that is more similar to their users.

In the ASSISTANCE aspect, all the previous work is used to support and train human creativity, creative reasoning and creative problem solving. To close the loop, the assistive systems learns from the human creative activity, from feedback or from human performance.

The METRICS aspect is used to optimize, evaluate and protect cognitive social machine set-ups.

As future work, the authors intend to provide a more formal description of (i) information flow, (ii) process and KR replication and inspiration and (iii) cognitive social machine metrics. A set of case studies will also be observed in depth through the lens of this approach, in all its five aspects.

Acknowledgements

Ana-Maria Olte¸teanu gratefully acknowledges the support of the German Research Foundation (Deutsche Forschungsgemeinschaft - DFG) for the Creative Cognitive Systems2 (CreaCogs) project OL 518/1-1. 2 http://creacogcomp.com/

References

1. Bowden , E.M. , Jung-Beeman , M. : Normative data for 144 compound remote associate problems . Behavior Research Methods, Instruments, & Computers 35 ( 4 ), 634 - 639 ( 2003 )

2. Carpenter , J.: Electronic text composition project . The Slought Foundation ( 2004 )

3. Cohen , H. : The further exploits of AARON, painter . Stanford Humanities Review 4 ( 2 ), 141 - 158 ( 1995 )

4. Colton , S. : Automated theory formation in pure mathematics . Springer Science & Business Media ( 2012 )

5. Colton , S.: The painting fool: Stories from building an automated painter . In: Computers and creativity , pp. 3 - 38 . Springer ( 2012 )

6. Colton , S. , Bundy , A. , Walsh , T. : On the notion of interestingness in automated mathematical discovery . International Journal of Human-Computer Studies 53 ( 3 ), 351 - 375 ( 2000 )

7. Colton , S. , Goodwin , J. , Veale , T. : Full-FACE poetry generation . In: Proceedings of the Third International Conference on Computational Creativity . pp. 95 - 102 ( 2012 )

8. Colton , S. , Pease , A. , Charnley , J.: Computational creativity theory: The FACE and IDEA descriptive models . In: Proceedings of the Second International Conference on Computational Creativity . pp. 90 - 95 ( 2011 )

9. Cook , M. , Colton , S. : Ludus ex machina: Building a 3D game designer that competes alongside humans . In: Proceedings of the 5th International Conference on Computational Creativity ( 2014 )

10. Duncker , K. : On problem solving . Psychological Monographs 58 ( 5 , Whole No . 270 ) ( 1945 )

11. Eppe , M. , Confalonieri , R. , Maclean , E. , Kaliakatsos , M. , Cambouropoulos , E. , Schorlemmer , M. , Ku¨hnberger, K.U.: Computational invention of cadences and chord progressions by conceptual chord-blending ( 2015 )

12. Falkenhainer , B. , Forbus , K.D. , Gentner , D.: The structure-mapping engine: Algorithm and examples . Artificial intelligence 41(1) , 1 - 63 ( 1989 )

13. Fauconnier , G. , Turner , M. : Conceptual integration networks . Cognitive science 22 ( 2 ), 133 - 187 ( 1998 )

14. Gentner , D. : Structure-mapping: A theoretical framework for analogy . Cognitive science 7 ( 2 ), 155 - 170 ( 1983 )

15. Gerv´as, P.: Engineering linguistic creativity: Bird flight and jet planes . In: Proceedings of the NAACL HLT 2010 Second Workshop on Computational Approaches to Linguistic Creativity. pp. 23 - 30 . Association for Computational Linguistics ( 2010 )

16. Greene , E. , Bodrumlu , T. , Knight , K. : Automatic analysis of rhythmic poetry with applications to generation and translation . In: Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing . pp. 524 - 533 . Association for Computational Linguistics ( 2010 )

17. Guilford , J.P.: The nature of human intelligence . McGraw-Hill ( 1967 )

18. Hendler , J. , Berners-Lee , T. : From the semantic web to social machines: A research challenge for ai on the world wide web . Artificial Intelligence 174 ( 2 ), 156 - 161 ( 2010 )

19. Jordanous , A. : A standardised procedure for evaluating creative systems: Computational creativity evaluation based on what it is to be creative . Cognitive Computation 4 ( 3 ), 246 - 279 ( 2012 )

20. Kim , K.H. : Can we trust creativity tests? A review of the Torrance Tests of Creative Thinking (TTCT) . Creativity Research Journal 18 ( 1 ), 3 - 14 ( 2006 )

21. Langley , P. : Bacon. 1: A general discovery system . In: Proc. 2nd Biennial Conf. of the Canadian Society for Computational Studies of Intelligence . pp. 173 - 180 ( 1978 )

22. Langley , P. : Data-driven discovery of physical laws . Cognitive Science 5 ( 1 ), 31 - 54 ( 1981 )

23. Langley , P. , Bradshaw , G.L. , Simon , H.A. : Bacon . 5: The discovery of conservation laws . In: IJCAI . vol. 81 , pp. 121 - 126 ( 1981 )

24. Lenat , D.B.: AM : An artificial intelligence approach to discovery in mathematics as heuristic search . Tech. rep., DTIC Document ( 1976 )

25. Maier , N.R. : Reasoning in humans. II. The solution of a problem and its appearance in consciousness . Journal of Comparative Psychology 12 ( 2 ), 181 ( 1931 )

26. Mednick , S.A. , Mednick , M. : Remote associates test: Examiner's manual . Houghton Mi✏ in ( 1971 )

27. O 'Hara , S. , Indurkhya , B. : Incorporating (re) -interpretation in case-based reasoning . Topics in case-based reasoning pp. 246 - 260 ( 1994 )

28. Olte¸teanu, A.M.: Publications of the Institute of Cognitive Science , vol. 01 - 2014 , chap. Two general classes in creative problem-solving? An account based on the cognitive processes involved in the problem structure - representation structure relationship . Institute of Cognitive Science , Osnabru¨ck ( 2014 )

29. Olte¸teanu, A.M.: Publications of the Institute of Cognitive Science , vol. 02 - 2015 , chap. The Input, Coherence, Generativity (ICG) Factors . Towards a Model of Cognitive Informativity Measures for Productive Cognitive Systems . Institute of Cognitive Science, Osnabru¨ck ( 2015 )

30. Olte¸teanu, A.M. : In: Proceedings of the Workshop on Computational Creativity , Concept Invention, and General Intelligence (C3GI2016) . vol. 1767 . CEUR-Ws , Osnabru¨ck ( 2016 )

31. Olte¸teanu, A.M. , Falomir , Z. : comRAT-C: A computational compound remote associate test solver based on language data and its comparison to human performance . Pattern Recognition Letters 67 , 81 - 90 ( 2015 )

32. Olte¸teanu, A.M. , Falomir , Z. : Object replacement and object composition in a creative cognitive system. towards a computational solver of the Alternative Uses Test . Cognitive Systems Research 39 , 15 - 32 ( 2016 )

33. Olte¸teanu, A.M. , Falomir , Z. , Freksa , C. : Artificial cognitive systems that can answer human creativity tests: An approach and two case studies . IEEE Transactions on Cognitive And Developmental Systems pp. 1 - 7 ( 2016 )

34. Olte¸teanu, A.M. , Gautam , B. , Falomir , Z. : Towards a visual remote associates test and its computational solver . In: Proceedings of the Third International Workshop on Artificial Intelligence and Cognition 2015 . vol. 1510 , pp. 19 - 28 . CEUR-Ws ( 2015 )

35. Olte¸teanu, A.M. , Schultheis , H. , Dyer , J.B. : Constructing a repository of compound Remote Associates Test items in American English with comRAT-G . Behavior Research Methods, Instruments, & Computers (accepted)

36. Olte ¸teanu, A.M.: From simple machines to Eureka in four not-so-easy steps.Towards creative visuospatial intelligence . In: Mu¨ller, V. (ed.) Fundamental Issues of Artificial Intelligence, Synthese Library , vol. 376 , pp. 159 - 180 . Springer ( 2016 )

37. Olte¸teanu, A.M. : Towards using cognitive word associates to create functional remote associates test problems . In: Signal-Image Technology & Internet-Based Systems (SITIS), 2016 12th International Conference on. pp. 612 - 617 . IEEE ( 2016 )

38. Pachet , F. : Musical virtuosity and creativity . In: Computers and Creativity , pp. 115 - 146 . Springer ( 2012 )

39. Pearce , M. , Wiggins , G.: Improved methods for statistical modelling of monophonic music . Journal of New Music Research 33 ( 4 ), 367 - 385 ( 2004 )

40. Pease , A. , Winterstein , D. , Colton , S.: Evaluating machine creativity . In: Workshop on Creative Systems, 4th International Conference on Case Based Reasoning . pp. 129 - 137 ( 2001 )

41. Ritchie , G. : Some empirical criteria for attributing creativity to a computer program . Minds and Machines 17 ( 1 ), 67 - 99 ( 2007 )

42. Schneider , S. , Fischer , J.R. , Ko¨nig, R.: Rethinking automated layout design: developing a creative evolutionary design method for the layout problems in architecture and urban design . In: Design Computing and Cognition'10 , pp. 367 - 386 . Springer ( 2011 )

43. Shadbolt , N.R. , Smith , D.A. , Simperl , E., Van Kleek , M. , Yang , Y. , Hall , W. : Towards a classification framework for social machines . In: Proceedings of the 22nd International Conference on World Wide Web . pp. 905 - 912 . ACM ( 2013 )

44. Smith , B.D. , Garnett , G.E. : Reinforcement learning and the creative, automated music improviser . In: Evolutionary and Biologically Inspired Music, Sound, Art and Design , pp. 223 - 234 . Springer ( 2012 )

45. Ventura , D. : Mere generation: Essential barometer or dated concept? In: Proceedings of the Seventh International Conference on Computational Creativity ( 2016 )

46. Wallach , M.A. , Kogan , N.: Modes of thinking in young children: A study of the creativity-intelligence distinction . Holt, Rinehart & Winston ( 1965 )

47. Wiggins , G.A. : Towards a more precise characterisation of creativity in AI . In: Case-based reasoning: Papers from the workshop programme at ICCBR . vol. 1 , pp. 113 - 120 ( 2001 )

48. Williams , H. , McOwan , P.W. : Manufacturing magic and computational creativity . Frontiers in Psychology 7 ( 2016 )