The framework of a cognitive architecture for music perception is presented. The architecture extends and completes a similar architecture for computer vision developed during the years. The extended architecture takes into account many relationships between vision and music perception. The focus of the architecture resides in the intermediate area between the subsymbolic and the linguistic areas, based on conceptual spaces. A conceptual space for the perception of notes and chords is discussed along with its generalization for the perception of music phrases. A focus of attention mechanism scanning the conceptual space is also outlined. The focus of attention is driven by suitable linguistic and associative expectations on notes, chords and music phrases. Some problems and future works of the proposed approach are also outlined.
Garderfors [
The correspondences between vision and music perception have been
discussed in details by Tanguiane [
Several cognitive models of music cognition have been proposed in the
literature based on di erent symbolic or subsymbolic approaches, see Pearce and
Wiggins [
Here, we sketch a cognitive architecture for music perception that extends
and completes an architecture for computer vision developed during the years.
The proposed cognitive architecture integrates the symbolic and the sub
symbolic approaches and it has been employed for static scenes analysis [
In analogy with Tanguiane, we distinguish between \static" perception related with the perception of chords in analogy with perception of static scenes, and \dynamic" perception related with the perception of musical phrases, in analogy with perception of dynamic scenes.
The considered cognitive architecture for music perception is organized in
three computational areas - a term which is reminiscent of the cortical areas
in the brain - that follows the Gardenfors theory of conceptual spaces [
In the following, Section 2 outlines the cognitive architecture for music perception, while Section 3 describes the adopted music conceptual space for the perception of tones. Section 4 presents the linguistic area of the cognitive architecture and Section 5 presents the related operations of the focus of attention. Section 6 outlines the generalization of the conceptual space for tones perception to the case of perception of music phrases, and nally Section 7 discusses some problems of the proposed approach and future works. 2
The proposed cognitive architecture for music perception is sketched in Figure 1. The areas of the architecture are concurrent computational components working together on di erent commitments. There is no privileged direction in the ow of information among them: some computations are strictly bottom-up, with data owing from the subconceptual up to the linguistic through the conceptual area; other computations combine top-down with bottom-up processing.
Conceptual
Area
Linguistic
Area Subconceptual
Area
The subconceptual area of the proposed architecture is concerned with the processing of data directly coming from the sensors. Here, information is not yet organized in terms of conceptual structures and categories. In the linguistic area, representation and processing are based on a logic-oriented formalism.
The conceptual area is an intermediate level of representation between the subconceptual and the linguistic areas and based on conceptual spaces. Here, data is organized in conceptual structures, that are still independent of linguistic description. The symbolic formalism of the linguistic area is then interpreted on aggregation of these structures.
It is to be remarked that the proposed architecture cannot be considered as a model of human perception. No hypotheses concerning its cognitive adequacy from a psychological point of view have been made. However, various cognitive results have been taken as sources of inspiration. 3
The conceptual area, as previously stated, is the area between the subconceptual and the linguistic area, and it is based on conceptual spaces. We adopt the term knoxel (in analogy with the term pixel ) to denote a point in a conceptual space CS. The choice of this term stresses the fact that a point in CS is the knowledge primitive element at the considered level of analysis.
The conceptual space acts as a workspace in which low-level and high-level processes access and exchange information respectively from bottom to top and from top to bottom. However, the conceptual space has a precise geometric structure of metric space and also the operations in CS are geometric ones: this structure allows us to describe the functionalities of the cognitive architecture in terms of the language of geometry.
In particular, inspired by many empirical investigations on the perception of
tones (see Oxenham [
It should be noticed that the partials of a tone are related both with the
pitch and the timbre of the perceived note. Roughly, the fundamental frequency
is related with the pitch, while the amplitude of the remaining partials are also
related with the timbre of the note. By an analogy with the case of static scenes
analysis, a knoxel changes its position in CS when a perceived 3D primitive
changes its position in space or its shape [
A chord is a set of two or more tones perceived at the same time. The chord is treated as a complex object, in analogy with static scenes analysis where a complex object is an object made up by two or more 3D primitives. A chord is then represented in music CS as the set of the knoxels [ka; kb; : : : ] related with the constituent tones. It should be noticed that the tones of a chord may di er not only in pitch, but also in timbre. Figure 2 is an evocative representation of a chord in the music CS made up by knoxel ka corresponding to tone C and knoxel kb corresponding to the tone G.
In the case of perception of complex objects in vision, their mutual positions and shapes are important in order to describe the perceived object: e.g., in the case of an hammer, the mutual positions and the mutual shapes of the handle and the head are obviously important to classify the composite object as an hammer. In the same way, the mutual relationships between the pitches (and the timbres) of the perceived tones are important in order to describe the perceived chord. Therefore, spatial relationships in static scenes analysis are in some sense analogous to sounds relationships in music CS.
It is to be noticed that this approach allows us to represent a chord as a set of knoxels in music CS. In this way, the cardinality of the conceptual space does not change with the number of tones forming the chord. In facts, all the tones of the chord are perceived at the same time but they are represented as di erent points in the same music CS; that is, the music CS is a sort of snapshot of the set of the perceived tones of the chord.
In the case of a temporal progression of chords, a scattering occur in the music CS: some knoxels which are related with the same tones between chords will remain in the same position, while other knoxels will change their position in CS, see Figure 3 for an evocative representation of scattering in the music CS. In the gure, the knoxels ka, corresponding to C, and kb, corresponding to E,
Ax(3)
Ay(0) Ay(1) Ay(2) Ay(3) change their position in the new chord: they becomes A and D, while knoxel kc, corresponding to G, maintains its position. The relationships between mutual positions in music CS could then be employed to analyze the chords progression and the relationships between subsequent chords.
A problem may arise at this point. In facts, in order to analyze the
progression of chords, the system should be able to nd the correct correspondences
between subsequent knoxels: i.e., k0a should correspond to ka and not to, e.g.,
kb. This is a problem similar to the correspondence problem in stereo and in
visual motion analysis: a vision system analyzing subsequent frames of a moving
object should be able to nd the correct corresponding object tokens among the
motion frames; see the seminal book by Ullman [
The described representation is well suited for the recognition of chords: for
example we may adopt the algorithms proposed by Tanguiane [
The proposed representation is also suitable for the analysis of the e ciency
in voice leading, as described by Tymoczko [
Ax(3)
Ay(0) Ay(1) Ay(2) Ay(3) i.e., a one note circular space, a two note space, a three note space, and so on. Instead, the cardinality of the considered conceptual space does not change, as previously remarked. 4
In the linguistic area, the representation of perceived tones is based on a high
level, logic oriented formalism. The linguistic area acts as a sort of long term
memory, in the sense that it is a semantic network of symbols and their
relationships related with musical perceptions. The linguistic area also performs
inferences of symbolic nature. In preliminary experiments, we adopted a
linguistic area based on a hybrid KB in the KL-ONE tradition [
In the domain of perception of tones, the terminological component contains the description of relevant concepts such as chords, tonic, dominant and so on. The assertional component stores the assertions describing speci c situations. Figure 4 shows a fragment of the terminological knowledge base along with its mapping into the corresponding entities in the conceptual space. has-dominant
has-tonic Dominant
Tonic
A generic Chord is described as composed of at least two knoxels. A SimpleChord is a chord composed by two knoxels; a Complex-Chord is a chord composed of more than two knoxels. In the considered case, the concept Chord has two roles: a role has-dominant, and a role has-tonic both lled with speci c tones.
In general, we assume that the description of the concepts in the symbolic KB is not exhaustive. We symbolically represent the information necessary to make suitable inferences.
The assertional component contains facts expressed as assertions in a predicative language, in which the concepts of the terminological components correspond to one argument predicates, and the roles (e.g., part of) correspond to two argument relations. For example, the following predicates describe that the instance f7#1 of the F7 chord has a dominant which is the constant ka corresponding to a knoxel ka and a tonic which is the constant k#b corresponding to a knoxel kb of the current CS: ChordF7(f7#1) has-dominant(f7#1,ka) has-tonic(f7#1,kb)
By means of the mapping between symbolic KB and conceptual spaces, the linguistic area assigns names (symbols) to perceived entities, describing their structure with a logical-structural language. As a result, all the symbols in the linguistic area nd their meaning in the conceptual space which is inside the system itself.
A deeper account of these aspects can be found in Chella et at. [
A cognitive architecture with bounded resources cannot carry out a one-shot, exhaustive, and uniform analysis of the perceived data within reasonable resource constraints. Some of the perceived data (and of the relations among them) are more relevant than others, and it should be a waste of time and of computational resources to detect true but useless details.
In order to avoid the waste of computational resources, the association between symbolic representations and con gurations of knoxels in CS is driven by a sequential scanning mechanism that acts as some sort of internal focus of attention, and inspired by the attentive processes in human perception.
In the considered cognitive architecture for music perception, the perception model is based on a focus of attention that selects the relevant aspects of a sound by sequentially scanning the corresponding knoxels in the conceptual space. It is crucial in determining which assertions must be added to the linguistic knowledge base: not all true (and possibly useless) assertions are generated, but only those that are judged to be relevant on the basis of the attentive process.
The recognition of a certain component of a perceived con guration of knoxels in music CS will elicit the expectation of other possible components of the same chord in the perceived conceptual space con guration. In this case, the mechanism seeks for the corresponding knoxels in the current CS con guration. We call this type of expectation synchronic because it refers to a single con guration in CS.
The recognition of a certain con guration in CS could also elicit the
expectation of a scattering in the arrangement of the knoxels in CS; i.e., the
mechanism generates the expectations for another set of knoxels in a subsequent CS
con guration. We call this expectation diachronic, in the sense that it involves
subsequent con gurations of CS. Diachronic expectations can be related with
progression of chords. For example, in the case of jazz music, when the system
recognized the Cmajor key (see Rowe [
Actually, we take into account two main sources of expectations. On the one
side, expectations could be generated on the basis of the structural
information stored in the symbolic knowledge base, as in the previous example of the
jazz chord sequence. We call these expectations linguistic. Several sources may
be taken into account in order to generate linguistic expectations, for example
the ITPRA theory of expectation proposed by Huron [
On the other side, expectations could be generated by purely Hebbian, associative mechanisms. Suppose that the system learnt that typically a jazz player adopts the tritone substitution when performing the previous described jazz progression. The system could learn to associate this substitution to the progression: in this case, when a compatible chord is recognized, the system will generate also expectations for the sequence ii [II I. We call these expectations associative.
Therefore, synchronic expectations refer to the same con guration of knoxels at the same time; diachronic expectations involve subsequent con gurations of knoxels. The linguistic and associative mechanisms let the cognitive architecture generate suitable expectations related to the perceived chords progressions. 6
So far we adopted a \static" conceptual space where a knoxel represents the partials of a perceived tone. In order to generalize this concept and in analogy with the di erences between static and dynamic vision, in order to represent a music phrase, we now adopt a \dynamic" conceptual space in which each knoxel represents the whole set of partials of the Short Time Fourier Transform of the corresponding music phrase. In other words, a knoxel in the dynamic CS now represents all the parameters of the spectrogram of the perceived phrase.
Therefore, inspired by empirical results (see Deutsch [
The operations in the dynamic CS are largely similar to the static CS, with the main di erence that now a knoxel is a whole perceived phrase.
A con guration of knoxels in CS occurs when two or more phrases are perceived at the same time. The two phrases may be related with two di erent sequences of pitches or it may be the same sequence played for example, by two di erent instruments. This is similar to the situation depicted in Figure 2, where the knoxels ka and kb are interpreted as music phrases perceived at the same time.
A scattering of knoxels occurs when a change occurs in a perceived phrase. We may represent this scattering in a similar way to the situation depicted in Figure 3, where the knoxels also in this case are interpreted as music phrases: knoxels ka and kb are interpreted as changed music phrases while knoxels kc corresponds to the same perceived phrase.
As an example, let us consider the well known piece In C by Terry Riley.
The piece is composed by 53 small phrases to be performed sequentially; each
player may decide when to start playing, how many times to repeat the same
phrase, and when to move to the next phrase (see the performing directions of
In C [
Let us consider the case in which two players, with two di erent instruments, start with the rst phrase. In this case, two knoxels ka and kb will be activated in the dynamic CS. We remark that, although the phrase is the same in terms of pitch and duration, it corresponds to two di erent knoxels because of di erent timbres of the two instruments. When a player will decide at some time to move to next phrase, a scattering occur in the dynamic CS, analogously with the previous analyzed static CS: the corresponding knoxel, say ka, will change its position to k0 .
a
The focus of attention mechanism will operate in a similar way as in the static case: the synchronous modality of the focus of attention will take care of generation of expectations among phrases occurring at the same time, by taking into account, e.g., the rules of counterpoint. Instead, the asynchronous modality will generate expectations concerning, e.g., the continuation of phrases.
Moreover, the static CS and the dynamic CS could generate mutual expectations: for example, when the focus of attention recognizes a progression of chords in the static CS, this recognized progression will constraint the expectations of phrases in the dynamic CS. As another example, the recognition of a phrase in the dynamic CS could constraint as well the recognition of the corresponding progression of chords in the static CS. 7
The paper sketched a cognitive architecture for music perception extending and completing a computer vision cognitive architecture. The architecture integrates symbolic and the sub symbolic approaches by means of conceptual spaces and it takes into account many relationships between vision and music perception.
Several problems arise concerning the proposed approach. A rst problem,
analogously with the case of computer vision, concerns the segmentation step.
In the case of static CS, the cognitive architecture should be able to segment
the Fourier Transform signal coming from the microphone in order to
individuate the perceived tones; in the case of dynamic CS the architecture should be
able to individuate the perceived phrases. Although many algorithms for music
segmentation have been proposed in the computer music literature and some
of them are also available as commercial program, as the AudioSculpt program
developed by IRCAM1, this is a main problem in perception. Interestingly,
empirical studies concur in indicating that the same Gestalt principles at the basis
of visual perception operate in similar ways in music perception, as discussed by
Deutsch [
The expectation generation process at the basis of the focus of attention
mechanism can be employed to help solving the segmentation problem: the
linguistic information and the associative mechanism can provide interpretation
1 http://forumnet.ircam.fr/product/audiosculpt/
contexts and high level hypotheses that help segmenting the audio signal, as
e.g., in the IPUS system [
Another problem is related with the analysis of time. Currently, the proposed
architecture does not take into account the metrical structure of the perceived
music. Successive development of the described architecture will concern a
metrical conceptual space; interesting starting points are the geometric models of
metrical-rhythmic structure discussed by Forth et al. [
However, we maintain that an intermediate level based on conceptual spaces could be a great help towards the integration between the music cognitive systems based on subsymbolic representations, and the class of systems based on symbolic models of knowledge representation and reasoning. In facts, conceptual spaces could o er a theoretically well founded approach to the integration of symbolic musical knowledge with musical neural networks.
Finally, as stated during the paper, the synergies between music and vision are multiple and multifaceted. Future works will deal with the exploitation of conceptual spaces as a framework towards a sort of uni ed theory of perception able to integrate in a principled way vision and music perception.