The cognitive framework of conceptual spaces bridges the gap between symbolic and subsymbolic AI by proposing an intermediate conceptual layer where knowledge is represented geometrically. There are two main approaches for obtaining the dimensions of this conceptual similarity space: using similarity ratings from psychological experiments and using machine learning techniques. In this paper, we propose a combination of both approaches by using psychologically derived similarity ratings to constrain the machine learning process. This way, a mapping from stimuli to conceptual spaces can be learned that is both supported by psychological data and allows generalization to unseen stimuli. The results of a rst feasibility study support our proposed approach.
The cognitive framework of conceptual spaces [
The framework of conceptual spaces has been highly in uential in the last
15 years within cognitive science [
The dimensions of a conceptual similarity space are usually obtained based on either psychological experiments or machine learning methods. The rst approach has a clear psychological grounding, but is only applicable to items for which human similarity ratings are available. The second approach, in contrast, can work on large amounts of unlabeled data, but lacks psychological validity. In this paper, we propose a combination of both approaches that uses psychologically derived similarity spaces as a target for machine learning algorithms. This way, a mapping from stimuli to conceptual spaces can be found that is both supported by psychological data and able to generalize to unseen stimuli. The results of a rst feasibility study support our proposed approach.
The remainder of this paper is structured as follows: Section 2 introduces the framework of conceptual spaces. Section 3 summarizes techniques for deriving similarity spaces from psychological experiments and Section 4 describes two recently proposed types of neural networks from the area of representation learning. In Section 5, we formulate our proposal of combining arti cial neural networks with psychological data. Section 6 presents a rst feasibility study for our approach and Section 7 provides an outlook on future work. 2
A conceptual space [
The framework distinguishes properties like \red", \round", and \sweet" from full- eshed concepts like \apple" or \dog": Properties are represented as convex sets within individual domains (e.g., color, shape, taste), whereas fulleshed concepts span multiple domains. Reasoning within a conceptual space can be done based on geometric relationships (e.g., betweenness and similarity) and geometric operations (e.g., intersection and projection).
In his book [12, Chapter 1.7, Chapter 6.5], Gardenfors describes three ways for identifying the dimensions spanning a conceptual space: Handcrafting, multidimensional scaling, and machine learning.
Handcrafting is only applicable to domains with a well known structure such
as the temperature domain, which can be described by a single dimension and
measured with a single sensor. However, handcrafting is not possible for more
complex domains (e.g., shapes) that are based on complex sensors (e.g.,
cameras). We therefore exclude it from our further considerations.
If handcrafting is not applicable, one can conduct a psychological experiment
to elicit human similarity ratings for pairs of stimuli and then apply
\multidimensional scaling" (MDS) [
The third approach for obtaining the dimensions of a conceptual space is to use machine learning techniques for dimensionality reduction. Gardenfors argues that raw perceptual input is too rich and too unstructured for direct processing. It is thus necessary to lift the input to a more economic form of representation, which typically involves a drastic reduction in the number of dimensions. There are a variety of dimensionality reduction algorithms in the machine learning eld, among which arti cial neural networks (ANNs) are a promising candidate for implementing a multi-layered and non-linear dimensionality reduction. In Section 4 we will introduce two recently proposed network architectures. 3
Several data collection methods have been proposed for collecting similarity judgments from human participants. The traditional technique is called \dissimilarity ratings". In this approach, all possible pairs from a set of stimuli are presented to participants (one pair at a time), and participants rate the dissimilarity of each pair on a continuous or categorical scale. This method is however quite ine cient, due to the large number of judgments required by humans.
A faster alternative is the \sorting method", where participants are given the
whole set of stimuli and are asked to assign them into groups of similar stimuli.
There are two versions of the sorting method: In \constrained sorting" the
number of groups is pre-de ned by the experimenters, whereas in \free sorting" the
number of groups is at the discretion of each participant. The resulting pairwise
ratings derived from the sorting method are binary { either the two stimuli
belong to the same group or not. Sorting methods, although faster, lack accuracy
compared to dissimilarity ratings [
A more recent technique, named \Spatial Arrangement Method" (SpAM), is
proposed by Goldstone [
After having elicited psychological similarity ratings, MDS can be used to convert them into a geometric representation which can be interpreted as a conceptual space. MDS takes as input the pair-wise average similarity scores for all pairs of stimuli and the desired number n of dimensions. It then represents each stimulus as a point in an n-dimensional space and tries to arrange these points in such a way that the distance of two points accurately re ects the similarity rating of the stimuli they represent. The optimal number of dimensions is usually determined by repeatedly running the MDS algorithm with di erent values of n. The accuracy of each resulting space in representing the original similarity ratings is calculated. One typically selects a relatively small value n for which the accuracy is deemed good enough and where the accuracy of using n + 1 dimensions is not considerably better.
While providing an elegant way for deriving a conceptual space from human similarity ratings, this approach does have some disadvantages: First of all, there is no guarantee that the dimensions generated by the MDS algorithm are interpretable, so one typically needs to search for interpretable directions in the generated space. Moreover (and in our opinion more importantly), MDS is only applicable to a xed number of input stimuli. If a new, previously unseen stimulus is perceived, MDS does not provide a mapping of this stimulus onto a point in the similarity space. 4
In recent years, there has been substantial work on learning compressed
representations of a given feature space with neural networks [
The left part of Figure 1 shows the typical structure of an autoencoder. The overall network consists of two parts, an encoder and a decoder. The encoder takes an input image and compresses it into a low-dimensional \hidden" representation. This hidden representation is passed on to the decoder, which tries to reconstruct the original image from it.
While the general autoencoder structure has existed for decades, Kingma
and Welling [
Higgins et al [
InfoGAN [
Both of these recent approaches have shown promising early results in extracting meaningful dimensions from unlabeled data. They can both generalize to unseen inputs and generate example images from a given hidden/latent representation. However, choosing the correct size of the hidden/latent representation is crucial and typically requires either good prior knowledge of the domain or extensive manual optimization. Neural networks in general require large amounts of training data, but as both presented approaches only need unlabeled data, this is not critical in practice. Finally, while the extracted dimensions might be useful from an AI perspective, they cannot claim any psychological validity. 5
As we have seen in Sections 3 and 4, both \traditional" approaches for obtaining the dimensions of a conceptual space have their individual strengths and weaknesses. We are not aware of an approach that can claim both to be psychologically valid and to generalize well to unseen stimuli. By combining MDS with neural networks, one can potentially obtain such an approach. We will focus our subsequent discussion on stimuli in the form of images. 5.1
After having determined the domain of interest (e.g., the domain of shapes), one
rst needs to acquire a data set of stimuli from this domain (e.g., ShapeNet [
This subset of stimuli is then used in a psychological study where similarity
judgments by humans are obtained, using one of the techniques described in
Section 3. The choice of the data collection method usually depends on the type
of stimuli, the size of the data set and the aims of the study. Choosing the right
method is crucial for the quality of the resulting space [
One can now apply MDS to the collected similarity judgments in order to extract a spatial representation of the underlying domain. As stated in Section 3, one needs to select the optimal number of dimensions { either based on prior knowledge or by manually optimizing the trade-o between high representational accuracy and a low number of dimensions. From a machine learning perspective, the stimulus-point mappings obtained from MDS can be interpreted as labeled training instances: The stimulus is identi ed with the input vector and the point in the MDS space is used as its label.
Now an ANN can be trained to map from stimuli (e.g., images) to points in the MDS space. As the number of stimulus-point pairs is quite low for a machine learning problem, one needs to introduce an additional training objective (e.g., minimizing the reconstruction error on the full data set) to avoid over tting and to ensure that the network is able to generalize to unseen inputs.
If training the ANN is successful, then it has learned a mapping from stimuli to points in the conceptual space. While MDS only maps from a xed set of stimuli to points in the conceptual space, the ANN is expected to generalize this mapping to previously unseen stimuli. Moreover, as psychologically derived data was used to train the network, it can claim at least some psychological validity. Figure 2 illustrates three potential network architectures for image-point mappings, where the dashed grey lines illustrate the reconstruction constraints used in the training procedure. In all cases, the respective network is trained on a secondary task for all images from the full data set and on the mapping task only for the images that were used in the psychological study. Using a secondary task with additional training data constrains the network's weights and can be seen as a form of regularization: These additional constraints are expected to counteract over tting tendencies, i.e., tendencies to memorize all given mapping examples without being able to generalize.
The left part of Figure 2 shows a standard feed-forward network which is supported by a secondary task of predicting the correct classes. This approach is however only applicable if the data set contains class labels.
The middle part of Figure 2 shows the network structure of an autoencoder: The network is trained to minimize the di erence between the input images and their reconstruction, while being encouraged to use the MDS space in its bottleneck layer. As the computation of the reconstruction error does not need any class labels, this is applicable also to unlabeled data sets. Of course, instead of using a regular autoencoder, one can also employ VAE or -VAE. The autoencoder structure has the additional advantage that one can use the decoder network to generate an image based on a point in the conceptual space.
The right part of Figure 2 shows an extended version of the InfoGAN
framework, where the discriminator network is also constrained to correctly extract
the MDS points from the images. This is in some sense reminiscent of the
ACGAN network [
Instead of learning a direct mapping from images to points in a conceptual space, one can also train a network to predict the similarity ratings for pairs of images. In this case, the neural network receives two images as input and predicts their similarity. The input dimensionality is twice as large in this case, but there are also more training examples available. Again, an additional training objective might be needed in order to prevent over tting. In principle, the three approaches discussed above can be adapted also to this learning problem. One can use such a network to indirectly map an image onto a point in the conceptual space as follows: The network predicts the similarity of the given image to a xed set of \anchor images" for which a mapping into the conceptual space is known. One can then use the points representing these anchor images, together with the predicted similarity ratings, to triangulate the point representing the new image. 5.3
What are the advantages of our proposed approach? As stated before, we expect to obtain a neural network that learns a mapping from input images to a psychological similarity space. We thus get a mapping that is both psychologically grounded and that generalizes to unseen inputs. Even if the mapping performance of the neural network is not perfect, a rough guess for the point representing a previously unseen stimulus might still be quite useful.
In general, one can distinguish two types of similarity: While perceptual similarity is exclusively based on the immediate perceptual features of the stimuli, conceptual similarity involves additional sources of knowledge, e.g., the expected use of a depicted object or the typical context in which it can be found.
Neural networks that are trained on a task such as reconstruction are likely to yield a hidden representation which encodes perceptual similarity. The human similarity ratings retrieved in psychological experiments might however also include conceptual similarity, and thus implicit features such as the object's perceived softness. If a neural network is not only trained to reconstruct images, but also to predict their positions in a psychological similarity space, it is incited to use also such implicit features in its internal representations. In some sense, these psychological constraints could thus help the network to \see behind the image" and to represent not only perceptual, but also conceptual features of the stimuli.
If the approach described above yields promising and useful results, one could use the observation that -VAE and InfoGAN tend to discover meaningful dimensions in order to devise a new MDS algorithm based on these networks. This algorithm would train a standard -VAE or InfoGAN network while ensuring through an additional term in the loss function that the hidden space extracted by the network accurately re ects the psychological similarity ratings. This overall algorithm would thus result in a spatial representation of psychological similarities which generalizes to unseen images, uses interpretable dimensions, and can be used to generate new images based on points in the conceptual space. 6
In order to validate whether our proposed approach is worth pursuing and support our claims with some initial empirical results, we conducted a feasibility study based on a simple setup: We used an existing data set of similarity ratings for images of novel objects and we trained a linear regression on the hidden activations of a pre-trained image classi cation network.3 6.1
For our feasibility study, we used existing similarity ratings reported for the
Novel Object and Unusual Name (NOUN) dataset [
Using the similarity matrix from the study of Horst and Hout [
As the number of images is quite low for a machine learning problem, we
augmented the data set by applying horizontal ips, random crops, a Gaussian
blur, manipulation of the image's contrast and brightness, additive Gaussian
noise, a ne transformations, and salt and pepper noise. For each of the original
64 images, we created 1,000 augmented versions, resulting in a data set of 64,000
images in total. We assigned the MDS point of the original image to each of the
1,000 augmented versions.
3 Code for reproducing this study can be found online at https://github.com/
lbechberger/LearningPsychologicalSpaces/ [
Our network architecture is a special variant of the feed-forward network proposed in Section 5.2: Instead of training both the mapping and the classi cation task simultaneously, we use an already pre-trained network, keep its weights xed, and augment it by an additional output layer which is trained separately.
We have implemented four baselines against which we compare our system: { Zero: Always predict the origin, i.e., the (0; 0; : : : ; 0) point. { Mean: Always predict the mean of all MDS points from the training set. { Distribution: Draw a random sample from a Gaussian distribution which was estimated from the MDS points in the training set. { Random Draw: Use a randomly selected MDS point from the training set as prediction.
We expect that our system is able to outperform all of these baselines. We would however also like to investigate whether learning a mapping into an MDS space is easier than learning a mapping into an arbitrary space of the same dimensionality. Therefore, we trained and evaluated two versions of our system: One of them was trained on the mappings from images to points derived by the MDS. For the other version, we used the same MDS points, but we shu ed the Con guration
2D 4D 8D
Training Testing Training Testing Training Testing Zero baseline 0.4408 Mean baseline 0.4408 Distribution baseline 0.6287 Random draw baseline 0.6233 assignment of images to points. We ensured that all augmented images created based on the same original image were still mapped onto the same MDS point. With this shu ing procedure, we aimed to destroy any semantic structure inherent in the original space. We expect that the regression works better for the original mapping than for the shu ed mapping, as it contains more semantic content and should therefore be easier to learn. By using a shu ed mapping rather than a mapping to randomly chosen points in the space, we ensure that the distribution of the target points remains constant. This is necessary in order to make a meaningful comparison between the two regression con gurations.
In order to evaluate both our system and the baselines, we used the root mean squared error (RMSE), which is a standard metric for regression problems. The RMSE metric has the same scale as the target space and can thus provide an intuition about the system's performance if interpreted as the average distance between the prediction and the ground truth. Due to the small amount of original images, we performed an image-based leave-one-out evaluation: All augmented images generated from one of the original images were used as test set, while all other images were to train the linear regression. This was repeated for each of the original images and the training and test errors were averaged across all of these runs. As all augmented images based on the same original image can be expected to still be quite similar to each other, this procedure ensures that the system cannot simply \memorize" all 64 images. 6.3
Table 1 shows the results obtained in our experiment for the three MDS spaces with two, four, and eight dimensions, respectively. Among the di erent baselines, the \zero baseline" consistently has the best performance on the test data, followed closely by the \mean baseline". The two other baselines perform considerably worse. Notably, the regression always has a much lower RMSE than any of the baselines, indicating that the system is indeed capable of learning a mapping from images to points in a psychological space. As expected, the RMSE for the correct mapping is always considerably lower than for the shu ed mapping. However, the regression always performs signi cantly better on the training data than on the test data, which is a clear sign for over tting.
Figure 4 illustrates the RMSE results during testing for the three di erent similarity spaces. As one can see, the performance of the best baseline is quite independent from the dimensionality of the similarity space. Although in theory the regression problem becomes harder as the number of dimensions increases (when predicting more coe cients of the output vector, one can make more mistakes), both regression approaches perform better in higher-dimensional spaces. The results presented in Section 6.3 con rm our hypotheses: Our system is able to perform better than the baselines, indicating that a mapping from images to MDS points can be learned. Moreover, the observed performance di erence on the test set between learning the correct mapping and learning the shu ed mapping indicates that the semantic structure of a similarity space facilitates learning a generalizable mapping. The small di erence on the training data indicates that both learning problems have a similar di culty.
The regression also seems to bene t from a higher-dimensional MDS space. As a higher-dimensional space derived with MDS can more accurately re ect the original similarity ratings, this might be interpreted as the system learning ner nuances of similarity if given the chance to do so. As all baselines have a similar performance on all MDS spaces, the distribution of the points in the space seems to have similar characteristics regardless of the number of dimensions. We assume that also the regression on the shu ed mapping can bene t from a higher number of dimensions, as the shu ing procedure may not be able to completely destroy the structure of the similarity space { even after shu ing, the mapping might still partially re ect the similarity ratings.
These overall results are in line with our expectations and indicate that our overall approach is sound and promising. The achieved performance is still far from perfect, probably because of the observed over tting tendencies. However, the goal of this early feasibility study was not to aim for optimal performance, but to evaluate the principle idea of our approach. We suspect that the relatively large amount of over tting is caused by the small set of original images and the large size of the extracted feature vector. Although we applied a variety of augmentation techniques to the original images, the resulting images might still be too similar to each other. Moreover, as the feature vector contains 2048 entries, the linear regression has to estimate a large number of parameters. Furthermore, as the network was pre-trained on a di erent data set with di erent characteristics, we cannot expect a perfect generalization.
Finally, one could argue that even humans might fail to achieve an RMSE
of approximately zero, as the given problem is inherently di cult, due to the
nature of the data set used. The MDS space was created based on the average
similarity ratings obtained in Horst and Hout's [
We have argued that the combination of neural networks with psychological similarity judgments o ers a promising way for extracting a conceptual space from data. This can help to make the framework of conceptual spaces more viable for arti cial intelligence: Our proposed approach can potentially provide a principled way of mapping sensory input to conceptual spaces, while still maintaining some psychological validity. The results of our feasibility study are encouraging and show that our approach is also feasible in practice.
In future work, we will implement and evaluate our proposal in more depth by
exploring the remaining proposed network structures. We will conduct a
psychological study on a subset of ImageNet [