Modern Information Science deals with tasks which include classifying, searching and browsing large numbers of digital objects. The problem today is that our computerized tools are poorly adapted to our needs as they are often too formal: we illustrate this matter in the rst section of this article with the example of multimedia collections. We then propose a software tool, ReCollection, for dealing with digital collections in a less formal and more sustainable manner. Finally, we explain how our software design is strongly backed up by both artistic and psychological knowledge concerning the ancient human activity of collecting, which we will see can be described as a metaphor for categorization in which two irreducible cognitive modes are at play: aspectual similarity and spatiotemporal proximity.
1.1 to name but a few technological trends. These are still evolving but the fact is that today more and more people are using their computers not only for editing and ling documents, but also for collecting music, lms, images, books... Large amounts of these can be stored on hard drives and DVD-ROMS. The contents can be downloaded from the internet, or imported from digital devices such as cameras, which have also become mainstream.
Not surprisingly, a huge market has emerged from these multimedia collections. We can now choose from a myriad of computerized tools which assist us in nding, retrieving, recording, creating, editing, browsing and classifying multimedia contents. The variety of tools at hand seems to t with the variety of uses involved in multimedia computing, from the most creative ones - such as graphic design, audio synthesis, etc - to the most formal ones - classication in particular. However, there doesn’t seem to be many tools bridging the gap between these two seemingly opposing polarities. 1.2
Let us illustrate this situation. First, let us suggest that looking for new material and classifying are two important processes involved in collecting. Indeed, when someone decides to start building a collection he usually already possesses a few items. Then, to extend this collection, new items must be added. In order to do so, the collector goes into the world and looks for these new items. Then as the collection builds up, the need to arrange the items into categories will become clearer, as the collection cannot simply remain a messy stack of unordered items.
So, in order to illustrate our point, let us describe a particular example: the music collector. As we have said, our collector will surely possess some initial items; these may be some CDs or vinyl records. His rst action involved in extending his collection could be a visit to the record shop for example. Here, the music is classied conformingly to the record companies’ desires, which can sometimes be confusing for our collector, who is a fan of Jimi Hendrix, and just does not know where to look for his albums: in the blues section? rock section? Is there a ’sixties’ section? Anyway, despite nding them rather practical at rst sight, our collector didn’t create these labels, and nds it dicult adapting to them. However, as he browses through the shop, he also notices some nicely illustrated records, and discovers new artists he is interested in because their records are sitting next to Jimi’s. Finally, when he has bought enough music records, and come back home, he will be able to start arranging his collection in a very personal and satisfying manner, which will be pleasing to the eyes, and also allow him to retrieve items quickly.
If he had decided to collect digital music, and go online to nd new items
for his collection, the process would have been rather similar. Commercial music
download sites allow the user to browse through predened music categories, thus
implementing a kind of virtual record shop with the same problems mentioned
earlier. The search tool however can come in handy, and allow the user to search
for the name of an artist, a song, an album or even musical genre. All these
are still editorial information, which aren’t necessarily the most useful to the
collector. Then, when the music is downloaded, the album consists of a group
of compressed audio les, containing preset meta-tags, again storing editorial
information. When browsing these les in his audio player, the songs are dened
and classied automatically, not always according to the collector’s desires. His
nal attempt is then to create a set of folders on his disk, and arrange his items in
these folders. But how does he name these folders? What if he wants to arrange
and browse the items in multiple ways? What if a particular item doesn’t t in
any folder, or could be placed in two or three dierent categories? Pachet has
also described many problems in the area of Electronic Music Distribution [
As we see from this example, the tools that the everyday user has at hand are too formal, and are poorly adapted to the growing activity of collecting multimedia contents. Indeed, what we have said for music can also be said for the other kinds of media, and can also be said for information research, le sharing, etc.
Attempts have been made at putting the human user back in control of the collecting process, rather than relying purely on predened categories and automated research algorithms. However, it has become obvious that the other extreme of handing complete control over to the user isn’t optimal either. Let us take a look at online content sharing sites, such as the famous FlickR. There is no categorization here, but there are three main strategies when looking for photos: date, location, tags. The rst two are self-explanatory, but the tags are more interesting here. When someone uploads a photo to the website, they can link a certain number of keywords, called tags, to this photo. Then, we can either browse through the most popular tags, or type a tag into a textbox for a more precise search. The users then have complete freedom on the way they choose to dene their photos. But the problem is that many photos aren’t tagged, and the photos that are, often have poorly named tags, making them dicult to retrieve. Therefore, we believe that an optimal solution to the problem of digital collections could lie somewhere between these two polarities: predened categories and total user creativity. 1.3
When digital sound les are imported into the system, they are analyzed, and a database of their acoustic properties is created / updated. Then the user can browse through the collection in a traditional manner, relying on editorial information. He can also create his own categories intuitively. He starts by creating a category, and giving it a name. This can be totally subjective if he wishes, he may call it "evening music", "happy music" or "favorite", etc. He then adds a few songs to this category, before asking the program to nish classifying, based on acoustic similarities. Of course, the more categories there are, and the more examples there are, the easier it is for the system to classify the entire collection. However, if there are mistakes, the user may simply move a song from one category to another, and ask the system to start again. This creative feedback loop, between user input and automated algorithms, will eventually lead to a satisfying classication for the user, who will have saved a lot of time in the process. He will then be able to create other classications of the same collection if he wishes, and switch instantly between any of them. He may also share these classications or download others.
IMEDIA is a research project focused on indexing large collections of photos,
and interactive searching and browsing [
In these two systems, we have noticed a creative feedback loop between the human user’s input (starting point, examples, relevance feedback...) and the computer (automated algorithms for classifying and searching). This helps the user build and browse his collection in a constructive process, leading to a result which neither he nor the computer could have achieved alone. Also, both editorial information and semantic information (invisible to the user) are taken into account. IMEDIA and MusicBrowser address the problems of music collections, and photo collections, but the same ideas may be applied to other media collections, such as texts or videos, for example. It is only a case of nding the appropriate descriptors. Also, both these ideas, interactive searching and browsing, can be transposed to dierent media.
We can even think further, and imagine a common environment for collecting multimedia les. This could be a system with a generic layout and set of functionalities that would give birth to dierent programs specialized in collecting certain types of media. In the next section, we shall present a software prototype that we have implemented in order to experiment with this idea. As we shall see in the next section, we have tried to create a program more suitable to the particular process of collecting, which has an element of subjectivity, evolves over time and doesn’t rely purely on similarities, as in the IMEDIA system for example. Indeed, we sometimes wish to expand our collection with something completely dierent, now how would we do that? We also believe that this process lies somewhere between formal classication/automated algorithms and total creativity. There are more and more examples of this, such as the two projects described previously, and we will try to take this process even further.
ReCollection: An Experimental Software For The Creation Of Multimedia Collections ReCollection is a computer program for searching, arranging and browsing digital content. As our collecting activities vary from one context to another, it is too ambitious to seek a general solution to the problem. Rather, particular application areas must be dened and isolated, in order for a specic answer to be given, however always relying on a set of basic principles. Here, we shall discuss the software prototype we have created for the digital opera / open form opera Alma Sola2. 2.1
Artists and philosophers have described some very particular characteristics of
collections. One of those, as noted by Wajcman[
The reserve allows us to handle the excess in collections, which is a problem in many of today’s computer applications. Our multimedia collections, for example, are becoming very large and we are often losing control over them.
On the other hand, objects which are currently exposed are found in the gallery. Here, the objects follow a spatio-temporal arrangement dening a nite number of visitation paths. The closeness in space of certain artworks and the chronological order in which they are approached are set carefully by the curator, as they strongly inuence the visitors’ experience. This aspect is also very important, and we shall discuss it later in detail. 2.2
The ReCollection software has two main modes: reserve and gallery. The reserve allows us to store our objects which aren’t exposed in the gallery. There are many objects in the reserve, and these are not always labeled; also they are rarely arranged in an orderly and tidy manner. So when we visit the reserve, we have no choice but to wander around, picking up objects, inspecting and identifying them one at a time. The reserve can also be compared to the attic, in which our family possessions are stored similarly. As we explore our attic, we 2 Designed by Alain Bonardi, IRCAM, Paris and performed at Le Cube, Issy les
Moulineaux, October 2005. can happen to pick up an old photo album, which we had completely forgotten about. This item will surely bring back memories and emotions. We can then choose to keep this album under our arm, as we continue to explore the attic, or we can leave straight away, and put it on our replace, for example, making it visible to visitors. It is all these pleasant and familiar experiences which we believe can be recreated thanks to the modeling of the reserve in our computer program.
The user can create any number of reserves. However, he must create at least one, and store at least one object in this reserve. When he is in reserve mode, he can only view one object at a time. When he decides to view another object, it is chosen randomly from the remaining items in reserve. During a visit, each object is viewed only once. If the user wants to view an item he has already visited, he may go through the history of items on the left side of the screen, as shown in Fig. ??. When he nds an object of interest, he can move it to the gallery. It will then be removed from the reserve, and saved in memory, with a group of objects waiting to be imported in the gallery. Then, in gallery mode, the user will see this heap of objects, and will be able to import it in the desired gallery, at the desired location. 2.3
The items in the Alma Sola collection are made up of three components: a photo of the performance, a sound recording of a few seconds of the singing, a text, the line which is sang in the corresponding sound le.
These are all regular les stored on disk (bitmap, wave and .txt formats). Each item also has a name. In a more general context, the objects can be made up of any one of these types of media, a video (though not implemented in this version), or any combination of these.
Also, each object has a set of descriptors attached. There is a specic set of descriptors for each type of media, which describe the contents of the object, for example the average volume of the sound, the brightness of the photo, the number of words, etc. Depending on the application, we could also include editorial information, such as date, author, etc.
These descriptors may be assimilated to the private properties of traditional computer objects. But in the context of collecting objects, we also need to account for other properties that come from the activities in which these objects collectively engage. 2.4
A collective activity involving a number of objects at a time is their relative arrangement in the gallery space. To the location of objects in this space, we have added their color; these two properties make up an extra conceptual layer which is the framework for the creation and management of our collections.
In ReCollection, there is always at least one gallery, and the user can create as many as he wishes. There is always at least one item in a gallery, some basic content that the user can interact with, a starting point for his collection.
The objects can be placed and arranged manually in the gallery space, using
click and move, just as in common user interfaces. The user can also rely on two
algorithms to automatically dispose the objects. The rst one, inspired by cataRT
software [
The arrangements resulting from the algorithmic calculations can always be modied manually in order to correct them (in the eventuality of rather subjective descriptors), to build up a global gure, or to bring items together. This way, through creative human-computer feedback loops, meaningful global gures can emerge through the arrangement in space of collected items, as well as local gures, soft pseudo-categories which are heaps of objects brought together by the system and/or the human user. These pseudo-categories are the building blocks for the classes the collection is implicitly aiming for. They are easily and constantly updated; items are added and removed instantly by being moved in space. They are loosely dened and never completely closed o from others, allowing some objects to be lost somewhere in between several heaps, when they cannot be placed in any one category. In a nutshell, this system allows for the creation of collections in which classes are in constant evolution, and are built by exploiting not only the objects’ degree of similarity, but also their relative location in space and time.
Furthermore, the user may wish to search for objects in the gallery or in the reserve, in order to build on these categories, look for new kinds, or even ll in gaps in the gallery space. For this, the ReCollection system has two search tools he can use. The rst is a simple ’keyword query’, which searches for a keyword within the text or names of the objects. The second is a ’search by similarity’. The user selects an object, or group of objects, and the system searches for items which are similar (according to the descriptors). In both cases, the search is carried out in both the gallery and reserve, and a list of results is displayed in the gallery, ordered by similarity.
Once all the items of interest have been imported from the reserve, through browsing or searching, and once they have been arranged in the gallery space, the user has a rst disposition he can play with. When he will browse the gallery space, his experience will be inuenced by the fact that certain objects are close in space, and in time of visitation. Although this is interesting in itself, the system can help the user go further, by dening a set of guided visits, which are simply an order of visitation of selected objects in the gallery.
The type of interface we have chosen to implement these functionalities is a
2D zoomable user interface (ZUI), inspired by Ken Perlin’s Pad [
In the next part of this article, we shall discuss in detail some key characteristics of collections, as identied by artists, philosophers and psychologists. This theory is at the foundation of our work, and it demonstrates the novelty and usefulness of the collections metaphor in computer science. 3
The Strange Epistemological Status of Collections Object-oriented computer science was invented to assist the task of classifying objects in a structure where dierent classes are distinguished[1012].
As we all know, this innovation quickly became a success. 3.1
Recently, an innovative trend is mobilizing computer objects for the organization of our collections, considered like a group of objects waiting to be organized in ad hoc classes that must be created simultaneously[1315].
Because our collections seem to be nearer to order than disorder, attempting to assimilate them in classes is not so surprising. At least, collections look like they are waiting for their completion within a classication order, with the aim of turning into canonic achieved structures made of objects and classes. But something is also resisting this assimilation, as artists and philosophers have always noticed. 3.2
Here, for example, is the analysis of GØrard Wajcman ( Catalog for the inaugural exhibit of the Maison Rouge ) on the status of excess in a collection: ‘Excess in a collection does not mean disordered accumulation; it is a fundamental principle: for a collection to exist as such-in the collector’s eyes the number of objects must exceed the physical possibilities of exposing and storing the entire collection at home. Therefore, someone who lives in a studio can have a collection: it is only necessary for him to have at least one work he cannot hang in his studio. That is why the reserve is an integral part of collections. Excess also applies to the capacity of memorization: for the collection to exist, it is necessary for the collector not to be able to remember all the works he owns. In fact, the number of objects he owns must be so important that it becomes too important, so that the collector can forget one of them or leave a part of his collection outside of his home. To say it dierently, for a collection to exist, the collector must not have full control over his collection anymore.’ 4
Undoubtedly impressed by artists and philosophers who considered the strange status of collections, ’object-oriented’ computer program designers understood that computer modeling of object collections would necessarily involve the creation of hybrid structures including private characteristics - by which the collected objects are usually referred to - but also including characteristics that come from the activities in which these objects collectively engage. 4.1
A Parsimonious, Conservative and indeed Seductive Approach Often, the approach implicitly chosen to characterize a collection is parsimonious and consists of over-determining the private referencing of the collected objects through a minimal description of the collective activity’s context, even if it means predicting that the collection shall become a class or set of classes.
This practice presents the unquestionable advantage of not fundamentally opposing the traditional modeling of objects. However, it does not always live up to the collectors’ high standards.
Here it is important to distinguish between gural and non-gural collections.
This subtle distinction, introduced in the 1970s by Piaget and his research teams
of child psychologists [
In their book La genŁse des structures logiques ØlØmentaires (lit: The Genesis of Elementary Logical Structures ), Jean Piaget and Brbel Inhelder provide a precise distinction between gural and non-gural collections, which are still called classes or categorical collections. For the authors, a class requires only two categories or relationships, both necessary and sucient, for its actual denition as a class (page 25): 1. The qualities common to its members and to those of the classes it belongs to, as well as the specic dierences that distinguish its own members from the members of other classes (comprehension); 2. The relationship of a part to the whole (membership and inclusion) determined by the quantiers all, some (including one ) and none applied to the members of the class in question and to other members of the classes it belongs to, dened as extensions of that class.
For example, cats share in common several qualities owned by all cats, some being specic and some others belonging also to other animals. But no spatial considerations ever enter into such a denition: cats may be grouped or not in the space without any change concerning their class denition and properties (1) and (2).
Piaget then introduces gural collections , in which meaning dened by properties (1) and (2) is linked to the spatial arrangement of its elements. A gural collection composes a gure, through the spatial relationships between its elements, whereas non-gural collections and classes are free of any gure. 4.3
It is precisely these gural collections that object-oriented computing is
promising more and more an eective modeling of, pushed by an ever-growing social
demand for on-line digital media browsing and information research amongst
multiple sources[
But as we now understand, gural collections adapt poorly to their assimilation into non-gural collections or classes. Although according to Piaget, collections are destined to become classes, in the same way as subjects will grow psychologically so as to improve their cognitive capacity to classify. Still referring to Piaget, it is a radical lack of dierentiation that nudges gural collections out of the classical modeling eld. 5
Husserl used to say that consciousness is always consciousness of something, that consciousness always pre-dates the subject and the object, and puts them together in the process. There are no subjects or objects already existing independently that meet in the world to ll out a journal of experiences (the subject) and perhaps adapt to each other by induction. In the same fashion, we could say that a collection is always a collection of something , in that the original process of categorization is the activity of collecting, implacably mixing abstraction and spatio-temporal arrangements, and producing as many metastable categories.
The current models for information search are too formal, and they assume that the function and variables dening the categorization are known in advance. In practice, however, when searching for information, experimentation plays a good part in the activity, not due to technological limits, but because the searcher does not know all the parameters of the class he wants to create. He has hints, but these evolve as he sees the results of his search. The procedure is dynamic, but not totally random, and this is where the collection metaphor is interesting.
The collector’s experimentation is always carried out by placing objects in temporary and metastable space/time. Here, the intension of the future category has an extensive gure in space/time. And this system of extension (the gure) gives as many ideas as it does constraints. What is remarkable is that when we collect something, we always have the choice between two systems of constraints, irreducible one to the other. This articial indierentiation for similarity/contiguity is the only possible kind of freedom allowing us to categorize by experimentation.
Our prototype implements these ideas by allowing the user to dispose his objects in 2D space. This arrangement may be manual, automated or both; it may be based on similarity, spatial proximity or both. A global gure may emerge from this arrangement, inuencing the browsing and also the extension of the collection. Local gures emerge, which are the temporary pseudo-classes illustrating the pre-categorization building process of collecting. The art gallery metaphor ts very well, as it adds further meaning to the arrangement of the collected items in space, and models the excess in collections thanks to the reserve.
Through exploiting space in this way, the software interface takes advantage of our cognitive abilities in dealing with spatial information, and also our ability to collect information and acquire knowledge. Our next step is experimentation in order to validate our work. This could simply take the form of a series of sessions in which both novice and experimented users are asked to build up collections using the software. Through user-feedback, we will have a rst idea of how well the interface is understood, how useful the users nd it and how easy it is to use. If this experiment is a success, as we believe it will be, we will continue our research and bring it to the next level. Through integrating new functionality focused on indierentiation for similarity/proximity, we will be able to build specic tools for a variety of applications in which the user’s activity may be - at least metaphorically - described as building a gural collection .