Since the late 1990's, digital repositories for learning purposes have gained ground. Such repositories with metadata and/or educational content have been set up on regional, national and international levels to offer digital learning resources for teachers and learners from K-12 to tertiary and vocational education [1,2]. Sharing, using and reusing the content are the main drivers of the learning object economy [3]. Participants of this economy are educational institutions, digital libraries & learning object repositories (LOR) and their diverse stake-holders such as managers, content providers, policy makers, educators and learners, each with their own needs, requirements and agendas. Users and the usage in the field of learning resource repositories and digital libraries have been studied by different means, such as using Web metrics [4,5,6], attention metadata [7], data mining techniques [8] and mixed and qualitative methods [9,10,11].
Social tags offer an interesting aspect to study learning resources, its metadata and how users interact with them. Tags, as opposed to conventional metadata description such as Learning Object Metadata (LOM) [12], are free, non-hierarchical keywords that end users associate with a digital artefact, for example a learning resource. Tags are formed by a triple of (user,item,tag). Tags and the resulting networks, folksonomies, are commonly modelled as tri-partite hypergraphs [13,14]. This ternary relational structure gives rise to the (item,user) relationship, which can be regarded as a parameter of the interaction between a user and a learning resource in question. By looking at the (user,tag) relation, tags can be regarded as part of user models that reflects user's interests and intentions. The full relational structure emphasises also the (item,tag) relations that allow tags to be part of describing the item that they are related to, in this case the learning resource. Additionally, the (item,tag) relation can also be extended to the whole metadata (e.g. LOM) that is used to describe the item, creating an additional relationship (tag,LOM). Figure 1 represents these relationships between a user, a learning resource, its metadata (LOM) and tags. The main interest in this study is to understand these relationships and their ramifications in Technology Enhanced Learning (TEL) and more specifically, for digital learning resources. In [15] a review of related work is given.
Learning Object Repositories (LOR) and digital libraries can be regarded as sociotechnological systems with complex combinations of people, content artefacts and technologies. A social tagging and bookmarking tool as a feature on a conventional LOR potentially adds a number of dynamical mechanisms in such system. The act of a user adding a tag to a resource, for example, can be regarded as a lower-level interaction on the portal that is executed on the basis of purely local information, e.g. the user has discovered a resource that is relevant to his information seeking task. This individual behaviour, however, also modifies its environment. The tag(s) added by the user now appear in the resource-related tagclouds and on the global tagcloud Tags and self-organisation in a multilingual educational context 3 creating patterns on the system level. This, in turn, has potent to modify the behaviour of other individuals, as they might be inclined to use the tag as a navigational aid or prompt for their own resource discovery process. Such phenomena is explained as stigmergy, it provides a general mechanism that relates individual and colony-level behaviours in the literature of social insects, first introduced by P. Grassé in 1959 [16], and for example, in Swarm Intelligence [17]. Swarm Intelligence is based on the idea that the design of adaptive, decentralised and robust artificial systems could be inspired by social insects (for self-organising applications in general, see [18]). Implementations of these ideas in technology enhanced learning exist, e.g. a collaborative filtering [19,20], designing lifelong learning networks [21], selforganising wayfinding support for lifelong learners [22], sequencing recommendations [23], and self-organising navigational support [24]. The grounding of these works relies in complexity theory [25,26]. By studying the behaviour of social insects such as ants, termites or certain wasps, the scientists have elicited three characteristics behind their success in carrying out complex tasks such as building a nest or finding a shortest route to a food source [27]. These are:
• Self-organisation (activities are neither centrally controlled nor locally supervised); • Flexibility (the colony can adapt to a changing environment); • Robustness (even when one or more individuals fail, the group can still perform its tasks). Self-organisation represents the idea that even if individuals follow simple rules, the resulting group behaviour can be surprising complex and effective. Selforganisagtion is explained as "a set of dynamical mechanisms whereby structures appear at the global level of a system from interactions among its lower-level components. The rules specifying these interactions are executed on the basis of purely local information, without reference to the global pattern, which is an emergent property of the system rather than a property imposed upon the system by an external ordering influence." [17, p.9]. According to the authors, the four basic ingredients of self-organisation are the following: 1. Positive feedback: simple behavioural "rules of thumb" that promote the creation of structures. An example of this is "recruitment" is by ants, i.e. when other ants start following a trail to a food source thanks to indirect interactions among insects. 2. Negative feedback counterbalances positive feedback and helps to stabilise the collective pattern. In the example of wayfinding among ants, this can be food source exhaustion, or competition between food sources. 3. Self-organisation (SO) relies on the amplification of fluctuations (e.g. random walks, errors). Randomness is often crucial since it enables the discovery of new solutions. An example of this is an ant that gets lost and finds a new, unexploited food sources. 4. Multiple interactions. SO generally requires a minimal density of mutually tolerant individuals who are able to make use of the results of their own activities as well as of others' activities. E.g. trail networks can self-organise and be used collectively if individuals use others' pheromone (a chemical substance that can be sensed by other ants).
A series of studies [28,29,30,31] has been conducted on a learning resource portal currently known as the Learning Resource Exchange, hereafter referred to as portal. The portal was developed by European Schoolnet and its partners in the MELT and Calibrate projects. A version of the LRE federation of repositories [32] was made available to a restricted number of schools with more than 30 000 open educational resources and nearly 90 000 assets from 19 content providers in Europe and elsewhere. These resources exist in different languages and conform to different national and local curricula. A common Learning Resource Exchange Application Profile [33] is used by content providers which make the use of classification keywords from the LRE Thesaurus mandatory [34]. This Thesaurus currently exists in 17 languages. The portal offers a social tagging tool, which allows users to add tags to resources so that they can easily find them later and share them with other users. In the following part, first a study on self-organisation aspects of a social tagging system is introduced. Then, the other two important aspects behind the success of social insects are studies, namely how the user-generated tags make the system more Tags and self-organisation in a multilingual educational context 5 flexible and robust. This part of the paper presents a trilogy of studies using empirical, behavioural data captured from log-files and users' attention metadata trails.
Attention metadata (e.g. how do users search, what do users click on, what do they bookmark) was collected from users on the portal and a model on users' search-playannotation behaviour was created, the logging schema is explained in details in [30].
The following observations were gathered:
• Users follow a simple rule: "Search resources using your preferred search method. When a resource is relevant, bookmark it with tag(s)". When a user discovers resources and provides annotations at the individual level, they are regarded as lower-level interactions that are executed on the basis of purely local information. These comprise 16% of all the actions on the portal (Figure 3). • This individual behaviour modifies the environment and creates spatiotemporal structures such as the Community browsing features, which are global patterns on the system level. These are tagclouds (e.g. global, resource-specific and personal ones) and lists of "most bookmarked resources". On average, 21% of users' search actions take advantage of these spatiotemporal structures. • Tagclouds are an example of the spatiotemporal structures which emerge as a result of self-organisation. When a tagcloud, for example, influences the behaviour of other individuals in discovering new resources and further tagging and rating them, this is considered as a sign of stigmergy. Bookmarks and ratings indicated in green boxes in Figure 3 show how 33% of all annotations were initiated through these structures creating an ongoing feedback loop in a self-organised system. • When other users start using these spatiotemporal structures as a social navigation aid, it can be understood as positive feedback to the system. This prompts convergence in the behaviour: it increases the frequency of use of the same resources and tags, and creates the emergence of patterns (e.g. "most bookmarked resources" and "top-used tags"). On average 29% of the all plays and 33% of annotations are generated through these structures. • Negative feedback is given to the system when a user, for example, does not find a relevant resource using a tag and thus chooses to use some other retrieval method. This is a control mechanism that counterbalances positive feedback in the system. • Amplification of fluctuations is a counter-measure against too much positive feedback, which can lead to 'suboptimal convergence' and kill innovation, result of which could be no new emerging behaviours. Discovery and annotations of new resources that have no previous annotations through "Explicit search" and "Browse by category" introduce new items to spatiotemporal structures, 67% of all annotations were produced this way. These annotations act as seeds from which new structures can nucleate and grow. • Multiple interactions (e.g. on search behaviour, clicks, annotations) from users, both authenticated and non-authenticated, are recorded on the back-end of the LOR using attention metadata schema designed for social discovery processes.
Individuals are able to make use of the results of their own activities (e.g. 2% of plays are generated by authenticated users as they replay the resources that they bookmarked), however, these emerging structures are also made available collectively to all the users which increases their use manifold (on average 28% of plays are generated through these structures). Lastly, we studied whether Social Information Retrieval strategies made users more efficient when discovering relevant learning resources. By Social Information Retrieval strategies we mean all the Community browsing features, and it also comprises the retrieved resources that contain user-generated Interest indicators. These are Interest indicators such as a rating on a scale 1 to 5 (1="of no use" to 5="very useful") or a bookmark with tags (called Favourites).
According the ideas of self-organisation, ants, for example, are attracted to the shorter path to a food source because of its higher concentration of "pheromone", a chemical that ants use to mark the path. Following the same logic, the users who are attracted by the annotations of other users should find the relevant resources with less effort. In [30] a measure for user's efficiency in finding relevant resources was defined. We showed that by taking advantage of the given SIR methods on the portal, the users spent less effort in finding relevant resources. The average efficiency ratio went down from 4.4:1 to 2.8:1, meaning that with SIR methods, 2.8 searches were needed to find one relevant resource. However, we were not able to show that by using Community browsing methods users were to discover more relevant crossboundary resources. By cross-boundary discovery we mean that the user and the learning resource discovered come from different countries, and/or that the content is in a language other than the user's mother tongue.
Following the idea of self-organisation and stigmergy, learning resources and their metadata on the one hand, and social tagging and its products, tags on the other hand, do not only create new ways to discover learning resources, but also create a learning resource metadata ecology. The term "metadata ecology" is used to mean the interrelation of conventional metadata and social tags, and their interaction with the Tags and self-organisation in a multilingual educational context 7 environment, which can be understood as the repository in the large sense (resources, their metadata, interfaces and underlying technology) and its community of users. In the following Section, the other two important factors in social insects success are studies, namely flexibility and robustness.
[27] describe flexibility of social insects as the capacity of a colony to adapt to a changing environment, and robustness meaning that even when one or more individuals fail, the group can still perform its task. Similarly, we are interested in these properties for the users of the learning resource portal on the one hand, and for the learning resource metadata ecology on the other hand, i.e. how tags can make the system of a learning resource portal more flexible and robust.
Studying the same portal we documented different user behaviour while interacting with the self-organised model such as ours. In this study [29], over a period of six months, empirical data from more than 200 users was gathered. We found that 33% of the users contributed tags, whereas 32% of users never contributed tags themselves, but used them for retrieval. Moreover, 35% of users did not interact with tags at all (Figure 3). Chi-Square test for these differences is significant (p< 0.001). We thus have seen that 59% users used the new emerging structures to discover resources, indicating that due to self-organisation on the portal, more flexible ways to access resources have been created. We can also argue that it is a robust system, as even if only 33% of users contribute tags, they are used by 59% of users for retrieval purposes.
For the resource discovery, we were interested whether all the tags were used in a similar way. Out of more than 3800 distinct tags, our logging analyses show that only 11% of the tags were clicked on and that they generated 2631 clicks. On average, each tag received 6.9 clicks; however, in reality, 20% of the top clicked tags generated 79.6% of the clickstream.
This led to study how the supply of tags in the system matches with the demand, i.e. how flexibly can the portal's offer to adapt to a chancing environment. A measure for "attractive tags" was introduced which compares the amount of clickstream on a tag against how many times it had been added to the system by teachers (i.e. supply). If the number is above one (1), it means that the tag has generated more clickstream than supply. This means that the tag is "attractive". If the number equals to one, it means that there is an equal amount of demand and supply, and below one indicates that there is supply, but no demand. We found that 21% of tags were "attractive" and 24% had an equal demand and supply. 55% of tags received less clicks than there were supply. Language-wise, within the "attractive" and "equal" tags, 28% are in another language than English. The flexibility of the tags to adapt to a chancing environment by accommodating users' demand was demonstrated in showing that 45% of tags attracted more or equal amount of demand than there was supply.
Moreover, we also studied the flexibility and robustness of the system from the point of view of interplay between social tags and conventional metadata, i.e. relationship (tag,LOM) [31]. Flexibility in this case can be regarded as the capacity of the metadata to adapt to a changing environment, and robustness can be interpreted meaning that even if one or more metadata elements of LOM fail, thanks to tags, the system can still perform its task, i.e. support teachers in discovering learning resources.
As the portal is made available to teachers from European countries and its interface is made available in multiple languages, it is normal that users tag in multiple languages. The tagging behaviour in a multilingual context is studied in [28]. Similarly to the previous study, we also found that users tag in multiple languages. In this study we found that 29% of the tags were in English, although a very few users had English as mother tongue. A medium correlation (r=0.57) was found between the language of the content and language of tags.
Tags and self-organisation in a multilingual educational context 9 Fig. 5. Attractive tags, i.e. the tags that proportionally received more clicks from users as opposed to tags that were added by users. The "wish list" of the users of an international learning resource portal.
We ran a database query against all the tags and the multilingual Thesaurus terms. We found that 11.3% of distinct user-generated tags exist in the LRE multilingual Thesaurus. We call these "Thesaurus tags", as they are end-user generated, but they also exist in the Thesaurus. The number of times "Thesaurus tags" were applied rises to 30.6% of all tags (i.e. the same tag added to many resources). On average, these tags were reused 11.8 times compared to other tags which were reused on average 2.4 times. It is interesting that, especially in a multilingual context, such a high percentage of overlap exists between natural language and controlled vocabularies. In [35] authors report that the folksonomy set overlapped with the indexer set on average 19.5%.
These "Thesaurus tags" by users can be used to improve the semantic interoperability of tags. First, they have a potential to be used as a "bridge" between existing descriptors and tags, and thus enhance the semantic interoperability within and across languages. One example is the resource "Change of State" in Figure 5, which has tags by endusers as well as the classification terms by the expert indexer. Table 1, on the other hand, shows the Thesaurus "descriptor 195" representing the concept of "chemistry" with its language equivalences. As we can now observe, the tag "kemia" is actually a "Thesaurus tag". Thanks to the multilingual Thesaurus, we can first of all recognise the similarity between a "Thesaurus tag" and the descriptor, and then assign properties to these tags from the Thesaurus, e.g. the tag "kemia" is related to the concept of "descriptor 195" and its language is Finnish. A similar idea of connecting tags to existing ontologies has been presented in [36], although the difference is that in our case, we use the resource and its existing descriptors as a proxy for the semantic link between the descriptor and tag, and that this process can be automated to take place at the back-end without being intrusive to the user.
The information gained from the link between the "Thesaurus tag" and descriptor can be used in various ways. It can be used, for example, in the tagcloud to show different translations of the tag "kemia". As for the retrieval purposes, the system could infer that other resources indexed with the "descriptor 195" are also relevant. Here, the user will get a chance to retrieve learning resources in multiple languages, thanks to the inter-language connection that the multilingual Thesaurus offers. Moreover, "Thesaurus tags" open up new options to navigate across multilingual resources, for example, a thematic multilingual tagcloud could be created by displaying all the tags that are added to resources which contain a given Thesaurus descriptor (e.g. tagcloud on physical sciences). Secondly, the "Thesaurus tags" can be suitable descriptors to be added to the original LOM description of the learning resource, particularly in cases where the original indexing has been poor or limited. In our example of "Change of State", we know from the Thesaurus hierarchies that the "descriptor 195" is a narrower term of the existing indexing term "physical sciences". As the "Thesaurus tag" narrows down the current classification of the learning resource in question, we can automatically add it as a new classification term for the resource.
Thirdly, the area of intra-language equivalence within the multilingual Thesaurus could be improved with tags, as in our evaluations they have been identified as a good source for non-descriptors [37]. A non-descriptor provides the intra-language equivalence that facilitates access to resources that are indexed by using the thesaurus terms that do not translate well to the language that the end-user uses. For example, the tag "efl" (= "English as foreign language") could be expressed in thesauri terms as "English language" + "foreign language". When the user types a text search "efl", not only tagged resources would be retrieved, but also the ones with the above descriptors. In this way the gap between natural language and controlled language could be reduced. The same could apply also for gathering better scope-notes, which deal with the meaning of terms and help the user to understand the term better. Especially in a multilingual context, where some differences occur from one language/culture to another, this feature is useful to understand cultural differences.
Tags and self-organisation in a multilingual educational context 11
Previous studies have shown that the reuse of learning resources is low [e.g. 6]. Barriers to reuse have been studied in [11], where the authors argue that a "repositorycentric perspective" of learning resource repositories create a barrier for the use and reuse of learning resources, as repositories are often introduced as a stand-alone tool to users. To improve the reuse, [6] show that improving even one of the steps in the reuse chain would improve the probability of reuse and therefore, the amount of reuse within the platform. The interplay between learning resource repositories is considered as a step in the reuse chain, and this study focuses on the mechanisms that could create such interplay. We studied the relationship of (tag, item) and how it can be used to create interplay between different contexts [31]. The research challenge was to demonstrate whether the end user generated tags can create cross-references between separate pieces of content that reside in heterogeneous content platforms in a multilingual context. We focused on tag-based interest structure on learning resources that teachers have tagged on a number of different educational platforms or tools. [38] argue that tag-based interest structures in social tagging systems are less segmented than item-based interest structures, which are typically used for social recommendation purposes.
To study the possibility of interplay more than 20 000 tag applications between five different educational resource platforms were collected (Calibrate, LeMill, OER Commons, LRE and delicious.com). We then analysed all the tag-item pairs from the datasets to find the overlap between tags in different services. The entire dataset comprises 21269 tag applications (Table 2). We found that 666 of the distinct tags (7.4% of all distinct tags) overlap at least in two out of five different tagging systems. They result in 6452 tag applications, which covers 30% of all the posts in our dataset. Using this tag-based interest structure, we can create an aggregated "cross-platform tagcloud". It filters 7.4% of all distinct tags and creates man-made bridges across two or more platforms taking advantage of the tag-based interest structure in an educational context. The idea of allowing users to access resources originating from different platforms through tags is complimentary to other forms of sharing learning resources and their metadata between repositories [32,39]. Our proposal of a "cross-platform tagcloud", though, introduces three new aspects. First, it builds on the social interactions among users in terms of co-construction of knowledge as tags, and secondly, it uses them as a way to offer interplay between learning resources platforms. Lastly, it introduces the idea of accessing both institutional resources (usually subject to some quality control within a closed information retrieval system) and private collections of resources from various sources. Such ideas are novel in the area of learning object repositories, where the de facto way of sharing resources is based on federating and harvesting metadata. Instead of accessing the entire set of "conventional" metadata, which can amount to thousands of resources (e.g. the LRE alone makes more than 35 000 resources available), "man-made" filters, i.e. tags, bridge between platforms and guide the user's choice of resources.
In this study we have focused on three different aspects of self-organisation and stigmergy on a learning resource platform with a social tagging feature. We have been able to show that the self-organisation aspect of a social tagging system helps users discover learning resources more efficiently. We also showed that both users and the metadata ecology benefited from the flexibility and robustness of such system.
Thanks to the triple (user,item,tag) when represented in a "cross-platform tagcloud", we have been able to show that content which comes from heterogeneous repositories that typically do not cross-reference each other via link-structures, has such cross-references. Therefore, the link-structures from our aggregated tagcloud opens more sophisticated avenues for resource discovery across contexts (e.g. repository, language, country, curriculum). Future work focusing on using these underlying connections to create measures of resources' importance will offer plenty of research challenges. Similarly to the Page-Rank algorithm [40], tags, creating underlying connections between seemingly random pieces of content in different languages (from repositories in different countries), rely on humans' subjective idea of their importance for a given information-seeking task. Using this new, emerging link-structure, and involving tags as "anchor texts", could offer totally new ways to "organise the world's learning resources and make them universally accessible and useful", similar to what Google claims its mission statement is for world's information. Moreover, resource's potential for crossing across different contexts could be detected from the same link-structure. Resources-specific tags, for example, that appear in many different languages could indicate that the resource is being used in different language contexts and thus has potential to be used across contexts. Similarly, resources with users from a number of different countries could indicate that these resources are being used in different country and curriculum contexts. Conversely, resources that have tags associated to them only in one language or only by users from the same country as the resource is, could be disregarded and given less importance for the across-context discovery.
Acknowledgments. Authors would like to thank the MELT (http://info.meltproject.eu) team supported by the European Commission. Especially warm thanks to Sylvia Hartinger for her endless help with log-files and tag exports, and to Frans Van Assche and Jim Ayre for all discussions and comments. Thanks to Matt for everything.
Tags and self-organisation in a multilingual educational context 13