=Paper=
{{Paper
|id=Vol-535/paper-3
|storemode=property
|title=Tags and Self-organisation in a Multilingual Educational Context
|pdfUrl=https://ceur-ws.org/Vol-535/Vuorikari_SIRTEL09.pdf
|volume=Vol-535
}}
==Tags and Self-organisation in a Multilingual Educational Context==
https://ceur-ws.org/Vol-535/Vuorikari_SIRTEL09.pdf
Tags and self-organisation
in a multilingual educational context
Riina Vuorikari1,2, Rob Koper2
1
European Schoolnet, Rue de Trèves, 61,
1040 Brussels, Belgium
{Riina Vuorikari}@eun.org
2
Open Universiteit Netherland
P.O. Box 2960, 6401 DL Heerlen (NL)
{Riina Vuorikari, Rob.Koper}@ou.nl
Abstract. Social tags offer a novel aspect to study learning resources, its
metadata and how users interact with them. This paper investigates the impact
of social tagging on the discovery of digital learning resources in a multilingual
context. The main hypothesis is that the self-organisation aspect of a social
tagging system helps users discover learning resources more efficiently and that
the user-generated tags make the system, which operates in a multilingual
context, more flexible and robust.
Keywords: Learning resource metadata, tags, self-organisation, discovery,
social information retrieval.
1 Introduction
Since the late 1990’s, digital repositories for learning purposes have gained ground.
Such repositories with metadata and/or educational content have been set up on
regional, national and international levels to offer digital learning resources for
teachers and learners from K-12 to tertiary and vocational education [1, 2]. Sharing,
using and reusing the content are the main drivers of the learning object economy [3].
Participants of this economy are educational institutions, digital libraries & learning
object repositories (LOR) and their diverse stake-holders such as managers, content
providers, policy makers, educators and learners, each with their own needs,
requirements and agendas. Users and the usage in the field of learning resource
repositories and digital libraries have been studied by different means, such as using
Web metrics [4, 5, 6], attention metadata [7], data mining techniques [8] and mixed
and qualitative methods [9, 10, 11].
Social tags offer an interesting aspect to study learning resources, its metadata and
how users interact with them. Tags, as opposed to conventional metadata description
such as Learning Object Metadata (LOM) [12], are free, non-hierarchical keywords
that end users associate with a digital artefact, for example a learning resource. Tags
are formed by a triple of (user,item,tag). Tags and the resulting networks,
2 Riina Vuorikari1,2, Rob Koper2
folksonomies, are commonly modelled as tri-partite hypergraphs [13, 14]. This
ternary relational structure gives rise to the (item,user) relationship, which can be
regarded as a parameter of the interaction between a user and a learning resource in
question. By looking at the (user,tag) relation, tags can be regarded as part of user
models that reflects user’s interests and intentions. The full relational structure
emphasises also the (item,tag) relations that allow tags to be part of describing the
item that they are related to, in this case the learning resource. Additionally, the
(item,tag) relation can also be extended to the whole metadata (e.g. LOM) that is used
to describe the item, creating an additional relationship (tag,LOM). Figure 1
represents these relationships between a user, a learning resource, its metadata (LOM)
and tags. The main interest in this study is to understand these relationships and their
ramifications in Technology Enhanced Learning (TEL) and more specifically, for
digital learning resources. In [15] a review of related work is given.
Fig. 1. Relational structures that emerge when a social tagging tool is introduced as a feature to
a conventional Learning Object Repository (LOR).
2. Self-organisation and social tagging
Learning Object Repositories (LOR) and digital libraries can be regarded as socio-
technological systems with complex combinations of people, content artefacts and
technologies. A social tagging and bookmarking tool as a feature on a conventional
LOR potentially adds a number of dynamical mechanisms in such system. The act of
a user adding a tag to a resource, for example, can be regarded as a lower-level
interaction on the portal that is executed on the basis of purely local information, e.g.
the user has discovered a resource that is relevant to his information seeking task.
This individual behaviour, however, also modifies its environment. The tag(s) added
by the user now appear in the resource-related tagclouds and on the global tagcloud
Tags and self-organisation
in a multilingual educational context 3
creating patterns on the system level. This, in turn, has potent to modify the behaviour
of other individuals, as they might be inclined to use the tag as a navigational aid or
prompt for their own resource discovery process. Such phenomena is explained as
stigmergy, it provides a general mechanism that relates individual and colony-level
behaviours in the literature of social insects, first introduced by P. Grassé in 1959
[16], and for example, in Swarm Intelligence [17]. Swarm Intelligence is based on the
idea that the design of adaptive, decentralised and robust artificial systems could be
inspired by social insects (for self-organising applications in general, see [18]).
Implementations of these ideas in technology enhanced learning exist, e.g. a
collaborative filtering [19, 20], designing lifelong learning networks [21], self-
organising wayfinding support for lifelong learners [22], sequencing
recommendations [23], and self-organising navigational support [24]. The grounding
of these works relies in complexity theory [25, 26].
By studying the behaviour of social insects such as ants, termites or certain wasps,
the scientists have elicited three characteristics behind their success in carrying out
complex tasks such as building a nest or finding a shortest route to a food source [27].
These are:
• Self-organisation (activities are neither centrally controlled nor locally
supervised);
• Flexibility (the colony can adapt to a changing environment);
• Robustness (even when one or more individuals fail, the group can still perform
its tasks).
Self-organisation represents the idea that even if individuals follow simple rules,
the resulting group behaviour can be surprising complex and effective. Self-
organisagtion is explained as “a set of dynamical mechanisms whereby structures
appear at the global level of a system from interactions among its lower-level
components. The rules specifying these interactions are executed on the basis of
purely local information, without reference to the global pattern, which is an emergent
property of the system rather than a property imposed upon the system by an external
ordering influence.” [17, p.9]. According to the authors, the four basic ingredients of
self-organisation are the following:
1. Positive feedback: simple behavioural “rules of thumb” that promote the creation
of structures. An example of this is “recruitment” is by ants, i.e. when other ants
start following a trail to a food source thanks to indirect interactions among
insects.
2. Negative feedback counterbalances positive feedback and helps to stabilise the
collective pattern. In the example of wayfinding among ants, this can be food
source exhaustion, or competition between food sources.
3. Self-organisation (SO) relies on the amplification of fluctuations (e.g. random
walks, errors). Randomness is often crucial since it enables the discovery of new
solutions. An example of this is an ant that gets lost and finds a new, unexploited
food sources.
4. Multiple interactions. SO generally requires a minimal density of mutually
tolerant individuals who are able to make use of the results of their own activities
as well as of others’ activities. E.g. trail networks can self-organise and be used
4 Riina Vuorikari1,2, Rob Koper2
collectively if individuals use others’ pheromone (a chemical substance that can
be sensed by other ants).
3 Studies on self-organisation in the context of multilingual
educational resource discovery
A series of studies [28, 29, 30, 31] has been conducted on a learning resource portal
currently known as the Learning Resource Exchange, hereafter referred to as portal.
The portal was developed by European Schoolnet and its partners in the MELT and
Calibrate projects. A version of the LRE federation of repositories [32] was made
available to a restricted number of schools with more than 30 000 open educational
resources and nearly 90 000 assets from 19 content providers in Europe and
elsewhere. These resources exist in different languages and conform to different
national and local curricula. A common Learning Resource Exchange Application
Profile [33] is used by content providers which make the use of classification
keywords from the LRE Thesaurus mandatory [34]. This Thesaurus currently exists in
17 languages. The portal offers a social tagging tool, which allows users to add tags to
resources so that they can easily find them later and share them with other users.
Fig. 2. The Learning Resource Exchange portal is available in different languages.
Figure 2 shows the front page of the portal. It offers different categories of
searches: “Explicit search” (text based and advanced search) and “Browse by
category” that take advantage of multilingual metadata. “Community browsing”, on
the other hand, takes advantage of the other users’ behaviour.
In the following part, first a study on self-organisation aspects of a social tagging
system is introduced. Then, the other two important aspects behind the success of
social insects are studies, namely how the user-generated tags make the system more
Tags and self-organisation
in a multilingual educational context 5
flexible and robust. This part of the paper presents a trilogy of studies using empirical,
behavioural data captured from log-files and users’ attention metadata trails.
3.1 Self-organisation
Attention metadata (e.g. how do users search, what do users click on, what do they
bookmark) was collected from users on the portal and a model on users’ search-play-
annotation behaviour was created, the logging schema is explained in details in [30].
The following observations were gathered:
• Users follow a simple rule: “Search resources using your preferred search method.
When a resource is relevant, bookmark it with tag(s)”. When a user discovers
resources and provides annotations at the individual level, they are regarded as
lower-level interactions that are executed on the basis of purely local information.
These comprise 16% of all the actions on the portal (Figure 3).
• This individual behaviour modifies the environment and creates spatiotemporal
structures such as the Community browsing features, which are global patterns on
the system level. These are tagclouds (e.g. global, resource-specific and personal
ones) and lists of “most bookmarked resources”. On average, 21% of users’ search
actions take advantage of these spatiotemporal structures.
• Tagclouds are an example of the spatiotemporal structures which emerge as a
result of self-organisation. When a tagcloud, for example, influences the behaviour
of other individuals in discovering new resources and further tagging and rating
them, this is considered as a sign of stigmergy. Bookmarks and ratings indicated in
green boxes in Figure 3 show how 33% of all annotations were initiated through
these structures creating an ongoing feedback loop in a self-organised system.
• When other users start using these spatiotemporal structures as a social navigation
aid, it can be understood as positive feedback to the system. This prompts
convergence in the behaviour: it increases the frequency of use of the same
resources and tags, and creates the emergence of patterns (e.g. “most bookmarked
resources” and “top-used tags”). On average 29% of the all plays and 33% of
annotations are generated through these structures.
• Negative feedback is given to the system when a user, for example, does not find a
relevant resource using a tag and thus chooses to use some other retrieval method.
This is a control mechanism that counterbalances positive feedback in the system.
• Amplification of fluctuations is a counter-measure against too much positive
feedback, which can lead to 'suboptimal convergence' and kill innovation, result of
which could be no new emerging behaviours. Discovery and annotations of new
resources that have no previous annotations through “Explicit search” and
“Browse by category” introduce new items to spatiotemporal structures, 67% of
all annotations were produced this way. These annotations act as seeds from which
new structures can nucleate and grow.
• Multiple interactions (e.g. on search behaviour, clicks, annotations) from users,
both authenticated and non-authenticated, are recorded on the back-end of the
LOR using attention metadata schema designed for social discovery processes.
6 Riina Vuorikari1,2, Rob Koper2
Individuals are able to make use of the results of their own activities (e.g. 2% of
plays are generated by authenticated users as they replay the resources that they
bookmarked), however, these emerging structures are also made available
collectively to all the users which increases their use manifold (on average 28% of
plays are generated through these structures).
Fig. 3. The search-play-annotate model of user behaviour on a learning resource portal
depicting the characteristics of self-organisation.
Lastly, we studied whether Social Information Retrieval strategies made users
more efficient when discovering relevant learning resources. By Social Information
Retrieval strategies we mean all the Community browsing features, and it also
comprises the retrieved resources that contain user-generated Interest indicators.
These are Interest indicators such as a rating on a scale 1 to 5 (1=”of no use” to
5=”very useful”) or a bookmark with tags (called Favourites).
According the ideas of self-organisation, ants, for example, are attracted to the
shorter path to a food source because of its higher concentration of “pheromone”, a
chemical that ants use to mark the path. Following the same logic, the users who are
attracted by the annotations of other users should find the relevant resources with less
effort. In [30] a measure for user’s efficiency in finding relevant resources was
defined. We showed that by taking advantage of the given SIR methods on the portal,
the users spent less effort in finding relevant resources. The average efficiency ratio
went down from 4.4:1 to 2.8:1, meaning that with SIR methods, 2.8 searches were
needed to find one relevant resource. However, we were not able to show that by
using Community browsing methods users were to discover more relevant cross-
boundary resources. By cross-boundary discovery we mean that the user and the
learning resource discovered come from different countries, and/or that the content is
in a language other than the user’s mother tongue.
Following the idea of self-organisation and stigmergy, learning resources and their
metadata on the one hand, and social tagging and its products, tags on the other hand,
do not only create new ways to discover learning resources, but also create a learning
resource metadata ecology. The term “metadata ecology” is used to mean the
interrelation of conventional metadata and social tags, and their interaction with the
Tags and self-organisation
in a multilingual educational context 7
environment, which can be understood as the repository in the large sense (resources,
their metadata, interfaces and underlying technology) and its community of users. In
the following Section, the other two important factors in social insects success are
studies, namely flexibility and robustness.
3.2 More flexible and robust system
[27] describe flexibility of social insects as the capacity of a colony to adapt to a
changing environment, and robustness meaning that even when one or more
individuals fail, the group can still perform its task. Similarly, we are interested in
these properties for the users of the learning resource portal on the one hand, and for
the learning resource metadata ecology on the other hand, i.e. how tags can make the
system of a learning resource portal more flexible and robust.
Flexibility and robustness regarding users
Studying the same portal we documented different user behaviour while interacting
with the self-organised model such as ours. In this study [29], over a period of six
months, empirical data from more than 200 users was gathered. We found that 33% of
the users contributed tags, whereas 32% of users never contributed tags themselves,
but used them for retrieval. Moreover, 35% of users did not interact with tags at all
(Figure 3). Chi-Square test for these differences is significant (p< 0.001). We thus
have seen that 59% users used the new emerging structures to discover resources,
indicating that due to self-organisation on the portal, more flexible ways to access
resources have been created. We can also argue that it is a robust system, as even if
only 33% of users contribute tags, they are used by 59% of users for retrieval
purposes.
For the resource discovery, we were interested whether all the tags were used in a
similar way. Out of more than 3800 distinct tags, our logging analyses show that only
11% of the tags were clicked on and that they generated 2631 clicks. On average,
each tag received 6.9 clicks; however, in reality, 20% of the top clicked tags
generated 79.6% of the clickstream.
This led to study how the supply of tags in the system matches with the demand,
i.e. how flexibly can the portal’s offer to adapt to a chancing environment. A measure
for “attractive tags” was introduced which compares the amount of clickstream on a
tag against how many times it had been added to the system by teachers (i.e. supply).
If the number is above one (1), it means that the tag has generated more clickstream
than supply. This means that the tag is “attractive”. If the number equals to one, it
means that there is an equal amount of demand and supply, and below one indicates
that there is supply, but no demand. We found that 21% of tags were “attractive” and
24% had an equal demand and supply. 55% of tags received less clicks than there
were supply. Language-wise, within the “attractive” and “equal” tags, 28% are in
another language than English. The flexibility of the tags to adapt to a chancing
8 Riina Vuorikari1,2, Rob Koper2
environment by accommodating users’ demand was demonstrated in showing that
45% of tags attracted more or equal amount of demand than there was supply.
Fig. 4. Different users interact differently with the social tagging system (n=234).
Flexibility and robustness regarding metadata ecology
Moreover, we also studied the flexibility and robustness of the system from the point
of view of interplay between social tags and conventional metadata, i.e. relationship
(tag,LOM) [31]. Flexibility in this case can be regarded as the capacity of the
metadata to adapt to a changing environment, and robustness can be interpreted
meaning that even if one or more metadata elements of LOM fail, thanks to tags, the
system can still perform its task, i.e. support teachers in discovering learning
resources.
As the portal is made available to teachers from European countries and its
interface is made available in multiple languages, it is normal that users tag in
multiple languages. The tagging behaviour in a multilingual context is studied in [28].
Similarly to the previous study, we also found that users tag in multiple languages. In
this study we found that 29% of the tags were in English, although a very few users
had English as mother tongue. A medium correlation (r=0.57) was found between the
language of the content and language of tags.
Tags and self-organisation
in a multilingual educational context 9
Fig. 5. Attractive tags, i.e. the tags that proportionally received more clicks from users as
opposed to tags that were added by users. The “wish list” of the users of an international
learning resource portal.
We ran a database query against all the tags and the multilingual Thesaurus terms.
We found that 11.3% of distinct user-generated tags exist in the LRE multilingual
Thesaurus. We call these “Thesaurus tags”, as they are end-user generated, but they
also exist in the Thesaurus. The number of times “Thesaurus tags” were applied rises
to 30.6% of all tags (i.e. the same tag added to many resources). On average, these
tags were reused 11.8 times compared to other tags which were reused on average 2.4
times. It is interesting that, especially in a multilingual context, such a high
percentage of overlap exists between natural language and controlled vocabularies. In
[35] authors report that the folksonomy set overlapped with the indexer set on average
19.5%.
These “Thesaurus tags” by users can be used to improve the semantic
interoperability of tags. First, they have a potential to be used as a “bridge” between
existing descriptors and tags, and thus enhance the semantic interoperability within
and across languages.
Fig. 5. Learning resource “Change of State” with tags (e.g. “kemia”) and indexing terms
“sciences” and “physical sciences” from the multilingual Thesaurus.
One example is the resource “Change of State” in Figure 5, which has tags by end-
users as well as the classification terms by the expert indexer. Table 1, on the other
hand, shows the Thesaurus “descriptor 195” representing the concept of “chemistry”
with its language equivalences. As we can now observe, the tag “kemia” is actually a
“Thesaurus tag”. Thanks to the multilingual Thesaurus, we can first of all recognise
the similarity between a “Thesaurus tag” and the descriptor, and then assign
10 Riina Vuorikari1,2, Rob Koper2
properties to these tags from the Thesaurus, e.g. the tag “kemia” is related to the
concept of “descriptor 195” and its language is Finnish. A similar idea of connecting
tags to existing ontologies has been presented in [36], although the difference is that
in our case, we use the resource and its existing descriptors as a proxy for the
semantic link between the descriptor and tag, and that this process can be automated
to take place at the back-end without being intrusive to the user.
The information gained from the link between the “Thesaurus tag” and descriptor
can be used in various ways. It can be used, for example, in the tagcloud to show
different translations of the tag “kemia”. As for the retrieval purposes, the system
could infer that other resources indexed with the “descriptor 195” are also relevant.
Here, the user will get a chance to retrieve learning resources in multiple languages,
thanks to the inter-language connection that the multilingual Thesaurus offers.
Moreover, “Thesaurus tags” open up new options to navigate across multilingual
resources, for example, a thematic multilingual tagcloud could be created by
displaying all the tags that are added to resources which contain a given Thesaurus
descriptor (e.g. tagcloud on physical sciences).
Table 1. Language equivalences for the Thesaurus “descriptor 195”, including also one user-
generated "Thesaurus tag" kemia.
Descriptor ID Language equivalences
195 Chemie fr
chemistry en
kemi sv
kemia (Thesaurus tag) fi
kémia hu
Secondly, the “Thesaurus tags” can be suitable descriptors to be added to the
original LOM description of the learning resource, particularly in cases where the
original indexing has been poor or limited. In our example of “Change of State”, we
know from the Thesaurus hierarchies that the “descriptor 195” is a narrower term of
the existing indexing term “physical sciences”. As the “Thesaurus tag” narrows down
the current classification of the learning resource in question, we can automatically
add it as a new classification term for the resource.
Thirdly, the area of intra-language equivalence within the multilingual Thesaurus
could be improved with tags, as in our evaluations they have been identified as a good
source for non-descriptors [37]. A non-descriptor provides the intra-language
equivalence that facilitates access to resources that are indexed by using the thesaurus
terms that do not translate well to the language that the end-user uses. For example,
the tag “efl” (= “English as foreign language”) could be expressed in thesauri terms as
“English language” + “foreign language”. When the user types a text search “efl”, not
only tagged resources would be retrieved, but also the ones with the above
descriptors. In this way the gap between natural language and controlled language
could be reduced. The same could apply also for gathering better scope-notes, which
deal with the meaning of terms and help the user to understand the term better.
Especially in a multilingual context, where some differences occur from one
language/culture to another, this feature is useful to understand cultural differences.
Tags and self-organisation
in a multilingual educational context 11
Flexibility and robustness regarding crossing contexts
Previous studies have shown that the reuse of learning resources is low [e.g. 6].
Barriers to reuse have been studied in [11], where the authors argue that a “repository-
centric perspective” of learning resource repositories create a barrier for the use and
reuse of learning resources, as repositories are often introduced as a stand-alone tool
to users. To improve the reuse, [6] show that improving even one of the steps in the
reuse chain would improve the probability of reuse and therefore, the amount of reuse
within the platform. The interplay between learning resource repositories is
considered as a step in the reuse chain, and this study focuses on the mechanisms that
could create such interplay.
We studied the relationship of (tag, item) and how it can be used to create interplay
between different contexts [31]. The research challenge was to demonstrate whether
the end user generated tags can create cross-references between separate pieces of
content that reside in heterogeneous content platforms in a multilingual context. We
focused on tag-based interest structure on learning resources that teachers have tagged
on a number of different educational platforms or tools. [38] argue that tag-based
interest structures in social tagging systems are less segmented than item-based
interest structures, which are typically used for social recommendation purposes.
To study the possibility of interplay more than 20 000 tag applications between
five different educational resource platforms were collected (Calibrate, LeMill, OER
Commons, LRE and delicious.com). We then analysed all the tag-item pairs from the
datasets to find the overlap between tags in different services. The entire dataset
comprises 21269 tag applications (Table 2). We found that 666 of the distinct tags
(7.4% of all distinct tags) overlap at least in two out of five different tagging systems.
They result in 6452 tag applications, which covers 30% of all the posts in our dataset.
Using this tag-based interest structure, we can create an aggregated “cross-platform
tagcloud”. It filters 7.4% of all distinct tags and creates man-made bridges across two
or more platforms taking advantage of the tag-based interest structure in an
educational context.
Table 2. Tags shared among five different tagging tools in an educational context.
Tags appear Distinct tags Applications % tag applications
Total in 2 or more
platforms 666 6452 30.3%
All tags 5 services 9036 21269
The idea of allowing users to access resources originating from different platforms
through tags is complimentary to other forms of sharing learning resources and their
metadata between repositories [32, 39]. Our proposal of a “cross-platform tagcloud”,
though, introduces three new aspects. First, it builds on the social interactions among
users in terms of co-construction of knowledge as tags, and secondly, it uses them as a
way to offer interplay between learning resources platforms. Lastly, it introduces the
idea of accessing both institutional resources (usually subject to some quality control
within a closed information retrieval system) and private collections of resources from
12 Riina Vuorikari1,2, Rob Koper2
various sources. Such ideas are novel in the area of learning object repositories, where
the de facto way of sharing resources is based on federating and harvesting metadata.
Instead of accessing the entire set of “conventional” metadata, which can amount to
thousands of resources (e.g. the LRE alone makes more than 35 000 resources
available), “man-made” filters, i.e. tags, bridge between platforms and guide the
user’s choice of resources.
4 Conclusion and future work
In this study we have focused on three different aspects of self-organisation and
stigmergy on a learning resource platform with a social tagging feature. We have been
able to show that the self-organisation aspect of a social tagging system helps users
discover learning resources more efficiently. We also showed that both users and the
metadata ecology benefited from the flexibility and robustness of such system.
Thanks to the triple (user,item,tag) when represented in a “cross-platform
tagcloud”, we have been able to show that content which comes from heterogeneous
repositories that typically do not cross-reference each other via link-structures, has
such cross-references. Therefore, the link-structures from our aggregated tagcloud
opens more sophisticated avenues for resource discovery across contexts (e.g.
repository, language, country, curriculum). Future work focusing on using these
underlying connections to create measures of resources’ importance will offer plenty
of research challenges. Similarly to the Page-Rank algorithm [40], tags, creating
underlying connections between seemingly random pieces of content in different
languages (from repositories in different countries), rely on humans’ subjective idea
of their importance for a given information-seeking task. Using this new, emerging
link-structure, and involving tags as “anchor texts”, could offer totally new ways to
“organise the world's learning resources and make them universally accessible and
useful”, similar to what Google claims its mission statement is for world’s
information. Moreover, resource’s potential for crossing across different contexts
could be detected from the same link-structure. Resources-specific tags, for example,
that appear in many different languages could indicate that the resource is being used
in different language contexts and thus has potential to be used across contexts.
Similarly, resources with users from a number of different countries could indicate
that these resources are being used in different country and curriculum contexts.
Conversely, resources that have tags associated to them only in one language or only
by users from the same country as the resource is, could be disregarded and given less
importance for the across-context discovery.
Acknowledgments. Authors would like to thank the MELT (http://info.melt-
project.eu) team supported by the European Commission. Especially warm thanks to
Sylvia Hartinger for her endless help with log-files and tag exports, and to Frans Van
Assche and Jim Ayre for all discussions and comments. Thanks to Matt for
everything.
Tags and self-organisation
in a multilingual educational context 13
References
1. Tzikopoulos, A., Manouselis, N., & Vuorikari, R. (2007). An overview of Learning Object
Repositories. In P. Northrup (Ed.), Learning Objects for Instruction, Design and Evaluation
(pp. 29-55). Information Science Publishing.
2. McGreal, R. (2008). A Typology of Learning Object Repositories. In H. Adelsberger,
Kinshuk, J. Pawlowski, & D. Sampson (Eds.), Handbook on Information Technologies for
Education and Training (pp. 5-28). Retrieved April 27, 2009, from
http://dx.doi.org/10.1007/978-3-540-74155-8_1.
3. Campbell, L. (2003). Engaging with the learning object economy. In A. Littlejohn (Ed.),
Reusing Online Resources: A Sustainable Approach to E-Learning (pp. 35-45). Kogan
Page, Limited.
4. Farooq, U., Kannampallil, T. G., Song, Y., Ganoe, C. H., Carroll, J. M., & Giles, L. (2007).
Evaluating tagging behavior in social bookmarking systems: metrics and design heuristics.
In Proceedings of the 2007 international ACM conference on Supporting group work (pp.
351-360). Sanibel Island, Florida, USA: ACM. doi: 10.1145/1316624.1316677.
5. Khoo, M., Pagano, J., Washington, A. L., Recker, M., Palmer, B., & Donahue, R. A.
(2008). Using web metrics to analyze digital libraries. In Proceedings of the 8th
ACM/IEEE-CS joint conference on Digital libraries (pp. 375-384). Pittsburgh PA, PA,
USA: ACM. doi: 10.1145/1378889.1378956.
6. Ochoa, X. (2008). Learnometrics: Metrics for Learning Objects. PhD dissertation.
KULeuven.
7. Wolpers, M., Najjar, J., Verbert, K., & Duval, E. (2007). Tracking actual usage: the
attention metadata approach. Educational Technology and Society Journal, Special Issue on
"Advanced Technologies for Life-Long Learning, 106.
8. Romero, C., & Ventura, S. (2007). Educational data mining: A survey from 1995 to 2005.
Expert Systems with Applications, 33(1), 135-146. doi: 10.1016/j.eswa.2006.04.005.
9. Harley, D., Henke, S., & Nasatir, D. (2006). Use and Users of Digital Resources: A Focus
on Undergraduate Education in the Humanities and Social Sciences . Digital Resource
Study. Center for Studies in Higher Education, UC Berkeley.
10. Petrides, L., Nguyen, L., Jimes, C., & Karaglani, A. (2008). Open educational resources:
inquiring into author use and reuse. Int. J. Technol. Enhanc. Learn., 1(1/2), 98-117.
11. Margaryan, A., & Littlejohn, A. (2008). Repositories and communities at cross-purposes:
issues in sharing and reuse of digital learning resources. Journal of Computer Assisted
Learning, 24(4), 333-347. doi: 10.1111/j.1365-2729.2007.00267.x.
12. IEEE LOM. (2002). Standard for Learning Object Metadata.
13. Marlow, C., Naaman, M., Boyd, D., & Davis, M. (2006). HT06, tagging paper, taxonomy,
Flickr, academic article, to read. In Proceedings of the seventeenth conference on Hypertext
and hypermedia (pp. 31-40). Odense, Denmark: ACM. doi: 10.1145/1149941.1149949.
14. Cattuto, C., Schmitz, C., Baldassarri, A., Servedio, V. D. P., Loreto, V., Hotho, A., et al.
15. Vuorikari, R., & Berendt, B. (2009). Study on contexts in tracking usage and attention
metadata in multilingual Technology Enhanced Learning. In Exploitation of Usage and
Attention Metadata (EUAM 09), Lecture Notes in Informatics (LNI). Lübeck, Germany:
Gesellschaft für Informatik.
16. Grassé, P. (1959). La reconstruction du nid et des coordinations inter-individuelles chez
Bellicositermes matalensis et Cubitermes sp. La théorie de la stigmérgie:: essai
d’interprétation des termites constructeurs.”. Insectes Sociaux, 6, 41-83.
17. Bonabeau, E., Dorigo, M., & Theraulaz, G. (1999). Swarm intelligence (p. 307). Oxford
University Press US.
14 Riina Vuorikari1,2, Rob Koper2
18. Mostefaoui, S., Rana, O. F., Foukia, N., Hassas, S., Di Marzo Serugendo, G., Van Aart, C.,
et al. (2003). Self-organising applications: a survey. In Engineering Self-Organising
Apllications, First International workshop. Melbourne, Australia.
19. Dron, J., Mitchell, R., Siviter, P., & Boyne, C. (2000). CoFIND - an experiment in N-
dimensional collaborative filtering. Journal of Network and Computer Applications, 23(2),
131-142.
20. Dron, J. (2004). Termites in the Schoolhouse: Stigmergy and Transactional Distance in an
E-learning Environment. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World
Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp.
263-269). Chesapeake, VA: AACE. Retrieved July 12, 2009, from
http://www.editlib.org/p/12942.
21. Koper, R., Rusman, E., & Sloep, P. (2005). Effective Learning Networks. Lifelong learning
in Europe, 1, 18-27.
22. Tattersall, C., Manderveld, J., van den Berg, B., van Es, R., Janssen, J., & Koper, R. (2005).
Self Organising Wayfinding Support for Lifelong Learners. Education and Information
Technologies, 10(1), 111-123. doi: 10.1007/s10639-005-6750-9.
23. Van den Berg, B., Tattersall, C., Janssen, J., Brouns, F., Kurvers, H., & Koper, R. (2005).
Swarm-based Sequencing Recommendations in E-learning. International Journal of
Computer Science & Applications, 3(3), 1-11. doi: http://hdl.handle.net/1820/510.
24. Janssen, J., Tattersall, C., Waterink, W., Berg, B. V. D., Es, R. V., Bolman, C., et al.
(2007). Self-organising navigational support in lifelong learning: How predecessors can
lead the way. Comput. Educ., 49(3), 781-793.
25. Kauffman, S. (1995). At Home in the Universe: The Search for the Laws of Self-
Organization and Complexity. Oxford: Oxford University Press. Retrieved July 5, 2009,
from http://www.amazon.com/gp/product/0195111303/ref=pd_luc_sim_01_01.
26. Waldrop, M. (1992). Complexity: The Emerging Science at the Edge of Order and Chaos.
New York: Simons & Schuster.
27. Bonabeau, E., & Meyer, C. (2001). Swarm Intelligence: A Whole New Way to Think
About Business. Harward Business Review.
28. Vuorikari, R., & Ochoa, X. (2009). Exploratory Analysis of the Main Characteristics of
Tags and Tagging of Educational Resources in a Multi-lingual Context. Journal of Digital
Information, 10(2).
29. Vuorikari, R., Sillaots, M., Panzavolta, S., & Koper, R. (2009). Are tags from Mars and
descriptors from Venus? A study on the ecology of educational resource metadata. In
Advances in Web-Based Learning - ICWL 2009, Lecture Notes in Computer Science
(Vol. 5686). Springer.
30. Vuorikari, R., & Koper, R. (2009). Ecology of social search for learning resources.
Campus-Wide Information Systems, 26(4). doi: 10.1108/10650740910946783.
31. Vuorikari, R., Poldoja, H., & Koper, R. (in press). Comparison of educational tagging
systems – any chances of interplay? IJTEL. Retrieved May 6, 2009, from
http://dspace.ou.nl/handle/1820/1798.
32. Massart, D. (2009). Towards a pan-European learning resource exchange infrastructure. In
Proceedings of the 7th conference on Next Generation Information Technologies and
Systems (NGITS’2009). Haifa, Israel.
33. Learning Resource Exchange LOM Application Profile. (2007). European Schoolnet.
Retrieved April 27, 2009, from http://fire.eun.org/.
34. Learning Resource Exchange Thesaurus. (2002). Project deliverable, European Schoolnet.
Retrieved from
http://insight.eun.org/ww/en/pub/insight/interoperability/learning_resource_exchange/meta
data.htm.
Tags and self-organisation
in a multilingual educational context 15
35. Al-Khalifa, H. S., & Davis, H. C. (2006). Folksonomies versus Automatic Keyword
Extraction: An Empirical Study. IADIS International Journal on Computer Science and
Information Systems (IJCSIS), 1(2), 132-143.
36. Bateman, S., Brooks, C., & McCalla, G. (2006). Collaborative Tagging Approaches for
Ontological Metadata in Adaptive E-Learning Systems. In 4th International Workshop on
Applications of Semantic Web Technologies for E-Learning (SW-EL 05) (pp. 3-12).
Dublin, Ireland. Retrieved May 6, 2009, from http://www.cs.usask.ca/~cab938/.
37. Vuorikari, R., Ayre, J. (2009). Melt deliverable 5.2. European Schoolnet, Brussels.
Accessible: http://info.melt-project.eu/ww/en/pub/melt_project/project_deliverables.htm
38. Santos-Neto, E., Condon, D., Nazareno, A., & Ripeanu, M. (2009). Individual and Social
Behavior in Tagging Systems. In 20th ACM Conference on Hypertext and Hypermedia.
Torino, Italy.
39. Ternier, S., Massart, D., Campi, A., Guinea, S., Ceri, S., & Duval, E. (2008).
Interoperability for Searching Learning Object Repositories, The ProLearn Query Language
.D-Lib Magazine, 14(1/2).
40. Brin, S., & Page, L. (1998). The Anatomy of a Large-Scale Hypertextual Web Search
Engine. COMPUTER NETWORKS AND ISDN SYSTEMS, 30, 107-117.