=Paper=
{{Paper
|id=None
|storemode=property
|title=A Cognitive Perspective on Emergent Semantics in Collaborative Tagging: The Basic Level Effect
|pdfUrl=https://ceur-ws.org/Vol-590/sasweb10_2.pdf
|volume=Vol-590
}}
==A Cognitive Perspective on Emergent Semantics in Collaborative Tagging: The Basic Level Effect==
https://ceur-ws.org/Vol-590/sasweb10_2.pdf
A Cognitive Perspective on Emergent Semantics in
Collaborative Tagging: The Basic Level Effect
Tobias Ley1,2, Paul Seitlinger2
1
Know-Center1, Inffeldgasse 21a, 8010 Graz, Austria,
tley@know-center.at
2
Cognitive Science Section, University of Graz, Universitätsplatz 2, 8010 Graz, Austria
paulchristian.seitlinger@edu.uni-graz.at
Abstract. Researching the emergence of semantics in social systems needs to
take into account how users process information in their cognitive system. We
report results of an experimental study in which we examined the interaction
between individual expertise and the basic level advantage in collaborative tag-
ging. The basic level advantage describes availability in memory of certain pre-
ferred levels of taxonomic abstraction when categorizing objects and has been
shown to vary with level of expertise. In the study, groups of students tagged
internet resources for a 10-week period. We measured the availability of tags in
memory with an association test and a relevance rating and found a basic level
advantage for tags from more general as opposed to specific levels of the tax-
onomy. An interaction with expertise also emerged. Contrary to our expecta-
tions, groups that spent less time to develop a shared understanding shifted to
more specific levels as compared to groups that spent more time on a topic. We
attribute this to impaired collaboration in the groups. We discuss implications
for personalized tag and resource recommendations.
Keywords: Tagging, Categorization, Personalized Recommendation
1 The Basic Level Effect in Collaborative Tagging
Emerging semantics in social systems is a topic that has sparked significant interest in
the research community. In collaborative tagging systems [5], for example, it has been
suggested that a community of users negotiates meaning in a collaborative sensemak-
ing process [4] that would lead to a stabilization of the used vocabulary over time [5].
Some have suggested that this process could be an alternative to the usually top down
driven engineering of ontologies [3] [8] [11]. An example for this is the Software
SOBOLEO (Social Bookmarking and Lightweight Engineering of Ontologies [1])
1 The Know-Center is funded within the Austrian COMET Program (Competence Centers for
Excellent Technologies) under the auspices of the Austrian Federal Ministry of Transport,
Innovation and Technology, the Austrian Federal Ministry of Economy, Family and Youth and
by the State of Styria. COMET is managed by the Austrian Research Promotion Agency FFG.
Work on this paper has been partially funded within the MATURE project (www.mature-ip.eu)
by the European Community.
that was used in our study. Here, users collaboratively tag bookmarks and then use the
tags to build a shared vocabulary and a taxonomic structure.
Our conjecture is that besides an understanding of the social (e.g. [2]) and prag-
matic processes (e.g. 6], it is equally important to understand the underlying cognitive
processes in collaborative tagging for offering effective recommendations. For exam-
ple, it has been shown that human categorization processes are highly variable and
adaptive. Categorization does, for instance, vary on the level of specificity depending
on a number of factors. Therefore, our intention with the study reported here is to look
at the temporal dynamics in the collaborative tagging environment both in terms of
the tagging activities and the associated cognitive processes over time. By doing so,
we would like to gain a better understanding of the variability in human categoriza-
tion as it can be observed in such an environment, and thereby enhance current per-
sonalized tag recommendation mechanisms provided both in the process of tagging
and in the process of browsing tag clouds and resource collections. This should en-
hance the emergence of stable patterns in these environments.
The term basic level advantage [10] has been introduced to describe a preferred
level of taxonomic abstraction when classifying objects of the real world (e.g. a pref-
erence for the term “dog” as opposed to “mammal” or “poodle”). In human commu-
nication, the basic level has an important role as it contains categories that are most
easily retrieved from memory and have a high degree of information value in describ-
ing objects. Among many others, an advantage for the basic level has been shown
when people verify the categories of pictures of objects [10], or in a free naming
paradigm [12]. While the role of the basic level advantage in collaborative tagging is
often acknowledged [1] [5] [6], surprisingly little empirical research exists to inform
design decisions. In their study of delicious, Golder & Huberman [5] suggest that
popular tags which are introduced very early for a certain bookmark correspond to
categories of the basic level. The authors also find that the tag distribution for a cer-
tain bookmark quickly stabilizes over time suggesting an emerging consensus.
The authors also point to a potential problem with the basic level advantage that
arises with differing levels of expertise. They hypothesize that there should be sys-
tematic variations across individuals of “what constitutes a basic level”. In collabora-
tive tagging, this basic level variation is a potential drawback. When resources are
described on varying levels of specificity, it makes retrieval of information more
difficult both for experts and for novices. While for the former, the information value
of a basic level category is too low, for the latter the specific categories are not suffi-
ciently well represented in memory, and, hence, their labels difficult to comprehend.
The hypothesized basic level variation is in line with cognitive research which has
found a basic level shift in various categorization paradigms, such as generating at-
tributes of category objects, free naming of category labels or verifying category
membership [9] [10] [12]. Basic level shift for more experienced persons leads to
better availability in memory of category members and their attributes on more spe-
cific levels of the taxonomy. Following sensemaking research, we expect that in a
collaborative tagging environment a growing expertise in the domain can be observed
over time. Therefore, we hypothesize that users will use more specific categories, will
show better availability of these in memory and will ascribe more importance to more
specific categories, when they collaboratively tag for a longer as compared to a
shorter duration of time.
2 An Experimental Study
To test this hypothesis, we asked four groups of students to collaboratively collect
bookmarks related to their course subject and describe them with tags. Two of the
groups had to work on a topic for the whole duration of the semester (10 weeks), the
other two groups switched their topic at half time. Our hypothesis was that the long
duration (ld) groups would form a stronger representation in memory of the more
specific tags and that they would rate their relevance higher than the short duration
(sd) groups. Collaborative tagging among the students was realized through the social
bookmarking system SOBOLEO. In SOBOLEO, the tags and the tag taxonomy that is
collaboratively created are shared by all users of the system.
2.1 Participants and Procedure
The study took place in the context of a university course on cognitive models in
technology enhanced learning at the University of Graz. Subjects (N=25, mean age
M=23.3, SD=1.2) were psychology students participating for course credit. After an
introduction to SOBOLEO, a computer literacy questionnaire and a word association
test eliciting participants’ knowledge about central concepts of the given topics were
administered to the participants. Subjects were then assigned to four groups of 6 or 7
participants which were equivalent according to their scores on the word association
test and computer literacy questionnaire. Each group was provided with their own
SOBOLEO instantiation only accessible by personal usernames and passwords.
E-mails were then sent out to inform the participants of the topic they had to work
on together with access details for their SOBOLEO environment. Two groups were
asked to research the topic “the use of Wikis in enterprises”, the other two groups “the
use of Weblogs in universities”. They were asked to prepare these topics as if they
were collaboratively working on a report of presentation. Both topics were chosen
because they were related to the course subject and because we expected the partici-
pants to have only little prior knowledge about them.
During the whole duration of the study (ten weeks) each student was expected to
post two relevant bookmarks per week to the SOBOLEO environment and describe
them with meaningful tags. The students were also required to collaboratively organ-
ize their tag collection with the help of the SOBOLEO taxonomy editor. To facilitate
the emergence of consensus, the students were also encouraged to utilize the SOBO-
LEO chat and an external discussion forum.
After five weeks (at halftime), the SOBOLEO environments of two of the four
groups were cleared. They had to start from scratch and to work on the other topic for
another five weeks, making them the short duration (sd) groups. The other two groups
continued with their prior topic, making them the long duration (ld) groups. Right
before this topic switch, we also controlled for the fact that there still were no differ-
ences between the two conditions in the word association test. At the end of the se-
mester, the association test and the relevance rating were administered to the 25 stu-
dents in a group setting using a sample of tags they had created so far.
2.3 Tag Samples, Tag Specificity and Dependent Measures
By the end of the 10-week period, the four groups had created N=213 tags from
which n=76 tags were drawn as a sample. To yield the independent variable tag speci-
ficity, tags were drawn from three different levels of the SOBOLEO taxonomies the
students had created: General tags were drawn from the taxonomy levels 1, medium
tags from level 2, and all tags below level 2 were allocated to the specific tags. From
each of the four SOBOLEO environments, 19 tags were randomly drawn: three gen-
eral (e.g. “weblogs”, “e-learning by collaborating”), eight medium (e.g. “kinds of
weblogs”, “psychology of weblogs”) and eight specific tags (e.g. “videoblogs”, “mi-
croblogging”). Hence, the entire sample consisted of 76 tags: 12 general, 32 medium
and 32 specific tags.
As a dependent measure, a relevance rating was collected at the end of the semes-
ter asking subjects to rate each tag sampled from their own SOBOLEO environment
on a five-point Likert scale ranging from strongly relevant to strongly irrelevant for
describing and organizing resources. By averaging the ratings of all group members a
mean relevance rating for each tag was obtained. An association test was also con-
ducted at the end of the semester. This test elicits implicit knowledge about concepts
underlying verbal representations. Subjects were confronted with tags as stimulus
words and asked to write down all associations coming to their mind. Response time
was confined to 30 seconds. By counting the number of associations, the test informs
about the strength of representation of concepts in memory. Stimulus words were the
same tags used for the relevance rating. Again we averaged the number of associa-
tions of all group members to obtain a mean number of associations for each tag.
3 Results
Figure 1 displays the mean number of associations (left) and the mean relevance
rating (right) as a function of tag specificity and duration obtained at the end of the
study. These results indicate a basic level advantage, i.e., a strong representation of
categories represented by general tags. Independent of duration, general tags at level 1
seem to evoke more associations (M=4.43, SD=0.21) than medium tags at level 2
(M=2.95, SD=0.13) and specific tags at level 3 (M=2.99, SD=0.13).
Secondly, a level - group interaction is emerging, but it is in the opposite direction
than we had expected. Contrary to our expectations, sd groups achieved more associa-
tions and higher relevance ratings than ld groups for medium and specific tags. This
was confirmed by a duration (ld and sd) × tag specificity (medium and specific) mul-
tivariate analyses (MANOVA) on the variables number of associations and relevance
rating. The main effect for duration proved highly significant (F2,58=9.82, p<.01)
explaining 25% of variance in the dependent variables and indicating a strong effect
∧
( f > .40 ). Neither the main effect tag specificity nor the interaction between duration
and tag specificity were significant. To further determine the nature of the significant
effect, two univariate analyses (ANOVAs) for each of the dependent variables were
conducted. Both results match our descriptive pattern. Averaging over medium and
specific tags, the sd groups achieve more associations (M=3.29, SD=0.61) than the ld
groups (M=2.65, SD=0.81; F1,59=13.01, p<.01). The same applies to the relevance
rating (F1,59=9.12, p<.001): the judged relevance of medium and specific tags is higher
in sd groups (M=2.56, SD=0.60) than in ld groups (M=2.22, SD=0.69).
Fig. 1. Number of associations (left) and relevance rating (right) for general, medium and
specific tags in long duration (10 weeks) and short duration (five weeks) groups
Results of a post-hoc questionnaire that had been administered to the students at
the end of the semester give insight into these counterintuitive findings. First, all
groups indicated they had been dissatisfied with the communication mechanisms (the
SOBOLEO Chat and discussion forum). Albeit having worked on their topic for a
longer time, groups of the ld condition gave significantly lower ratings when asked
for the understanding of the topic (M=1.67 on a 5-point Likert scale, SD=1.23) than
sd groups (M=2.69, SD=0.75; F1,23=6.44, p<.05). Additionally, ld groups (M=1.92,
SD=1.00) perceived a lower quality of their taxonomy than sd groups (M=2.92,
SD=0.86; F1,23=7.33, p<.05). Free text answers indicate that especially students in ld
groups found it more difficult to collaboratively work on the shared taxonomy in
SOBOLEO and they felt that the exercise had resulted in a chaotic collection of
bookmarks and tags where it was rather difficult to keep an overview.
4 Discussion and Outlook
We conclude from the study that a strong basic level effect could be observed for an
implicit memory measure (number of associations) as well as an explicit measure
(relevance rating), where for the latter this only showed for one of the groups. How-
ever, our manipulation (duration of engagement with a topic) was obviously not effec-
tive in producing a stronger representation in memory. Quite to the contrary, the fact
that environments of students in sd groups were cleared after half time actually helped
them to build a more effective and shared external knowledge representation. The
negative effect for ld groups was exacerbated by missing effective communication
mechanisms in the SOBOLEO system. Similarly, we assume that it was students from
sd groups that developed a more shared and stronger internal representation. If this
was the case, then there is clear evidence for a shift in the basic level. This already
showed after a comparatively little time (5 weeks), and produced a strong and also
practically significant effect (an increase of 0.64 associations on average).
Results of this study have practical significance for tag and resource recommenda-
tion in collaborative environments (e.g. [2]) as they suggest that effective tag recom-
mendations need to take tag specificity into account. Experts in a domain would bene-
fit from more specific tag recommendations or from recommendations of resources
with more specific tag assignments. The study also suggests that temporal dynamics
need to be taken into account where shifts in basic level already take place after a few
weeks of collaboration. Finally, in case tag specificity could be captured, this would
also have implications for user modelling as the level of expertise pertaining to a
certain topic could be derived for any user from his or her tag assignments.
A limitation of our results relates to the manual creation of the taxonomy by stu-
dents which extends the (normally flat) folksonomy by a taxonomic relation. For our
future work, we plan to draw on statistical approaches, such as [3] who found differ-
ent tag similarity measures (tag co-occurrence vs. distributional measures) to corre-
spond to different taxonomic relationships between tags. Moreover, these results seem
to be moderated by particular behavioural tendencies of users using the tagging sys-
tem [6].
References
1. Braun, S., Schmidt, A., Walter, A., Nagypal, G., Zacharias, V.: Ontology Maturing: a Col-
laborative Web 2.0 Approach to Ontology Engineering. In: 16th int’l WWW conference,
pp. 1-10 (2007).
2. Carmagnola, F., Vernero, F., Grillo, P.: SoNARS: A Social Networks-Based Algorithm for
Social Recommender Systems. In: 17th int’l UMAP conference, pp. 223-234 (2009).
3. Cattuto, C., Benz, D., Hotho, A., Stumme, G.: Semantic Grounding of Tag Relatedness in
Social Bookmarking Systems. In: 7th ISWC Conference, pp. 615-631 (2008).
4. Fu, Wai-Tat.: The microstructures of social tagging: a rational model. In: Proceedings of
the ACM 2008 conference on Computer supported cooperative work, pp.229-238 (2008).
5. Golder, S., Huberman, B.A. The Structure of Collaborative Tagging Systems. J. Informa-
tion Sciences 32, 198-208 (2006).
6. Körner, C., Benz, D., Hotho, A., Strohmaier, M., Stumme, G.: Stop Thinking, Start Tag-
ging: Tag Semantics Emerge From Collaborative Verbosity. In: 19th Int’l WWW Confer-
ence, ACM, New York, 2010.
7. Marlow, C., Naaman, M., Boyd, D., Davis, M: HT06, tagging paper, taxonomy, Flickr,
academic article, to read. In Proc. HYPERTEXT '06, ACM Press, 31-40 (2006).
8. Mika, P.: Ontologies Are Us: A Unified Model of Social Networks and Semantics. J. Web
Semantics: Science, Services and Agents on the World Wide Web 5, 5-15 (2007).
9. Rogers, T.T., Patterson, K.: Object categorization: reversals and explanations of the basic-
level advantage J. Experimental Psychology: General 136, 451-469 (2007).
10. Rosch, E., Mervis, C.B., Gray, W.D., Johnson, D.M., Boyes-Braem, P.: Basic objects in
natural categories. J. Cognitive Psychology 8, 382-439 (1976).
11. Specia, L., Motta, E.: Integrating Folksonomies with the Semantic Web. In: 4th ESWC2007
European Semantic Web Conference, pp. 624--639. Springer, Heidelberg (2007).
12. Tanaka, J.W., Taylor, M.: Object Categories and Expertise: Is the Basic Level in the Eye of
the Beholder? J. Cognitive Psychology 23, 457--482 (1991).