Introduction
The popularity of Social Tagging has rapidly increased since 2004 [
52, 62
]. Social
Tagging is mostly known from private application context. Once evolved from Social
Bookmarking Service (delicious)4, social tagging systems have become one of the
bestknown web-based [
42
] social software applications. One reason for its ongoing
4 http:// delicious.com
popularity is their simplicity and ease of use: Everybody can be a tagger, who interacts
collaboratively, or collectively with other taggers in a web-based community for the
purpose to administer, describe, share, structure and maintain several kinds of digital
objects, e.g. pictures (Flickr5), videos (YouTube6) or WebPages (delicious) using
selfcreated keywords called tags [
42
]. Taggers do not have to obey any rules or a bound to
a controlled vocabulary; so, social tagging systems are anarchic decentralized social
indexing systems as well.
Apart from private usage social tagging has also been applied for corporate purposes,
e.g. to support and facilitate customers‟ navigation, e.g. product search (Amazon7) by
means of product-related tags created by the customers themselves. Meanwhile, it is
also possible to use social tagging systems to describe and augment personal profiles
using tags. This special form of social tagging is also meant as people-tagging [
13, 15,
16
]. So far, it has been used for both private and corporate communities e.g.
organizations to gather, acquire and retrieve characterizing tags [
16, 46
].
Moreover, social tagging systems have also become relevant for e-HRM tasks. So far
there already exist few approaches which mainly focus on augmenting employees‟
profiles with characterizing tags [
13
] contributed by the employees themselves. First
promising results have already shown that social tagging systems are useful to facilitate
the corporate search for knowledge management, to discover employees‟ connections
and support the expert finding, e.g. IBM Lotus connections8 [
12, 13
]. Another approach
combines people-tagging and ontology maturing to support competence management
mainly focused on augmenting competency models with tags [
6
].
However, the applicability of social tagging for competence management still seems not
to be exhausted. Social Tagging systems might also be a promising method to support
competency acquisition, which belongs to the main functions of competency
management as well. It pursues the purpose to provide reliable and valid personal
related information, gathered by means of measuring, observing and describing
methods. Having both kind of information it gets possible to align required job-related
target-competencies with actual competences. Alignment results are for instance needed
in human resource management to schedule and control e-HRM tasks, such as strategy,
planning, acquisition, requirement, deployment and development.
Although previous researches confirmed social tagging systems enable the provision of
characterizing tags; there is need of further research to detect systematically appropriate
variants of social tagging systems to support competency acquisition. Further it still
lacks evidence if they are also able to ensure the provisioning of reliable and valid
information [
5
]. Hence, we focus on design characteristics and quality of those systems
from the classical testing theories‟ point of view. In short, the following questions are
answered:
Q1: Which possibilities do social tagging systems offer to gather competency
related information?
5 http://www.flickr.com/
6 http://www.youtube.com/
7 http://www.amazon.com/gp/tagging/cloud?redirect=true
8 http://www-01.ibm.com/software/de/lotus/wdocs/connection/
Q2: Do they ensure the provision of reliable and valid information?
To answer the first question Chapter 2 regards several variants of social tagging systems
from an external view and to filter appropriate ones. In a second step the internal view
focuses on the dimensions social tagging systems consists of, and presents a conceptual
framework to detect possible design characteristics to gather competency-related
information.
To answer the second question Chapter 3 introduces the quality criteria of the classical
testing theory, on which in Chapter 4 the social tagging systems are analyzed if they are
able to ensure reliable and valid data information. Difficulties and benefits social
tagging systems offer are discussed. Chapter 5 summarizes the results and gives an
outlook on future research.
Conceptual Framework
Social tagging systems (social classification systems [
57
], collaborative tagging systems
[
62
]) is a collective term that comprises different system variants. It is at present still
unclear if every social tagging system version is appropriate to support competency
acquisition. Hence, we give a short overview on existing system versions detecting
appropriate social tagging system variants.
1.1
Social tagging systems – System variants (External view)
Social Tagging Systems allow a categorization from different perspectives. Firstly, the
stability distinct between closed and open systems. In open systems the taggers
fluctuation is very high, because the taggers are not bound to the system (delicious);
whereas in closed systems the same taggers stay for a longer time and the tagger group
remains stable (IBM lotus connections) [
10, 13
]. Secondly, the taggers‟ transparency
within the system separates transparent systems from anonymously ones. In the former
taggers use their real names, whereas in the latter taggers act anonymously and hide
their identity using “nicknames”. Thirdly, social tagging systems, based on their entry
barriers, can be split in systems with minor entry barriers [
43
] and systems with major
entry barriers. Fourthly, their purpose separates privately used from corporate ones.
System variants
Open Systems are wide spread mainly in private usage. Everybody can become a
member of such an open community, because taggers just have to sign in with their
email address, first name and last name. It is the taggers decision to use real names or
fictive ones, so transparency cannot be ensured. They are not bound to the system;
hence, fluctuation level is high. Further, there also exist open social tagging systems
which are used for corporate purpose, e.g. Amazon. The entry barrier is higher than the
first variant, because only customers are allowed to tag, who are transparent for the
company, because of their customer profile. Closed social tagging systems can be found
in both private and corporate application [
58
]. In private application context Collabio
[
4
] has to be mentioned a “Facebook”9 application which allows persons to be
characterized by other persons with the help of tags in a playful way. Entry barrier is the
profile owner decides who is allowed to tag e.g. friends or colleagues. Taggers are
consequently transparent to the profile owner and other taggers. Those taggers are
bound to the system for reasons of social reputation, consequently their fluctuation is
low.
For the context of competence acquisition which takes places in a corporate
environment a closed social tagging system is required. The opportunity to tag is
restricted to a special tagger group, the organizational members. Taggers interact
transparently within the system, and can be identified by their personal ID and real
names as well. The entrance to those systems is bound to the employment contract that
limit and regulate the period tagger belongs to the systems and how long they obtain a
special role and job. Normally, there is low fluctuation within a closed organization.
Hence, a corporate purpose is given, so we narrow the variety of all social tagging
systems for this paper to closed ones that provide a high transparency of the tagger, low
fluctuation and the corporate purpose as well. In the next step we focus on the elements
social tagging systems consist of from an internal view.
9 http://apps.facebook.com/collabio/
1.2
Social Tagging Systems – Dimensions (Internal view)
Previous external view has narrowed the number of several social tagging systems to
special closed ones with special attributes. Now, we have also to narrow the number of
tagger, digital objects and tags, which are required for competency acquisition.
Social Tagging Systems consist of three related dimensions: tagger, digital objects and
tags [
9, 42
]. Taggers are the persons who interact within the closed community. They
obtain several roles at the same time, e.g. they are producer and consumer of their tags
[
55
]. Digital objects are the resources to tag, and tags are used to describe, augment and
structure several kinds of them; thereby the same tag can be added to one or many
objects.
In context of competency acquisition the variety of those dimensions is restricted.
Taggers get additional attributes; they are organizational members, employees,
superiors, subordinates and work mates as well. Further, not every digital object is
needed to be tagged. We just focus on competency profiles [
6
]. Finally, we regard only
competency related information as special content [
63
] so the tags are also narrowed to
those which contain and competency-related information. However, this seems not to be
enough to acquire all facets competency acquisition is composed of. Further filtering
views on the limited closed social tagging systems and its dimensions seems be
sufficient to define them more detailed. So each single element is regarded in the
following from several sub dimensions that originate from competency acquisition.
In detail, the profile dimension focuses on appropriate profile types and several
characteristics of transparency [
16
]. The tagger dimension focuses on taggers rights
with the closed system, hierarchical level, and taggers‟ perspectives. Further the number
of taggers, their incentive to contribute, their independence and visibility is regarded.
Finally, the tag dimension has a focus on suggested tags, permitted tag-types, number of
equal and different tags for single taggers, the acquisition of a temporal dimension,
weighted tags, scope of tags, granularity of tags, tag structuring and font size. All
dimensions, sub dimensions and combinable characteristics are composed in the figure
below and presented in detail in subsequent paragraphs.
Conceptual Framework - Internal View
Dimensions Subdimensions
Profiles
Tagger
Tags
Type
Transparency
Rights
Hierarchcal level
Perspective
Number
Incentive
Independence
Visibility
Sugesstions
Characteristics
Individual
Transparent
Use
Create
Equal
Self
Single
Voluntary
Given
Transparent
Given
Delete
Job
Non Transparent
Change
Unequal
Others
Multiple
Compulsory
Not Given
Anonymous
Not Given
Tag Types
Use of the same Tag
Number
of
different
Unlimited
Taegmsporal dimension
Weight
Scope
Granularity
Structure
Fontsize
Unrestricted
Single
Given
Given
Professional
Predefined
Given
Equal
Restricted
Actual competency profiles reflect individuals‟ (e.g. employees‟) competency stock,
whereas target competency profiles point out required job related competencies [
11
]. A
comparison of both helps to detect competency gaps [
11
]. Based on this information,
measures of personnel requirement planning, recruiting or training can be adopted.
Consequently, the alignment of actual and target competencies represents a main
function of competency acquisition. Hence, social tagging systems might support both
competency profiles, individuals „and job related ones.
Transparency (Transparent, Non-Transparent)
Individuals „competence is sensitive personal-related information; therefore, a selected
group of experts has to acquire and assess individuals‟ competence. A transparency of
individuals‟ competency profiles, contributed tags and tag creators for all tagger seems
to be debatable against the background of data protection. Some people-tagging systems
[
13, 15
] just allow a transparent view on individuals profiles [
42
], provided that the
profile owner and participating tagger agree with that. However, research results show
taggers tend to a non-transparent, private solution [
49
] when it refers to the tags
attached to their profile. Thereby it is for the tagger to decide on who and how many
taggers are allowed to have a look on their profile.
1.2.2
Tagger
Rights (Create, Use, Change, Delete)
Taggers obtain several rights and roles in social tagging systems [
62
]. Taggers are
consequently entitled to create and use tags for profile description. Due to the context of
competency acquisition and in particular the augmentation of individuals‟ profiles
taggers have to obtain additional rights, e.g. changing or deleting tags, if they are
inappropriate or false [
13, 49
]. These rights might also be helpful to eliminate obsolete
tags keeping the profiles up to date.
Hierarchical level (Equal, Unequal)
A specialty of social tagging systems is that all taggers are treated equally; everybody
can tag and there are no hierarchical differences [
53
]. However, for the purpose of
competency acquisition taggers action is embedded into a closed organizational system,
where taggers from several hierarchical levels interact. Every hierarchical level also
reflects a special power of decision and expertise [
35
], e.g. not every organizational
member is currently allowed to assess and ascertain competencies. Mutually tagging
already exists in social tagging systems; however mutual assessing within an
organization is possible but not common [
39
]. Hence, social tagging allows mutual
tagging over several hierarchy levels where all taggers are considered unequal or
without the hierarchical restriction, where all taggers are considered equal.
Perspective (Self, Other)
Self-description and assessment by others represent two well-known aptitude testing
methods [
6
], which are often used for competency acquisition and assessment. Social
Tagging also offers taggers the opportunity to describe their own profiles (both
jobrelated and personal-related) as well as foreign ones [
15
]. Social Tagging seems to be
particularly suitable and accepted by taggers in purpose of self-assessment and
selfreference as previous research results show [
14, 21, 22, 42, 46, 64
]. For the reason that
taggers can have both points of view social tagging systems offer both perspectives:
tagging themselves or others [
3
].
Number (Single, Multiple)
There are several methods to acquire competencies. One of them is the single-appraisal
such as a self-description or the single appraisal by the supervisor [
37, 39
]. A
comparison of both represents a common method to gather competency-related
information. Apart from those methods there are further methods that include the
appraisal of multiple raters, which differ from each other by their perspectives [
39
].
Social Tagging Systems also allows a single tagger to describe profiles and a
description of the same profile by a group of tagger from several perspectives as well
[
61
]. So the number of tagger can vary between one (single -assessment) and many
(multiple appraisal).
Incentives (Voluntary, Compulsory)
Social Tagging Systems base on the principle of voluntary participation of taggers. This
principle has led to a high acceptance [
43
]. But Social Tagging System can only then be
effective when a minimum of tags and profiles is given. More important become the
taggers‟ incentives. So, the question is if the competence acquisition should be carried
out through social tagging systems on a volunteer or compulsory base. Research
findings show that taggers can be split by their motivation [
13, 42, 56
]. So, for instance,
some of them tag for their own sake or for the sake the others [
13
]. Some tag for reasons
of self-presentation [
42, 64
] or just to store tags [
15, 50, 51
]. Further motivation has also
been detected in the users need to be a part of a community. However, compulsory
incentives have also been detected [
6
], e.g. Social pressure can also be a reason why
taggers tag to get not excluded from the community [
8
]. So, voluntary and compulsory
incentives can be distinguished. Both are relevant for competency assessment, because
in closed social tagging systems with corporate purposes a contribution of tags serves
predominantly corporate and non-private purposes, for which voluntary contribution or
commitment cannot be ensured [
33
].
Independence (Given, Not Given)
Social tagging is based on the principle of collaborative object description and taggers
interexchange. While the description of a profile just by one single tagger might be time
consuming and incomplete, social tagging systems use taggers collaborative
participation to get multiple perspectives and a broad description. However, taggers in
most cases do not acts independent from each other. It is more like a transparence and
mutual influence between them [
50
]. They swap tags as ideas through a transparent
visualization in order to collect multiple descriptions, synonyms or alternative
descriptions.
Those can be improved if taggers are influenced by tags from others, and an internal
vocabulary evolves and gets more stable over time [
36
]. However, there are few
approaches in which taggers act independent from each other in order to filter the best
describing tags for a digital object [
61
]. So, one can decide for dependence or
independence over the taggers, but due to the requirements of data protection
independence over the taggers shall be recommended.
Visibility (Transparent, Anonym)
Once a tag has been added to an object, its source cannot be traced anymore. In most
cases the tag creator remains invisible. Because of its collaborative sharing character
tags become common property [
50
] and can be reused by other taggers, which means
one tag can be related with many taggers. However, to avoid inappropriate tags or tag
spam [
38
] it might become important to identify the tagger who causes the false tag
[
49
]. Two cases remain disputable. The first is the transparency and visibility of the
related tagger for all others, the second is an anonymous solution, where the tagger
remains invisible for reason of data protection and to ensure taggers freedom of opinion.
1.2.3
Tags
Suggestions (Given, Not Given)
A vexed characteristic for social tagging systems is the taggers‟ free choice of
vocabulary without being bound to controlled vocabularies. Ambiguity of language has
often been discussed as a main disadvantage of social tagging systems [
26,
53
].Therefore, existing approaches tend to get social tagging more structured and
recommend to support taggers vocabulary choice by suggested tags [
18, 42,63
]. There
are already several approaches to suggest tags, e.g. previously used tags [
13
], tags with
similar spell, frequently used ones or the latest ones. Each kind of suggestion aims to
reduce the ambiguity of language [
18, 53, 59
]. It is the taggers‟ choice to accept the
suggest tags or ignore them. Hence, suggestions can be given or not.
Types (Unrestricted, Restricted)
Apart from suggestions a variety of tag-types results from the ambiguity of language
[
21, 22
]. However, it still lacks a complete categorization. We can roughly separate
single-word-tags from compounded-tags [
59
] as well as objective tags from subjective
ones [
34
]. But there are also tag-types only made for the tagger himself to retrieve its
own information, which are meaningless for other taggers. For the purpose of
competency acquisition just tag-types that contain competency related information are
required, not all tag types can be used [
5
]. Hence, two cases remain to decide for, a
reduction of allowed tag-types [
63
] or tagging without constraints.
Use of the same tags (Single, Multiple)
Normally, competency acquisition methods include a scale to measure the degree each
competence has got or is required. However, in social tagging systems a rating scale is
missing [
31, 55
]. Social rating systems represent another social software category. To
express the importance a tag has got for one profile, taggers sometimes use the same tag
multiple times. But in some social tagging systems the multiple use of the same tag for
one tagger counts once even if it is added twice or multiple times to the same object. A
reuse of tags by one tagger might also a method to express the importance the tag has
got for the profile or the degree to which a competency is given or needed.
Number of different tags (Unlimited, Limited)
In social tagging systems taggers are not limited in the number of tags they add to an
object [
22
]. It is the taggers‟ choice how many tags he or she wants to contribute, that‟s
why the number of tags varies among the tagger. Using social tagging systems for
competency acquisition regulations to determine the number of tags might be necessary
to avoid an assessment bias within a profile and tag spam [
38
]. However, regulations
towards a fixed number of tags might also cause spam [
38
] tags, when taggers just add
tag because they have to. Hence one has to decide for a limited or unlimited number of
different tags.
Temporal dimension (Given, Not Given)
Some social tagging systems consist of more than three elements; some also include a
temporal dimension to acquire the date a tag occurs for the first time or every time it has
been changed. With the additional temporal dimension changes over the time could be
measured. The additional temporal dimension might also useful for competency
acquisition to depict changes in the profiles over time [
13
].
Weight (Equal, Unequal)
Another specialty of social tagging systems is the equality in weight every tag has
within the system. This is related with the equal treatment of each tagger. However, a
weighted tag might be a method to underline the tags‟ importance [
63
], e.g. the
superiors‟ tags might be more important for individuals profiles than the subordinates‟
one. Otherwise latest added tags might be more important than those which were added
a year ago. So, one can decide for unequally or equally weighted tags.
Scope (Professional Competence, Personal Competence)
Current people tagging approaches allow taggers to describe profiles in an unstructured
manner [
13, 28
]. Focusing on competency acquisition there are several dimensions
respectively facets competence consists of [
24, 37, 60
]. According to the DQR
competence can be subdivided in two sections: professional and personal competence
[19]. Hence, social tagging might cover the whole scope or just one of both sections
[
60
].
Granularity (Predefined, Not Predefined)
Apart from the dimensions competence consists of there are also differences
considering the granularity a competence is ascertained. The more granular a
competency is acquired the easier it is to align actual and target competencies.
However, ambiguity of language and the absence of rules in social tagging systems
allow every hierarchical level [
63
] within the tags [
10
]. To gather more accurate
information we suggest predefining granular levels. Alternatively to this suggestion a
non predefined characteristic is also possible.
Structure (Given, Not Given)
Unlike taxonomies, where terms are clearly kept in strict mono- hierarchical
parentchild relations in social tagging systems each tagger can create his or her own structure
[
43, 55
]. In most cases those structures are individual and cannot be matched with
others. Tags can also been aggregated, e.g. compounded as tag bundles [
42, 53, 59
].
Further there exist approaches, which recommend a predefined structure, e.g. by means
of given metadata, to get tags more accurate [
1
]. Another research recommends
predefined facets or dimensions, where tags can be sub ordered [
48
]. In order to get a
more aggregated view on competencies a given structure through predefined facets or
metadata might be helpful. However it depends on number and quality of facets or
metadata whether this is effective [
1
]. So, one can decide for an integrated structure or a
structure less characteristic.
Font size (Equal, Unequal))
Finally, tag visualization as one of the main social tagging characteristics remains to
discuss. There are various ways to visualize tags. Tags can be ordered as tag-clouds or
they can be listed both horizontally and vertically. It‟s further on possible to order them
either alphabetically, or semantically, or visualize them as unordered [
27
]. Each
visualization aims at supporting social navigation [
38, 50, 52
]. Apart from structure the
font sized can vary. In most cases a big font size represents a high frequency [
30
],
upto-datedness or occurrence. But it is also possible to visualize all tags in an equal font
size.
2
Theoretical Foundation - Quality Criteria
Competency acquisition deals with measuring, elicitation and collection of competency
related information; thereby, it gets usually supported by electronically diagnostic
instruments for aptitude testing. Creating a scientific fundament each instrument used
for competency acquisition requires a theoretical foundation that in most cases is based
on a measuring theory. Each theory or model has several assumptions and is founded to
quality criteria which have to be observed. Competence is a construct of aptitude testing
[
60
]. Normally, methods to measure competencies for competency acquisition are
evaluated by quality criteria, particularly reliability and validity that result from
classical testing theory [
25
]. So, classical testing theory seems to be an appropriate
theory to evaluate social tagging systems. In the following assumptions of classical
testing theory are presented in short.
2.1
Assumptions
Classical testing bases on three main assumptions (axioms) and additional assumptions
as well. All of them concern the measuring process and in particular the measured
values [
44
]. Firstly, the existence axiom declares that the real value exists as the
expected value of a measurement. Secondly, the connectivity axiom implies that each
measured value consists of both a true and an error value. It implies each measurement
is defected with errors. Thirdly, the independence axiom precludes that there is
dependence between error values and true values among several persons. It is
additionally assumed that error values within a person are not related. Further
independence over participating raters is assumed [
25
]. Classical testing theory mainly
considers the quality of a measurement or elicitation from the measurement errors point
of view; therefore it is also called “measurement error theory”. Measurement errors or
error values can be subdivided in coincidental errors and systematical errors [
23
]. The
former results from internal and external influences a person is affected with. They
appear infrequently and are non predictable. The latter appears in pattern and results
from errors within the theoretical or empirical measuring model. According to the
classical testing theory error values result from the lack of quality. Thereby the degree
of quality can be estimated with quality criteria, in particular, reliability and validity
which will be subsequently introduced.
2.1.1
Reliability
Reliability is „ (…) the degree of accuracy a procedure has with regard to the
characteristic to be measured.“ [35, p. 250][
29
]. It is also a degree for the stability a
measuring instrument has got. [
44
]. Reliability requires two temporal distanced
measurements which have to congruent in their measured values. So, reliability is a
measure how prone a measuring method is for coincidental error values. Getting reliable
results or measured values rules, regulations, norms and structures are necessary
required.
The estimation of reliability embraces stability over time (re-test reliability), internal
stability and at last the agreement over several raters in their interpretation of measured
values. In detail re-test reliability can be estimated if there are two measurements at
different times in which the same group uses the same method to measure or ascertain
the same thing. A special kind of re-test reliability is the intra-rater reliability, a measure
for the stability of measured values within one rater over time [
2
]. Estimating internal
consistency it requires a measuring method to be divisible into many equal small
measurements, which count as a measurement for their own [
29
]. If all measurements
deliver the same measured values, internal consistency is given. Thirdly,
inter-raterreliability can be estimated if a fixed group of rater is ensured, who interpret the same
measured value independent from each other in the same way. It implies a measuring
method to be clear and accurate so that coincident interpretation errors can be
minimized.
2.2
Validity
Validity is a measure how trustworthy, complete and valid a measuring method is. It is
given if a measuring method exactly measures what it is supposed to do and nothing
else. Validity can be assumed as given if theoretical argument and empirical results
underline this. Validity represents further a measure, how prone a measuring method is
from both systematical and coincidental error values [
2
].
There are several ways to estimate validity. Firstly, content validity is a measure if the
applied measuring method actually measures the whole content respectively every facet
of the regarded construct. It implies that a construct is measurable and a fixed definition
exists which contains every facets the construct exits of it [
60
]. To estimate or test the
content validity normally experts are interviewed, respectively, consulted. This is what
we call face validity, which is given if the majority of experts agree with the definition.
Secondly, criterion validity is a measure to what extent measured values match with
present or future external criterion. Simultaneously or present comparison is regarded
by concurrent validity, whereas predictive validity focuses on delayed comparisons.
Thirdly, construct validity is given if actually the contrast is measured and nothing else.
To put that in our perspective construct validity is given if all measured values refer to
competence and not to intelligence. Convergence validity is given if several measuring
methods get the same measured values. Discriminate validity is given if measuring
different constructs provide different measured values.
3
Discussion
Based on conceptual and theoretical assumptions we answer the question if social
tagging systems, as limited before, are able to ensure the provision of reliable and valid
competency related information. In the following at the first step we examine to what
extent social tagging systems ensure the measuring of both quality criteria. At the
second step we show up the sources of coincidental and systematical errors and give
recommendations to minimize them referring to our conceptual framework.
Estimating intra- rater reliability or re-test reliability a temporal dimension is required to
compare tags within one tagger over time. Further, it has to be ensured that a tagger
remains for a fixed period within a social tagging system. It is necessary that at least
two measurements at different times can be done to compare changes within the tags.
This is only to ensure in closed systems, because in open ones there is nothing that
bounds a tagger to a system, whereas in closed systems the employment contract
regulates the length of a period a tagger is bound to the organization. It still lacks
research if competency related tags within one tagger are stable over time. Tags
represent the taggers vocabulary, which develops over time as well as the tagger‟s
personality and competence. Although previous findings show that some patterns of
stability in the taggers vocabulary choice and spelling exist [
50
], it remains unclear if
their understanding of the tag content remains also the same.
However, just the fact that the taggers belong to the organization does not guarantee
their contribution yet. Previous findings show taggers can be distinguished in power
user respectively normal taggers who tag frequently and lurkers who tag infrequently or
just profit of existing information [
50, 51
]. This aggravates the validation of re-rest
reliability and intra-rater-reliability for all taggers.
Currently, there is just a voluntary incentive; taggers are free to contribute tags driven
by their own motivation. A compulsory incentive for now does not exist. However, if
taggers are not expected to contribute, re-test reliability respectively
intra-raterreliability might be hardly to estimate. But a compulsory incentive might also increase
the spam tag [
63
] because taggers just tag because they have to.
Estimating internal consistency implies that the measurement method, e.g. social
tagging systems can be divided into many smaller measurements which measure the
construct competence in an equal manner. Social tagging systems consist of multiple
taggers, profiles and tags and it is possible to form smaller social tagging systems within
one system. However, there are differences between each single tagger, profile and tags,
so equality of each dimension is hardly to attest. For instance, there are several tag types
which vary in their accurateness, objectivity and content. Not all of them are appropriate
to ascertain competencies [
5
]. Further, spam tags exist [
38, 54
] that does not contain any
relevant content. So consistency within the measurement method is hardly to attest.
Estimating inter-rater-reliability implies that at least two taggers tag the same profile. In
social tagging systems multiple taggers describe the same digital object in a
collaborative manner. However, the number of tagger varies depending on the digital
object, respectively, profile. It is possible that just one or all taggers describe a profile,
because there are no rules that restrict a minimum of taggers. Referring to the
conceptual approach there are several possibilities to estimate inter-rater-reliability
exist, e.g. estimating inter-rater-reliability over several perspectives (self-assessment
and foreign appraisal) [
15
], several hierarchical level or within one hierarchical level
can be done.
Secondly, inter-rater-reliability requires congruence over taggers‟ interpretation of the
same measured value. In social tagging systems there are neither rules nor regulations
concerning the vocabulary choice. This causes the ambiguity of language social tagging
systems are known for, in particular tags are imprecise [
26
]. Synonyms, homonyms,
abbreviations and so on are not excluded because taggers are free in their vocabulary
choice. However, taggers differ from each other in their linguistic power of expression,
cognitive talents and domain knowledge [
17
]. Hence, ambiguity can be avoided neither
in the spelling, nor in the understanding or sense a tag has [
31, 53, 54
]. For example,
although an individual‟s profile is tagged multiple times by different taggers with
„leadership ability“, inter-rater-reliability cannot be attested at all because each tagger
might have its own understanding, what “leadership ability” is [
16, 45
]. Tags are
consequently not accurate to interpret, in particular single-word tags offer polysemy
[
3,57
]. But can be used as a start tag which can be augmented with more specific tags.
Further there tags with multiple or additional hidden meanings that are just
understandable for a special group, called socio-semantic tags [
34, 57
]. Apart from
objective tags there subject related ones [
32
], which can just be understood or
interpreted by the tagger himself [
6, 13, 32, 51, 53
]. So, we recommend limiting the
tagtypes to those tag types which are clear for all taggers. Problems to interpret tags in an
accurate manner might further result from different granularities within the tags.
Finegranular tags might be more accurate than large-grained ones, e.g. the tag „C++“ is
more defining than “computerlanguage”. We recommend a predefined granularity level;
however, it needs further research to detect which granularity-level is the best.
Independence from the participating taggers for each tagger has to be observed for
competency acquisition and assessment procedures [
35
] according to the German
requirements for proficiency assessment procedures and classical testing theory as well.
In detail, it concerns the processes of acquisition, interpretation and evaluation of
competency related information respectively tags. It is especially relevant for personal
related information and individual‟s competency profile description due to the fact that
competencies are sensitive personal-related data. However, social tagging systems
follow the principle of collaborative tag sharing and a mutual transparency of all
contributed tags [
13, 15, 16
]. So it requires a special characteristic as we presented in
the conceptual framework.
Assuming each tagger tags independent from others the same profile, multiple single
descriptions of one profile are given to estimate their inter-rater reliability. In this
combination multiple tagger are involved in a collective way. To ensure independence
the transparency we recommend to keep profiles and the foreign related tags
nontransparent for the tagger that he or she just see own contributed tags [
61
].Thus
coincidental errors that result from external influences might be minimized.
Nevertheless, coincidental errors result from both influence sources externals and
internals as well. This has been scarcely confirmed in [
30, 50
], which show taggers are
influenced by their own subjective point of view and other taggers‟ influence as well.
Internal coincidental errors result from the taggers‟ current temper and personality. So,
every tagger is influenced by its own idiosyncratic subjective point of view [
20
].
Socially desired tags cannot be avoided at all [
6
]. Otherwise, tags can consciously been
avoided [
16
], e.g. to strip other taggers a special competence which tagger do not have
themselves or just to save another tagger to be not connected with an expertise they
want not to related with [
14
]. Underestimation or overestimation of their own or foreign
competencies might occur as well [
23
].
However, classical testing theory assumes the true values are the expected values, so the
mean of multiple measure values might compensate error values. Social tagging
systems already use the wisdom of multiple taggers to describe the same object. The
congruence over several taggers in his or her profile description helps to detect relevant
tags. Research results from social indexing show, taggers agree on core terms [
60
],
which are mostly defining for a digital object. Nevertheless, even if tags just appear
once ,they can be valuable [
4
].Therefore, social tagging seems to be an appropriate
method to minimize internal coincidental error values. The more taggers are involved,
the more objective a profile description might become [
36
], it might also be helpful to
improve inter indexer inconsistency [
21, 22, 30
]. Further subjective coincidental errors
in competency acquisition could be reduced [
23
].
However non-transparent solutions act against the collaborative character social tagging
systems has. Hence, we recommend a transparent solution, in which a tagger is
influenced by external criteria (foreign tags) but all taggers interact invisibly with each
just other over their tags. Thereby, all tags are in an equal font size to avoid halo effects
[
23, 27
]. So, every tagger gets more objective information, foreign tags work as
suggestions and it is the taggers decision to use the same tags or create new ones. So,
taggers might be inspired by other tags to find additional rich deep information. It
requires further research whether this approach observes the requirements for
proficiency assessment and data protection.
Finally, due to the accuracy and trustworthiness [
35
] to estimate inter-rater-reliability it
requires equal expertise or domain knowledge for all taggers. In social tagging systems
an expertise is not required [
17, 43, 52
], each tagger is allowed to participate [
51
]
However, it requires empirical research to test if non-experts tags are less defining than
experts [
31, 51
].Taggers differ in their expertise and knowledge, especially if we
assume that they are from several hierarchical levels. But every single tagger has got
special domain knowledge and might contribute hidden but important competency
related information [
5, 31
]. For example, a tagger who has no special expertise in
competency acquisition might know in detail which competencies his job requires. On
the other hand, work mates might also know each other from another perspective than
superiors or subordinates do, similarly to the multiple-rater-assessments. So, we
recommend reaching equality in the taggers expertise as required to weight tags
corresponding with the hierarchical level or by the distance a tagger has to the profile
owner or job.
In sum, it remains debatable if social tagging systems provide hard, reliable and
accurate competency-related information. But using the recommended characteristics
coincidental error sources might be minimized, which required further research.
However, rich, deep competency-related information from many different perspectives
can be gained [
4, 7
]. Social tagging systems seem to be appropriate to detect hidden
information [
5
], which is hardly to collect over all facets with current methods in such a
simple manner. Especially for self-description they seem to be an appropriate method
[
16, 46,50
] because taggers can describe their own competencies in their own words as
detailed as required.
3.2
Estimating content validity it requires a fix measuring model that defines all facets
dimensions competence consists of, in a special granularity [
2
]. However, social tagging
systems do not provide any guidelines or definitions. They rather aim at the collection
of all possible descriptions a digital object or construct might have. Social Tagging
systems are foreign from controlled vocabularies, in which a single group of experts
defines what competence is, the facets it consist of and how granular it is to ascertain.
Instead of consulting experts to evaluate the face validity, social tagging systems use the
collective knowledge of taggers to get a broad, rich and extensive definition. So content
validity in social tagging systems is not based on a fixed definition but it is rather a
continuous evolutionary defining process. This procedure is already used in
combination with ontologies to augment competency models supported by employee‟s
commitment [
6, 33
]. So, content validity is difficult to estimate. Estimating criterion
validity needs external criteria, e.g. empirically measured values to which tags can be
compared with. This could be difficult to prove because competence is hardly to
measure directly [
6
]. Hence, the estimation of concurrent and predictive validity
requires further research. Estimating construct validity implies that, firstly, the construct
is measurable and it, secondly, can be clearly distinguished from other constructs [
2
].
This seems to be difficult for multifaceted constructs such as competence [
24
], because
there are several definitions and in part overlapping understandings what competence is
[
37,41
]. Competence is a latent construct that consist depending on the situation of more
or less facets [
2, 41
]. The harder it is to ascertain all facets with one measurement
method [
2
]. But competencies are everywhere to detect, so each tag might be able to
ascertain a small facet of competence [
6
]. To estimate convergent validity we
recommend comparing tags with existing definitions and measured values gained by
other conventional methods. If they are congruent concurrent validity is given.
Estimating discriminate validity requires additional social tagging systems that ascertain
other construct profiles with the same tagger. Both require further research.
4