If non-technical end users are to contribute to the Web of Data as they have to the Web of Documents, they must employ tools that enable them to do so. This challenge is not easy to meet, as formal knowledge representation is a daunting task for the uninitiated. Indeed, we have empirically observed that expressing anything but the most straightforward of facts in RDF-compatible format is extremely difficult for newcomers to do reliably. This paper reports on a controlled experiment in which novices attempted to use a prototype Linked Data interface to both find and encode bits of everyday knowledge. The application presents a user-friendly veneer to the Semantic Web, manifesting the essential graph-based nature of the data model while shielding the user from the complexity of syntax. This allows us to study user behavior in attacking the deep, cognitive problem: breaking down knowledge into the triple-based structure required by RDF Linked Data. Our study sheds light on some of the key aspects of knowledge formulation that novices struggle with, and suggests several specific design approaches for Linked Data authoring environments that our experiment makes clear beneficially address crucial issues.
A successful, global-scale Semantic Web presupposes large
amounts of instance data available for machines to process. As
Tom Mitchell summarized during his ISWC 2009 keynote
address[
One might suppose that the only major impediment to (1) is
convincing the masses that they have an incentive to do this. But
in addition to the issue of motivation, serious questions arise
about novices' ability to generate Linked Data in the format
required by the Semantic Web. Formal knowledge
representation is difficult and error-prone for most non-technical
people. It is a very different activity from writing in natural
language, which is the way that most laypeople have contributed
to the Web to date. Authoring Linked Data demands an
unswervingly consistent naming scheme, an unprecedented level
of exactitude, fluency with a new suite of concepts, and an
adherence to a set of rigid and (to the layperson) seemingly
arbitrary rules that run counter to the way most people think, let
alone converse. Though some psychologists (e.g., [
We believe that for non-specialists to be successful in contributing to the Web of Data, they must use tools designed to compensate for their weaknesses. The design of such tools should be informed by empirical studies that illuminate how target users actually go about generating Linked Data, so that strengths can be maximized, weaknesses complemented, and unfruitful trends redirected.
The immediate goal of the work presented in this paper is not so much to design the ultimate Linked Data authoring environment as to empirically verify which aspects of such environments might be beneficial or harmful. By studying user behavior under simulated conditions, and observing which specific aspects of the Linked Data authoring process prove to be obstacles, we illuminate the nature of the problem and offer experimentally driven guidance on how to make end users successful. The remainder of this paper is organized as follows. First, we describe related work in user studies of knowledge formulation processes and tools. Then, we introduce OKM1, the prototype Linked Data authoring tool used in our experiments, highlighting key features whose viability we focused on in our study. We then describe the nature of our usability experiment, and present and interpret a quantitative analysis of the results. Finally, we summarize our findings and make generalizations and recommendations for future interfaces to Linked Data applications. 1 OKM is a recursive acryonym which stands for “OKM Knowledge Management,” and is pronounced as “Occam.” The prototype application is open-source and publicly accessible at http://sourceforge.net/projects/okm.
A wide array of tools have appeared in the last several years to
help users in the RDF generation process. These include
everything from semantic wikis (e.g., Platypus[
We mention here two notable efforts which did include
illuminating usability studies. One was conducted by Staab et
al.[
The setting we explore is more reminiscent of Staab et al.'s study, since we are focusing on laypeople (not domain experts) who are tasked with formulating generalized, open-ended knowledge. Our work differs from each of these efforts in that we are examining the effect and usage of specific user interface features, with the goal of discovering how a general Linked Data editor would best be designed. In particular, we analyze user behavior in choosing resources versus literals to represent information, the efficacy of employing types and templates in the interface to steer users towards data consistency, and alternative ways to express n-ary relations. None of these UI aspects has, to our knowledge, been empirically studied in a focused, experimental setting.
OKM’s primary purpose is to serve as a testing bed for analyzing how laypeople interact with Linked Data tools, and its basic design is common to many state-of-the-art RDF and ontology editors. This commonality is key in relating OKM to tools currently in use by the Semantic Web community; with it, we hope to generalize the results we obtain from empirical testing to Linked Data authoring as a whole.
For instance, like OntoWiki[
Users can add datatype or object properties to a resource directly from its page. In the interface, OKM refers to datatype properties (whose values are literals) as “attributes” and object properties (whose values are resources) as “statements.”2 (We will use this terminology throughout the remainder of this paper.) The use of two terms (instead of calling everything a “triple”) is intended to help the user better appreciate the distinction between them, since they are created, presented, and navigated differently. If the user chooses to add an “attribute,” the property value will be interpreted as a primitive data type. If the user chooses to add a “statement,” the property value will be interpreted as the HRN of another resource. For statements, the user can specify an existing resource in the system as the object – at which point the new resource is effectively “stitched in” to the rest of the graph – or else refer to a resource which does not yet exist, which will implicitly create that resource.
Users can also search the system for resources by typing in a
search box that autocompletes based on HRNs, or any portion
thereof (e.g., typing “lin” will match a resource whose HRN is
Abraham Lincoln.) This functionality is of course common to
innumerable tools today, from Freebase[
If the user presses the “Publish” link while viewing the OKM home page, Linked Data for all resources in the local system will be generated. The entire knowledge base will thus be globally exposed to Linked Data consumers.
In this way, Semantic Web amateurs can be empowered to contribute to the Linked Data movement by utilizing a tool with a low barrier to entry and which shields them from the syntactic complexities of RDF. Note that the current version of OKM does not support “round-trip” knowledge creation whereby existing Linked Data (and ontologies) can be imported into the tool. This feature was postponed since it did not bear upon our immediate experimental concern; in future studies, however, we plan to implement this and study user behavior in interacting with a larger, pre-existing knowledge space (in which there is greater urgency to find and re-us existing resources.)
Supplementing this normative user interface are three atypical features, which formed the focus for most of the investigative effort described in this paper. We hypothesized that each of these changes to the pseudo-standard user interaction paradigm would prove beneficial to novices attempting to interact with Semantic Web data, and for different reasons.
Rather than presenting all properties of a given resource in one long display, OKM encourages – and in fact, mandates – organization of these properties according to the resource's “roles.” A role is essentially an rdfs:Class to which the resource belongs, and which acts as the rdfs:domain (or rdfs:range) of the properties relevant to that class. Consider the screenshot in Figure 1. Here, the “Leonardo da Vinci” resource (which of course has a unique URI but which is presented to the user in terms of its HRN, as described above) has three roles: Artist, Person, and Scientist. Each role is manifested as its own box, with the relevant statements as contents. A given triple about Leonardo Da Vinci will appear in the role box which represents the domain for that triple (or, if Leonardo Da Vinci is the object rather than the subject of the triple, in the role box which represents the range. The “Giorgio Vasari” triple is an example of this latter case.) In order to add an RDF triple to the system, the user must choose one of the resource's roles (or add a new role) which will serve as the domain of the triple, and then add the triple in the corresponding role box. The user begins this process by clicking on the “Edit” link at the top of the page, thereby putting the page in “edit mode.” (See Figure 2 for an example.) The role boxes then acquire buttons labeled “+Attribute” and “+Statement,” which can be used to add attributes or statements to that role box. The user can then type the name of a predicate and a value. An autocomplete function assists the user with both inputs, offering to match predicates already in the system, and (in the case of statements) HRNs of resources already in the system. It is perfectly permissible, however, for the user to type the name of a new predicate and/or the name of a new resource, in which case the new item is implicitly created. The new predicate is automatically given a domain based on the role box it was added to, and a range based on the role of the object value it was given. (For object resources with multiple roles, the “Set Role” button can be used to select which of the resource's roles should be the range of the predicate.) From that point forward, the system incorporates the new predicate into its ever-evolving schema.
One important aspect of roles is that when in edit mode, a
template appears within each role box that displays the
predicates already known to have that role as a domain. Using
these templates is similar to inserting data in Freebase's
typebased editing model [
In summary, then, roles are intended to provide three benefits:
information) than novice users would ordinarily produce. It seems likely that when authoring only simple triples, most novice users would not bother to assign types to their resources, nor domains and ranges to their predicates. (At the least, it is unlikely that they would consistently do this.) With OKM, however, the act of assigning types, domains, and ranges is built in to the very process of creating triples, making it convenient to do and impossible to avoid.
We present benefits 1 and 2 without proof. Later in this paper, we provide an in-depth empirical analysis to judge the efficacy of benefit 3.
The flexibility that RDF offers in supporting both resources and literals as object values is a mixed blessing. On the one hand, it presents an expressive modeling device. The objects of the triples “John marriedTo Sally” and “John weightInPounds 175.4” seem inherently different: “Sally” is presumably a bona fide resource in her own right, with other triples expressing information about her, whereas “175.4” intuitively seems like a primitive piece of raw data, undeserving of resource status. Allowing authors to designate an object as one or the other affords the opportunity to express this subtle aspect of the object.
On the other hand, the existence of the distinction means that authors are forced to choose between the two alternatives, and the choice is not always easy to make. Consider triples like “BeverlysToyota color red,” “Charlie bornIn 1982,” and “Candice schoolYear sophomore.” The object values “red,” “1982,” and “sophomore” might be considered literal pieces of data, as with the above weight example, or as first-class resources. Anyone who has composed RDF for any length of time knows that this choice presents itself at every turn, and that in some cases it feels almost arbitrary.
Our work presents two contributions toward better understanding this phenomenon and how to best handle it. First, by creating a system that lowers the barrier of entry for the creation of RDF, as well as a system for creating both statements and attributes, we can observe how uninitiated users tend to differentiate between the two in practice. Later in this paper we present findings that reveal user tendencies in choosing between statements and attributes for specific types of information, and an analysis of the degree of consistency laypeople exhibit in this choice.
Second, we explore the effects of an RDF editor in which attributes are simply eliminated. It is possible, of course, to completely do away with the concept of literals if one is prepared to accept elements like “175.4” as resources. This is one way of dispensing with both the angst users face in making the decision, and also the inconsistency that can result when users make different choices: simply take away the choice altogether. This may seem like a heavy-handed solution, but it is not without theoretical merit. Consider that more than one prominent cognitive psychologist (e.g., [1, pp.125-7; 10, pp.3492; 23]) has formulated a knowledge representation theory based on something akin to semantic networks, yet found no need to differentiate between resouces and literals. One kind of node is all that comprises these knowledge structures, which suggests that a “resources only” network is in fact sufficient to encode human knowledge. And it places the burden of proof rather on those who argue for the existence of two distinct kinds of entities.
As described below, we deployed to experimental subjects not only the version of OKM depicted in Figures 1 and 2, but also versions in which attributes were completely eliminated. The “+Attribute” button was removed from all displays, which effectively forced users to model everything as statements. We then compared accuracy, consistency, and user satisfaction between the different versions.
Lastly, OKM allows users to construct n-ary relations without
explicitly using reification. This feature was inspired by a recent
project in which we conducted a pencil-and-paper based
experiment[
Guided by these findings, we implemented a predicate modifier scheme in OKM. An example is the “Leonardo da Vinci painted Mona Lisa” fact in Figure 1. Note that “with: oils” is an attribute, and “for Lisa del Giocondo” is a statement, and that both are indented underneath the “painted” triple. Users create such indented facts by pressing the “+Attribute” or “+Statement” buttons next to a triple, rather than at the top of the role box (refer to Figure 2.) When generating Linked Data, OKM converts these indented facts into traditionally reified triples, so that the knowledge is compatible with all current Semantic Web tools. From a user interface perspective, however, users never see the complexities of reification: they view and edit n-ary relations in terms of the much more intuitive predicate modifiers.
We formulated the experimentally. following
properly by selecting appropriate roles (and thus incidentally contribute meaningful domain, range, and type information.)
H1D – Users will in general select roles consistently with one another (i.e., if two users separately encode the same bit of knowledge, they are very likely to select the same role under which to create the triple.) Regarding the “elimination of attributes” feature: • • • • • • • • relations, users will choose the latter significantly more often, and have more success in doing so.
Our participant group consisted of 71 college students ranging from 18 to 22 years of age and contained roughly an even split between genders. All students were enrolled at the University of Mary Washington during the Spring 2010 semester and were of many diverse majors.
Participants took the one-hour experiment using the Firefox Internet browser on either a Windows or UNIX workstation. A ten-minute demonstration and explanation of OKM was given, and then each participant received an experiment packet and was directed to a URL (unique for each participant) that housed an OKM deployment with a pre-fabricated knowledge base containing about 130 resources and 150 predicates. The packet included 10 questions to be answered using this knowledge base (Part 1) and 24 facts to be added to it (Part 2). The final part of the packet (Part 3) was a survey to help us better analyze how the participants reacted to the system.
Part 1 questions ranged from easy to difficult depending on how difficult it was to find the information in the system. Easy questions were ones where the participant had to locate a specific resource page in the system and the answer was directly on that page. For example, “How tall is Jason Thompson?” The “Jason Thompson” resource existed in the system and the answer could be found on that page. More difficult questions forced the participant to view multiple pages and traverse links within the pages to locate the answer. For example, “What ballpark does Todd Helton’s baseball team play in?” The participant had to first find the “Todd Helton” resource page, and then find and click the link to the “Colorado Rockies” page in order to find the name of the sports facility in which the team played. Part 1 also acted as practice to help the participants become more comfortable and aware of the system and how it was organized.
Regarding hypothesis H3A, it is important to note that the last two Part 1 items involved n-ary relations, but that the prefabricated knowledge base had encoded one of them using predicate modifiers, and the other using predicate reification. The two items had nearly identical structure: “For what novel did Ernest Hemingway win the Pulitzer Prize?” and “For what film did Martin Scorcese win the Academy Award for Best Director?” Hence in answering this question, all participants witnessed properly encoded examples of both predicate reification and predicate modifiers. They were then presumably not biased in either direction when beginning Part 2, which required the encoding of five n-ary relations among its 24 items. Part 2 had the participants add data to the knowledge base. This part had a range of difficulty levels just as Part 1 did. The facts were presented as sentences with each sentence having one to three small facts within it. For example, “Madison Square Garden is located in New York City” has one fact: the fact that Madison Square Garden is in New York City. “Mark David Chapman assassinated John Lennon on December 8, 1980 at the Dakota Apartment Complex,” on the other hand, has three facts: the fact that Mark David Chapman assassinated John Lennon,
statement for a given bit of knowledge, there will be no predictable consensus among of a group of laypeople. They will very often make inconsistent choices with one another, leading to gross inconsistencies in a collaborative knowledge base.
H3A – Given examples of both predicate reification (traditional) and predicate modifiers (as described above), and the choice to use either technique to express n-ary and the date and the place of the assassination. (Note that this is an n-ary relation.) Resources referred to in part 2 did not always exist in the pre-fabricated knowledge base, requiring the participant to create a resource before adding the fact. Our participants were split into four groups based on which version of the program they used. The four versions were: ST (18 participants) – A “statements only” interface (i.e., no attributes) that provided role-based template information when editing resources (as described above.) This is the version of the interface which we hypothesized would be the most effective, since it incorporated all three of the experimental features described above.
S (19 participants) – A “statements only” interface with no templates. This version was identical to ST, except that when in edit mode, the greyed-out “suggested” predicates (such as “dimensions,” “period,” and “influenced” in Figure 2) would not appear.
SAT (18 participants) – A “statements + attributes” interface with templates. This version was identical to ST, except that the “+Attribute” buttons were included so that users could choose between creating attributes or statements. (This is the version of the interface depicted in Figures 1 and 2.) SA (16 participants) – Finally, in order to test hypothesis H2A, a number of participants received a more “traditional” version of OKM that permitted both statements and attributes, but provided no templates.
We then evaluated our hypotheses by judging the contents of the Linked Data knowledge bases that users produced while carrying out the actions required in Part 2. We did this in the following way:
chose to put a triple under was conceptually correct. This was admittedly somewhat subjective, but in practice there was very little debate among the graders (the four authors of this paper) as to whether a role was correct.
H3B – for the nineteen Part 2 items that did not require n-ary relations, count the number of times participants mistakenly used the predicate modifiers feature and generated nonsensical Linked Data as a result. • • • • • • • •
We evaluated our template-related hypotheses using nine specific items. For each of these items, the system contained a predicate, associated with an appropriate role, that users should have noticed and could have selected using the template. However, these items were in two groups. For six of them (items J, K, L, P, W, and X) the already existing predicate was in fact semantically appropriate for the item. For instance, for item J, “David Beckham scored 27 goals,” the pre-fabricated knowledge base contained the predicate “goals” for the “Soccer Player” role. Therefore, it would have been appropriate and consistent for ST users to select “goals” from the template to record this item (as opposed to creating their own equivalent predicate such as “scored” or “numberOfGoals.”) For the other three items, however, the already existing predicate was not semantically appropriate for the item. We called these items “traps.” For example, the pre-existing predicate that ST users saw for item O, “The Matrix's gross earnings were $90 million,” was “net earnings.” We included these three items because we wanted to measure the degree of danger templates may introduce in leading users to choose preexisting predicates that are in fact not appropriate.
We evaluated templates in two ways. First (hypothesis H1A) we compared the total correct responses between the ST and S groups for the nine items, treating each users' response for each item as a seperate trial. Considering all nine, 70.37% of ST users' representations (114 out of 162) were correct, as opposed to 67.25% (115 out of 171) of S users' representations. This difference was not statistically significant (p > 0.01 by Fisher's exact test). This indicates that templates do not improve correctness of user-generated data, thereby refuting hypothesis H1A. (Interestingly, when considering only the six non-trap items, the ST group got 65.74% correct compared with S's 61.40%; for the trap items, ST got 79.63% and S 78.95%, neither of which was statistically significant. It appears, then, that templates have no impact on correctness, regardless of whether the predicates in question are in fact appropriate.) We also studied the impact templates have on consistency of data (H1B); i.e., the likelihood that data authors would re-use an appropriate predicate already existing in the system as opposed to creating a synonymous one. The effects of templates on the predicates used for the nine items are summarized in Table 1. The results show that users in the ST group were significantly (p < 0.01 by Fisher's exact test) more likely to (correctly) use the existing predicate in three of the six non-trap cases, and to (incorrectly) use it in one of the three trap cases. This is a mixed result. Evidently, templates effectively promote consistency for some facts but not for others, and they mislead users into semantic incorrectness for some facts but not for others. Hypothesis H1B appears to be confirmed only in certain cases. When we probe the specific items to discover which kinds of items templates assist, we discover that it greatly depends on word choice. Templates were not shown to be helpful for items P (“The song 'Stairway to Heaven' featured Jimmy Page on guitar”, with predicate “plays” defined for role “Musician”), W (“John Entwistle played bass on the song 'Behind Blue Eyes'”, with predicate “plays” defined for role "Musician"), and X (“Paul McCartney wrote the song 'Maybe I'm Amazed'”, with predicate “composed” defined for role “Musician”). The wording of items P and X differs from the defined predicates, while the tense differs for item W. These results suggest that users are less likely to use templates when it would require restructuring the sentence or using different terminology. In a real-world setting, of course, users are not translating sentences into Linked Data, but “mental knowledge” into Linked Data. We can only speculate as to the size of this effect for mental knowledge, but it seems reasonable to assume that if a user wants to encode a fact, and has a certain phrasing in mind, they will succumb to the same pitfall that our testers did. Note that two requirements must be met in order for a user to take advantage of a template: they must (1) select the proper role (i.e., the role that is the domain for the predicate), and they must (2) observe and decide to use the relevant predicate for that template. In cases where the user failed to use templates, neither of these factors was entirely to blame. For items P and J, for example, 81.8% and 82.4% of the failures were due to choosing the wrong role; for items W and X, on ther other hand, 100.0% and 86.7% were due to not using the right predicate within the (correct) role. It appears possible to fail at either.
Item O (“The Matrix's gross earnings were $90 million”) illustrates the potential negative impact of templates. Six out of eighteen ST users chose to use the existing predicate “net earnings.” This result indicates a risk that users may select incorrect predicates when they are lexically similar but semantically different from the phrases they intend to use. However, this risk can be expected to diminish with substantial domain knowledge, which equips users to correctly differentiate between similar terms.
Overall, these findings suggest that templates assist with both consistency and correctness in predicate usage, despite not being effective in all cases. Templates are helpful when the user selects the appropriate role and when the existing predicate is consistent with the user's intended phrasing of the information.
In order to evaluate the viability of roles, we examine both correctness (H1C) and consistency (H1D) of roles chosen for 20 items. We judged correctness by the relevance of the chosen roles to the information entered. For example, we considered appropriate roles for item L – “Deion Sanders has stolen 35 bases” – to include “Baseball Player” and “Athlete” but not “Football Player” or “Person.” Inconsistency is measured using Simpson's Index of Diversity (D = 1 - Σ(pi2), where pi is the proportion of users who chose role i) for each item. An index of 0 indicates that all users chose the same role. Larger values indicate more distinct roles chosen and a more even distribution between roles. Both are shown in Table 2, divided into two groups: items for which the resource already had an appropriate role, and items for which the user would have to add a role to the resource in order to make a reasonable choice.
The results show that users were very likely (92.81%) to make a reasonable choice if the relevant objects already had an appropriate role, but less likely (73.32%) to add one themselves (p < 0.05 by a t-test). The diversity of roles was high for both kinds of facts, although the second group was more diverse (p < 0.05). This indicates that users are often unable to choose correct roles, and are not reliably consistent with one another. The fact that templates were successful at helping users enter semantically correct data despite these difficulties suggests that users better guided by ontologies might experience greater benefits from a system that incorporates type and schema information.
To determine whether the presence of attributes in the system influences users' ability to represent data, each item was rated for correctness. A response was considered correct if it accurately conveyed the information given in the text, was consistent with the graph-based data model, and was associated with a reasonable role. Table 3 compares the SAT and ST groups for overall correctness of data entry.
The results show no significant difference between the two groups (by Fisher's exact test, α = .01). (It is also the case that no significant difference existed on any one item.) This finding is consistent with our hypothesis that the presence of attributes in the system would not influence the quality of data produced by users. Thus hypothesis H2B is confirmed.
We also hypothesized that, when forced to choose whether to model a fact as a statement or as an attribute, users would behave inconsistently with one another. Table 4 shows the data we collected to evaluate this hypothesis. The 24 sentences contained 30 atomic facts, which are divided into five categories based on whether their objects are: proper nouns (14 facts, such as “New York City”), common nouns (5 facts, such as “piano”), numeric values (9 facts, such as “186 pounds”), dates (3 facts, such as “April 4, 2008”), and years (1 fact, “2004”). Our users demonstrated the highest consistency for proper nouns and the lowest for numeric values. However, it is clear that for none of the five types can consistency be counted on. No matter what type of fact is being represented, different novice users will encode it in different ways – some as attributes, some as statements – leading to basic inconsistencies in the resulting structure of the Linked Data.
Evaluating consistency in the abstract is difficult, but we note certain items that show a surprising lack of consensus. For instance:
Item F - “Kelly Witt is a freshman.” (14 attrs, 18 stmnts) Item S - “Michael Abram stabbed George Harrison on Dec. 30th, 1999” (the date: 19 attrs, 12 stmnts) Item Q - “Deion Sanders hit 7 home runs” (24 attrs, 9 stmnts) Users exhibit no strong consensus regarding how best to model these pieces of information, and many others, confirming hypothesis H2A.
For some items, a substantial number of users made very unintuitive choices. Item E, clearly a numeric value (“Ryan Medina's GPA is 2.79”) was represented as a statement 11 out of 33 times (33.3%). Even more problematic is the tendency to represent proper nouns as attributes. For item A (“Madison Square Garden is located in New York City”) 9 out of 30 users chose to represent New York City as an attribute (30.0%) And for item N (“Mark David Chapman assassinated John Lennon on December 8, 1980 at the Dakota Apartment Complex”), 17 out of 31 users represented Dakota Apartment Complex as an attribute (54.8%). We argue that representing proper nouns like “New York City,” about which many things on the Semantic Web are likely to be said, as anything other than resources is a mistake, and that untrained users are likely to make that mistake often when given the choice. In the absence of a compelling reason to do so, we recommend that systems not force users to make the choice between resources and literals.
To evaluate the effectiveness of predicate modifiers as a tool for
expressing n-ary relations, we exposed users to both predicate
modifiers and the traditional method of predicate reification.
None of the 71 users employed predicate reification in
representations of any of the five facts containing n-ary
relations. Table 5 shows the overall correctness of the user's
representations of those facts (using predicate modifiers).
Users were less likely to express these facts correctly than
simpler items. However, in light of our previous study[
For the vast majority of test items, which contained only a single fact each, users did not attempt to (incorrectly) represent them using predicate modifiers. (No more than 2 out of 71 users tried this for any of those items.) However, for item K (“Peyton Manning passed for 206 yards, while Brett Favre threw for 315”) this did prove to be a common problem. This item actually contains two separate binary relations, but 25 out of 71 total users (35.2%) incorrectly applied predicate modifiers to try and express it. This finding may suggest that novice users can have difficulty determining whether a complex thought represents a single n-ary relation, or a series of binary relations. If so, we argue that this only emphasizes the need to investigate more intuitive techniques for representing complex information. In any case, for simple sentences, hypothesis H3B is confirmed.
Finally, our experiment ended with a 12-question survey in which participants answered reaction questions on a 6-point Likert scale. These measured user satisfaction with the system, the ease with which they could locate information, etc. Only two of the items demonstrated any significance between groups (to an α of 0.05): “It was easy to use the system to add new information.” The average responses on this item were: Group S=4.9, Group ST=5.5, Group SAT=4.1. Considering “templates” and “attributes” to be two independent variables, a univariate ANOVA test confirms that a “statements only” interface has a beneficial effect on user perception of how easy it is to add data. (p < 0.05). “I was confident that I added the information correctly.” The average responses were: Group S=3.8 Group ST=4.6, Group SAT=3.9. The ANOVA test confirms that the template feature has a beneficial effect on user confidence in adding data. (p < 0.05) (Note that since we had no survey information from an “SA” group – i.e., attributes, but no templates – we could not detect any possible interaction between variables.) Although we had no a priori hypotheses regarding user preferences, this survey information seems significant. Users appear to have a preference for a “statements only” interface with templates. This type of interface modestly enchances the user experience and raises confidence.
Novice end users, who are potential contributors to the Web of Linked Data, have substantial difficulties formulating knowledge in the format the Semantic Web requires. User interface design, therefore, is paramount. Our empirical testing has shed light on certain aspects of how such interfaces are used in practice, and should be best designed. These include:
Requiring users to group information about a resource according its roles (types), and displaying a template of previously used predicates for each of those types, can help channel users towards the re-use of predicates that already exist, avoiding undesirable proliferation of synonyms. We also believe that roles and templates provide benefits in terms of more facile navigation and the implicit creation of domain, range, and type information. However, users are not consistent in their choice of roles, suggesting that some mechanism for encouraging consistency would be wise. Users are very inconsistent in choosing to model an object as either a resource or a literal, and this appears to be true across a wide variety of types of objects. In many cases they simply make inappropriate choices. Moreover, users appear to be just as successful generating Linked Data when they use an interface that only supports resources. We therefore recommend that tools for authoring Linked Data not include literals in the interface.
A scheme for allowing users to express n-ary relations with modified predicates, rather than with traditional predicate reification, can enormously increase their success in modeling such information.
We would like to thank Trillane Burlar and Christopher (Shane) Voisard for their inimitable development work, without which this project would not have been possible.