This paper discusses the use of semantic web technology to realize the vision of future scienti c publishing and scholarly communication. It introduces a novel knowledge system that builds on the popularity of social tagging and collaboration systems as well as on the idea of \Annotations Anywhere, Annotations Anytime" (hence 4A). Key technical characteristics of the realized components are presented. User experience observations and the results of a preliminary experiment are also reported.
Recent years brought various proposals for changes in the model of scienti c
publishing and scholar communication. Most radical ones call for a complete
shift from pre-publication peer review and impact factor measures towards social
popularity of papers [
The 4A framework introduced in this paper adds a piece to the mosaic of
tools that enable such changes. It can be seen as a modularization and
unittesting system in the parallel between scienti c knowledge artifacts and software
artifacts [
Transformations in scholar communication are often motivated by bene ts they could bring to a research community. On the other hand, business models applicable in the new publishing era need serious consideration [21]. Publishers obviously try to nd their position in the evolving environment. It can be demonstrated by their e ort to engage authors and get their feedback or to support sharing of scienti c citations among researchers in the form of reference manager
services (e. g., Connotea3 or CiteULike4). The 4A framework is also intended to
encourage the publishers to see opportunities in making money by adding value
to open access data, rather than restricting the access to outcomes of research [
The framework emphasizes the role of annotation in the process of knowledge
creation. It is crucial to realize that tags can be associated not only with a
scienti c publication, but also with any other material relevant for the research.
If the experimental data referred to in a paper is made available (as required in
some disciplines today) and also annotated, it is easier to verify its interpretation
presented in the text. The annotation of data that is employed in experiments by
other researchers also simpli es tracing its reuse and thus provides an alternative
to the current state characterized by reusing and not appraising [
A special attention has been also paid to the means of tracking down particular ideas described in papers to the source code of computer programs described in the text. One can employ the concept of program documentation generation (for example, by means of the popular Doxygen tool5) and interconnect particular pieces of code with the description of their functionality on the higher level.
Last but not least, the concept of annotation is general enough to cover not only the textual material related to the presentation of a particular research result (slides from talks, blogs, tweets or other formats of messages referring to the results) but also related audio- or video recordings. For example, the proof-of-concept case study involved a material from the Speaker and Language Recognition Workshop Oddysey 20106 and it employed the search-in-speech service Superlectures7. Many ideas from referring texts are not easy to link to the material available in textual form so that working with a multimedia content is often necessary.
To summarize the directions of motivation mentioned in this section, the new scienti c publishing and scholarly communication model should be accompanied by an annotation framework supporting the whole life-cycle of scienti c papers { from ideas, hypotheses, identi cation of related research, and data collection, through setting and running experiments, implementing solutions and interpreting results, to submitting, reviewing, re ecting reviews, preparing nal versions, publishing complementary material, getting feedback, discussing content and re ecting previous results in a new work. Annotations on all levels pave 3 http://www.connotea.org/ 4 http://www.citeulike.org/ 5 http://www.stack.nl/~dimitri/doxygen/
the way for shared knowledge understanding. The social dimension of ubiquitous annotations of knowledge artifacts can also bring immediate bene ts to research communities in terms of better models of ne-grained impact characteristics. The elaborated annotation system helps to pinpoint an expertise in particular elds. Researchers can also bene t from instant grati cation { the annotation is immediately available for others, it can be shared and re-used in other contexts.
This paper discusses the use of semantic technologies to realize the abovementioned vision. The name of the annotation framework { 4A { refers to the concept of \annotation anywhere, annotation anytime". Key features of the realized solution are introduced in Section 2. A proof-of-concept implementation of the 4A annotation client and its application in experiments are presented in Section 3. The paper concludes by discussing the directions of future work. 2
Semantic search and other advanced functionality of the future web o er clear bene ts for the semantic annotation of knowledge artifacts. On the other hand, the semantic enrichment of resources presents a tedious work for users. It is therefore crucial to lower the barrier to annotate by means of immersing the annotation into everyday work of users.
We address this requirement by de ning a general interaction schema and protocols that can be supported in various contexts. An ultimate goal of the 4A framework is therefore to let users annotate naturally in any application used. The current implementation focuses mainly on the textual resources and implements the functionality of server components as well as several clients.
From the annotation support perspective, it is crucial to distinguish two types of environments users work in viewer- and editor applications. As viewers do not modify the source content, changes in annotations need to be transferred only. On the other hand, it is tricky to synchronize editing annotation sessions as the changes in text may invalidate annotations.
To serve both types of clients, a general annotation exchange protocol has been de ned. The 4A synchronization server implements the protocol and enables coordinated work of 4A clients. An annotation extension to general Javascriptbased editors has been developed as the rst 4A client. A PDF reader add-on and a Firefox browser extension are being implemented.
The 4A framework goes beyond the current practice of simple keyword
tagging and knowledge structuring curated in advance. It introduces an intuitive
knowledge structuring schema of structured tags [
Unfortunately, none of the annotation formats applied in existing tools is
general enough to suit our purposes without modi cations. Even Annotea [
The position of an annotated fragment is given by the path in the document object model (DOM), the o set and the size. The representation is robust to changes in general formatting. It is usually not necessary to process the whole document. For example, a web page boilerplate and other parts that are not in a DOM node on the path to an annotated fragment will be ignored.
The annotation types form a hierarchical structure. They include common comments (a note, a description) and basic types of entities (a thing, a person, etc.). Users can add new types and create complex type hierarchies.
The URI of an annotated document identi es a copy of the document that is stored on the server. The annotation process starts with a synchronization step in which the client sends the document URI and its content to the server. The server returns the URI of the local copy of the document which will be used in annotations. This procedure enables annotating documents that the server could not access directly. Processing the original document on the server side also enables removing irrelevant attributes (e. g., session ID) and applying changes to the correct version of a document as the stored version is updated together with all annotations at every access.
A new annotation interchange protocol has been de ned for the communication between 4A clients and the server(s). In addition to actual annotations, it can be used for simple authentication, synchronization of annotated documents, subscription to annotations from de ned sources, annotation suggestions, interchange of knowledge structures (annotation and attribute types) and various annotation-related settings.
The protocol enables two-way asynchronous communication between clients and servers. If a user adds an annotation, the server sends it immediately to all other users that annotate the same document and are subscribed to a given channel (de ned by an author, a group or an annotation type). Changes of annotation types, of the document content and of relevant settings are distributed immediately as well.
Messages are de ned in XML. They can be therefore easily sent over various protocols on a lower level and parsed on the client side. It is also possible to combine the messages into one XML and make the communication even more e cient.
Session management messages include the protocol version negotiation, logins and log-outs. Subscription management enables specifying what types of annotations from what sources should be sent to a particular client. Annotations can come from another user or a URI representing an automatic annotation server, user group or other general source (e. g., an external service).
The server gets a copy of the current version of an annotated document by means of a document synchronization process. If it is the rst time the document is sent, the server just stores it. If there is already a copy of the document, the server compares the new version with the stored one and updates the stored version together with all annotations. If the new version impacts no previous annotation, the operation is con rmed instantly. If there are annotations that could be invalid, the server informs users who can consequently correct possible errors.
To support clients that are not able to work with structured texts, the server can linearize the text from documents. The client transforms the document to plain text and sends it to the server. If there is a structured form of the document at the server, it is linearized and compared to the received version. If they are the same, it is possible to start the annotation process. The server will then adapt all incoming annotations for the structured version of the document. The linearization also enables cooperating with clients working with other structured formats than the server. The client will adapt positions to the linearized text, the server will adapt it to its structured form. This way it is possible to annotate the same text synchronously e. g. in HTML and in PDF. 3
As a proof of concept, we employed the current version of the JavaScript annotation editor in experiments . The tool is a universal component which can be easily integrated into various JavaScript-based editors such as TinyMCE8. It implements the functionality of a 4A client { it enables editing complex annotations, synchronizes tagging with the server side and presents annotation suggestions provided by information extraction components.
The client makes the annotation process manageable. The type of annotation can be speci ed in a text eld. Instant search in type names is supported { the input serves as an intelligent lter. It is also possible to browse the hierarchical structure of types. This reduces diversity of annotation types.
It is possible to add new attributes to identi ed relations. A name, a type and a value are associated with each attribute. The selection of an attribute type is similar to that of the annotation type. The way information is presented also corresponds to the types. An attribute can be of a simple data type, of an extended type (e. g., a geographic location), or of an annotation type. Simple and extended data types are added as new branches of the type tree. If an annotation type is utilized, it is possible to choose one of the existing annotations (an annotation reference) or to create directly a nested annotation.
It is possible to select more textual fragments in a document. Annotating more fragments simultaneously enables identifying all occurrences of a relation
in a document. When storing it, a separate annotation is generated for each fragment. The same procedure can be applied for attributes (nested annotations) with the same name, type and content which are then displayed as a list. Clients employ suggestions to facilitate the tagging process (see Figure 1 for an example).
One senior researcher (the rst author of this paper) and six PhD students (including the second author) participated in the annotation experiments. Source texts included not only scienti c papers, but also blog messages and tweets related to speci c research topics and the material referred from these sources (e. g., a particular part of a dataset, a PowerPoint presentation, a source-code le). The papers dealt mainly with natural language processing, information extraction and machine learning (correspondingly to the expertise of the participants). However, there were also general topics discussed (especially in the blog messages).
The experiments aimed at identifying particular pieces of text (selecting textual fragments) that correspond to speci c types of content patterns (e. g., statements showing equivalence of two approaches or suggesting a resource for a speci c group of readers). A part of the task also involved tagging the fragments supplying evidence that a particular method is really applied for a particular task (not just referred to as an alternative the paper does not really deal with). For citations, the task was to nd the part of a referred text illustrating the referring context.
The primary objective of the analysis was to learn what form of tags (simple or structured ones) users prefer in given situations and how tag suggestions help to annotate the content (even if they are far from being perfect). Annotation suggestions based on automatic information extraction were switched o to not in uence the answer to the rst part of the question. This setting simulates specialized tagging where the set of available annotated examples is too limited to be used for learning-based methods or too complex to be described by a handful set of rules. The participants have been instructed that the annotation should be usable for future knowledge acquisition from annotated resources. Suggestions were turned on for the second part. Figure 2 shows a typical setting of the environment.
Results of the rst part (annotation form preference) are conclusive. All annotators resort to structured tags in the task of method comparison. On the other hand, they prefer at tags over structured ones when dealing with simple annotations of individual methods (\conjugate gradient" is a \ML method"). As a side e ect of the advanced functionality of the tool, it was also observed that the annotation form will probably stay unchanged if a user de nes the structure of a tag for a speci c task and others are able just to follow and re-use the de nition.
The second part of the question turned to be hard to answer. Automatic annotation suggestions accelerate the tagging process but they also distract annotator's attention. The level of acceptance is subjective and signi cantly varies with respect to the precision and recall of the information extraction process generating suggestions. It is also questionable what the granularity of structured-tag suggestions should be, i. e., what information (that does not need to be correct) should be combined and potentially con rmed by one click. For example, one of the annotators pointed out that \momentum" should not be referred to as \a method" in the context shown in Figure 2 so that he could not accept the suggestion as a whole. 4
A lot of previous work has been done in elds related to the presented research.
The general topic of future research, scienti c publishing and scholarly
communication is discussed in various contexts { linking scholarly literature with
research data [
First tools supporting general annotations on the web date back to
midnineties [
PREP Editor [
Popular web-based editors and other o ce applications such as Google Docs10 or Microsoft O ce Live11 show current trends in the development of collaborative tools. Even though there is either no or very limited annotation functionality, they need to be considered as \opinion makers" in terms of user interface simplicity.
Google Docs is also noteworthy as being a successor to Google Wave12 { an envisioned distributed platform of Email 2.0 where clients should be connected to their particular servers and the servers should communicate by means of the Google Wave Protocol13. Compatible solutions, such as Novell Vibe14, only started to appear when Google turned away from a strong support of the platform. Nevertheless, the vision of a distributed platform transferring user actions with a very low granularity stimulated the development of the 4A framework.
10 https://docs.google.com 11 http://www.officelive.com/ 12 https://wave.google.com 13 http://www.waveprotocol.org/ 14 https://vibe.novell.com/
The area of social tagging is also relevant for general topics discussed in this paper. Various tools exist for the speci c subdomain of collaborative scienti c citation managers such as CiteULike, Connotea, Bibsonomy15, or Mendeley16. The systems o er advanced support for the particular task (automatic extraction of relevant metadata, export to various formats, etc.) but they are usually not able to link references to particular pieces of text and do not explicitly deal with knowledge structuring.
On the other side of the knowledge processing support scale, there is a
family of ontology, topic maps and other knowledge representation format editors.
Protege17 de nes a kind of standard, other tools such as Neon Toolkit18,
Ontolingua19, or TopBraid Composer20 stress the collaborative-, user support-, or
integration aspects, respectively. Anchoring created knowledge structures in real
data is often limited to examples or glosses that can be stored together with
concepts and relations. In this context, resources resulting from semantic
annotation e orts in computation linguistics, such as FrameNet21 or OntoNotes22, as
well as enriching folksonomies by formal knowledge sources [
The presented work also extends the concept of semantic wikis. Several
systems have been created around the idea of semantic web technologies enhancing
the wiki way of content creation [
Last but not least, one of the key components of the 4A framework { the
information extraction module { relates to previous research on automatic knowledge
extraction. In particular, we take advantage of the KiWi extraction elements [
The 4A framework presented in this paper incorporates annotation into knowledge acquisition and knowledge sharing processes. The proof-of-concept implementation and the use of an annotation client in tagging experiments proved validity of the idea but also revealed imperfections of the model and aws in the user interface of the realized tools. The future work will focus on removing these de ciencies. The concept of knowledge emergence and knowledge structuring needs to be re ned. We will esh out the support for social knowledge-creation processes. New information extraction modules will employ advanced machine learning methods to provide better suggestions in situations when only a few training examples are available.
The number of 4A clients will also grow. The development of the JavaScript annotation component and its integration into various web-based editors will continue. The Firefox browser extension and the PDF reader add-on will be nished. We will initiate the work on other extensions for LibreO ce and the Semantic Desktop. The annotation format will be proposed as an extension of the o cial W3C Annotea system.
New experiments will explore advanced tagging collaboration patterns. The question on the acceptable error-rate for the automatic suggestions will be also tackled. A higher number of participants and a more advanced setting of experiments are necessary to prove the e ciency of the knowledge structuring processes in the 4A framework. Last but not least, consistency and adequacy of the knowledge representation resulting from the use of the 4A tools will be studied.
The research leading to these results has received funding from the European Community's 7th Framework Programme FP7/2007-2013 under grant agreement number 270001 { Decipher. 21. Waltham, M. Why does one size not t all in journal publishing?, 2010. Presentation at the June 2010 Society for Scholarly Publishing meeting, slides available online: http://www.marywaltham.com/SSP_Seminar_2010.pdf. 22. Wimalasuriya, D. C., and Dou, D. Ontology-based information extraction: An introduction and a survey of current approaches. Journal of Information Science 36, 3 (2010), 306{323. 23. Zheng, Q. Structured annotations to support collaborative writing work ow. Master's thesis, The Faculty of Graduate Studies (Computer Science), The University of British Columbia, Dec. 2005. (accessed online March 14, 2011). 24. Zheng, Q., Booth, K., and McGrenere, J. Co-authoring with structured annotations. Department of Computer Science, University of British Columbia, 2006.