In this poster, we present an annotation system as an abstraction layer to enrich the collaborative knowledge creation and curation experiences by structuring data extracted from the exchanges between users, between users and AI services, and from users' input on content. It supports the de nition of more meaningful relations between concepts and richer discussion processes among users, contributing to the expansion and evolution of knowledge bases that feed o the aforementioned structured data. It is also capable of yielding relevant results to semantic queries by which users can retrieve content and knowledge they contributed to creating. Our results show that users found this method of joint knowledge building to be useful and that it could optimize tasks, mainly because a) it allows access to fresh insights, correlations, and valuable knowledge exchange, and b) it supports data retrieval via semantic queries.
This poster1 presents the ongoing work around the Hyperknowledge Annotation System (HAS), by describing a qualitative approach to understanding user needs following the speculative development of a proposed system. The work disclosed in this piece was elaborated in the context of the di culties around interacting with knowledge bases to curate and enrich them. It can be a tiring and complex activity, especially for those who are not familiar with the eld of knowledge engineering, which can be further complicated when there are multiple inputs from users from di erent backgrounds.
One of the many ways one may interact with such bases is by means of
annotation systems. In general, these focus on one type of media (text, image,
video or audio) [
Our proposed approach, the HAS, provides an abstraction layer that allows users to collaborate (with each other and with arti cial intelligence services) when creating and curating knowledge to enrich knowledge bases. The system supports multimodal annotations over multimedia content segments so that annotators can use diverse types of content to create annotations, and the retrieval of information from the knowledge bases, which represents a reward for engaging in the activity in the rst place.
It is all made possible by extracting and structuring concepts from
annotations and their anchors (selected piece of content), as well as by o ering
suggestions through understanding annotators' discourse with the support of arti
cial intelligence (AI) algorithms. To structure the annotation content, the HAS
uses its own conceptual model called Hyperknowledge [
Our results showed that participants were able to make annotations, curate suggestions, understand how to collaborate, and make queries; understanding that in order to obtain results to their queries, the information needed to have been previously added to the base. When the sheer volume of information one operates with becomes ine ective to deal with in other methods such as keyword search, users stated that the HAS is a better alternative as it allows semantic queries that are a clear advantage to the process. 2
The HAS systems design was rst introduced by Moreno et al. [
First, the architecture is composed of four layers, each de ning a level of abstraction: layout structure layer; syntactic layer; semantic layer; and pragmatic layer. The layout structure layer supports information extraction in a document by identifying semantically related structure (e.g. bullets and headers). The syntactic layer handles the grammatical structure of sentences (e.g. identifying a substructure in a sentence as its subject). The semantic layer is in charge of specifying content meaning (e.g. of a given word or concept). Lastly, the pragmatic layer provides support to annotation at a natural language level (e.g. manually annotating a concept).
The human-machine collaborative scope relates automated annotations from AI services with users annotation. It captures part of the contextual information from a users annotation and provides an automatically generated annotation. The users annotation can be done on and in a range of media types, and to support that, the HAS establishes contracts between a media type and the appropriate AI services.
Fig. 1. shows an example of the aforementioned annotation interaction on
an image, in which the white rectangle is the users annotation anchor referring
to the concept of a player and the red rectangle refers the AI services output
which identi es the player as Neymar. The media node Image A represents the
image you see to the left (Neymar playing). It contains one more anchor besides
the default anchor. The anchor anchor 1 can be linked with connectors of
type depicts to nodes of type instance (sprint 17) of concept (M ove). In this
example, sprint 17 is an instance of the class M ove and is linked to an anchor
of Image. For the ontology in question, a M ove (such as sprint 17) is executed
by a P layer, which, in this case, is the instance N eymar. Finally, the facts are
inside a context called M atch 3, but the nodes N eymar, Player, and Move, are
in dashed lines, which indicates they are being reused. In other words, reusing
allows entities that belong to di erent contexts to be linked without having to
de ne them once more. How to proceed with the de nition of the entities in
contexts is up to their application. To structure the annotations and store them
on the knowledge base, the conceptual model behind HAS - Hyperknowledge
uses domain-based ontologies. In this particular example, the chosen domain was
soccer [
The methods we used to test user interactions and assess the system's value to them were small-scale and qualitative in nature, but enough to drive investigations into our two main questions: (a) Do people understand and perform according to their role of calibrating the AI algorithms and enriching the knowledge base? (b) Do users perceive the advantage in contributing to the system in order to reap the bene ts of knowledge retrieval via semantic queries?
We interviewed scientists of di erent backgrounds, all of which engaged in research and development activities that require a lot of information to consume, analyze, share, and build upon, about which they often needed to retrieve speci c information such as the temperature used in a speci c experimental setting. In this case, digital or physical notes have to be associated with a digital image le. To nd such information as the temperature used in experimental settings that had returned a particular type of result, they would have to parse physical documents in a binder; or, in the case of digital documents, type in the appropriate keywords in a document nder or software, and then look for the speci c data among all the results that the keyword search returns. Saving a digital picture of the physical note does not help search for words that might be in the content itself, and not the le's metadata. In that sense, it would be of great bene t to them if they could use di erent media types to directly annotate on multimedia content.
In testing the HAS, they were instructed to simulate uploading a le (which,
in this case, was an image), annotating on it, reviewing the suggestions made
by the AI (in that case, IBM's Watson Image Recognition), saving that
annotation, and then going over that annotation's details and properties in order
to contribute to it via replies and further annotations. Finally, we gave them 2
minutes to freely pose the system with queries that they would like to be able
to make, and they came up with twenty of them, a few of which were: show
videos of test 4; show highlighted points of interest in a content; how did the
colorimetric response of a given indicator vary over time. All of the queries they
wished to make were feasably supported by the HAS, provided that the relevant
data was present in the base and appropriately structured; that parameters for
properties such as "colorimetric response" were de ned; and that query inputs
were adapted to one of the supported query languages, such as SPARQL [
We were able to successfully answer the research questions posed beforehand in the following manner:
(a) Do people understand and perform according to their role of calibrating the AI algorithms and enriching the knowledge base? We were able to conclude through observations and direct user quotes ("you have to keep in mind that all that you might want to ask it depends on what has been annotated") that users did indeed understand where inputs came from (themselves, AI suggestions, and mutual feedback/curation between users and between users and AI); their roles in providing these inputs and curating them; that the enrichment of the knowledge base depended on that; and that the possibility of querying did as well.
(b) Do users perceive the advantage in contributing to the system in order to reap the bene ts of knowledge retrieval via semantic queries? We reached the conclusion that indeed they do. That same quote we highlighted as part of the answer to question above encapsulates a fundamental factor to answering this one. If they could understand and accept that they had to make an e ort in order to be able to make queries, it is clear that that is something they wish to able to do. They realized that the queries a orded by the HAS could signi cantly optimize their process, and so they represent an advantage to which the e ort required to contribute to the base did not seem disproportionate (and neither, as stated before, more laborious than their current process), especially as they greatly reduced the number of tasks required to get access to speci c knowledge during a research and development project, in a way that contributes to reducing cognitive exhaustion.
Furthermore, users stated that working alongside other users and an AI annotator's suggestions in the HAS provided them with fresh insights into relationships between concepts, which helped them establish other correlations they might not have thought of, and which brought them new ideas to, in turn, bring to discussions, and to further enrich their knowledge building process (and even if they're not in direct contact with it, the corresponding knowledge base).