Images depicting key findings of research papers contain rich sets of information derived from a wide range of biomedical experiments. Biomedical imaging [1] employs numerous modalities such as X-Rays (CT scans), sound (ultrasound), magnetism (MRI), radioactive pharmaceuticals (nuclear medicine: SPECT, PET) or light (endoscopy, OCT) to evaluate the status of an organ or tissue. Unlike text or other non-imaging data, image data poses a number of idiosyncratic issues rendering them mainly opaque for reuse without significant manual intervention. Current practices related to the extraction of implicit knowledge provide annotations that are neither anchored with an image, nor documented in a machine-readable fashion. As a consequence images cannot be readily discovered or categorized based on their contents. In the case of biomedical images that contain some type of biological sequence data summarizing the atomic composition of biological molecules [2] a combination of optical character recognition and text extraction techniques can provide better searchability over these images such that questions like “display of all the sequence images that show proteins from the same protein family”- [3] could be asked, provided that annotations could be made available to a search or query engine. However, image repositories in use today restrict the features that users can search with to those described in text based image captions and predominantly encourage the syntactic keyword based search, which constitutes a significant limitation [4]. In contrast images with semantic annotation can be automatically and/or semi-automatically discovered and linked to new information. The resulting enriched images are readily reusable based on their semantic annotations and can be used in semantic search and ad-hoc data integration activities. Overall, to achieve a greater degree of reusability and interoperability over image data certain core infrastructure is required, including automated image annotation pipelines and semantic vocabularies that can anticipate and represent image related content unambiguously. Existing ontologies and vocabularies describing biomedical images, particularly sequence images, are not sufficient to fulfill the requirements mentioned above and for our use case (SEBI) [4]. This motivated us to build the BIM ontology described in this paper which was designed and modeled with the following purposes in mind: formal representation of image annotation, enhanced reusability of image related data, depiction of pre and post image processing phases, design of context aware image search engines and semantics enabled bioimaging applications. 2

BIM ontology has been created to provide the standardized semantic representation of the annotations generated to describe a biomedical image by SEBI. BIM can further be used for annotating the associated text references by a machine or human. In order to collect the relevant terms, relationships / properties for sequence related images, we reviewed literature mentioning sequence analysis algorithms [ 9 ] such as BLAST, HMMER, Prosite, and the conserved domain database. A total number of 23 papers published from 2006 to 2015 were selected from different journals. We focused on actual depictions and discussion of sequence alignment outputs, rather than the algorithms, to distill the typical terms, concept and relations used. In order to accumulate terminologies associated with non-sequence image types such as: X-Rays, ultrasound, MRI, radioactive pharmaceuticals endoscopy, we selected a random sample set of papers from the Journal of Bioimaging and applied the SNOMED-CT1 and BioNLP web services [ 10 ] to expedite the knowledge elicitation process. The SNOMED-CT and NLP web services provided the exact annotation location (e.g. start and stop annotation word) wherever a term existed in the paper. Manual evaluation of the outputs extracted from papers was performed, whenever relevant terms were found they were categorized and documented. While modeling the BIM ontology, a number of ontologies relating to annotation and biomedical imaging were also consulted and where appropriate, classes and properties were reused.

Table 1 depicts the ontologies, prefixes and namespaces of the existing ontologies that have been employed in the modeling of BIM ontology. We have reused the vocabularies defined in Annotation Ontology (AO) [ 11 ] to model the biological concepts mentioned in an image caption. AO is an open-source ontology for annotating the scientific documents on the web. In AO, all the annotations are regarded as resources and fall under the instance category of the Annotation class. Each annotation has some hasTopic, context predicates and object class. Objects can be a particular entity such as protein or chemical name, a disease, or reified fact, while the context refers to a certain text segment inside the sentence (see Fig.3). This simple reference model makes it possible to integrate the extracted information semantically. The provenance of annotations is modeled with Provenance, Authoring and Versioning (PAV) ontology [ 12 ] e.g. predicates such as createdBy, createdOn describe the annotation creator and date of creation. PAV provides the terminologies for tracing provenance of the digital entities that have been published on the web and then accessed, transformed and consumed. To cover highlevel scientific research concepts, terms from the Semanticscience Integrated Ontology (SIO) were imported [ 13 ]. SIO provides a simple, integrated ontology of types and relations to describe objects, processes and their attributes. SIO behaves as an upper level ontology and supplies many high-level biomedical concepts. To represent the structural information of a biological sequence semantically, we incorporated a number of classes and relationships from Sequence Ontology (SO) [ 14 ] ontology such as transcript, primary-transcript, intron, mRNA, insertion sequence.

The Exif2 ontology [ 15 ] mainly describes the Exif format of picture data semantically, and provides useful vocabularies supporting the pre-processing and usage of Exif images. In BIM ontology, we used the Exif terminologies to define image orientation and size using ExAnnotation Ontology Provenance Authoring & Versioning Ontology SemanticScience Integrated Ontology Sequence Ontology Friend Of A Friend SIOC Ontology SKOS ontology Exif Ontology Time Ontology

AO PAV SIO SO FOAF SIOC SKOS exif TIME http://purl.org/ao/ http://purl.org/pav/ http://semanticscience.org/ontology/si o.owl http://purl.obolibrary.org/obo/so.owl http://xmlns.com/foaf/0.1/ http://rdfs.org/sioc/ns# http://www.w3.org/2004/02/skos/core http://www.kanzaki.com/ns/exif# http://www.w3.org/TR/owl-time/ http://purl.bioontology.org/ontology/S

EDI if:Orientation, Exif:ImageWidth, Exif:ImageHeight and corresponding vocabularies to represent the stages of image processing e.g Exif:WhiteBalance. DICOM (Digital Imaging and Communications in Medicine) [ 16 ] is a standard to represent the medical image information worldwide. Most of the available medical images modalities follow the DICOM standards to capture, store and disseminate the medical image information. However, the DO (DICOM Ontology) [ 17 ] serves the purpose of integrating and explicitly representing the concepts and relationships of DICOM in machine readable and human understandable format. In BIM ontology, we imported DO classes to represent the information associated with radiology images and to represent image capturing detail semantically. The FOAF [ 18 ] vocabulary describes people, their relations with other people, and objects that are related to a person-to-person connection.

We also leveraged the DBpedia ontology [ 19 ], a multidomain ontology that is mainly designed to cover the Wikipedia infoboxes. In version 3.2, there are roughly 359 classes and 1775 properties, which cover a vast range of common and life science concepts. In contrast, the Dublin core Metadata [ 20 ] vocabulary was used to represent general meta-data attributes for documents such as titles, authors, subjects, descriptions, date, type, and format. Core concepts from time and relationship ontologies were imported to describe concepts relating to time units (e.g. minutes, seconds) and relations between objects. The Semantically-Interlinked Online Communities (SIOC pronounced as “shock”) [ 21 ] is a domain ontology, which perfectly defines and interlinks all the online communities’ concepts such as posts, comments, and users. Similarly, the Simple Knowledge Organization System (SKOS) [ 22 ] is a generalized model written in RDF for sharing and interlinking organizational knowledge with semantic description. We reused the terms SKOS:prefLabel, SKOS:Concept, SIOC:Item and SIOC:userAccount from SIOC and SKOS ontologies. To assemble the BIM ontology model, we used the Protégé, editor [ 23 ]. However, to efficiently manage and utilize the BIM vocabularies, an ontology-publishing server called UNBvps (http://cbakerlab.unbsj.ca/unbvps/) was set up.

This section demonstrates the BIM ontology modeling with three different use cases. 4.1

This paper introduces Biomedical Image ontology (BIM) that supports the publication of annotations on, features identified within a biomedical image generated by SEBI tools. BIM was created to address the dearth of appropriate ontologies and appropriately integrated semantic metadata targeted to annotating diverse biomedical images, particularly images depicting biological sequences. BIM supports both the creation of machine generated and human curated annotations which can be reused in multiple knowledge discovery tasks or resources. These include; image mashups, linked image data, semantic image search and the computing of image similarity, which along with provenance annotations indicating an image’s source publication permits the linking of publications containing related images. The SEBI framework is designed to facilitate all these goals.