Plant Image Segmentation and Annotation with Ontologies in BisQue Justin Preece, Justin Elser, Pankaj Jaiswal Kris Kvilekval, Dmitri Fedorov, B.S. Manjunath Botany & Plant Pathology Department of Electrical and Computer Engineering Oregon State University University of California, Santa Barbara Corvallis, OR, USA Santa Barbara, CA, USA Ryan Kitchen, Xu Xu, Dmitrios Trigkakis, Sinisa Seth Carbon Todorovic Environmental Genomics and Systems Biology Division Electrical Engineering & Computer Science Lawrence Berkeley National Laboratory Oregon State University Berkeley, CA, USA Corvallis, OR, USA Keywords—image analysis; segmentation; ontology; II. IMPLEMENTATION annotation; machine learning The Planteome project [2] has partnered with BisQue and I. INTRODUCTION CyVerse to take advantage of their image analysis, storage and authentication features. BisQue (Bio-Image Semantic Query The field of computer vision has experienced much User Environment), a platform hosted at the UC-Santa Barbara progress in the last two decades. Image analysis of Center for Bio-Image Informatics, is designed to store, photography and video has moved out of computer science visualize and analyze a wide range of multidimensional research labs and into a wide range of applications. One biological images [3]. CyVerse (formerly iPlant) provides a example of progress in image analysis concerns the computational infrastructure for all manner of data-driven segmentation of images on the basis of gray scale, color hue, discovery projects in academic research [4]. BisQue is texture, geometry, and other features. Such image integrated specifically with the CyVerse authentication and segmentation allows for increasingly refined classification of storage systems. images and their components. In a parallel development, semantic computing has pursued the creation of ontologies in The BisQue environment allows external development hopes of capturing and defining what it is we “know” about the groups to build and contribute image analysis modules. world, and presenting it in the form of a terminology network Planteome is developing such a module to provide a connected by defined relationships. This knowledge network is segmentation feature utilizing a Dynamic Graph Cuts computable, and makes it possible to make logical inferences algorithm [5]. Planteome is working with BisQue to specify about facts and data annotated with ontology terms. and develop service brokers and a user interface within BisQue that will give end-users the ability to label images and portions By combining these two innovations: image analysis and of images with ontology terms [Fig. 1]. ontology annotation, we can imbue images with structured meaning and enable the inferential computability of image III. RESULTS data. For example, it may be possible to segment an image of a We have established development servers hosting the plant leaf into diseased and undiseased tissue, and then to BisQue engine, and have developed initial specifications for annotate these segments with ontology terms describing the the module’s user interface and backend segmentation disease state and associated phenotypes. Once a database of processing. Our server-side module consists of a MatLab such images is developed, machine-learning algorithms can be package implementing the Dynamic Graph Cuts algorithm, and applied to the data and predictive models can be developed. In is heavily modeled on the preexisting Image Matting module this scenario, new images of plant leaves may be “tagged” with [6], with notable modifications. User-guided segmentation a disease state based on earlier examples. requires lines (“markup”) drawn to indicate foreground and We have already explored the segmentation and ontological background elements relative to the desired segment. Our annotation components in the desktop application AISO [1], module user interface has been enhanced to allow multiple but would like to see this functionality available in an online foreground and background markup lines. The Planteome format that allows for better processing scalability, storage, project now has a running module in our development security, and collaborative feature sharing. We also want to environment that successfully processes and returns a apply machine learning to a collection of segmented, annotated segmented image (http://bisque-dev.planteome.org/. (NOTE: images, thereby automating future image processing. This is an active development environment, and the module may not be available at all times). Our module source code is Planteome project (OSU) supported by NSF Award #1340112. Center for Bioimage Informatics (UCSB) supported by NSF ABI Award #1356750. [1] Proposed interface for Planteome segmentation and ontology annotation in BisQue. The image viewer on the left contains a mock-up design for displaying segmented image results labeled with an ontology term. The data panel on the right contains a hypothetical key-value pair listing for ontology term data associated with the segment on the left. Our project module is currently able to accept user-guided foreground and background markup, segment an image, and return that segment data to the viewer. Ontology APIs and annotation interface are still under development. available on GitHub (https://github.com/Planteome/planteome- services, and future machine learning can be a powerful tool in image-annotator). an era where high-throughput, high-quality digital images of biological phenotypes are readily available and ripe for With regard to ontology service development, the computational analysis. The Planteome segmentation module Planteome team has enabled an API from the AmiGO platform and accompanying BisQue ontology annotation integration [7] that serves out multiple Planteome-developed ontologies in may point the way to an effective suite of auto-segmentation JSON. For example, API requests may be made for term and auto-annotation tools built to meet this need. details and autocomplete suggestions. Research is also underway on novel feature detection and prediction (i.e. leaf ACKNOWLEDGMENTS orientation and characterization) that may be incorporated into the machine-learning aspects of this project. J.P. thanks Planteome project personnel Laurel Cooper, Austin Meier, and Chris Mungall for advice and support, and IV. DISCUSSION Nirav Merchant (U. of Ariz.) for access to the CyVerse infrastructure and systems architecture advice. We are currently in the process of defining specifications for ontology integration and annotation with the BisQue team. REFERENCES We believe it will be beneficial to allow multiple ontology [1] Lingutla N*, Preece J*, Todorovic S, Cooper L, Moore L, Jaiswal P. terms to be applied to the same segment in the same image; 2014. AISO: Annotation of Image Segments with Ontologies. Journal that is a feature to be added at some point. Other key topics of Biomedical Semantics. 5:50. (*: Co-authors) under discussion include whether to make the ontology [2] Project description: http://planteome.org/about annotation interface in BisQue configurable and extensible to [3] Kristian Kvilekval, Dmitry Fedorov, Boguslaw Obara, Ambuj Singh and external ontology services provided by multiple sources in B.S. Manjunath, “Bisque: A Platform for Bioimage Analysis and different formats, and whether to allow user customization and Management”, Bioinformatics, vol. 26, no. 4, pp. 544-552, Feb. 2010. localization of ontologies. These features will benefit the [4] Merchant, Nirav, et al., "The iPlant Collaborative: Cyberinfrastructure broader image analysis community, as all users of the BisQue for Enabling Data to Discovery for the Life Sciences," PLOS Biology (2016), doi: 10.1371/journal.pbio.1002342. platform will be able to take advantage of the ontology [5] P. Kohli and P. H. S. Torr, "Dynamic Graph Cuts for Efficient Inference annotation functionality. Further user interface enhancements in Markov Random Fields," in IEEE Transactions on Pattern Analysis to the segmentation feature will include differential markup and Machine Intelligence, vol. 29, no. 12, pp. 2079-2088, Dec. 2007. line coloring (red = foreground, blue = background); currently, doi: 10.1109/TPAMI.2007.1128 all markup lines are colored red and may confuse the end-user. [6] Vignesh Jagadeesh, Utkarsh Gaur, “Graph Cuts-based Image Matting / Segmentation”, unpublished. Online BisQue module: V. CONCLUSIONS http://bisque.iplantcollaborative.org/module_service/ImageMatting/ The combination of a robust online image analysis [7] Carbon S, Ireland A, Mungall CJ, Shu S, Marshall B, Lewis S, AmiGO platform, an efficient segmentation algorithm, ontology Hub, Web Presence Working Group. AmiGO: online access to ontology and annotation data. Bioinformatics. Jan 2009;25(2):288-9.