=Paper=
{{Paper
|id=Vol-1747/D101_ICBO2016
|storemode=property
|title=Plant Image Segmentation and Annotation with Ontologies in BisQue
|pdfUrl=https://ceur-ws.org/Vol-1747/D101_ICBO2016.pdf
|volume=Vol-1747
|authors=Justin Preece,Justin Elser,Pankaj Jaiswal,Kris Kvilekval,Dmitry Fedorov,B.S. Manjunath,Ryan Kitchen,Xu Xu,Dmitrios Trigkakis,Sinisa Todorovic,Seth Carbon
|dblpUrl=https://dblp.org/rec/conf/icbo/PreeceEJKFMKXTT16
}}
==Plant Image Segmentation and Annotation with Ontologies in BisQue ==
https://ceur-ws.org/Vol-1747/D101_ICBO2016.pdf
Plant Image Segmentation and Annotation with
Ontologies in BisQue
Justin Preece, Justin Elser, Pankaj Jaiswal Kris Kvilekval, Dmitri Fedorov, B.S. Manjunath
Botany & Plant Pathology Department of Electrical and Computer Engineering
Oregon State University University of California, Santa Barbara
Corvallis, OR, USA Santa Barbara, CA, USA
Ryan Kitchen, Xu Xu, Dmitrios Trigkakis, Sinisa Seth Carbon
Todorovic Environmental Genomics and Systems Biology Division
Electrical Engineering & Computer Science Lawrence Berkeley National Laboratory
Oregon State University Berkeley, CA, USA
Corvallis, OR, USA
Keywords—image analysis; segmentation; ontology; II. IMPLEMENTATION
annotation; machine learning The Planteome project [2] has partnered with BisQue and
I. INTRODUCTION CyVerse to take advantage of their image analysis, storage and
authentication features. BisQue (Bio-Image Semantic Query
The field of computer vision has experienced much User Environment), a platform hosted at the UC-Santa Barbara
progress in the last two decades. Image analysis of Center for Bio-Image Informatics, is designed to store,
photography and video has moved out of computer science visualize and analyze a wide range of multidimensional
research labs and into a wide range of applications. One biological images [3]. CyVerse (formerly iPlant) provides a
example of progress in image analysis concerns the computational infrastructure for all manner of data-driven
segmentation of images on the basis of gray scale, color hue, discovery projects in academic research [4]. BisQue is
texture, geometry, and other features. Such image integrated specifically with the CyVerse authentication and
segmentation allows for increasingly refined classification of storage systems.
images and their components. In a parallel development,
semantic computing has pursued the creation of ontologies in The BisQue environment allows external development
hopes of capturing and defining what it is we “know” about the groups to build and contribute image analysis modules.
world, and presenting it in the form of a terminology network Planteome is developing such a module to provide a
connected by defined relationships. This knowledge network is segmentation feature utilizing a Dynamic Graph Cuts
computable, and makes it possible to make logical inferences algorithm [5]. Planteome is working with BisQue to specify
about facts and data annotated with ontology terms. and develop service brokers and a user interface within BisQue
that will give end-users the ability to label images and portions
By combining these two innovations: image analysis and of images with ontology terms [Fig. 1].
ontology annotation, we can imbue images with structured
meaning and enable the inferential computability of image III. RESULTS
data. For example, it may be possible to segment an image of a We have established development servers hosting the
plant leaf into diseased and undiseased tissue, and then to BisQue engine, and have developed initial specifications for
annotate these segments with ontology terms describing the the module’s user interface and backend segmentation
disease state and associated phenotypes. Once a database of processing. Our server-side module consists of a MatLab
such images is developed, machine-learning algorithms can be package implementing the Dynamic Graph Cuts algorithm, and
applied to the data and predictive models can be developed. In is heavily modeled on the preexisting Image Matting module
this scenario, new images of plant leaves may be “tagged” with [6], with notable modifications. User-guided segmentation
a disease state based on earlier examples. requires lines (“markup”) drawn to indicate foreground and
We have already explored the segmentation and ontological background elements relative to the desired segment. Our
annotation components in the desktop application AISO [1], module user interface has been enhanced to allow multiple
but would like to see this functionality available in an online foreground and background markup lines. The Planteome
format that allows for better processing scalability, storage, project now has a running module in our development
security, and collaborative feature sharing. We also want to environment that successfully processes and returns a
apply machine learning to a collection of segmented, annotated segmented image (http://bisque-dev.planteome.org/. (NOTE:
images, thereby automating future image processing. This is an active development environment, and the module
may not be available at all times). Our module source code is
Planteome project (OSU) supported by NSF Award #1340112. Center for
Bioimage Informatics (UCSB) supported by NSF ABI Award #1356750.
[1] Proposed interface for Planteome segmentation and ontology annotation in BisQue. The image viewer on the left contains a mock-up design for
displaying segmented image results labeled with an ontology term. The data panel on the right contains a hypothetical key-value pair listing for
ontology term data associated with the segment on the left. Our project module is currently able to accept user-guided foreground and background
markup, segment an image, and return that segment data to the viewer. Ontology APIs and annotation interface are still under development.
available on GitHub (https://github.com/Planteome/planteome- services, and future machine learning can be a powerful tool in
image-annotator). an era where high-throughput, high-quality digital images of
biological phenotypes are readily available and ripe for
With regard to ontology service development, the computational analysis. The Planteome segmentation module
Planteome team has enabled an API from the AmiGO platform
and accompanying BisQue ontology annotation integration
[7] that serves out multiple Planteome-developed ontologies in may point the way to an effective suite of auto-segmentation
JSON. For example, API requests may be made for term
and auto-annotation tools built to meet this need.
details and autocomplete suggestions. Research is also
underway on novel feature detection and prediction (i.e. leaf ACKNOWLEDGMENTS
orientation and characterization) that may be incorporated into
the machine-learning aspects of this project. J.P. thanks Planteome project personnel Laurel Cooper,
Austin Meier, and Chris Mungall for advice and support, and
IV. DISCUSSION Nirav Merchant (U. of Ariz.) for access to the CyVerse
infrastructure and systems architecture advice.
We are currently in the process of defining specifications
for ontology integration and annotation with the BisQue team. REFERENCES
We believe it will be beneficial to allow multiple ontology
[1] Lingutla N*, Preece J*, Todorovic S, Cooper L, Moore L, Jaiswal P.
terms to be applied to the same segment in the same image; 2014. AISO: Annotation of Image Segments with Ontologies. Journal
that is a feature to be added at some point. Other key topics of Biomedical Semantics. 5:50. (*: Co-authors)
under discussion include whether to make the ontology [2] Project description: http://planteome.org/about
annotation interface in BisQue configurable and extensible to [3] Kristian Kvilekval, Dmitry Fedorov, Boguslaw Obara, Ambuj Singh and
external ontology services provided by multiple sources in B.S. Manjunath, “Bisque: A Platform for Bioimage Analysis and
different formats, and whether to allow user customization and Management”, Bioinformatics, vol. 26, no. 4, pp. 544-552, Feb. 2010.
localization of ontologies. These features will benefit the [4] Merchant, Nirav, et al., "The iPlant Collaborative: Cyberinfrastructure
broader image analysis community, as all users of the BisQue for Enabling Data to Discovery for the Life Sciences," PLOS Biology
(2016), doi: 10.1371/journal.pbio.1002342.
platform will be able to take advantage of the ontology
[5] P. Kohli and P. H. S. Torr, "Dynamic Graph Cuts for Efficient Inference
annotation functionality. Further user interface enhancements in Markov Random Fields," in IEEE Transactions on Pattern Analysis
to the segmentation feature will include differential markup and Machine Intelligence, vol. 29, no. 12, pp. 2079-2088, Dec. 2007.
line coloring (red = foreground, blue = background); currently, doi: 10.1109/TPAMI.2007.1128
all markup lines are colored red and may confuse the end-user. [6] Vignesh Jagadeesh, Utkarsh Gaur, “Graph Cuts-based Image Matting /
Segmentation”, unpublished. Online BisQue module:
V. CONCLUSIONS http://bisque.iplantcollaborative.org/module_service/ImageMatting/
The combination of a robust online image analysis [7] Carbon S, Ireland A, Mungall CJ, Shu S, Marshall B, Lewis S, AmiGO
platform, an efficient segmentation algorithm, ontology Hub, Web Presence Working Group. AmiGO: online access to ontology
and annotation data. Bioinformatics. Jan 2009;25(2):288-9.