Rich Annotation of Images Pramod Jain Innovative Decision Technologies, Inc. 9000 Cypress Green Drive, Jacksonville, FL 32256 USA +1 904 636 6374 pramod@indent.org ABSTRACT transparent layer placed on top of the image using Scalable In this paper, we describe a tool for rich annotation of Vector Graphics (SVG, a World Wide Web Consortium’s images. This tool allows creation of graphical annotations standard for 2-D graphics on the web). The transparent to identify and describe features in images. It also provides layer is called an annotation layer, and it can contain the capability to then catalog the annotation layer in several annotated features. We have developed Rich metadata repositories. Application of the annotation tool to Annotation Service for Images (RASI), a web-based tool to describe satellite images in the areas of meteorology and create rich annotations of images. Rich annotation is a vegetation will be presented. fusion of a graphical marker (to identify a feature) and a structured object (to describe the feature) that contains Keywords attribute names and attribute values, this is the reason for Annotations, Scalable vector graphics, SVG, Rich calling it a “rich” annotation. annotations, Images, Attributes, Metadata, DLESE (Digital Library for Earth Science Education), NASA-GCMD RICH ANNOTATION SERVICE FOR IMAGES (RASI) (Global Change Master Directory), content-based RASI is a web-based application for creating rich metadata, XIMA (XML for Image and Map Annotations) annotations of images, it may be accessed at http://www.indent.org/rasi.htm. The drawing capabilities INTRODUCTION in the browser are programmed in Scalable Vector The amount of image-based content has exploded on the Graphics (SVG, a World Wide Web Consortium standard web. However, there is a lack of effective tools to for 2-D graphics on the web). See Figure 1 for a screen describe, enhance, catalog, search and retrieve images. shot of RASI. This leads to poor use of image-based resources in general and for teaching in particular. The issues are: a) Limited RASI contains an annotation canvas, an annotation toolbar metadata to describe and find high quality and “content and section to manage properties. In the annotation toolbar, rich” images b) No methods for teachers and students to a palette of annotation markers is available to draw extend or customize an image, c) No tools to interact with, freeform or place icons in the annotation canvas. When a inquire about, experiment with and discuss content in feature is annotated in an image using an annotation images, and d) No “visual” methods to collaborate on or marker, a rich annotation is created. This rich annotation discuss an image. object contains the geometry of the annotation marker and a set of properties. See Figure 2 for relationship between In a typical image, the image as a whole is usually not the various entities in RASI. interesting; the features contained in the image are of primary interest. Features inside an image are the true As an example consider an Infrared (IR) image of a storm resources that need to be described, cataloged, searched with features like cold front, warm front, occluded front and retrieved. We adopt this approach. and low pressure point. These features are annotated with appropriate annotation markers. Cold front, warm front RICH ANNOTATIONS and occluded front feature have a property called Rich annotation (RA) is a combination of a) graphical “precipitation,” and low pressure point has properties: drawing or marking to identify the location and geometry surface pressure, surface temperature, precipitation, and of a feature in an image and b) attributes or properties to cloud top temperature. The annotation markers used for describe the annotated feature. RA is created in a the features are: cold front is drawn using the blue line with triangles, warm front with the red line with semicircle, and low pressure point is an icon with letter L in the center. The RASI tool is customizable to other domains—user with appropriate authorization can create a new problem domain, define new features for the domain, and create annotation marker and annotation properties for the new features. Metadata The process of creating annotations is the following, user: Services picks an annotation tool based on the feature they want to Web Middle-tier: annotate; creates an annotation to identify the feature; Server Aspire/J2EE chooses the created annotation and assigns values to the RASI running in Annotation properties associated with the feature; add more properties browser with Database SVGViewer plug-in XML, if required; create more features on the same annotation SVG, layer; saves the annotation layer; creates metadata for the HTML annotation layer and submits to DLESE or NASA-GCMD. Figure 3: Three tiers of the RASI system. Image CONCLUSIONS 0…* RASI has been built for creating rich annotations for Annotation ÙCollection of images. Several projects have been created in areas of Layer Annotated Features meteorology, vegetation and others to allow users to capture knowledge in a graphical manner on the image. In 0…* addition, this knowledge is used to generate metadata for ÙAnnotated images Rich Annotation Features ACKNOWLEDGMENTS We thank NASA for partially funding the development of 1 0…* RASI; NASA-GCMD and DLESE for cataloging the Annotation Annotation metadata records from RASI. Marker Properties REFERENCES Figure 2: Class diagram to illustrate the object model. An 1. Jain, P., et al. “Data Driven Web Graphics with SVG,” image can have multiple annotation layers, an annotation XML Journal, Vol 3, Issue 5, May 2002. layer can have multiple rich annotations, a rich annotation must have one annotation marker and can have multiple annotation properties. Technical Architecture The three components of its technical architecture are (see Figure 3): i) User-Agent, which includes HTML, Javascript, SVG and XML files that are delivered to the browser or to a metadata repository; ii) Middle-tier, which is ASPIRE (http://www.indent.org/aspire.htm, a Java-based Commercial Off The Shelf product of Innovative Decision Technologies, inc.) and is used to process SVG-XML data from browser and store into database, extract relational data from database and transform into SVG-XML, and metadata repository specific XML formats; iii) Database. The middle-tier generates the SVG, HTML and XML files based on templates that contain Aspire specific Annotation Properties replacement, loop and if tags. Annotation canvas RASI also contains a search facility that allows users to search for images based on advanced queries like: Annotation Toolbar Find an image with keywords = “cold front” AND (cloud top temperature < -50 Celsius OR pressure < 970 milli bars).