Spatio-Temporal Ontology for Change Analysis of Flood Affected Areas Using Remote Sensing Images Kuldeep R. KURTEa,1 and Surya S. DURBHAa,1 a Centre of Studies in Resources Engineering, Indian Institute of Technology, Bombay, India Abstract. This article presents ST-IIM (Spatio-Temporal Image Information Mining) – an ontology to model the spatio-temporal changes during a flood disaster event. It is integrated with the image ontology to represent the various characteristics of Remote Sensing (RS) images. The goal is to query, detect and monitor disaster affected areas from multi-temporal images by automatic reasoning. The spatial and temporal aspects were modeled using DL-safe rules based on qualitative topological relationships and Allen's interval algebra respectively. This enables to draw inferences about the temporal changes and evolution of the Land Use/Cover (LU/LC) classes during the disaster event for rapid disaster response and recovery activities. Keywords. Spatio-Temporal Ontology, RCC8 topological relations, Allen's interval algebra, Flood disaster ontology 1. Introduction Natural disasters such as, flood, earthquake have deep impact on human life. Such natural events cannot be prevented, but their impact can be mitigated and the loss of human life can be minimized by preparing an action plan for the entire disaster management cycle. Remote Sensing (RS) data, captured by various Earth Observation (EO) satellites plays a very important role in a disaster event, as it provides a synoptic view of the affected area. During a disaster event, a series of RS images obtained from various RS platforms can be processed and analyzed, and the information such as flood affected area (during response phase), and Land Use/Cover (LU/LC) change over the flood duration, flood recession pattern (during recovery phase) can be extracted. However, the low-level features obtained from RS imagery are unsuitable to capture the high- level semantic concept such as spatial patterns, spatio-temporal evolution of LU/LC classes. Hence, an ontology is useful to conceptualize the domain knowledge, and to reduce the semantic gap between low-level image features and high level spatio- temporal semantics. This paper describes the ontology development (ST-IIM) to model the spatio- temporal changes specifically for flood disaster situation. 2. Design and Development of ST-IIM ontology The ontology presented in this paper encodes the formal knowledge about the qualitative spatial relationships between objects in an image as well as their temporal 1 Contact Authors: kuldeep.iitb@iitb.ac.in; sdurbha@iitb.ac.in behavior in order to model, retrieve and detect the spatio-temporal changes. The Basic Formal Ontology (BFO2) is imported as a base ontology and the domain knowledge (LU/LC classes) and spatial and spatio-temporal knowledge is built upon it. The ontology (ST-IIM) is developed in Web Ontology Language (OWL-DL) and is openly available 3. The developed ontology is able to do reasoning about the subsumption relationships among LU/LC classes, reasoning about spatial relations and spatio- temporal changes during dynamic events such as flood disaster. The similar work has been presented in [1] where a 4D-fluent approach is used to represent the evolution of the temporal information in ontologies. 2.1. Encoding spatial knowledge To understand the spatial configuration, the spatial relationships among image regions need to be formally defined and encoded in an ontology. The Region Connection Calculus (RCC8) defines a set of eight jointly exhaustive and pair wise disjoint (JEPD) topological relationships (see Figure 1 (a)) [2]. These topological relationships are defined as a hierarchy of object properties in ST-IIM ontology. For example, Externally Connected (EC) subPropertyOf Connected (C) and Partially Overlap (PO) subPropertyOf Overlap (O) are defined. Moreover, the characteristics such as, symmetry, reflexivity, transitivity are defines for these topological object properties, e.g. the object property EC and PO are declared as symmetric. Figure 1. (a) RCC8 topological relations hierarchy [2] (b) Non-Tangential Proper Part (NTPP) example Also in this work, image regions are assumed to be represented by their Minimum Bounding Rectangles (MBRs) (see Figure 1(b)). The OWL classes, Region and BoundingBox are defined in an ontology and connected via hasBBox object property. The MBR representation of a region divides the space into 25 partitions (4 points, 12 lines and 9 region partitions) [3]. Hence, the MBR representation of two regions will generate 169 exhaustive spatial configurations. This spatial knowledge is defined in terms of topological relationships by developing their rule-based logical encoding using Semantic Web Rule Language (SWRL) in ST-IIM ontology. These spatial rules compares four datatype properties of two MBRs i.e. hasLeftLong (longitude of lower left corner of MBR), hasLowerLat (latitude of lower left corner of MBR), hasRightLong (longitude of upper right corner of MBR), hasUpperLat (latitude of upper right corner of MBR), which is presently not possible using OWL-DL constructs, and hence SWRL rules were used. A subset of 65 topological rules are developed and integrated into ST- 2 https://github.com/BFO-ontology/BFO/blob/v2.0/bfo.owl 3 http://home.iitb.ac.in/~kuldeep.iitb/ST-IIM.owl ; http://ontohub.org/fois2016_ontology_comp/ST-IIM IIM to infer all 169 topological relationships. Table 1 shows example SWRL rules for two topological relationships NTPP (Non-tangential Proper Part) and PO (Partially overlap) (refer rules 1, 2). 2.2. Encoding Temporal knowledge The time ontology published by W3C, available on-line4 is imported in ST-IIM (re- usability) as a base ontology, as it has the taxonomy of the time classes already defined. However, this basic ontology is not capable of performing reasoning over temporal intervals. For example, if events and timestamps of those events are given, this ontology will not be able to do automatic reasoning to retrieve image regions based on their temporal interval relations. Hence, SWRL rules were developed to address this issue. In this ontology, the major classes Instant, DateTimeInterval etc. are defined to capture the temporal event's instant (e.g. flood event) and the valid time duration of that event (e.g. flood duration). Figure 2. (a) Pictorial representation of Allen's temporal relations (b) Example RS images showing submergence and flood recession pattern of Road, River and other LU/LC classes In ST-IIM the Allen's temporal algebra [4] is defined as object properties (see Figure 2 (a)), which along with the encoded SWRL rules for temporal intervals can be used to state the temporal relationships among two intervals (refer rules 3-6 in Table 1). For example, whether an interval of submerged road in one part of a city is temporally overlapping with any other interval of submerged roads in some other part of the same city. Figure 2(b) shows example of such LU/LC evolution pattern. Table 1. Example SWRL rules to model spatio-temporal changes (shows only two rules for each category) No. SWRL Rules Topological Relation SWRL rules 1 hasBBox(?R1, ?B1), hasBBox(?R2, ?B2), hasLeftLong(?B1, ?B1LeftLong), hasLeftLong(?B2, ?B2LeftLong), hasLowerLat(?B1, ?B1LowerLat), hasLowerLat(?B2, ?B2LowerLat), hasRightLong(?B1, ?B1RightLong), hasRightLong(?B2, ?B2RightLong), hasUpperLat(?B1, ?B1UpperLat), NTPP hasUpperLat(?B2, ?B2UpperLat), greaterThan(?B1LeftLong, ?B2LeftLong), greaterThan(?B1LowerLat, ?B2LowerLat), lessThan(?B1RightLong, ?B2RightLong), lessThan(?B1UpperLat, ?B2UpperLat) -> NTPP(?B1, ?B2) 2 hasBBox(?R1, ?B1), hasBBox(?R2, ?B2), hasLeftLong(?B1, ?B1LeftLong), hasLeftLong(?B2, ?B2LeftLong), hasLowerLat(?B1, ?B1LowerLat), hasLowerLat(?B2, ?B2LowerLat), hasRightLong(?B1, ?B1RightLong), hasRightLong(?B2, ?B2RightLong), hasUpperLat(?B1, ?B1UpperLat), PO hasUpperLat(?B2, ?B2UpperLat), equal(?B1UpperLat, ?B2UpperLat), greaterThan(?B1LeftLong, ?B2LeftLong), lessThan(?B1LowerLat, ?B2LowerLat), lessThan(?B1RightLong, ?B2RightLong) -> PO(?B1, ?B2) Temporal relations SWRL rules 3 inXSDDateTime(?i1, ?t1), inXSDDateTime(?i2, ?t2), lessThan(?t1, ?t2) -> before(?i1, ?i2) 4 https://www.w3.org/2006/time 4 inXSDDateTime(?i1, ?t1), inXSDDateTime(?i2, ?t2), equal(?t1, ?t2) -> at(?i1, ?i2) 5 before(?i12,?i21), hasBeginning(?dt1,?i11), hasBeginning(?dt2,?i21), hasEnd(?dt1,?i12), hasEnd(?dt2, ?i22) -> intervalBefore(?dt1, ?dt2) 6 before(?i11,?i21),before(?i12,?i22),before(?i21,?i12),hasBeginning(?dt1,?i11), hasBeginning(?dt2, ?i21), hasEnd(?dt1, ?i12), hasEnd(?dt2, ?i22) -> intervalOverlaps(?dt1, ?dt2) Spatio-Temporal SWRL rules 7 LULCRegion(?r12), StagnatedFloodWater(?r21), DateTimeInterval(?dt1), DateTimeInterval(?dt2), hasFirstDefiningRegionOfInterval(?dt2, ?r21), hasLastDefiningRegionOfInterval(?dt1, ?r12), hasPreviousTemporalNeighborhood(?dt2, ?dt1) -> hasSubmergenceInterval(?r12, ?dt1), isSubmergedBy(?r12, ?r21) 8 NTPP(?b1,?b2), hasBBox(?r1,?b1), hasBBox(?r2,?b2), isSubmergedBy(?r1,?r2)-> isCompletelySubmergedBy(?r1, ?r2) 9 PO(?b1,?b2), hasBBox(?r1,?b1), hasBBox(?r2,?b2), isSubmergedBy(?r1,?r2) -> isPartiallySubmergedBy(?r1, ?r2) 3. Representing spatio-temporal changes in flood disaster using ST-IIM During flood disaster, in response phase, the pre-flood RS imagery can be used along post-flood RS images to identify the submerged areas and underlying LU/LC classes such as, flooded roads, flooded cropland, flooded built-up. During the recovery phase (post flood disaster), a series of multi-temporal RS images captured during and after the flood, and along with the few pre-flood images of same area can be analyzed together to get more insight about the temporal behavior of flood. For example, LU/LC submergence pattern, flood recession pattern etc (as shown in Figure 2(b)). Such a information mining during recovery phase, provides very important insights for the future flood mitigation and preparedness plans. In this work, the class DateTimeInterval is considered as an important class, which is used to denote a temporal interval during which an image region has been observed with a specific LU/LC class. The object property hasObservedLULCFirstTime and hasObservedLULCLastTime are defined as subPropertyOf hasBeginning and hasEnd respectively. These properties have a class DateTimeInterval as domain, whereas a class Instant as range. This interval is obtained by tracking an image object with a particular LU/LC class in a temporal stack of RS images. Hence, a class DateTimeInterval in ST-IIM holds the information about a persistence of an image object with a particular LU/LC class along with its temporal interval. To achieve this, two object properties hasFirstDefiningRegionOfInterval and hasLastDefiningRegionOfInterval are defined to describe the regions which are associated with the interval. These properties have range class LULCRegion, which will indicate the LU/LC observed over an interval. It may happen that, the LU/LC of a region in an image persists for some temporal interval and later changed to another class forming another temporal interval. Such temporal interval instances can be identified and connected via an object property hasPreviousTemporalNeighborhood, which connects current and previous temporal intervals of a region. 3.1. Example Scenario: Detecting the spatio-temporal interactions between a road and river during a flood event Figure 3 shows an example of some temporal intervals which are assumed to be observed during a flood event. As RS images are obtained at discrete time intervals, the temporal intervals shown for different LULC classes are discrete. The example shows, the temporal intervals of two image objects River and Road. After the flood has occurred the object road has disappeared and submerged by the spread of flooded river- a region with a new class called StagnatedFloodWater is evolved (refer Figure 2(b)). The subsequent intervals shows that the River object has started receding before the Road object resurfaces. Here, an interval I3 is connected with intervals I1, I2 via hasPreviousTemporalNeighborhood object property. The SWRL rules are developed and integrated into ST-IIM ontology to infer that, the Road and River regions in interval I1, I2 isSubmergedBy a region in an interval I3. Moreover, the topological relations among MBRs of interval defining region's can be used to find whether the region is partially or completely flooded (refer Table 1 (rules 7-9)). Figure 3. Shows the pre-post flood temporal intervals for Road, River and StagnatedFlood regions (vertical dotted lines shows the discrete time instants when RS images are assumed to be acquired) Representing a temporal interval as described above enables the use of Allen's temporal algebra for comparing two intervals and do reasoning5 over them. This is useful to find whether one interval is overlapping other or finishes with other. In the case of flood disaster, comparing two intervals of FloodAffectedBuilding at two different parts of the flood affected area can give the insight about the recession pattern of the flood. Similarly, the post flood evolution of the submerged roads in an affected area can be obtained by reasoning over these interval relations using rules defined in this ontology. Conclusion The ST-IIM ontology to model spatio-temporal changes during a flood disaster event is presented. Although flood disaster is chosen for demonstration, the formalism is applicable to other spatio-temporal domains, where there is a need to understand the dynamically changing events. The presented ontology can be further extended to model Allen's temporal interval relations between spatial relations of image objects to understand the evolution of spatial configuration over time. References [1] S. Batsakis and E. G. M. Petrakis, SOWL:Spatio-temporal Representation, Reasoning and Querying over the Semantic Web, in: Proc. Of Int. Conference on Semantic Systems, 2010, no. 15, pp. 1-9. [2] D. A. Randell, Z. Cui, and A. G. Cohn , A spatial logic based on regions and connection, in: Proc. of 3rd Int. Conference on Principles of Knowledge Representation and Reasoning, 1992, pp. 165-176. [3] D. Papadias and Y. Theodoridis, Spatial relations, minimum bounding rectangles, and spatial data structures, International Journal of Geographic Information System 11 (1997), 111-138. [4] J. F. Allen, Maintaining knowledge about temporal intervals, Communications of the ACM, (1983), 832– 843. 5 Pellet- OWL-DL reasoner is used to validate rules and developed ontology (http://clarkparsia.com/pellet)